├── .gitignore ├── 1-lxastro0 ├── code │ ├── Dockerfile │ ├── README.md │ ├── data_lib.py │ ├── download.sh │ ├── dummy.tif │ ├── figs │ │ ├── f1.png │ │ ├── f2.png │ │ ├── f3.png │ │ ├── f4.png │ │ ├── f5.png │ │ ├── f6.png │ │ ├── f7.png │ │ └── f8.png │ ├── hrnet_sn7.yaml │ ├── pdseg │ │ ├── LICENSE │ │ ├── __init__.py │ │ ├── __pycache__ │ │ │ ├── data_aug.cpython-36.pyc │ │ │ ├── data_aug.cpython-37.pyc │ │ │ ├── data_utils.cpython-36.pyc │ │ │ ├── data_utils.cpython-37.pyc │ │ │ ├── eval.cpython-36.pyc │ │ │ ├── eval.cpython-37.pyc │ │ │ ├── loss.cpython-36.pyc │ │ │ ├── loss.cpython-37.pyc │ │ │ ├── lovasz_losses.cpython-36.pyc │ │ │ ├── lovasz_losses.cpython-37.pyc │ │ │ ├── metrics.cpython-36.pyc │ │ │ ├── metrics.cpython-37.pyc │ │ │ ├── reader.cpython-36.pyc │ │ │ ├── reader.cpython-37.pyc │ │ │ ├── solver.cpython-36.pyc │ │ │ ├── solver.cpython-37.pyc │ │ │ ├── vis.cpython-36.pyc │ │ │ └── vis.cpython-37.pyc │ │ ├── check.py │ │ ├── data_aug.py │ │ ├── data_aug.pyc │ │ ├── data_utils.py │ │ ├── data_utils.pyc │ │ ├── demo_clinet.py │ │ ├── demo_server.py │ │ ├── eval.py │ │ ├── export_model.py │ │ ├── loss.py │ │ ├── loss.pyc │ │ ├── lovasz_losses.py │ │ ├── lovasz_losses.pyc │ │ ├── metrics.py │ │ ├── models │ │ │ ├── __init__.py │ │ │ ├── __init__.pyc │ │ │ ├── __pycache__ │ │ │ │ ├── __init__.cpython-36.pyc │ │ │ │ ├── __init__.cpython-37.pyc │ │ │ │ ├── model_builder.cpython-36.pyc │ │ │ │ └── model_builder.cpython-37.pyc │ │ │ ├── backbone │ │ │ │ ├── __init__.py │ │ │ │ ├── __init__.pyc │ │ │ │ ├── __pycache__ │ │ │ │ │ ├── __init__.cpython-36.pyc │ │ │ │ │ ├── __init__.cpython-37.pyc │ │ │ │ │ ├── mobilenet_v2.cpython-36.pyc │ │ │ │ │ ├── mobilenet_v2.cpython-37.pyc │ │ │ │ │ ├── resnet.cpython-36.pyc │ │ │ │ │ ├── resnet.cpython-37.pyc │ │ │ │ │ ├── xception.cpython-36.pyc │ │ │ │ │ └── xception.cpython-37.pyc │ │ │ │ ├── mobilenet_v2.py │ │ │ │ ├── mobilenet_v2.pyc │ │ │ │ ├── resnet.py │ │ │ │ ├── resnet.pyc │ │ │ │ ├── vgg.py │ │ │ │ ├── xception.py │ │ │ │ └── xception.pyc │ │ │ ├── libs │ │ │ │ ├── __init__.py │ │ │ │ ├── __init__.pyc │ │ │ │ ├── __pycache__ │ │ │ │ │ ├── __init__.cpython-36.pyc │ │ │ │ │ ├── __init__.cpython-37.pyc │ │ │ │ │ ├── model_libs.cpython-36.pyc │ │ │ │ │ └── model_libs.cpython-37.pyc │ │ │ │ ├── model_libs.py │ │ │ │ └── model_libs.pyc │ │ │ ├── model_builder.py │ │ │ ├── model_builder.pyc │ │ │ └── modeling │ │ │ │ ├── __init__.py │ │ │ │ ├── __init__.pyc │ │ │ │ ├── __pycache__ │ │ │ │ ├── __init__.cpython-36.pyc │ │ │ │ ├── __init__.cpython-37.pyc │ │ │ │ ├── deeplab.cpython-36.pyc │ │ │ │ ├── deeplab.cpython-37.pyc │ │ │ │ ├── fast_scnn.cpython-36.pyc │ │ │ │ ├── fast_scnn.cpython-37.pyc │ │ │ │ ├── hrnet.cpython-36.pyc │ │ │ │ ├── hrnet.cpython-37.pyc │ │ │ │ ├── icnet.cpython-36.pyc │ │ │ │ ├── icnet.cpython-37.pyc │ │ │ │ ├── pspnet.cpython-36.pyc │ │ │ │ ├── pspnet.cpython-37.pyc │ │ │ │ ├── unet.cpython-36.pyc │ │ │ │ └── unet.cpython-37.pyc │ │ │ │ ├── deeplab.py │ │ │ │ ├── deeplab.pyc │ │ │ │ ├── fast_scnn.py │ │ │ │ ├── fast_scnn.pyc │ │ │ │ ├── hrnet.py │ │ │ │ ├── hrnet.pyc │ │ │ │ ├── icnet.py │ │ │ │ ├── icnet.pyc │ │ │ │ ├── pspnet.py │ │ │ │ ├── pspnet.pyc │ │ │ │ ├── unet.py │ │ │ │ └── unet.pyc │ │ ├── planB.py │ │ ├── reader.py │ │ ├── reader.pyc │ │ ├── solver.py │ │ ├── solver.pyc │ │ ├── tools │ │ │ ├── __init__.py │ │ │ ├── __init__.pyc │ │ │ ├── __pycache__ │ │ │ │ ├── __init__.cpython-36.pyc │ │ │ │ ├── __init__.cpython-37.pyc │ │ │ │ ├── gray2pseudo_color.cpython-36.pyc │ │ │ │ └── gray2pseudo_color.cpython-37.pyc │ │ │ ├── create_dataset_list.py │ │ │ ├── gray2pseudo_color.py │ │ │ ├── gray2pseudo_color.pyc │ │ │ ├── jingling2seg.py │ │ │ └── labelme2seg.py │ │ ├── train.py │ │ ├── utils │ │ │ ├── __init__.py │ │ │ ├── __init__.pyc │ │ │ ├── __pycache__ │ │ │ │ ├── __init__.cpython-36.pyc │ │ │ │ ├── __init__.cpython-37.pyc │ │ │ │ ├── collect.cpython-36.pyc │ │ │ │ ├── collect.cpython-37.pyc │ │ │ │ ├── config.cpython-36.pyc │ │ │ │ ├── config.cpython-37.pyc │ │ │ │ ├── dist_utils.cpython-36.pyc │ │ │ │ ├── dist_utils.cpython-37.pyc │ │ │ │ ├── load_model_utils.cpython-36.pyc │ │ │ │ ├── load_model_utils.cpython-37.pyc │ │ │ │ ├── timer.cpython-36.pyc │ │ │ │ └── timer.cpython-37.pyc │ │ │ ├── collect.py │ │ │ ├── collect.pyc │ │ │ ├── config.py │ │ │ ├── config.pyc │ │ │ ├── dist_utils.py │ │ │ ├── fp16_utils.py │ │ │ ├── load_model_utils.py │ │ │ └── timer.py │ │ └── vis.py │ ├── postprocess.py │ ├── pretrained_model │ │ └── download_model.py │ ├── src │ │ ├── __init__.py │ │ ├── sn7_baseline_infer.py │ │ ├── sn7_baseline_postproc_funcs.py │ │ ├── sn7_baseline_prep_funcs.py │ │ └── sn7_baseline_train.py │ ├── test.sh │ ├── test │ │ ├── __pycache__ │ │ │ ├── test_utils.cpython-36.pyc │ │ │ └── test_utils.cpython-37.pyc │ │ ├── ci │ │ │ ├── check_code_style.sh │ │ │ └── test_download_dataset.sh │ │ ├── configs │ │ │ ├── deeplabv3p_xception65_cityscapes.yaml │ │ │ └── unet_pet.yaml │ │ ├── local_test_cityscapes.py │ │ ├── local_test_pet.py │ │ └── test_utils.py │ ├── test_list.txt │ ├── tools.py │ ├── train.sh │ ├── train_list.txt │ └── val_list.txt ├── report-final-lxastro0.docx └── weights │ └── README.txt ├── 2-cannab ├── cannab-Solution Description.docx ├── code │ ├── Dataset.py │ ├── Dockerfile │ ├── adamw.py │ ├── create_masks.py │ ├── create_submission.py │ ├── ddp_utils.py │ ├── download_models.sh │ ├── flags.txt │ ├── generate_json.py │ ├── losses.py │ ├── notes.txt │ ├── predict_double.py │ ├── test.sh │ ├── track_buildings.py │ ├── train.sh │ ├── train_b6_double_full.py │ ├── train_b6_full2.py │ ├── train_b7_double_full.py │ ├── train_b7_full2.py │ ├── tune_b6_full2.py │ ├── tune_b7_full.py │ ├── utils.py │ └── zoo │ │ ├── __init__.py │ │ ├── __pycache__ │ │ ├── __init__.cpython-37.pyc │ │ ├── dpn.cpython-37.pyc │ │ ├── models.cpython-37.pyc │ │ └── senet.cpython-37.pyc │ │ └── models.py └── weights │ └── README.txt ├── 3-selim_sef ├── code │ ├── .dockerignore │ ├── Dockerfile │ ├── configs │ │ ├── b6adam.json │ │ └── b7adam.json │ ├── ensemble.py │ ├── flags.txt │ ├── folds.csv │ ├── generate_submit.py │ ├── postprocessing │ │ ├── __init__.py │ │ ├── instance.py │ │ ├── labeling.py │ │ ├── polygonize.py │ │ └── tracking.py │ ├── predict_test.py │ ├── preprocessing │ │ ├── __init__.py │ │ ├── generate_folds.py │ │ ├── masks.py │ │ └── split_images.py │ ├── test.sh │ ├── train.sh │ ├── training │ │ ├── __init__.py │ │ ├── adam_fixed.py │ │ ├── augmentations.py │ │ ├── config.py │ │ ├── datasets.py │ │ ├── instance_datasets.py │ │ ├── losses.py │ │ ├── metrics.py │ │ ├── schedulers.py │ │ ├── train_instance.py │ │ ├── train_instance_no_offset.py │ │ ├── transforms.py │ │ └── utils.py │ └── zoo │ │ ├── __init__.py │ │ ├── dpn.py │ │ ├── senet.py │ │ └── unet.py ├── selim_sef - Solution Description .docx └── weights │ └── README.txt ├── 4-motokimura ├── code │ ├── Dockerfile │ ├── configs │ │ ├── test_to_val_images.yml │ │ ├── tta_resize.yml │ │ ├── tta_resize_flip.yml │ │ ├── unet-scse_timm-efficientnet-b3_v_01.yml │ │ ├── unet_timm-efficientnet-b0_scale-4.0_v_01.yml │ │ ├── unet_timm-efficientnet-b3_scale-2.0_v_01.yml │ │ ├── unet_timm-efficientnet-b3_scale-3.0_v_01.yml │ │ ├── unet_timm-efficientnet-b3_scale-3.0_v_02.yml │ │ ├── unet_timm-efficientnet-b3_scale-4.0_v_01.yml │ │ ├── unet_timm-efficientnet-b3_scale-4.0_v_02.yml │ │ └── unet_timm-efficientnet-b5_v_01.yml │ ├── docker │ │ ├── Dockerfile.dev │ │ ├── build.sh │ │ ├── exec.sh │ │ ├── jupyter.sh │ │ ├── run.sh │ │ └── tensorboard.sh │ ├── docs │ │ ├── ALGORITHM_DEVELOPMENT.md │ │ └── FINAL_SCORING.md │ ├── flags.txt │ ├── licenses │ │ └── README.md │ ├── scripts │ │ ├── test │ │ │ ├── postprocess.sh │ │ │ └── test_cnns.sh │ │ └── train │ │ │ ├── preprocess.sh │ │ │ └── train_cnns.sh │ ├── settings.sh │ ├── spacenet7_model │ │ ├── __init__.py │ │ ├── configs │ │ │ ├── __init__.py │ │ │ ├── defaults.py │ │ │ └── load_config.py │ │ ├── datasets │ │ │ ├── __init__.py │ │ │ └── spacenet7.py │ │ ├── evaluations │ │ │ └── __init__.py │ │ ├── models │ │ │ └── __init__.py │ │ ├── solvers │ │ │ ├── __init__.py │ │ │ └── losses.py │ │ ├── transforms │ │ │ ├── __init__.py │ │ │ ├── augmentations.py │ │ │ └── preprocesses.py │ │ └── utils │ │ │ ├── __init__.py │ │ │ ├── checkpoint.py │ │ │ └── utils.py │ ├── static │ │ └── .gitkeep │ ├── test.sh │ ├── tools │ │ ├── _init_path.py │ │ ├── ensemble_models.py │ │ ├── filter_small_polygons.py │ │ ├── geojson_to_mask.py │ │ ├── pred_mask_to_poly.py │ │ ├── refine_pred_mask.py │ │ ├── split_dataset.py │ │ ├── split_dataset_random.py │ │ ├── test_spacenet7_model.py │ │ ├── track_polys.py │ │ └── train_spacenet7_model.py │ └── train.sh ├── detection_pipeline_01.png ├── motokimura-solution-description.md └── weights │ └── README.txt ├── 5-MaksimovKA ├── MaksimovKA Solution Description.docx ├── code │ ├── Adam.py │ ├── Dockerfile │ ├── callbacks.py │ ├── config.py │ ├── create_folds.py │ ├── create_masks.py │ ├── dataset.py │ ├── flags.txt │ ├── losses.py │ ├── map_jsons.py │ ├── masks_to_jsons.py │ ├── mean_folds.py │ ├── model.py │ ├── optimizers.py │ ├── predict.py │ ├── predict2.py │ ├── predict3.py │ ├── predict4.py │ ├── requirements.txt │ ├── senet.py │ ├── submit.py │ ├── test.sh │ ├── train.py │ ├── train.sh │ └── tta.py └── weights │ └── README.txt ├── LICENSE ├── README.md └── _figs ├── scot_rate_plot.png ├── sn_logo.png └── table1.png /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | _extra/ 3 | -------------------------------------------------------------------------------- /1-lxastro0/code/Dockerfile: -------------------------------------------------------------------------------- 1 | # Adapted from: https://raw.githubusercontent.com/CosmiQ/solaris/master/docker/gpu/Dockerfile 2 | # docker build -t sn7_baseline_image /path_to_docker/ 3 | # NV_GPU=0 nvidia-docker run -it -v /local_data:/local_data -v --rm -ti --ipc=host --name sn7_baseline_gpu0 sn7_baseline_image 4 | 5 | FROM nvidia/cuda:10.2-devel-ubuntu16.04 6 | LABEL maintainer="avanetten " 7 | 8 | ENV CUDNN_VERSION 7.6.0.64 9 | LABEL com.nvidia.cudnn.version="${CUDNN_VERSION}" 10 | ARG solaris_branch='master' 11 | 12 | # prep apt-get and cudnn 13 | RUN apt-get update && apt-get install -y --no-install-recommends \ 14 | apt-utils \ 15 | libcudnn7=$CUDNN_VERSION-1+cuda10.0 \ 16 | libcudnn7-dev=$CUDNN_VERSION-1+cuda10.0 && \ 17 | apt-mark hold libcudnn7 && \ 18 | rm -rf /var/lib/apt/lists/* 19 | 20 | # install requirements 21 | RUN apt-get update \ 22 | && apt-get install -y --no-install-recommends \ 23 | bc \ 24 | bzip2 \ 25 | ca-certificates \ 26 | curl \ 27 | git \ 28 | libgdal-dev \ 29 | libssl-dev \ 30 | libffi-dev \ 31 | libncurses-dev \ 32 | libgl1 \ 33 | jq \ 34 | nfs-common \ 35 | parallel \ 36 | python-dev \ 37 | python-pip \ 38 | python-wheel \ 39 | python-setuptools \ 40 | unzip \ 41 | vim \ 42 | wget \ 43 | build-essential \ 44 | && apt-get clean \ 45 | && rm -rf /var/lib/apt/lists/* 46 | 47 | SHELL ["/bin/bash", "-c"] 48 | ENV PATH /opt/conda/bin:$PATH 49 | 50 | # install anaconda 51 | RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh -O ~/miniconda.sh && \ 52 | /bin/bash ~/miniconda.sh -b -p /opt/conda && \ 53 | rm ~/miniconda.sh && \ 54 | /opt/conda/bin/conda clean -tipsy && \ 55 | ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \ 56 | echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \ 57 | echo "conda activate base" >> ~/.bashrc 58 | 59 | # prepend pytorch and conda-forge before default channel 60 | RUN conda update conda && \ 61 | conda config --prepend channels conda-forge && \ 62 | conda config --prepend channels pytorch 63 | 64 | # get dev version of solaris and create conda environment based on its env file 65 | WORKDIR /tmp/ 66 | RUN git clone https://github.com/cosmiq/solaris.git && \ 67 | cd solaris && \ 68 | git checkout ${solaris_branch} && \ 69 | conda env create -f environment-gpu.yml 70 | ENV PATH /opt/conda/envs/solaris/bin:$PATH 71 | 72 | RUN cd solaris && pip install . 73 | 74 | # install various conda dependencies into the space_base environment 75 | RUN conda install -n solaris \ 76 | jupyter \ 77 | jupyterlab \ 78 | ipykernel 79 | 80 | # add a jupyter kernel for the conda environment in case it's wanted 81 | RUN source activate solaris && python -m ipykernel.kernelspec \ 82 | --name solaris --display-name solaris 83 | 84 | # ensure solaris is activated 85 | # RUN conda activate solaris 86 | 87 | # Need imagecodecs for Planet files 88 | RUN pip install imagecodecs 89 | 90 | # Paddle 91 | RUN python3 -m pip install paddlepaddle-gpu==1.8.2.post107 -i https://mirror.baidu.com/pypi/simple 92 | RUN pip install scipy==1.3.2 93 | 94 | WORKDIR /work 95 | COPY . /work/ 96 | RUN chmod 777 train.sh 97 | RUN chmod 777 test.sh 98 | RUN chmod 777 download.sh 99 | RUN chmod 777 pretrained_model/download_model.py 100 | 101 | # open ports for jupyterlab and tensorboard 102 | EXPOSE 8888 6006 103 | 104 | RUN [ "/bin/bash" ] 105 | -------------------------------------------------------------------------------- /1-lxastro0/code/download.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | mkdir -p /wdata/saved_model/hrnet/ 4 | wget https://www.dropbox.com/s/krtl5tmrkf4qv56/prefix.tar.gz?dl=1 -O prefix.tar.gz 5 | tar -zxf prefix.tar.gz 6 | cp -r prefix /wdata/saved_model/hrnet/best_model 7 | -------------------------------------------------------------------------------- /1-lxastro0/code/dummy.tif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/dummy.tif -------------------------------------------------------------------------------- /1-lxastro0/code/figs/f1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/figs/f1.png -------------------------------------------------------------------------------- /1-lxastro0/code/figs/f2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/figs/f2.png -------------------------------------------------------------------------------- /1-lxastro0/code/figs/f3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/figs/f3.png -------------------------------------------------------------------------------- /1-lxastro0/code/figs/f4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/figs/f4.png -------------------------------------------------------------------------------- /1-lxastro0/code/figs/f5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/figs/f5.png -------------------------------------------------------------------------------- /1-lxastro0/code/figs/f6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/figs/f6.png -------------------------------------------------------------------------------- /1-lxastro0/code/figs/f7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/figs/f7.png -------------------------------------------------------------------------------- /1-lxastro0/code/figs/f8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/figs/f8.png -------------------------------------------------------------------------------- /1-lxastro0/code/hrnet_sn7.yaml: -------------------------------------------------------------------------------- 1 | # 数据集配置 2 | DATASET: 3 | DATA_DIR: "/data/SN7_buildings/train/" 4 | NUM_CLASSES: 2 5 | TEST_FILE_LIST: "test_list.txt" 6 | TRAIN_FILE_LIST: "train_list.txt" 7 | VAL_FILE_LIST: "val_list.txt" 8 | DATALOADER: 9 | NUM_WORKERS: 16 10 | # 预训练模型配置 11 | MODEL: 12 | MODEL_NAME: "hrnet" 13 | DEFAULT_NORM_TYPE: "bn" 14 | HRNET: 15 | STAGE2: 16 | NUM_CHANNELS: [48, 96] 17 | STAGE3: 18 | NUM_CHANNELS: [48, 96, 192] 19 | STAGE4: 20 | NUM_CHANNELS: [48, 96, 192, 384] 21 | 22 | # 其他配置 23 | TRAIN_CROP_SIZE: (512, 512) 24 | EVAL_CROP_SIZE: (512, 512) 25 | AUG: 26 | AUG_METHOD: "rangescaling" 27 | FIX_RESIZE_SIZE: (512, 512) 28 | INF_RESIZE_VALUE: 512 29 | MAX_RESIZE_VALUE: 600 30 | MIN_RESIZE_VALUE: 400 31 | MIRROR: True 32 | BATCH_SIZE: 16 33 | TRAIN: 34 | PRETRAINED_MODEL_DIR: "./pretrained_model/hrnet_w48_bn_imagenet/" 35 | MODEL_SAVE_DIR: "/wdata/saved_model/hrnet/" 36 | SNAPSHOT_EPOCH: 1 37 | TEST: 38 | TEST_MODEL: "/wdata/saved_model/hrnet/best_model/" 39 | TEST_AUG: False 40 | TEST_AUG_FLIP_OPS: ['v'] 41 | TEST_AUG_ROTATE_OPS: [] 42 | SOLVER: 43 | NUM_EPOCHS: 70 44 | LR: 0.01 45 | LR_POLICY: "piecewise" 46 | DECAY_EPOCH: [40, 60] 47 | GAMMA: 0.1 48 | OPTIMIZER: "sgd" 49 | VIS: 50 | ADD_LABEL: False 51 | RAW_PRED: True 52 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__init__.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | import models 16 | import utils 17 | from . import tools 18 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/data_aug.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/data_aug.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/data_aug.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/data_aug.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/data_utils.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/data_utils.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/data_utils.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/data_utils.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/eval.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/eval.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/eval.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/eval.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/loss.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/loss.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/loss.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/loss.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/lovasz_losses.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/lovasz_losses.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/lovasz_losses.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/lovasz_losses.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/metrics.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/metrics.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/metrics.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/metrics.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/reader.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/reader.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/reader.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/reader.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/solver.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/solver.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/solver.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/solver.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/vis.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/vis.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/__pycache__/vis.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/__pycache__/vis.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/data_aug.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/data_aug.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/data_utils.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/data_utils.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/loss.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/loss.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/lovasz_losses.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/lovasz_losses.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/metrics.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | import os 17 | import sys 18 | import numpy as np 19 | from scipy.sparse import csr_matrix 20 | 21 | 22 | class ConfusionMatrix(object): 23 | """ 24 | Confusion Matrix for segmentation evaluation 25 | """ 26 | 27 | def __init__(self, num_classes=2, streaming=False): 28 | self.confusion_matrix = np.zeros([num_classes, num_classes], dtype=np.float) 29 | self.num_classes = num_classes 30 | self.streaming = streaming 31 | 32 | def calculate(self, pred, label, ignore=None): 33 | # If not in streaming mode, clear matrix everytime when call `calculate` 34 | if not self.streaming: 35 | self.zero_matrix() 36 | 37 | label = np.transpose(label, (0, 2, 3, 1)) 38 | ignore = np.transpose(ignore, (0, 2, 3, 1)) 39 | mask = np.array(ignore) == 1 40 | 41 | label = np.asarray(label)[mask] 42 | pred = np.asarray(pred)[mask] 43 | one = np.ones_like(pred) 44 | # Accumuate ([row=label, col=pred], 1) into sparse matrix 45 | spm = csr_matrix((one, (label, pred)), 46 | shape=(self.num_classes, self.num_classes)) 47 | spm = spm.todense() 48 | self.confusion_matrix += spm 49 | 50 | def zero_matrix(self): 51 | """ Clear confusion matrix """ 52 | self.confusion_matrix = np.zeros([self.num_classes, self.num_classes], dtype=np.float) 53 | 54 | def _iou(self): 55 | """ 56 | Intersection over Union (IoU) 57 | """ 58 | cm_diag = np.diag(self.confusion_matrix) 59 | iou = cm_diag / (np.sum(self.confusion_matrix, axis=1) + np.sum(self.confusion_matrix, axis=0) - cm_diag) 60 | return iou 61 | 62 | def mean_iou(self): 63 | """ 64 | Mean Intersection over Union (MIoU) 65 | """ 66 | iou = self._iou() 67 | m_iou = np.mean(iou) 68 | return iou, m_iou 69 | 70 | def frequency_weighted_iou(self): 71 | """ 72 | Frequency Weighted Intersection over Union (FWIoU) 73 | """ 74 | frequency = np.sum(self.confusion_matrix, axis=1) / np.sum(self.confusion_matrix) 75 | iou = self._iou() 76 | fw_iou = np.sum(frequency * iou) 77 | return fw_iou 78 | 79 | def accuracy(self): 80 | """ 81 | Mean Pixel Accuracy (MPA) 82 | """ 83 | pa = np.diag(self.confusion_matrix) / np.sum(self.confusion_matrix, axis=1) 84 | mpa = np.mean(pa) 85 | return pa, mpa 86 | 87 | def kappa(self): 88 | """ 89 | Kappa coefficient 90 | """ 91 | cm_sum = np.sum(self.confusion_matrix) 92 | po = np.sum(np.diag(self.confusion_matrix)) / cm_sum 93 | pe = np.dot(np.sum(self.confusion_matrix, axis=0), np.sum(self.confusion_matrix, axis=1)) / (cm_sum ** 2) 94 | kappa = (po - pe) / (1 - pe) 95 | return kappa 96 | 97 | 98 | 99 | 100 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/__init__.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | import models.modeling 17 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/__init__.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/__pycache__/__init__.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/__pycache__/__init__.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/__pycache__/model_builder.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/__pycache__/model_builder.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/__pycache__/model_builder.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/__pycache__/model_builder.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/__init__.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/__init__.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/__pycache__/__init__.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/__pycache__/__init__.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/__pycache__/mobilenet_v2.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/__pycache__/mobilenet_v2.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/__pycache__/mobilenet_v2.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/__pycache__/mobilenet_v2.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/__pycache__/resnet.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/__pycache__/resnet.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/__pycache__/resnet.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/__pycache__/resnet.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/__pycache__/xception.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/__pycache__/xception.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/__pycache__/xception.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/__pycache__/xception.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/mobilenet_v2.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/mobilenet_v2.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/resnet.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/resnet.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/vgg.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | from __future__ import absolute_import 17 | from __future__ import division 18 | from __future__ import print_function 19 | 20 | import paddle 21 | import paddle.fluid as fluid 22 | from paddle.fluid import ParamAttr 23 | 24 | __all__ = ["VGGNet"] 25 | 26 | 27 | def check_points(count, points): 28 | if points is None: 29 | return False 30 | else: 31 | if isinstance(points, list): 32 | return (True if count in points else False) 33 | else: 34 | return (True if count == points else False) 35 | 36 | 37 | class VGGNet(): 38 | def __init__(self, layers=16): 39 | self.layers = layers 40 | 41 | def net(self, input, class_dim=1000, end_points=None, decode_points=None): 42 | short_cuts = dict() 43 | layers_count = 0 44 | layers = self.layers 45 | vgg_spec = { 46 | 11: ([1, 1, 2, 2, 2]), 47 | 13: ([2, 2, 2, 2, 2]), 48 | 16: ([2, 2, 3, 3, 3]), 49 | 19: ([2, 2, 4, 4, 4]) 50 | } 51 | assert layers in vgg_spec.keys(), \ 52 | "supported layers are {} but input layer is {}".format(vgg_spec.keys(), layers) 53 | 54 | nums = vgg_spec[layers] 55 | channels = [64, 128, 256, 512, 512] 56 | conv = input 57 | for i in range(len(nums)): 58 | conv = self.conv_block( 59 | conv, channels[i], nums[i], name="conv" + str(i + 1) + "_") 60 | layers_count += nums[i] 61 | if check_points(layers_count, decode_points): 62 | short_cuts[layers_count] = conv 63 | if check_points(layers_count, end_points): 64 | return conv, short_cuts 65 | 66 | return conv 67 | 68 | def conv_block(self, input, num_filter, groups, name=None): 69 | conv = input 70 | for i in range(groups): 71 | conv = fluid.layers.conv2d( 72 | input=conv, 73 | num_filters=num_filter, 74 | filter_size=3, 75 | stride=1, 76 | padding=1, 77 | act='relu', 78 | param_attr=fluid.param_attr.ParamAttr( 79 | name=name + str(i + 1) + "_weights"), 80 | bias_attr=False) 81 | return fluid.layers.pool2d( 82 | input=conv, pool_size=2, pool_type='max', pool_stride=2) 83 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/backbone/xception.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/backbone/xception.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/libs/__init__.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/libs/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/libs/__init__.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/libs/__pycache__/__init__.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/libs/__pycache__/__init__.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/libs/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/libs/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/libs/__pycache__/model_libs.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/libs/__pycache__/model_libs.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/libs/__pycache__/model_libs.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/libs/__pycache__/model_libs.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/libs/model_libs.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/libs/model_libs.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/model_builder.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/model_builder.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__init__.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__init__.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/__init__.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/__init__.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/deeplab.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/deeplab.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/deeplab.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/deeplab.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/fast_scnn.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/fast_scnn.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/fast_scnn.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/fast_scnn.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/hrnet.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/hrnet.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/hrnet.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/hrnet.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/icnet.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/icnet.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/icnet.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/icnet.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/pspnet.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/pspnet.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/pspnet.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/pspnet.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/unet.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/unet.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/__pycache__/unet.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/__pycache__/unet.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/deeplab.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/deeplab.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/fast_scnn.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/fast_scnn.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/hrnet.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/hrnet.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/icnet.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/icnet.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/pspnet.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/pspnet.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/models/modeling/unet.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/models/modeling/unet.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/reader.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/reader.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/solver.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/solver.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/tools/__init__.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/tools/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/tools/__init__.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/tools/__pycache__/__init__.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/tools/__pycache__/__init__.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/tools/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/tools/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/tools/__pycache__/gray2pseudo_color.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/tools/__pycache__/gray2pseudo_color.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/tools/__pycache__/gray2pseudo_color.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/tools/__pycache__/gray2pseudo_color.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/tools/gray2pseudo_color.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/tools/gray2pseudo_color.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__init__.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__init__.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/__init__.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/__init__.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/collect.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/collect.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/collect.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/collect.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/config.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/config.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/config.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/config.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/dist_utils.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/dist_utils.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/dist_utils.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/dist_utils.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/load_model_utils.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/load_model_utils.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/load_model_utils.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/load_model_utils.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/timer.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/timer.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/__pycache__/timer.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/__pycache__/timer.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/collect.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/collect.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/config.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/pdseg/utils/config.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/dist_utils.py: -------------------------------------------------------------------------------- 1 | #Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 2 | # 3 | #Licensed under the Apache License, Version 2.0 (the "License"); 4 | #you may not use this file except in compliance with the License. 5 | #You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | #Unless required by applicable law or agreed to in writing, software 10 | #distributed under the License is distributed on an "AS IS" BASIS, 11 | #WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | #See the License for the specific language governing permissions and 13 | #limitations under the License. 14 | 15 | from __future__ import absolute_import 16 | from __future__ import division 17 | from __future__ import print_function 18 | import os 19 | import paddle.fluid as fluid 20 | 21 | 22 | def nccl2_prepare(args, startup_prog, main_prog): 23 | config = fluid.DistributeTranspilerConfig() 24 | config.mode = "nccl2" 25 | t = fluid.DistributeTranspiler(config=config) 26 | 27 | envs = args.dist_env 28 | 29 | t.transpile( 30 | envs["trainer_id"], 31 | trainers=','.join(envs["trainer_endpoints"]), 32 | current_endpoint=envs["current_endpoint"], 33 | startup_program=startup_prog, 34 | program=main_prog) 35 | 36 | 37 | def pserver_prepare(args, train_prog, startup_prog): 38 | config = fluid.DistributeTranspilerConfig() 39 | config.slice_var_up = args.split_var 40 | t = fluid.DistributeTranspiler(config=config) 41 | envs = args.dist_env 42 | training_role = envs["training_role"] 43 | 44 | t.transpile( 45 | envs["trainer_id"], 46 | program=train_prog, 47 | pservers=envs["pserver_endpoints"], 48 | trainers=envs["num_trainers"], 49 | sync_mode=not args.async_mode, 50 | startup_program=startup_prog) 51 | if training_role == "PSERVER": 52 | pserver_program = t.get_pserver_program(envs["current_endpoint"]) 53 | pserver_startup_program = t.get_startup_program( 54 | envs["current_endpoint"], 55 | pserver_program, 56 | startup_program=startup_prog) 57 | return pserver_program, pserver_startup_program 58 | elif training_role == "TRAINER": 59 | train_program = t.get_trainer_program() 60 | return train_program, startup_prog 61 | else: 62 | raise ValueError( 63 | 'PADDLE_TRAINING_ROLE environment variable must be either TRAINER or PSERVER' 64 | ) 65 | 66 | 67 | def nccl2_prepare_paddle(trainer_id, startup_prog, main_prog): 68 | config = fluid.DistributeTranspilerConfig() 69 | config.mode = "nccl2" 70 | t = fluid.DistributeTranspiler(config=config) 71 | t.transpile( 72 | trainer_id, 73 | trainers=os.environ.get('PADDLE_TRAINER_ENDPOINTS'), 74 | current_endpoint=os.environ.get('PADDLE_CURRENT_ENDPOINT'), 75 | startup_program=startup_prog, 76 | program=main_prog) 77 | 78 | 79 | def prepare_for_multi_process(exe, build_strategy, train_prog): 80 | # prepare for multi-process 81 | trainer_id = int(os.environ.get('PADDLE_TRAINER_ID', 0)) 82 | num_trainers = int(os.environ.get('PADDLE_TRAINERS_NUM', 1)) 83 | if num_trainers < 2: return 84 | 85 | build_strategy.num_trainers = num_trainers 86 | build_strategy.trainer_id = trainer_id 87 | # NOTE(zcd): use multi processes to train the model, 88 | # and each process use one GPU card. 89 | startup_prog = fluid.Program() 90 | nccl2_prepare_paddle(trainer_id, startup_prog, train_prog) 91 | # the startup_prog are run two times, but it doesn't matter. 92 | exe.run(startup_prog) 93 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/fp16_utils.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | import os 17 | from paddle import fluid 18 | 19 | 20 | def load_fp16_vars(executor, dirname, program): 21 | load_dirname = os.path.normpath(dirname) 22 | 23 | def _if_exist(var): 24 | name = var.name[:-7] if var.name.endswith('.master') else var.name 25 | b = os.path.exists(os.path.join(load_dirname, name)) 26 | if not b and isinstance(var, fluid.framework.Parameter): 27 | print("===== {} not found ====".format(var.name)) 28 | return b 29 | 30 | load_prog = fluid.Program() 31 | load_block = load_prog.global_block() 32 | vars = list(filter(_if_exist, program.list_vars())) 33 | 34 | for var in vars: 35 | new_var = fluid.io._clone_var_in_block_(load_block, var) 36 | name = var.name[:-7] if var.name.endswith('.master') else var.name 37 | file_path = os.path.join(load_dirname, name) 38 | load_block.append_op( 39 | type='load', 40 | inputs={}, 41 | outputs={'Out': [new_var]}, 42 | attrs={ 43 | 'file_path': file_path, 44 | 'load_as_fp16': var.dtype == fluid.core.VarDesc.VarType.FP16 45 | }) 46 | 47 | executor.run(load_prog) 48 | -------------------------------------------------------------------------------- /1-lxastro0/code/pdseg/utils/timer.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | import time 17 | 18 | 19 | def calculate_eta(remaining_step, speed): 20 | if remaining_step < 0: 21 | remaining_step = 0 22 | remaining_time = int(remaining_step / speed) 23 | result = "{:0>2}:{:0>2}:{:0>2}" 24 | arr = [] 25 | for i in range(2, -1, -1): 26 | arr.append(int(remaining_time / 60**i)) 27 | remaining_time %= 60**i 28 | return result.format(*arr) 29 | 30 | 31 | class Timer(object): 32 | """ Simple timer class for measuring time consuming """ 33 | 34 | def __init__(self): 35 | self._start_time = 0.0 36 | self._end_time = 0.0 37 | self._elapsed_time = 0.0 38 | self._is_running = False 39 | 40 | def start(self): 41 | self._is_running = True 42 | self._start_time = time.time() 43 | 44 | def restart(self): 45 | self.start() 46 | 47 | def stop(self): 48 | self._is_running = False 49 | self._end_time = time.time() 50 | 51 | def elapsed_time(self): 52 | self._end_time = time.time() 53 | self._elapsed_time = self._end_time - self._start_time 54 | if not self.is_running: 55 | return 0.0 56 | 57 | return self._elapsed_time 58 | 59 | @property 60 | def is_running(self): 61 | return self._is_running 62 | -------------------------------------------------------------------------------- /1-lxastro0/code/src/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/src/__init__.py -------------------------------------------------------------------------------- /1-lxastro0/code/src/sn7_baseline_infer.py: -------------------------------------------------------------------------------- 1 | import solaris as sol 2 | import os 3 | config_path = '../yml/sn7_baseline_infer.yml' 4 | config = sol.utils.config.parse(config_path) 5 | print('Config:') 6 | print(config) 7 | 8 | # make infernce output dir 9 | os.makedirs(os.path.dirname(config['inference']['output_dir']), exist_ok=True) 10 | 11 | inferer = sol.nets.infer.Inferer(config) 12 | inferer() 13 | -------------------------------------------------------------------------------- /1-lxastro0/code/src/sn7_baseline_prep_funcs.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Tue Aug 25 14:11:02 2020 5 | 6 | @author: avanetten 7 | """ 8 | 9 | import multiprocessing 10 | import pandas as pd 11 | import numpy as np 12 | import skimage 13 | import gdal 14 | import os 15 | 16 | import solaris as sol 17 | from solaris.raster.image import create_multiband_geotiff 18 | from solaris.utils.core import _check_gdf_load 19 | 20 | 21 | def map_wrapper(x): 22 | '''For multi-threading''' 23 | return x[0](*(x[1:])) 24 | 25 | 26 | def make_geojsons_and_masks(name_root, image_path, json_path, 27 | output_path_mask, output_path_mask_fbc=None): 28 | ''' 29 | Make the stuffins 30 | mask_fbc is an (optional) three-channel fbc (footbrint, boundary, contact) mask 31 | ''' 32 | 33 | print(" name_root:", name_root) 34 | 35 | # filter out null geoms (this is always a worthy check) 36 | gdf_tmp = _check_gdf_load(json_path) 37 | if len(gdf_tmp) == 0: 38 | gdf_nonull = gdf_tmp 39 | else: 40 | gdf_nonull = gdf_tmp[gdf_tmp.geometry.notnull()] 41 | try: 42 | im_tmp = skimage.io.imread(image_path) 43 | except: 44 | print("Error loading image %s, skipping..." %(image_path)) 45 | return 46 | 47 | # handle empty geojsons 48 | if len(gdf_nonull) == 0: 49 | # create masks 50 | # mask 1 has 1 channel 51 | # mask_fbc has 3 channel 52 | print(" Empty labels for name_root!", name_root) 53 | im = gdal.Open(image_path) 54 | proj = im.GetProjection() 55 | geo = im.GetGeoTransform() 56 | im = im.ReadAsArray() 57 | # set masks to 0 everywhere 58 | mask_arr = np.zeros((1, im.shape[1], im.shape[2])) 59 | create_multiband_geotiff(mask_arr, output_path_mask, proj, geo) 60 | if output_path_mask_fbc: 61 | mask_arr = np.zeros((3, im.shape[1], im.shape[2])) 62 | create_multiband_geotiff(mask_arr, output_path_mask_fbc, proj, geo) 63 | return 64 | 65 | # make masks (single channel) 66 | # https://github.com/CosmiQ/solaris/blob/master/docs/tutorials/notebooks/api_masks_tutorial.ipynb 67 | f_mask = sol.vector.mask.df_to_px_mask(df=gdf_nonull, out_file=output_path_mask, 68 | channels=['footprint'], 69 | reference_im=image_path, 70 | shape=(im_tmp.shape[0], im_tmp.shape[1])) 71 | 72 | # three channel mask (takes awhile) 73 | # https://github.com/CosmiQ/solaris/blob/master/docs/tutorials/notebooks/api_masks_tutorial.ipynb 74 | if output_path_mask_fbc: 75 | fbc_mask = sol.vector.mask.df_to_px_mask(df=gdf_nonull, out_file=output_path_mask_fbc, 76 | channels=['footprint', 'boundary', 'contact'], 77 | reference_im=image_path, 78 | boundary_width=5, contact_spacing=10, meters=True, 79 | shape=(im_tmp.shape[0], im_tmp.shape[1])) 80 | 81 | return 82 | 83 | -------------------------------------------------------------------------------- /1-lxastro0/code/src/sn7_baseline_train.py: -------------------------------------------------------------------------------- 1 | import solaris as sol 2 | import os 3 | config_path = '../yml/sn7_baseline_train.yml' 4 | config = sol.utils.config.parse(config_path) 5 | print('Config:') 6 | print(config) 7 | 8 | # make model output dir 9 | os.makedirs(os.path.dirname(config['training']['model_dest_path']), exist_ok=True) 10 | 11 | trainer = sol.nets.train.Trainer(config=config) 12 | trainer.train() 13 | -------------------------------------------------------------------------------- /1-lxastro0/code/test.sh: -------------------------------------------------------------------------------- 1 | source activate solaris 2 | test_data_path=$1 3 | output_path=$2 4 | 5 | if [ ! -d /wdata/saved_model/hrnet/best_model ]; then 6 | bash download.sh 7 | fi 8 | 9 | rm -r /wdata/test 10 | cp -r $test_data_path /wdata/test 11 | rm /wdata/test/* 12 | 13 | python tools.py /wdata/test test 14 | cp dummy.tif /wdata/test 15 | 16 | python pdseg/eval.py --use_gpu --vis --vis_dir vis/test_org --cfg hrnet_sn7.yaml DATASET.DATA_DIR /wdata/test DATASET.VAL_FILE_LIST test_list.txt VIS.VISINEVAL True TEST.TEST_AUG True 17 | 18 | python tools.py vis/test_org compose 19 | 20 | python postprocess.py /wdata/test vis/test_org_compose "$output_path" 21 | 22 | rm -r vis 23 | -------------------------------------------------------------------------------- /1-lxastro0/code/test/__pycache__/test_utils.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/test/__pycache__/test_utils.cpython-36.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/test/__pycache__/test_utils.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/test/__pycache__/test_utils.cpython-37.pyc -------------------------------------------------------------------------------- /1-lxastro0/code/test/ci/check_code_style.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | function abort(){ 3 | echo "Your change doesn't follow PaddleSeg's code style." 1>&2 4 | echo "Please use pre-commit to check what is wrong." 1>&2 5 | exit 1 6 | } 7 | 8 | trap 'abort' 0 9 | set -e 10 | 11 | cd $TRAVIS_BUILD_DIR 12 | export PATH=/usr/bin:$PATH 13 | pre-commit install 14 | 15 | if ! pre-commit run -a ; then 16 | git diff 17 | exit 1 18 | fi 19 | 20 | trap : 0 21 | -------------------------------------------------------------------------------- /1-lxastro0/code/test/ci/test_download_dataset.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -o errexit 3 | 4 | base_path=$(cd `dirname $0`/../..; pwd) 5 | cd $base_path 6 | 7 | python dataset/download_pet.py 8 | -------------------------------------------------------------------------------- /1-lxastro0/code/test/configs/deeplabv3p_xception65_cityscapes.yaml: -------------------------------------------------------------------------------- 1 | EVAL_CROP_SIZE: (2049, 1025) # (width, height), for unpadding rangescaling and stepscaling 2 | TRAIN_CROP_SIZE: (769, 769) # (width, height), for unpadding rangescaling and stepscaling 3 | AUG: 4 | AUG_METHOD: "stepscaling" # choice unpadding rangescaling and stepscaling 5 | FIX_RESIZE_SIZE: (640, 640) # (width, height), for unpadding 6 | INF_RESIZE_VALUE: 500 # for rangescaling 7 | MAX_RESIZE_VALUE: 600 # for rangescaling 8 | MIN_RESIZE_VALUE: 400 # for rangescaling 9 | MAX_SCALE_FACTOR: 2.0 # for stepscaling 10 | MIN_SCALE_FACTOR: 0.5 # for stepscaling 11 | SCALE_STEP_SIZE: 0.25 # for stepscaling 12 | MIRROR: True 13 | BATCH_SIZE: 4 14 | DATASET: 15 | DATA_DIR: "./dataset/cityscapes/" 16 | IMAGE_TYPE: "rgb" # choice rgb or rgba 17 | NUM_CLASSES: 19 18 | TEST_FILE_LIST: "dataset/cityscapes/val.list" 19 | TRAIN_FILE_LIST: "dataset/cityscapes/train.list" 20 | VAL_FILE_LIST: "dataset/cityscapes/val.list" 21 | VIS_FILE_LIST: "dataset/cityscapes/vis.list" 22 | SEPARATOR: " " 23 | IGNORE_INDEX: 255 24 | FREEZE: 25 | MODEL_FILENAME: "__model__" 26 | PARAMS_FILENAME: "__params__" 27 | MODEL: 28 | DEFAULT_NORM_TYPE: "gn" 29 | MODEL_NAME: "deeplabv3p" 30 | DEEPLAB: 31 | ASPP_WITH_SEP_CONV: True 32 | DECODER_USE_SEP_CONV: True 33 | TEST: 34 | TEST_MODEL: "./saved_model/cityscape_v5/final/" 35 | TRAIN: 36 | MODEL_SAVE_DIR: "./saved_model/cityscape_v5/" 37 | PRETRAINED_MODEL_DIR: "pretrained_model/deeplabv3plus_gn_init" 38 | SNAPSHOT_EPOCH: 10 39 | SOLVER: 40 | LR: 0.001 41 | LR_POLICY: "poly" 42 | OPTIMIZER: "sgd" 43 | NUM_EPOCHS: 700 44 | -------------------------------------------------------------------------------- /1-lxastro0/code/test/configs/unet_pet.yaml: -------------------------------------------------------------------------------- 1 | TRAIN_CROP_SIZE: (512, 512) # (width, height), for unpadding rangescaling and stepscaling 2 | EVAL_CROP_SIZE: (512, 512) # (width, height), for unpadding rangescaling and stepscaling 3 | AUG: 4 | AUG_METHOD: "unpadding" # choice unpadding rangescaling and stepscaling 5 | FIX_RESIZE_SIZE: (512, 512) # (width, height), for unpadding 6 | 7 | INF_RESIZE_VALUE: 500 # for rangescaling 8 | MAX_RESIZE_VALUE: 600 # for rangescaling 9 | MIN_RESIZE_VALUE: 400 # for rangescaling 10 | 11 | MAX_SCALE_FACTOR: 1.25 # for stepscaling 12 | MIN_SCALE_FACTOR: 0.75 # for stepscaling 13 | SCALE_STEP_SIZE: 0.25 # for stepscaling 14 | MIRROR: True 15 | BATCH_SIZE: 4 16 | DATASET: 17 | DATA_DIR: "./dataset/mini_pet/" 18 | IMAGE_TYPE: "rgb" # choice rgb or rgba 19 | NUM_CLASSES: 3 20 | TEST_FILE_LIST: "./dataset/mini_pet/file_list/test_list.txt" 21 | TRAIN_FILE_LIST: "./dataset/mini_pet/file_list/train_list.txt" 22 | VAL_FILE_LIST: "./dataset/mini_pet/file_list/val_list.txt" 23 | VIS_FILE_LIST: "./dataset/mini_pet/file_list/test_list.txt" 24 | IGNORE_INDEX: 255 25 | SEPARATOR: " " 26 | FREEZE: 27 | MODEL_FILENAME: "__model__" 28 | PARAMS_FILENAME: "__params__" 29 | MODEL: 30 | MODEL_NAME: "unet" 31 | DEFAULT_NORM_TYPE: "bn" 32 | TEST: 33 | TEST_MODEL: "./saved_model/unet_pet/final/" 34 | TRAIN: 35 | MODEL_SAVE_DIR: "./saved_model/unet_pet/" 36 | PRETRAINED_MODEL_DIR: "./test/models/unet_coco_init/" 37 | SNAPSHOT_EPOCH: 10 38 | SOLVER: 39 | NUM_EPOCHS: 100 40 | LR: 0.005 41 | LR_POLICY: "poly" 42 | OPTIMIZER: "adam" 43 | -------------------------------------------------------------------------------- /1-lxastro0/code/test/local_test_cityscapes.py: -------------------------------------------------------------------------------- 1 | # coding: utf8 2 | # Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserve. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | from test_utils import download_file_and_uncompress, train, eval, vis, export_model 17 | import os 18 | import argparse 19 | 20 | LOCAL_PATH = os.path.dirname(os.path.abspath(__file__)) 21 | DATASET_PATH = os.path.join(LOCAL_PATH, "..", "dataset") 22 | MODEL_PATH = os.path.join(LOCAL_PATH, "models") 23 | 24 | 25 | def download_cityscapes_dataset(savepath, extrapath): 26 | url = "https://paddleseg.bj.bcebos.com/dataset/cityscapes.tar" 27 | download_file_and_uncompress( 28 | url=url, savepath=savepath, extrapath=extrapath) 29 | 30 | 31 | def download_deeplabv3p_xception65_cityscapes_model(savepath, extrapath): 32 | url = "https://paddleseg.bj.bcebos.com/models/deeplabv3p_xception65_cityscapes.tgz" 33 | download_file_and_uncompress( 34 | url=url, savepath=savepath, extrapath=extrapath) 35 | 36 | 37 | if __name__ == "__main__": 38 | download_cityscapes_dataset(".", DATASET_PATH) 39 | download_deeplabv3p_xception65_cityscapes_model(".", MODEL_PATH) 40 | 41 | model_name = "deeplabv3p_xception65_cityscapes" 42 | test_model = os.path.join(LOCAL_PATH, "models", model_name) 43 | cfg = os.path.join(LOCAL_PATH, "configs", "{}.yaml".format(model_name)) 44 | freeze_save_dir = os.path.join(LOCAL_PATH, "inference_model", model_name) 45 | vis_dir = os.path.join(LOCAL_PATH, "visual", model_name) 46 | saved_model = os.path.join(LOCAL_PATH, "saved_model", model_name) 47 | 48 | parser = argparse.ArgumentParser(description="PaddleSeg loacl test") 49 | parser.add_argument( 50 | "--devices", 51 | dest="devices", 52 | help="GPU id of running. if more than one, use spacing to separate.", 53 | nargs="+", 54 | default=[0], 55 | type=int) 56 | args = parser.parse_args() 57 | 58 | devices = [str(x) for x in args.devices] 59 | 60 | export_model( 61 | flags=["--cfg", cfg], 62 | options=[ 63 | "TEST.TEST_MODEL", test_model, "FREEZE.SAVE_DIR", freeze_save_dir 64 | ], 65 | devices=devices) 66 | 67 | # Final eval results should be #image=500 acc=0.9615 IoU=0.7804 68 | eval( 69 | flags=["--cfg", cfg, "--use_gpu"], 70 | options=["TEST.TEST_MODEL", test_model], 71 | devices=devices) 72 | 73 | vis(flags=["--cfg", cfg, "--use_gpu", "--local_test", "--vis_dir", vis_dir], 74 | options=["TEST.TEST_MODEL", test_model], 75 | devices=devices) 76 | 77 | train( 78 | flags=["--cfg", cfg, "--use_gpu", "--log_steps", "10"], 79 | options=[ 80 | "SOLVER.NUM_EPOCHS", "1", "TRAIN.PRETRAINED_MODEL_DIR", test_model, 81 | "TRAIN.MODEL_SAVE_DIR", saved_model 82 | ], 83 | devices=devices) 84 | -------------------------------------------------------------------------------- /1-lxastro0/code/test_list.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/test_list.txt -------------------------------------------------------------------------------- /1-lxastro0/code/tools.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | import data_lib as dlib 4 | 5 | path = sys.argv[1] 6 | mode = sys.argv[2] 7 | 8 | if mode == "train": 9 | dlib.create_label(path, f3x=False) 10 | dlib.enlarge_3x(path) 11 | dlib.create_label(path, f3x=True) 12 | dlib.divide(path) 13 | dlib.create_trainval_list(path) 14 | elif mode == "test": 15 | dlib.enlarge_3x(path) 16 | dlib.divide(path) 17 | dlib.create_test_list(path) 18 | else: 19 | dlib.compose(path) 20 | -------------------------------------------------------------------------------- /1-lxastro0/code/train.sh: -------------------------------------------------------------------------------- 1 | source activate solaris 2 | train_data_path=$1 3 | 4 | rm -r /wdata/saved_model/hrnet/best_model/ 5 | 6 | rm -r /wdata/train 7 | cp -r $train_data_path /wdata/train 8 | rm /wdata/train/* 9 | 10 | python tools.py /wdata/train train 2>err.log 11 | 12 | cd pretrained_model 13 | python download_model.py hrnet_w48_bn_imagenet 14 | cd .. 15 | 16 | python pdseg/train.py --do_eval --use_gpu --cfg hrnet_sn7.yaml DATASET.DATA_DIR /wdata/train DATASET.TEST_FILE_LIST val_list.txt 17 | -------------------------------------------------------------------------------- /1-lxastro0/code/train_list.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/train_list.txt -------------------------------------------------------------------------------- /1-lxastro0/code/val_list.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/code/val_list.txt -------------------------------------------------------------------------------- /1-lxastro0/report-final-lxastro0.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/1-lxastro0/report-final-lxastro0.docx -------------------------------------------------------------------------------- /1-lxastro0/weights/README.txt: -------------------------------------------------------------------------------- 1 | See s3://spacenet-dataset/spacenet-model-weights/spacenet-7/ -------------------------------------------------------------------------------- /2-cannab/cannab-Solution Description.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/2-cannab/cannab-Solution Description.docx -------------------------------------------------------------------------------- /2-cannab/code/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04 2 | 3 | # Use a fixed apt-get repo to stop intermittent failures due to flaky httpredir connections, 4 | # as described by Lionel Chan at http://stackoverflow.com/a/37426929/5881346 5 | RUN sed -i "s/httpredir.debian.org/debian.uchicago.edu/" /etc/apt/sources.list && \ 6 | apt-get update && apt-get install -y build-essential 7 | 8 | RUN apt-get update --fix-missing && apt-get install -y wget bzip2 ca-certificates \ 9 | libglib2.0-0 libxext6 libsm6 libxrender1 \ 10 | git mercurial subversion zip unzip 11 | 12 | # install Anaconda3 13 | ENV LANG=C.UTF-8 LC_ALL=C.UTF-8 14 | 15 | RUN echo 'export PATH=/opt/conda/bin:$PATH' > /etc/profile.d/conda.sh && \ 16 | wget --quiet https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh -O ~/anaconda.sh && \ 17 | /bin/bash ~/anaconda.sh -b -p /opt/conda && \ 18 | rm ~/anaconda.sh 19 | 20 | RUN apt-get install -y curl grep sed dpkg && \ 21 | TINI_VERSION=`curl https://github.com/krallin/tini/releases/latest | grep -o "/v.*\"" | sed 's:^..\(.*\).$:\1:'` && \ 22 | curl -L "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini_${TINI_VERSION}.deb" > tini.deb && \ 23 | dpkg -i tini.deb && \ 24 | rm tini.deb && \ 25 | apt-get clean 26 | 27 | ENV PATH /opt/conda/bin:$PATH 28 | 29 | RUN apt-get update && apt-get install -y libglu1 30 | 31 | RUN pip install opencv-python 32 | 33 | RUN conda install tqdm 34 | 35 | RUN conda install shapely 36 | 37 | RUN pip install imgaug 38 | 39 | RUN pip install geopandas 40 | 41 | RUN pip install rasterio 42 | 43 | RUN conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2 -c pytorch 44 | 45 | RUN pip install efficientnet_pytorch==0.6.3 46 | 47 | 48 | WORKDIR /work 49 | 50 | # copy entire directory where docker file is into docker container at /work 51 | COPY . /work/ 52 | 53 | RUN chmod 777 train.sh 54 | RUN chmod 777 test.sh 55 | RUN chmod 777 download_models.sh 56 | 57 | ENTRYPOINT [ "/usr/bin/tini", "--" ] 58 | CMD [ "/work/download_models.sh" ] -------------------------------------------------------------------------------- /2-cannab/code/adamw.py: -------------------------------------------------------------------------------- 1 | # Based on https://github.com/pytorch/pytorch/pull/3740 2 | import torch 3 | import math 4 | 5 | 6 | class AdamW(torch.optim.Optimizer): 7 | """Implements AdamW algorithm. 8 | 9 | It has been proposed in `Fixing Weight Decay Regularization in Adam`_. 10 | 11 | Arguments: 12 | params (iterable): iterable of parameters to optimize or dicts defining 13 | parameter groups 14 | lr (float, optional): learning rate (default: 1e-3) 15 | betas (Tuple[float, float], optional): coefficients used for computing 16 | running averages of gradient and its square (default: (0.9, 0.999)) 17 | eps (float, optional): term added to the denominator to improve 18 | numerical stability (default: 1e-8) 19 | weight_decay (float, optional): weight decay (L2 penalty) (default: 0) 20 | 21 | .. Fixing Weight Decay Regularization in Adam: 22 | https://arxiv.org/abs/1711.05101 23 | """ 24 | 25 | def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, #1e-8 2e-5 26 | weight_decay=0): 27 | defaults = dict(lr=lr, betas=betas, eps=eps, 28 | weight_decay=weight_decay) 29 | super(AdamW, self).__init__(params, defaults) 30 | 31 | def step(self, closure=None): 32 | """Performs a single optimization step. 33 | 34 | Arguments: 35 | closure (callable, optional): A closure that reevaluates the model 36 | and returns the loss. 37 | """ 38 | loss = None 39 | if closure is not None: 40 | loss = closure() 41 | 42 | for group in self.param_groups: 43 | for p in group['params']: 44 | if p.grad is None: 45 | continue 46 | grad = p.grad.data 47 | if grad.is_sparse: 48 | raise RuntimeError('AdamW does not support sparse gradients, please consider SparseAdam instead') 49 | 50 | state = self.state[p] 51 | 52 | # State initialization 53 | if len(state) == 0: 54 | state['step'] = 0 55 | # Exponential moving average of gradient values 56 | state['exp_avg'] = torch.zeros_like(p.data) 57 | # Exponential moving average of squared gradient values 58 | state['exp_avg_sq'] = torch.zeros_like(p.data) 59 | 60 | exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq'] 61 | beta1, beta2 = group['betas'] 62 | 63 | state['step'] += 1 64 | 65 | # according to the paper, this penalty should come after the bias correction 66 | # if group['weight_decay'] != 0: 67 | # grad = grad.add(group['weight_decay'], p.data) 68 | 69 | # Decay the first and second moment running average coefficient 70 | exp_avg.mul_(beta1).add_(1 - beta1, grad) 71 | exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad) 72 | 73 | denom = exp_avg_sq.sqrt().add_(group['eps']) 74 | 75 | bias_correction1 = 1 - beta1 ** state['step'] 76 | bias_correction2 = 1 - beta2 ** state['step'] 77 | step_size = group['lr'] * math.sqrt(bias_correction2) / bias_correction1 78 | 79 | # w = w - wd * lr * w 80 | if group['weight_decay'] != 0: 81 | p.data.add_(-group['weight_decay'] * group['lr'], p.data) 82 | 83 | # w = w - lr * w.grad 84 | p.data.addcdiv_(-step_size, exp_avg, denom) 85 | 86 | # w = w - wd * lr * w - lr * w.grad 87 | # See http://www.fast.ai/2018/07/02/adam-weight-decay/ 88 | 89 | return loss -------------------------------------------------------------------------------- /2-cannab/code/ddp_utils.py: -------------------------------------------------------------------------------- 1 | import pickle 2 | 3 | import torch 4 | import torch.distributed as dist 5 | from torch import Tensor 6 | 7 | 8 | def reduce_tensor(tensor): 9 | rt = tensor.clone() 10 | dist.all_reduce(rt, op=dist.ReduceOp.SUM) 11 | rt /= dist.get_world_size() 12 | return rt 13 | 14 | 15 | 16 | def all_gather(data): 17 | """ 18 | Run all_gather on arbitrary picklable data (not necessarily tensors) 19 | Args: 20 | data: any picklable object 21 | Returns: 22 | list[data]: list of data gathered from each rank 23 | """ 24 | world_size = dist.get_world_size() 25 | if world_size == 1: 26 | return [data] 27 | 28 | # serialized to a Tensor 29 | buffer = pickle.dumps(data) 30 | storage = torch.ByteStorage.from_buffer(buffer) 31 | tensor = torch.ByteTensor(storage).to("cuda") 32 | 33 | # obtain Tensor size of each rank 34 | local_size = torch.tensor([tensor.numel()], device="cuda") 35 | size_list = [torch.tensor([0], device="cuda") for _ in range(world_size)] 36 | dist.all_gather(size_list, local_size) 37 | size_list = [int(size.item()) for size in size_list] 38 | max_size = max(size_list) 39 | 40 | # receiving Tensor from all ranks 41 | # we pad the tensor because torch all_gather does not support 42 | # gathering tensors of different shapes 43 | tensor_list = [] 44 | for _ in size_list: 45 | tensor_list.append(torch.empty((max_size,), dtype=torch.uint8, device="cuda")) 46 | if local_size != max_size: 47 | padding = torch.empty(size=(max_size - local_size,), dtype=torch.uint8, device="cuda") 48 | tensor = torch.cat((tensor, padding), dim=0) 49 | dist.all_gather(tensor_list, tensor) 50 | 51 | data_list = [] 52 | for size, tensor in zip(size_list, tensor_list): 53 | buffer = tensor.cpu().numpy().tobytes()[:size] 54 | data_list.append(pickle.loads(buffer)) 55 | 56 | return data_list -------------------------------------------------------------------------------- /2-cannab/code/download_models.sh: -------------------------------------------------------------------------------- 1 | echo "Downloading model weights" 2 | wget https://vdurnov.s3.amazonaws.com/sn7_models.zip -O /wdata/sn7_models.zip 3 | unzip -o /wdata/sn7_models.zip -d /wdata/ 4 | /bin/bash -------------------------------------------------------------------------------- /2-cannab/code/flags.txt: -------------------------------------------------------------------------------- 1 | --gpus all --ipc=host -------------------------------------------------------------------------------- /2-cannab/code/notes.txt: -------------------------------------------------------------------------------- 1 | test.sh takes ~2 hours. 2 | 3 | train.sh takes ~1 day. 4 | 5 | All logs saved to /wdata/logs folder 6 | 7 | It's necessary to add '--ipc=host' option to run docker to test train.sh. Otherwise multithreaded pytorch's dataloader will crash. 8 | 9 | Use download.sh to download trained weight used for final submission. (Automaticly downloaded on docker start) -------------------------------------------------------------------------------- /2-cannab/code/test.sh: -------------------------------------------------------------------------------- 1 | mkdir -p foo /wdata/logs 2 | 3 | rm /wdata/test_pred_4k_double -r -f 4 | echo "Predicting NNs" 5 | python predict_double.py "$@" | tee /wdata/logs/predict_double.out 6 | 7 | echo "Generating JSON" 8 | python generate_json.py "$@" | tee /wdata/logs/generate_json.out 9 | 10 | echo "Tracking buildings" 11 | python track_buildings.py "$@" | tee /wdata/logs/track_buildings.out 12 | 13 | echo "Creating submission" 14 | python create_submission.py "$@" | tee /wdata/logs/create_submission.out 15 | 16 | echo "Submission created!" -------------------------------------------------------------------------------- /2-cannab/code/train.sh: -------------------------------------------------------------------------------- 1 | mkdir -p foo /wdata/logs 2 | 3 | echo "Creating masks..." 4 | python create_masks.py "$@" > /wdata/logs/create_masks.out 2>&1 5 | 6 | echo "Training b7..." 7 | python -m torch.distributed.launch --nproc_per_node=4 train_b7_full2.py --train_dir "$@" > /wdata/logs/train_b7_full2.out 2>&1 8 | python -m torch.distributed.launch --nproc_per_node=4 tune_b7_full.py --train_dir "$@" > /wdata/logs/tune_b7_full.out 2>&1 9 | python -m torch.distributed.launch --nproc_per_node=4 train_b7_double_full.py --train_dir "$@" > /wdata/logs/train_b7_double_full.out 2>&1 10 | 11 | echo "Training b6..." 12 | python -m torch.distributed.launch --nproc_per_node=4 train_b6_full2.py --train_dir "$@" > /wdata/logs/train_b6_full2.out 2>&1 13 | python -m torch.distributed.launch --nproc_per_node=4 tune_b6_full2.py --train_dir "$@" > /wdata/logs/tune_b6_full2.out 2>&1 14 | python -m torch.distributed.launch --nproc_per_node=4 train_b6_double_full.py --train_dir "$@" > /wdata/logs/train_b6_double_full.out 2>&1 15 | 16 | echo "All models trained!" -------------------------------------------------------------------------------- /2-cannab/code/zoo/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/2-cannab/code/zoo/__init__.py -------------------------------------------------------------------------------- /2-cannab/code/zoo/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/2-cannab/code/zoo/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /2-cannab/code/zoo/__pycache__/dpn.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/2-cannab/code/zoo/__pycache__/dpn.cpython-37.pyc -------------------------------------------------------------------------------- /2-cannab/code/zoo/__pycache__/models.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/2-cannab/code/zoo/__pycache__/models.cpython-37.pyc -------------------------------------------------------------------------------- /2-cannab/code/zoo/__pycache__/senet.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/2-cannab/code/zoo/__pycache__/senet.cpython-37.pyc -------------------------------------------------------------------------------- /2-cannab/weights/README.txt: -------------------------------------------------------------------------------- 1 | See s3://spacenet-dataset/spacenet-model-weights/spacenet-7/ -------------------------------------------------------------------------------- /3-selim_sef/code/.dockerignore: -------------------------------------------------------------------------------- 1 | venv/ 2 | weights/ 3 | logs/ -------------------------------------------------------------------------------- /3-selim_sef/code/configs/b6adam.json: -------------------------------------------------------------------------------- 1 | { 2 | "network": "timm_effnet_dragon", 3 | "encoder": "tf_efficientnet_b6_ns", 4 | "fp16": true, 5 | "use_last_decoder": true, 6 | "sigma": 2, 7 | "data_multiplier": 12, 8 | "num_classes": [3, 1], 9 | "optimizer": { 10 | "batch_size": 6, 11 | "type": "AdamW", 12 | "eps": 1e-5, 13 | "weight_decay": 5e-4, 14 | "learning_rate": 0.0001, 15 | "schedule": { 16 | "type": "cosine", 17 | "mode": "step", 18 | "epochs": 40, 19 | "params": { 20 | "T_max": 25400, 21 | "eta_min": 7e-6 22 | } 23 | } 24 | }, 25 | "input": 512, 26 | "mask_loss": { 27 | "type": "ComboLoss", 28 | "params": { 29 | "weights": { 30 | "dice": 1, 31 | "bce": 0.5, 32 | "focal": 1 33 | }, 34 | "channel_weights": [1, 0.1, 0.4] 35 | } 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /3-selim_sef/code/configs/b7adam.json: -------------------------------------------------------------------------------- 1 | { 2 | "network": "timm_effnet_dragon", 3 | "encoder": "tf_efficientnet_b7_ns", 4 | "fp16": true, 5 | "use_last_decoder": true, 6 | "sigma": 2, 7 | "data_multiplier": 10, 8 | "num_classes": [3, 1], 9 | "optimizer": { 10 | "batch_size": 5, 11 | "type": "AdamW", 12 | "eps": 5e-6, 13 | "weight_decay": 5e-4, 14 | "learning_rate": 0.0001, 15 | "schedule": { 16 | "type": "cosine", 17 | "mode": "step", 18 | "epochs": 40, 19 | "params": { 20 | "T_max": 25400, 21 | "eta_min": 7e-6 22 | } 23 | } 24 | }, 25 | "input": 512, 26 | "mask_loss": { 27 | "type": "ComboLoss", 28 | "params": { 29 | "weights": { 30 | "dice": 1, 31 | "bce": 0.5, 32 | "focal": 1 33 | }, 34 | "channel_weights": [1, 0.1, 0.4] 35 | } 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /3-selim_sef/code/ensemble.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | 4 | from tqdm import tqdm 5 | 6 | os.environ["MKL_NUM_THREADS"] = "1" 7 | os.environ["NUMEXPR_NUM_THREADS"] = "1" 8 | os.environ["OMP_NUM_THREADS"] = "1" 9 | 10 | from multiprocessing.pool import Pool 11 | 12 | import numpy as np 13 | import cv2 14 | cv2.setNumThreads(0) 15 | 16 | 17 | def average_strategy(images): 18 | return np.average(images, axis=0) 19 | 20 | 21 | def hard_voting(images): 22 | rounded = np.round(images / 255.) 23 | return np.round(np.sum(rounded, axis=0) / images.shape[0]) * 255. 24 | 25 | 26 | def ensemble_image(params): 27 | file, dirs, ensembling_dir, strategy = params 28 | images = [] 29 | for dir in dirs: 30 | file_path = os.path.join(dir, file) 31 | if "center" in file: 32 | im = cv2.imread(file_path, cv2.IMREAD_GRAYSCALE) 33 | else: 34 | im = cv2.imread(file_path, cv2.IMREAD_COLOR) 35 | images.append(im) 36 | images = np.array(images).astype(np.float32) 37 | 38 | if strategy == 'average': 39 | ensembled = average_strategy(images) 40 | elif strategy == 'hard_voting': 41 | ensembled = hard_voting(images) 42 | else: 43 | raise ValueError('Unknown ensembling strategy') 44 | cv2.imwrite(os.path.join(ensembling_dir, file), ensembled) 45 | 46 | 47 | def ensemble(dirs, strategy, ensembling_dir, n_threads): 48 | files = os.listdir(dirs[0]) 49 | params = [] 50 | 51 | for file in files: 52 | params.append((file, dirs, ensembling_dir, strategy)) 53 | with Pool(n_threads) as pool: 54 | with tqdm(total=len(params)) as pbar: 55 | for _ in pool.imap_unordered(ensemble_image, params): 56 | pbar.update() 57 | 58 | if __name__ == '__main__': 59 | parser = argparse.ArgumentParser("Ensemble masks") 60 | arg = parser.add_argument 61 | arg('--ensembling_cpu_threads', type=int, default=24) 62 | arg('--ensembling_dir', type=str, default='../test_results/ensemble') 63 | arg('--strategy', type=str, default='average') 64 | arg('--folds_dir', type=str, default='../test_results/folds') 65 | args = parser.parse_args() 66 | 67 | folds_dir = args.folds_dir 68 | dirs = [os.path.join(folds_dir, d) for d in os.listdir(folds_dir)] 69 | print(list(os.listdir(folds_dir))) 70 | for d in dirs: 71 | if not os.path.exists(d): 72 | raise ValueError(d + " doesn't exist") 73 | os.makedirs(args.ensembling_dir, exist_ok=True) 74 | ensemble(dirs, args.strategy, args.ensembling_dir, args.ensembling_cpu_threads) 75 | -------------------------------------------------------------------------------- /3-selim_sef/code/flags.txt: -------------------------------------------------------------------------------- 1 | --ipc=host --shm-size=8G --gpus=all --net=host -------------------------------------------------------------------------------- /3-selim_sef/code/postprocessing/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/3-selim_sef/code/postprocessing/__init__.py -------------------------------------------------------------------------------- /3-selim_sef/code/postprocessing/labeling.py: -------------------------------------------------------------------------------- 1 | import warnings 2 | 3 | import numpy as np 4 | from skimage import measure 5 | from skimage.segmentation import watershed 6 | 7 | warnings.simplefilter("ignore") 8 | def label_mask(pred, main_threshold=0.4, seed_threshold=0.75, w_pixel_t=50, pixel_t=90): 9 | 10 | av_pred = pred 11 | av_pred = av_pred[..., 0] * (1 - 0.7 * av_pred[..., 2]) * (1 - 0.2 * av_pred[..., 1]) 12 | av_pred = 1 * (av_pred > seed_threshold) 13 | av_pred = av_pred.astype(np.uint8) 14 | 15 | y_pred = measure.label(av_pred, neighbors=8, background=0) 16 | props = measure.regionprops(y_pred) 17 | for i in range(len(props)): 18 | if props[i].area < w_pixel_t: 19 | y_pred[y_pred == i + 1] = 0 20 | y_pred = measure.label(y_pred, neighbors=8, background=0) 21 | 22 | nucl_msk = (255 - pred[..., 0] * 255) 23 | nucl_msk = nucl_msk.astype('uint8') 24 | y_pred = watershed(nucl_msk, y_pred, mask=(pred[..., 0] > main_threshold), watershed_line=True, compactness=3) 25 | 26 | props = measure.regionprops(y_pred) 27 | 28 | for i in range(len(props)): 29 | if props[i].area < pixel_t: 30 | y_pred[y_pred == i + 1] = 0 31 | y_pred = measure.label(y_pred, neighbors=8, background=0) 32 | return y_pred -------------------------------------------------------------------------------- /3-selim_sef/code/postprocessing/polygonize.py: -------------------------------------------------------------------------------- 1 | from typing import Dict 2 | 3 | import numpy as np 4 | import rasterio 5 | import rasterio.features 6 | import shapely 7 | import shapely.affinity 8 | import shapely.geometry 9 | import shapely.ops 10 | import shapely.wkt 11 | from shapely.geometry import MultiPolygon 12 | 13 | 14 | def mask_to_poly(mask: np.ndarray) -> Dict: 15 | """Generates geojson polygon from a single instance binary mask 16 | Parameters 17 | ---------- 18 | mask : binary masks representing single object 19 | 20 | Returns 21 | ------- 22 | geo : dict 23 | Geo polygon as dict 24 | """ 25 | mp = mask_to_shapely_polygon(mask) 26 | geo = shapely.geometry.mapping(mp) 27 | 28 | return geo 29 | 30 | 31 | def mask_to_shapely_polygon(mask: np.ndarray) -> MultiPolygon: 32 | """Generates shapely polygon from a single instance binary mask 33 | Parameters 34 | ---------- 35 | mask : binary masks representing single object 36 | 37 | Returns 38 | ------- 39 | 40 | mp : shapely.geometry.MultiPolygon 41 | Geo polygon as shapely object 42 | """ 43 | 44 | shapes = rasterio.features.shapes(mask.astype(np.int16), mask > 0) 45 | mp = shapely.ops.cascaded_union( 46 | shapely.geometry.MultiPolygon([ 47 | shapely.geometry.shape(shape) 48 | for shape, value in shapes 49 | ])) 50 | 51 | return mp 52 | -------------------------------------------------------------------------------- /3-selim_sef/code/predict_test.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | import re 4 | import warnings 5 | 6 | from apex import amp 7 | 8 | import zoo 9 | from training.config import load_config 10 | from training.instance_datasets import SpacenetTestDataset 11 | 12 | os.environ["MKL_NUM_THREADS"] = "1" 13 | os.environ["NUMEXPR_NUM_THREADS"] = "1" 14 | os.environ["OMP_NUM_THREADS"] = "1" 15 | 16 | import cv2 17 | 18 | cv2.ocl.setUseOpenCL(False) 19 | cv2.setNumThreads(0) 20 | import numpy as np 21 | import torch 22 | from torch.utils.data import DataLoader 23 | from tqdm import tqdm 24 | 25 | warnings.simplefilter("ignore") 26 | 27 | 28 | def load_model(config_path, weights_path): 29 | conf = load_config(config_path) 30 | model = zoo.__dict__[conf['network']](seg_classes=conf["num_classes"], backbone_arch=conf['encoder']) 31 | print("=> loading checkpoint '{}'".format(weights_path)) 32 | checkpoint = torch.load(weights_path, map_location="cpu") 33 | print("best_dice", checkpoint['dice_best']) 34 | print("epoch", checkpoint['epoch']) 35 | state_dict = {re.sub("^module.", "", k): w for k, w in checkpoint['state_dict'].items()} 36 | model.load_state_dict(state_dict) 37 | model.eval() 38 | return model.cuda() 39 | 40 | 41 | if __name__ == '__main__': 42 | parser = argparse.ArgumentParser("Spacenet Test Predictor") 43 | arg = parser.add_argument 44 | arg('--config', metavar='CONFIG_FILE', default='configs/b7adam.json', help='path to configuration file') 45 | arg('--data-path', type=str, default='/mnt/datasets/spacenet/test_public/', 46 | help='Path to test images') 47 | arg('--gpu', type=str, default='0', help='List of GPUs for parallel training, e.g. 0,1,2,3') 48 | arg('--dir', type=str, default='../test_results/3k/b70') 49 | arg('--model', type=str, default='weights/3k_mask_center_timm_effnet_dragon_tf_efficientnet_b7_ns_0_best_dice') 50 | 51 | args = parser.parse_args() 52 | os.makedirs(args.dir, exist_ok=True) 53 | 54 | os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID' 55 | os.environ["CUDA_VISIBLE_DEVICES"] = args.gpu 56 | 57 | model = load_model(args.config, args.model) 58 | model = amp.initialize(model) 59 | 60 | model = torch.nn.DataParallel(model).cuda() 61 | 62 | data_val = SpacenetTestDataset(data_path=args.data_path, size=3072) 63 | val_data_loader = DataLoader(data_val, batch_size=1, num_workers=8, 64 | shuffle=False, 65 | pin_memory=False) 66 | with torch.no_grad(): 67 | for sample in tqdm(val_data_loader): 68 | img_name = sample["img_name"][0] 69 | imgs = sample["image"].cpu().float() 70 | output, center, *_ = model(imgs) 71 | output = torch.sigmoid(output) 72 | 73 | o, c, *_ = model(torch.flip(imgs, dims=[2])) 74 | o = torch.sigmoid(o) 75 | output += torch.flip(o, dims=[2]) 76 | center += torch.flip(c, dims=[2]) 77 | 78 | o, c, *_ = model(torch.flip(imgs, dims=[3])) 79 | o = torch.sigmoid(o) 80 | output += torch.flip(o, dims=[3]) 81 | center += torch.flip(c, dims=[3]) 82 | 83 | o, c, *_ = model(torch.flip(imgs, dims=[2, 3])) 84 | o = torch.sigmoid(o) 85 | output += torch.flip(o, dims=[2, 3]) 86 | center += torch.flip(c, dims=[2, 3]) 87 | 88 | output /= 4 89 | center /= 4 90 | center = np.clip(center.cpu().numpy()[0][0], 0, 1) 91 | binary_pred = np.moveaxis(output.cpu().numpy()[0], 0, -1) 92 | cv2.imwrite(os.path.join(args.dir, sample["img_name"][0] + "_mask.png"), binary_pred * 255) 93 | cv2.imwrite(os.path.join(args.dir, sample["img_name"][0] + "_centers.png"), center * 255) -------------------------------------------------------------------------------- /3-selim_sef/code/preprocessing/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/3-selim_sef/code/preprocessing/__init__.py -------------------------------------------------------------------------------- /3-selim_sef/code/preprocessing/generate_folds.py: -------------------------------------------------------------------------------- 1 | import os 2 | import random 3 | from glob import glob 4 | from random import shuffle 5 | 6 | import cv2 7 | import pandas as pd 8 | import skimage.io 9 | train_dir = "/mnt/datasets/spacenet/train/" 10 | groups = [g for g in os.listdir(train_dir) if "L15" in g] 11 | random.seed(777) 12 | shuffle(groups) 13 | data = [] 14 | for i, group in enumerate(groups): 15 | for f in os.listdir(os.path.join(train_dir, group, "images_masked")): 16 | if not os.path.exists(os.path.join(train_dir, "masks", f.replace(".tif", "_Buildings.png"))): 17 | continue 18 | if f.endswith("tif"): 19 | id = f.replace(".tif", "") 20 | for y in range(2): 21 | for x in range(2): 22 | data.append([id + "_{}_{}".format(y, x), i//6]) 23 | pd.DataFrame(data, columns=["id", "fold"]).to_csv("../folds.csv", index=False) -------------------------------------------------------------------------------- /3-selim_sef/code/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA=$1 4 | OUT_FILE=$2 5 | 6 | echo "Predicting masks..." 7 | python predict_test.py --gpu 0 --data-path $DATA --dir /wdata/predictions/1 --config configs/b7adam.json \ 8 | --model weights/run1_mask_center_timm_effnet_dragon_tf_efficientnet_b7_ns_0_best_dice & 9 | 10 | python predict_test.py --gpu 1 --data-path $DATA --dir /wdata/predictions/2 --config configs/b7adam.json \ 11 | --model weights/run2_mask_center_timm_effnet_dragon_tf_efficientnet_b7_ns_0_best_dice & 12 | 13 | python predict_test.py --gpu 2 --data-path $DATA --dir /wdata/predictions/3 --config configs/b7adam.json \ 14 | --model weights/run3_mask_center_timm_effnet_dragon_tf_efficientnet_b7_ns_0_best_dice & 15 | 16 | python predict_test.py --gpu 3 --data-path $DATA --dir /wdata/predictions/b61 --config configs/b6adam.json \ 17 | --model weights/run1_mask_center_timm_effnet_dragon_tf_efficientnet_b6_ns_0_best_dice & 18 | 19 | wait 20 | echo "Ensembling masks..." 21 | python ensemble.py --folds_dir /wdata/predictions/ --ensembling_dir /wdata/ensemble 22 | echo "Generating solution..." 23 | 24 | python generate_submit.py --preds-dir /wdata/ensemble/ \ 25 | --preds-dir /wdata/ensemble \ 26 | --workers 24 \ 27 | --json-dir /wdata/geojsons \ 28 | --out-csv-dir /wdata/csvs \ 29 | --image-dir $DATA \ 30 | --out-file $OUT_FILE 31 | 32 | echo "Done!!!" -------------------------------------------------------------------------------- /3-selim_sef/code/train.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA=$1 4 | GPUS=4 5 | mkdir -p /wdata/train_data/oof_preds 6 | LOG_DIR=/wdata/logs/ 7 | mkdir -p $LOG_DIR 8 | 9 | 10 | echo "Preparing masks..." 11 | python preprocessing/masks.py --train-dir $DATA --out-dir /wdata/train_data/masks --workers 24 12 | 13 | echo "Resize train images and split if needed..." 14 | python preprocessing/split_images.py --train-dir $DATA --out-dir /wdata/train_data/ --workers 24 15 | 16 | echo "Training B7 run 1..." 17 | PYTHONPATH=. python -u -m torch.distributed.launch --nproc_per_node=$GPUS \ 18 | --master_port 9901 training/train_instance_no_offset.py --distributed \ 19 | --data-dir /wdata/train_data/ --config configs/b7adam.json --predictions /wdata/train_data/oof_preds \ 20 | --workers 8 --opt-level O1 --freeze-epochs 2 --prefix run1_mask_center_ --fold 0 > $LOG_DIR/run1 & 21 | wait 22 | echo "Training B7 run 2..." 23 | PYTHONPATH=. python -u -m torch.distributed.launch --nproc_per_node=$GPUS \ 24 | --master_port 9901 training/train_instance_no_offset.py --distributed \ 25 | --data-dir /wdata/train_data/ --config configs/b7adam.json --predictions /wdata/train_data/oof_preds \ 26 | --workers 8 --opt-level O1 --freeze-epochs 2 --prefix run2_mask_center_ --fold 0 > $LOG_DIR/run2 & 27 | wait 28 | 29 | echo "Training B7 run 3..." 30 | PYTHONPATH=. python -u -m torch.distributed.launch --nproc_per_node=$GPUS \ 31 | --master_port 9901 training/train_instance_no_offset.py --distributed \ 32 | --data-dir /wdata/train_data/ --config configs/b7adam.json --predictions /wdata/train_data/oof_preds \ 33 | --workers 8 --opt-level O1 --freeze-epochs 2 --prefix run3_mask_center_ --fold 0 > $LOG_DIR/run3 & 34 | wait 35 | 36 | echo "Training B6 run 1..." 37 | PYTHONPATH=. python -u -m torch.distributed.launch --nproc_per_node=$GPUS \ 38 | --master_port 9901 training/train_instance_no_offset.py --distributed \ 39 | --data-dir /wdata/train_data/ --config configs/b6adam.json --predictions /wdata/train_data/oof_preds \ 40 | --workers 8 --opt-level O1 --freeze-epochs 2 --prefix run1_mask_center_ --fold 0 > $LOG_DIR/run1b6 & 41 | wait 42 | echo "Completed!!!" 43 | -------------------------------------------------------------------------------- /3-selim_sef/code/training/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/3-selim_sef/code/training/__init__.py -------------------------------------------------------------------------------- /3-selim_sef/code/training/augmentations.py: -------------------------------------------------------------------------------- 1 | import albumentations as A 2 | import cv2 3 | from albumentations import RandomCrop, CenterCrop 4 | 5 | from training.transforms import RandomSizedCropAroundBbox 6 | 7 | 8 | class InstanceAugmentations: 9 | def __init__(self): 10 | super().__init__() 11 | 12 | def create_train_transforms(self, crop_size): 13 | transforms = [ 14 | A.OneOf( 15 | [ 16 | RandomCrop(crop_size, crop_size, p=0.3), 17 | RandomSizedCropAroundBbox(min_max_height=(int(crop_size * 0.65), int(crop_size * 1.4)), 18 | height=crop_size, 19 | width=crop_size, p=0.7) 20 | ], p=1), 21 | A.Rotate(20, p=0.2, border_mode=cv2.BORDER_CONSTANT), 22 | A.HorizontalFlip(), 23 | A.VerticalFlip(), 24 | A.RandomRotate90(), 25 | A.RandomBrightnessContrast(), 26 | A.RandomGamma(), 27 | A.FancyPCA(p=0.2) 28 | ] 29 | return A.Compose(transforms, additional_targets={'labels': 'mask'}) 30 | 31 | def create_val_transforms(self): 32 | transforms = [ 33 | CenterCrop(2048, 2048) 34 | ] 35 | return A.Compose(transforms) 36 | -------------------------------------------------------------------------------- /3-selim_sef/code/training/config.py: -------------------------------------------------------------------------------- 1 | import json 2 | 3 | DEFAULTS = { 4 | "network": "dpn", 5 | "encoder": "dpn92", 6 | "model_params": {}, 7 | "optimizer": { 8 | "batch_size": 32, 9 | "type": "SGD", # supported: SGD, Adam 10 | "momentum": 0.9, 11 | "weight_decay": 0, 12 | "clip": 1., 13 | "learning_rate": 0.1, 14 | "classifier_lr": -1, 15 | "nesterov": True, 16 | "schedule": { 17 | "type": "constant", # supported: constant, step, multistep, exponential, linear, poly 18 | "mode": "epoch", # supported: epoch, step 19 | "epochs": 10, 20 | "params": {} 21 | } 22 | }, 23 | "normalize": { 24 | "mean": [0.485, 0.456, 0.406], 25 | "std": [0.229, 0.224, 0.225] 26 | } 27 | } 28 | 29 | 30 | def _merge(src, dst): 31 | for k, v in src.items(): 32 | if k in dst: 33 | if isinstance(v, dict): 34 | _merge(src[k], dst[k]) 35 | else: 36 | dst[k] = v 37 | 38 | 39 | def load_config(config_file, defaults=DEFAULTS): 40 | with open(config_file, "r") as fd: 41 | config = json.load(fd) 42 | _merge(defaults, config) 43 | return config 44 | -------------------------------------------------------------------------------- /3-selim_sef/code/training/datasets.py: -------------------------------------------------------------------------------- 1 | import os 2 | import traceback 3 | 4 | import cv2 5 | import numpy as np 6 | import pandas as pd 7 | import skimage.io 8 | import torch 9 | from torch.utils.data import Dataset 10 | from torchvision.transforms.functional import normalize 11 | 12 | 13 | class SpacenetLocDataset(Dataset): 14 | def __init__(self, data_path, mode, fold=0, folds_csv='folds.csv', transforms=None, 15 | normalize={"mean": [0.485, 0.456, 0.406], 16 | "std": [0.229, 0.224, 0.225]}, multiplier=1): 17 | super().__init__() 18 | self.data_path = data_path 19 | self.mode = mode 20 | 21 | df = pd.read_csv(folds_csv) 22 | self.df = df 23 | self.normalize = normalize 24 | self.fold = fold 25 | if self.mode == "train": 26 | ids = df[df['fold'] != fold]['id'].tolist() 27 | else: 28 | ids = sorted(list(set(df[(df['fold'] == fold)]['id'].tolist()))) 29 | self.transforms = transforms 30 | self.names = ids 31 | if mode == "train": 32 | self.names = self.names * multiplier 33 | print("names ", len(self.names)) 34 | 35 | def __len__(self): 36 | return len(self.names) 37 | 38 | def __getitem__(self, idx): 39 | name = self.names[idx] 40 | group = name.split("mosaic_")[-1][:-4] 41 | img_path = os.path.join(self.data_path, group, "images_masked_split", name + ".tif") 42 | 43 | try: 44 | image = cv2.imread(img_path, cv2.IMREAD_UNCHANGED) 45 | alpha = image[..., 3] 46 | image = image[..., :3] 47 | image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) 48 | 49 | mask_path = os.path.join(self.data_path, "masks_split", name + ".png") 50 | mask = cv2.imread(mask_path) 51 | mask = cv2.cvtColor(mask, cv2.COLOR_BGR2RGB) 52 | mask[alpha == 0, :] = 0 53 | except: 54 | traceback.print_exc() 55 | print(img_path) 56 | sample = self.transforms(image=image, mask=mask) 57 | sample['img_name'] = name 58 | sample['mask'] = torch.from_numpy(np.ascontiguousarray(np.moveaxis(sample["mask"], -1, 0))).float() / 255. 59 | image = torch.from_numpy(np.moveaxis(sample["image"], -1, 0)).float() / 255 60 | image = normalize(image, **self.normalize) 61 | sample['image'] = image 62 | return sample 63 | -------------------------------------------------------------------------------- /3-selim_sef/code/training/metrics.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | 4 | def miou_score(y, p): 5 | return calc_score(np.expand_dims(y, 0), np.expand_dims(p, 0)) 6 | 7 | 8 | def f_score(tp, fp, fn): 9 | precision = tp / (tp + fp) 10 | recall = tp / (tp + fn) 11 | f_score = 2 * precision * recall / (precision + recall) 12 | return f_score 13 | 14 | def calc_score(labels, y_pred): 15 | if y_pred.sum() == 0 and labels.sum() == 0: 16 | return 1. 17 | if labels.sum() == 0 and y_pred.sum() > 0 or y_pred.sum() == 0 and labels.sum() > 0: 18 | return 0. 19 | 20 | true_objects = len(np.unique(labels)) 21 | pred_objects = len(np.unique(y_pred)) 22 | 23 | intersection = np.histogram2d(labels.flatten(), y_pred.flatten(), bins=(true_objects, pred_objects))[0] 24 | 25 | # Compute areas (needed for finding the union between all objects) 26 | area_true = np.histogram(labels, bins=true_objects)[0] 27 | area_pred = np.histogram(y_pred, bins=pred_objects)[0] 28 | area_true = np.expand_dims(area_true, -1) 29 | area_pred = np.expand_dims(area_pred, 0) 30 | 31 | # Compute union 32 | union = area_true + area_pred - intersection 33 | 34 | # Exclude background from the analysis 35 | intersection = intersection[1:, 1:] 36 | union = union[1:, 1:] 37 | union[union == 0] = 1e-9 38 | 39 | # Compute the intersection over union 40 | iou = intersection / union 41 | return precision_at(0.25, iou) 42 | 43 | 44 | # Precision helper function 45 | def precision_at(threshold, iou): 46 | matches = iou > threshold 47 | true_positives = np.sum(matches, axis=1) == 1 # Correct objects 48 | false_positives = np.sum(matches, axis=0) == 0 # Missed objects 49 | false_negatives = np.sum(matches, axis=1) == 0 # Extra objects 50 | tp, fp, fn = np.sum(true_positives), np.sum(false_positives), np.sum(false_negatives) 51 | return tp, fp, fn 52 | -------------------------------------------------------------------------------- /3-selim_sef/code/training/schedulers.py: -------------------------------------------------------------------------------- 1 | from bisect import bisect_right 2 | 3 | from torch.optim.lr_scheduler import _LRScheduler 4 | 5 | 6 | class LRStepScheduler(_LRScheduler): 7 | def __init__(self, optimizer, steps, last_epoch=-1): 8 | self.lr_steps = steps 9 | super().__init__(optimizer, last_epoch) 10 | 11 | def get_lr(self): 12 | pos = max(bisect_right([x for x, y in self.lr_steps], self.last_epoch) - 1, 0) 13 | return [self.lr_steps[pos][1] if self.lr_steps[pos][0] <= self.last_epoch else base_lr for base_lr in self.base_lrs] 14 | 15 | 16 | class PolyLR(_LRScheduler): 17 | """Sets the learning rate of each parameter group according to poly learning rate policy 18 | """ 19 | def __init__(self, optimizer, max_iter=90000, power=0.9, last_epoch=-1): 20 | self.max_iter = max_iter 21 | self.power = power 22 | super(PolyLR, self).__init__(optimizer, last_epoch) 23 | 24 | def get_lr(self): 25 | self.last_epoch = (self.last_epoch + 1) % self.max_iter 26 | return [base_lr * ((1 - float(self.last_epoch) / self.max_iter) ** (self.power)) for base_lr in self.base_lrs] 27 | 28 | class ExponentialLRScheduler(_LRScheduler): 29 | """Decays the learning rate of each parameter group by gamma every epoch. 30 | When last_epoch=-1, sets initial lr as lr. 31 | 32 | Args: 33 | optimizer (Optimizer): Wrapped optimizer. 34 | gamma (float): Multiplicative factor of learning rate decay. 35 | last_epoch (int): The index of last epoch. Default: -1. 36 | """ 37 | 38 | def __init__(self, optimizer, gamma, last_epoch=-1): 39 | self.gamma = gamma 40 | super(ExponentialLRScheduler, self).__init__(optimizer, last_epoch) 41 | 42 | def get_lr(self): 43 | if self.last_epoch <= 0: 44 | return self.base_lrs 45 | return [base_lr * self.gamma**self.last_epoch for base_lr in self.base_lrs] 46 | 47 | -------------------------------------------------------------------------------- /3-selim_sef/code/training/transforms.py: -------------------------------------------------------------------------------- 1 | import random 2 | import time 3 | 4 | import albumentations as A 5 | 6 | 7 | class RandomSizedCropAroundBbox(A.RandomSizedCrop): 8 | @property 9 | def targets_as_params(self): 10 | return ['rectangles', 'image'] 11 | 12 | def get_params_dependent_on_targets(self, params): 13 | rectangles = params['rectangles'] 14 | img_height, img_width = params['image'].shape[:2] 15 | rm = random.Random() 16 | rm.seed(time.time_ns()) 17 | crop_height = rm.randint(self.min_max_height[0], self.min_max_height[1]) 18 | crop_width = int(crop_height * self.w2h_ratio) 19 | 20 | if rectangles: 21 | x, y, w, h = rm.choice(rectangles) 22 | min_x_start = max(x + (w / 2 if w >= crop_width else w) - crop_width, 0) 23 | min_y_start = max(y + (h / 2 if h >= crop_height else h) - crop_height, 0) 24 | max_x_start = min(x + (w / 2 if w >= crop_width else 0), img_width - crop_width) 25 | max_y_start = min(y + (h / 2 if h >= crop_height else 0), img_height - crop_height) 26 | if max_x_start < min_x_start: 27 | min_x_start, max_x_start = max_x_start, min_x_start 28 | if max_y_start < min_y_start: 29 | min_y_start, max_y_start = max_y_start, min_y_start 30 | start_y = rm.randint(int(min_y_start), int(max_y_start)) / img_height 31 | start_x = rm.randint(int(min_x_start), int(max_x_start)) / img_width 32 | else: 33 | start_y = rm.random() 34 | start_x = rm.random() 35 | return {'h_start': (start_y * img_height) / (img_height - crop_height), 36 | 'w_start': (start_x * img_width) / (img_width - crop_width), 37 | 'crop_height': crop_height, 38 | 'crop_width': int(crop_height * self.w2h_ratio)} -------------------------------------------------------------------------------- /3-selim_sef/code/zoo/__init__.py: -------------------------------------------------------------------------------- 1 | from zoo.unet import * -------------------------------------------------------------------------------- /3-selim_sef/selim_sef - Solution Description .docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/3-selim_sef/selim_sef - Solution Description .docx -------------------------------------------------------------------------------- /3-selim_sef/weights/README.txt: -------------------------------------------------------------------------------- 1 | See s3://spacenet-dataset/spacenet-model-weights/spacenet-7/ -------------------------------------------------------------------------------- /4-motokimura/code/configs/test_to_val_images.yml: -------------------------------------------------------------------------------- 1 | PREDICTION_ROOT: '/val_predictions' 2 | ENSEMBLED_PREDICTION_ROOT: '/val_ensembled_predictions' 3 | REFINED_PREDICTION_ROOT: '/val_refined_predictions' 4 | POLY_ROOT: '/val_polygons' 5 | TRACKED_POLY_ROOT: '/val_tracked_polygons' 6 | TEST_TO_VAL: True 7 | -------------------------------------------------------------------------------- /4-motokimura/code/configs/tta_resize.yml: -------------------------------------------------------------------------------- 1 | TTA: 2 | RESIZE: [[928, 928], [1152, 1152]] 3 | RESIZE_WEIGHTS: [0.125, 0.25] 4 | 5 | -------------------------------------------------------------------------------- /4-motokimura/code/configs/tta_resize_flip.yml: -------------------------------------------------------------------------------- 1 | TTA: 2 | RESIZE: [[928, 928], [1152, 1152]] 3 | RESIZE_WEIGHTS: [0.125, 0.25] 4 | HORIZONTAL_FLIP: True 5 | HORIZONTAL_FLIP_WEIGHT: 0.25 6 | VERTICAL_FLIP: True 7 | VERTICAL_FLIP_WEIGHT: 0.25 8 | -------------------------------------------------------------------------------- /4-motokimura/code/configs/unet-scse_timm-efficientnet-b3_v_01.yml: -------------------------------------------------------------------------------- 1 | MODEL: 2 | ARCHITECTURE: unet 3 | BACKBONE: 'timm-efficientnet-b3' 4 | UNET_ENABLE_DECODER_SCSE: True 5 | DATALOADER: 6 | TRAIN_BATCH_SIZE: 16 7 | VAL_BATCH_SIZE: 12 8 | TEST_BATCH_SIZE: 12 9 | TRAIN_NUM_WORKERS: 8 10 | VAL_NUM_WORKERS: 8 11 | TEST_NUM_WORKERS: 8 -------------------------------------------------------------------------------- /4-motokimura/code/configs/unet_timm-efficientnet-b0_scale-4.0_v_01.yml: -------------------------------------------------------------------------------- 1 | MODEL: 2 | ARCHITECTURE: unet 3 | BACKBONE: 'timm-efficientnet-b0' 4 | DATALOADER: 5 | TRAIN_BATCH_SIZE: 8 6 | VAL_BATCH_SIZE: 1 7 | TEST_BATCH_SIZE: 1 8 | TRAIN_NUM_WORKERS: 8 9 | VAL_NUM_WORKERS: 8 10 | TEST_NUM_WORKERS: 8 11 | TRANSFORM: 12 | SIZE_SCALE: 4.0 13 | TRAIN_RANDOM_CROP_SIZE: (160, 160) 14 | -------------------------------------------------------------------------------- /4-motokimura/code/configs/unet_timm-efficientnet-b3_scale-2.0_v_01.yml: -------------------------------------------------------------------------------- 1 | MODEL: 2 | ARCHITECTURE: unet 3 | BACKBONE: 'timm-efficientnet-b3' 4 | DATALOADER: 5 | TRAIN_BATCH_SIZE: 12 6 | VAL_BATCH_SIZE: 4 7 | TEST_BATCH_SIZE: 4 8 | TRAIN_NUM_WORKERS: 8 9 | VAL_NUM_WORKERS: 8 10 | TEST_NUM_WORKERS: 8 11 | TRANSFORM: 12 | SIZE_SCALE: 2.0 13 | -------------------------------------------------------------------------------- /4-motokimura/code/configs/unet_timm-efficientnet-b3_scale-3.0_v_01.yml: -------------------------------------------------------------------------------- 1 | MODEL: 2 | ARCHITECTURE: unet 3 | BACKBONE: 'timm-efficientnet-b3' 4 | DATALOADER: 5 | TRAIN_BATCH_SIZE: 8 6 | VAL_BATCH_SIZE: 1 7 | TEST_BATCH_SIZE: 1 8 | TRAIN_NUM_WORKERS: 8 9 | VAL_NUM_WORKERS: 8 10 | TEST_NUM_WORKERS: 8 11 | TRANSFORM: 12 | SIZE_SCALE: 3.0 13 | TRAIN_RANDOM_CROP_SIZE: (160, 160) 14 | -------------------------------------------------------------------------------- /4-motokimura/code/configs/unet_timm-efficientnet-b3_scale-3.0_v_02.yml: -------------------------------------------------------------------------------- 1 | MODEL: 2 | ARCHITECTURE: unet 3 | BACKBONE: 'timm-efficientnet-b3' 4 | SOLVER: 5 | LOSSES: ['dice', 'focal'] 6 | DATALOADER: 7 | TRAIN_BATCH_SIZE: 8 8 | VAL_BATCH_SIZE: 1 9 | TEST_BATCH_SIZE: 1 10 | TRAIN_NUM_WORKERS: 8 11 | VAL_NUM_WORKERS: 8 12 | TEST_NUM_WORKERS: 8 13 | TRANSFORM: 14 | SIZE_SCALE: 3.0 15 | TRAIN_RANDOM_CROP_SIZE: (160, 160) 16 | -------------------------------------------------------------------------------- /4-motokimura/code/configs/unet_timm-efficientnet-b3_scale-4.0_v_01.yml: -------------------------------------------------------------------------------- 1 | MODEL: 2 | ARCHITECTURE: unet 3 | BACKBONE: 'timm-efficientnet-b3' 4 | DATALOADER: 5 | TRAIN_BATCH_SIZE: 8 6 | VAL_BATCH_SIZE: 1 7 | TEST_BATCH_SIZE: 1 8 | TRAIN_NUM_WORKERS: 8 9 | VAL_NUM_WORKERS: 8 10 | TEST_NUM_WORKERS: 8 11 | TRANSFORM: 12 | SIZE_SCALE: 4.0 13 | TRAIN_RANDOM_CROP_SIZE: (160, 160) 14 | -------------------------------------------------------------------------------- /4-motokimura/code/configs/unet_timm-efficientnet-b3_scale-4.0_v_02.yml: -------------------------------------------------------------------------------- 1 | MODEL: 2 | ARCHITECTURE: unet 3 | BACKBONE: 'timm-efficientnet-b3' 4 | SOLVER: 5 | LOSSES: ['dice', 'focal'] 6 | DATALOADER: 7 | TRAIN_BATCH_SIZE: 8 8 | VAL_BATCH_SIZE: 1 9 | TEST_BATCH_SIZE: 1 10 | TRAIN_NUM_WORKERS: 8 11 | VAL_NUM_WORKERS: 8 12 | TEST_NUM_WORKERS: 8 13 | TRANSFORM: 14 | SIZE_SCALE: 4.0 15 | TRAIN_RANDOM_CROP_SIZE: (160, 160) 16 | -------------------------------------------------------------------------------- /4-motokimura/code/configs/unet_timm-efficientnet-b5_v_01.yml: -------------------------------------------------------------------------------- 1 | MODEL: 2 | ARCHITECTURE: unet 3 | BACKBONE: 'timm-efficientnet-b5' 4 | DATALOADER: 5 | TRAIN_BATCH_SIZE: 16 6 | VAL_BATCH_SIZE: 16 7 | TEST_BATCH_SIZE: 16 8 | TRAIN_NUM_WORKERS: 8 9 | VAL_NUM_WORKERS: 8 10 | TEST_NUM_WORKERS: 8 -------------------------------------------------------------------------------- /4-motokimura/code/docker/Dockerfile.dev: -------------------------------------------------------------------------------- 1 | # Adapted from: https://github.com/CosmiQ/CosmiQ_SN7_Baseline/blob/master/docker/Dockerfile 2 | 3 | FROM nvidia/cuda:10.2-devel-ubuntu16.04 4 | LABEL maintainer="motokimura " 5 | 6 | ENV CUDNN_VERSION 7.3.0.29 7 | LABEL com.nvidia.cudnn.version="${CUDNN_VERSION}" 8 | ARG solaris_branch='master' 9 | 10 | # prep apt-get and cudnn 11 | RUN apt-get update && apt-get install -y --no-install-recommends \ 12 | apt-utils \ 13 | libcudnn7=$CUDNN_VERSION-1+cuda9.0 \ 14 | libcudnn7-dev=$CUDNN_VERSION-1+cuda9.0 && \ 15 | apt-mark hold libcudnn7 && \ 16 | rm -rf /var/lib/apt/lists/* 17 | 18 | # install requirements 19 | RUN apt-get update \ 20 | && apt-get install -y --no-install-recommends \ 21 | bc \ 22 | bzip2 \ 23 | ca-certificates \ 24 | curl \ 25 | git \ 26 | sudo \ 27 | libx11-6 \ 28 | libgdal-dev \ 29 | libssl-dev \ 30 | libffi-dev \ 31 | libncurses-dev \ 32 | libgl1 \ 33 | jq \ 34 | nfs-common \ 35 | parallel \ 36 | python-dev \ 37 | python-pip \ 38 | python-wheel \ 39 | python-setuptools \ 40 | unzip \ 41 | vim \ 42 | wget \ 43 | build-essential \ 44 | && apt-get clean \ 45 | && rm -rf /var/lib/apt/lists/* 46 | 47 | SHELL ["/bin/bash", "-c"] 48 | 49 | # install anaconda 50 | RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh -O ~/miniconda.sh && \ 51 | /bin/bash ~/miniconda.sh -b -p /opt/conda && \ 52 | rm ~/miniconda.sh && \ 53 | /opt/conda/bin/conda clean -tipsy && \ 54 | ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \ 55 | echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc 56 | ENV PATH /opt/conda/bin:$PATH 57 | 58 | # prepend pytorch and conda-forge before default channel 59 | RUN conda update conda && \ 60 | conda config --prepend channels conda-forge && \ 61 | conda config --prepend channels pytorch 62 | 63 | # get dev version of solaris and create conda environment based on its env file 64 | WORKDIR /tmp/ 65 | RUN git clone https://github.com/motokimura/solaris.sn7 solaris && \ 66 | cd solaris && \ 67 | git checkout ${solaris_branch} && \ 68 | conda env create -f environment-gpu.yml 69 | ENV PATH /opt/conda/envs/solaris/bin:$PATH 70 | 71 | RUN cd solaris && pip install . 72 | 73 | # install various conda dependencies into the conda environment 74 | RUN conda install -n solaris \ 75 | jupyter \ 76 | jupyterlab \ 77 | ipykernel 78 | 79 | # install segmentation_models.pytorch 80 | RUN pip install git+https://github.com/motokimura/segmentation_models.pytorch.sn7 # install latest master 81 | RUN pip install yacs # required to use spacenet7_model library 82 | RUN pip install gitpython tensorboard tensorboardX lightgbm==2.3.1 geomet==0.2.1-1 # required to run scripts under tools/ 83 | RUN pip install seaborn # required to run notebooks under notebooks/ 84 | RUN pip install imagecodecs # required for Planet files 85 | 86 | # XXX: temporally commented out to avoid nodejs error 87 | # to show tqdm progress bar on jupyter lab nicely 88 | #RUN conda install -n solaris -y -c conda-forge nodejs 89 | #RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager 90 | 91 | # add a jupyter kernel for the conda environment in case it's wanted 92 | RUN source activate solaris && python -m ipykernel.kernelspec \ 93 | --name solaris --display-name solaris 94 | 95 | # activate conda environment immediately after entering container 96 | RUN echo "conda activate solaris" >> ~/.bashrc 97 | 98 | WORKDIR /work 99 | -------------------------------------------------------------------------------- /4-motokimura/code/docker/build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # set image name 4 | IMAGE="spacenet7:dev" 5 | 6 | # get project root dicrectory 7 | THIS_DIR=$(cd $(dirname $0); pwd) 8 | PROJ_DIR=`dirname ${THIS_DIR}` 9 | 10 | # build docker image from project root directory 11 | cd ${PROJ_DIR} && \ 12 | docker build -t ${IMAGE} -f ${THIS_DIR}/Dockerfile.dev . 13 | -------------------------------------------------------------------------------- /4-motokimura/code/docker/exec.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # exec 4 | CONTAINER="spacenet7_dev" 5 | 6 | docker exec -it ${CONTAINER} /bin/bash 7 | -------------------------------------------------------------------------------- /4-motokimura/code/docker/jupyter.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # launch jupyter from spacenet7_dev container 3 | 4 | # set jupyter port 5 | JUPYTER_PORT=8889 6 | if [ $# -eq 1 ]; then 7 | JUPYTER_PORT=$1 8 | fi 9 | 10 | echo "access jupyter sever via port: ${JUPYTER_PORT}" 11 | 12 | # get project root dicrectory 13 | THIS_DIR=$(cd $(dirname $0); pwd) 14 | PROJ_DIR=`dirname ${THIS_DIR}` 15 | 16 | # build docker image from project root directory 17 | cd ${PROJ_DIR} && \ 18 | jupyter notebook --port ${JUPYTER_PORT} --ip=0.0.0.0 --allow-root --no-browser 19 | -------------------------------------------------------------------------------- /4-motokimura/code/docker/run.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ENV=desktop1 # "desktop1", "desktop2", "mac", or "aws" 4 | if [ $# -eq 1 ]; then 5 | ENV=$1 6 | fi 7 | 8 | # set environment specific parameters 9 | if [ $ENV = desktop1 ]; then 10 | RUNTIME="--runtime nvidia" 11 | FEATURE_ROOT=/mnt/sdb1/spacenet7/ 12 | elif [ $ENV = desktop2 ]; then 13 | RUNTIME="--runtime nvidia" 14 | FEATURE_ROOT=${HOME}/data/spacenet7/ 15 | elif [ $ENV = mac ]; then 16 | RUNTIME="" 17 | FEATURE_ROOT=${HOME}/data/spacenet7/ 18 | elif [ $ENV = aws ]; then 19 | RUNTIME="--runtime nvidia" 20 | FEATURE_ROOT=/mnt/nfs/kimura/spacenet7/ 21 | else 22 | echo 'Usage: ./run.sh $ENV' 23 | echo '(ENV must be "desktop1", "desktop2", "mac" or "aws")' 24 | exit 1 25 | fi 26 | 27 | # set jupyter port 28 | JUPYTER_PORT=8889 29 | echo "mapping port docker:${JUPYTER_PORT} --> host:${JUPYTER_PORT}" 30 | 31 | # set image name 32 | IMAGE="spacenet7:dev" 33 | 34 | # set project root dicrectory to map to docker 35 | THIS_DIR=$(cd $(dirname $0); pwd) 36 | PROJ_DIR=`dirname ${THIS_DIR}` 37 | 38 | # set path to directories to map to docker 39 | DATA_DIR=${HOME}/data 40 | WEIGHTS_DIR=${FEATURE_ROOT}/weights 41 | LOG_DIR=${FEATURE_ROOT}/logs 42 | CHECKPOINT_DIR=${FEATURE_ROOT}/checkpoints 43 | PREDICTION_DIR=${FEATURE_ROOT}/predictions 44 | ENSEMBLED_PREDICTION_DIR=${FEATURE_ROOT}/ensembled_predictions 45 | REFINED_PREDICTION_DIR=${FEATURE_ROOT}/refined_predictions 46 | POLY_DIR=${FEATURE_ROOT}/polygons 47 | TRACKED_POLY_DIR=${FEATURE_ROOT}/tracked_polygons 48 | SOLUTION_DIR=${FEATURE_ROOT}/solutions 49 | VAL_PREDICTION_DIR=${FEATURE_ROOT}/val_predictions 50 | VAL_ENSEMBLED_PREDICTION_DIR=${FEATURE_ROOT}/val_ensembled_predictions 51 | VAL_REFINED_PREDICTION_DIR=${FEATURE_ROOT}/val_refined_predictions 52 | VAL_POLY_DIR=${FEATURE_ROOT}/val_polygons 53 | VAL_TRACKED_POLY_DIR=${FEATURE_ROOT}/val_tracked_polygons 54 | 55 | # run container 56 | CONTAINER="spacenet7_dev" 57 | 58 | docker run ${RUNTIME} -it --rm --ipc=host \ 59 | -p ${JUPYTER_PORT}:${JUPYTER_PORT} \ 60 | -p 6006:6006 \ 61 | -v ${PROJ_DIR}:/work \ 62 | -v ${DATA_DIR}:/data \ 63 | -v ${WEIGHTS_DIR}:/weights \ 64 | -v ${LOG_DIR}:/logs \ 65 | -v ${CHECKPOINT_DIR}:/checkpoints \ 66 | -v ${PREDICTION_DIR}:/predictions \ 67 | -v ${ENSEMBLED_PREDICTION_DIR}:/ensembled_predictions \ 68 | -v ${REFINED_PREDICTION_DIR}:/refined_predictions \ 69 | -v ${POLY_DIR}:/polygons \ 70 | -v ${TRACKED_POLY_DIR}:/tracked_polygons \ 71 | -v ${VAL_PREDICTION_DIR}:/val_predictions \ 72 | -v ${VAL_ENSEMBLED_PREDICTION_DIR}:/val_ensembled_predictions \ 73 | -v ${VAL_REFINED_PREDICTION_DIR}:/val_refined_predictions \ 74 | -v ${VAL_POLY_DIR}:/val_polygons \ 75 | -v ${VAL_TRACKED_POLY_DIR}:/val_tracked_polygons \ 76 | --name ${CONTAINER} \ 77 | ${IMAGE} /bin/bash 78 | -------------------------------------------------------------------------------- /4-motokimura/code/docker/tensorboard.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # launch tensorboard from spacenet7_dev container 3 | 4 | # set log_dir 5 | LOG_ROOT=/logs 6 | if [ $# -eq 1 ]; then 7 | LOG_ROOT=$1 8 | fi 9 | 10 | # launch tensorboard from project root directory 11 | tensorboard --logdir ${LOG_ROOT} #--bind_all 12 | -------------------------------------------------------------------------------- /4-motokimura/code/docs/FINAL_SCORING.md: -------------------------------------------------------------------------------- 1 | # Instructions for Final Scoring 2 | 3 | This document provides instructions for the final testing/scoring of motokimura's solution. 4 | 5 | ## Prepare SpaceNet-7 data 6 | 7 | ``` 8 | DATA_ROOT=${HOME}/data 9 | DATA_DIR=${DATA_ROOT}/SN7_buildings # path to download SpaceNet-7 dataset 10 | mkdir -p ${DATA_DIR} 11 | 12 | # download and extract train data 13 | cd ${DATA_DIR} 14 | aws s3 cp s3://spacenet-dataset/spacenet/SN7_buildings/tarballs/SN7_buildings_train.tar.gz . 15 | tar -xvf SN7_buildings_train.tar.gz 16 | 17 | # download and extract train csv data 18 | cd ${DATA_DIR} 19 | aws s3 cp s3://spacenet-dataset/spacenet/SN7_buildings/tarballs/SN7_buildings_train_csvs.tar.gz . 20 | tar -xvf SN7_buildings_train_csvs.tar.gz 21 | cp ${DATA_DIR}/csvs/sn7_train_ground_truth_pix.csv ${DATA_DIR}/train 22 | 23 | # download and extract test data 24 | cd ${DATA_DIR} 25 | aws s3 cp s3://spacenet-dataset/spacenet/SN7_buildings/tarballs/SN7_buildings_test_public.tar.gz . 26 | tar -xvf SN7_buildings_test_public.tar.gz 27 | 28 | # please prepare private test data under `DATA_DIR`! 29 | ``` 30 | 31 | ## Prepare temporal directory 32 | 33 | ``` 34 | WDATA_ROOT=${HOME}/wdata # path to directory for temporal files (train logs, prediction results, etc.) 35 | mkdir -p ${WDATA_ROOT} 36 | ``` 37 | 38 | ## Build image 39 | 40 | ``` 41 | cd ${CODE_DIR} # `code` directory containing `Dockerfile`, `train.sh`, `test.sh`, and etc. 42 | nvidia-docker build -t motokimura . 43 | ``` 44 | 45 | During the build, my home built models are downloaded 46 | so that `test.sh` can run without re-training the models. 47 | 48 | ## Prepare container 49 | 50 | ``` 51 | # launch container 52 | nvidia-docker run --ipc=host -v ${DATA_ROOT}:/data:ro -v ${WDATA_ROOT}:/wdata -it motokimura 53 | ``` 54 | 55 | It's necessary to add `--ipc=host` option when run docker (as written in [flags.txt](flags.txt)). 56 | Otherwise multi-threaded PyTorch dataloader will crash. 57 | 58 | ## Train 59 | 60 | **WARNINGS: `train.sh` updates my home built models downloaded during docker build.** 61 | 62 | ``` 63 | # start training! 64 | (in container) ./train.sh /data/SN7_buildings/train 65 | 66 | # if you need logs: 67 | (in container) ./train.sh /data/SN7_buildings/train 2>&1 | tee /wdata/train.log 68 | ``` 69 | 70 | Note that this is a sample call of `train.sh`. 71 | i.e., you need to specify the correct path to training data folder. 72 | 73 | ## Test 74 | 75 | ``` 76 | # start testing! 77 | (in container) ./test.sh /data/SN7_buildings/test_public /wdata/solution.csv 78 | 79 | # if you need logs: 80 | (in container) ./test.sh /data/SN7_buildings/test_public /wdata/solution.csv 2>&1 | tee /wdata/test.log 81 | ``` 82 | 83 | Note that this is a sample call of `test.sh`. 84 | i.e., you need to specify the correct paths to testing image folder and output csv file. 85 | -------------------------------------------------------------------------------- /4-motokimura/code/flags.txt: -------------------------------------------------------------------------------- 1 | --ipc=host -------------------------------------------------------------------------------- /4-motokimura/code/licenses/README.md: -------------------------------------------------------------------------------- 1 | Open Source Resources, Frameworks and Libraries used in this solution. 2 | 3 | ### Environment setup: 4 | 5 | - NVIDIA Docker, 6 | [https://github.com/NVIDIA/nvidia-docker](https://github.com/NVIDIA/nvidia-docker), 7 | Apache 2.0 8 | 9 | - Anaconda, 10 | [https://www.anaconda.com/](https://www.anaconda.com/), 11 | BSD 3-Clause 12 | 13 | ### CNN model tranning/inference: 14 | 15 | - PyTorch, 16 | [https://pytorch.org/](https://pytorch.org/), 17 | BSD 3-Clause 18 | 19 | - segmentation_models.pytorch, 20 | [https://github.com/qubvel/segmentation_models.pytorch](https://github.com/qubvel/segmentation_models.pytorch), 21 | MIT 22 | 23 | - pytorch-image-models, 24 | [https://github.com/rwightman/pytorch-image-models](https://github.com/rwightman/pytorch-image-models), 25 | Apache 2.0 26 | 27 | - torchvision, 28 | [https://github.com/pytorch/vision](https://github.com/pytorch/vision), 29 | BSD 3-Clause 30 | 31 | ### Pre-processing for SN7 dataset: 32 | 33 | - Solaris, 34 | [https://github.com/CosmiQ/solaris](https://github.com/CosmiQ/solaris), 35 | Apache 2.0 36 | 37 | - albumentations, 38 | [https://github.com/albumentations-team/albumentations](https://github.com/albumentations-team/albumentations), 39 | MIT 40 | 41 | - OpenCV, [https://opencv.org/](https://opencv.org/), 42 | BSD 3-Clause 43 | 44 | - scikit-image, 45 | [https://scikit-image.org/](https://scikit-image.org/), 46 | BSD 3-Clause 47 | 48 | - imagecodecs, 49 | [https://github.com/cgohlke/imagecodecs](https://github.com/cgohlke/imagecodecs), 50 | BSD 3-Clause 51 | 52 | ### Post-processing for CNN output: 53 | 54 | - NumPy, 55 | [https://numpy.org/](https://numpy.org/), 56 | BSD 3-Clause 57 | 58 | - pandas, 59 | [https://pandas.pydata.org/](https://pandas.pydata.org/), 60 | BSD 3-Clause 61 | 62 | - LightGBM, 63 | [https://github.com/microsoft/LightGBM](https://github.com/microsoft/LightGBM), 64 | MIT 65 | 66 | - GeoMet, 67 | [https://github.com/geomet/geomet](https://github.com/geomet/geomet), 68 | Apache 2.0 69 | 70 | - Rasterio, 71 | [https://github.com/mapbox/rasterio](https://github.com/mapbox/rasterio), 72 | BSD 3-Clause 73 | 74 | - Shapely, 75 | [https://github.com/Toblerity/Shapely](https://github.com/Toblerity/Shapely), 76 | BSD 3-Clause 77 | 78 | ### CNN model development (experiment management at local) 79 | 80 | - YACS, 81 | [https://github.com/rbgirshick/yacs](https://github.com/rbgirshick/yacs), 82 | Apache 2.0 83 | 84 | - tqdm, 85 | [https://github.com/tqdm/tqdm](https://github.com/tqdm/tqdm), 86 | MIT 87 | 88 | - tensorboard, 89 | [https://www.tensorflow.org/tensorboard](https://www.tensorflow.org/tensorboard), 90 | Apache 2.0 91 | 92 | - tensorboardX, 93 | [https://github.com/lanpa/tensorboardX](https://github.com/lanpa/tensorboardX), 94 | MIT 95 | 96 | - Jupyter, 97 | [https://jupyter.org/](https://jupyter.org/), 98 | BSD 3-Clause 99 | 100 | - seaborn, 101 | [https://github.com/mwaskom/seaborn](https://github.com/mwaskom/seaborn), 102 | BSD 3-Clause 103 | 104 | - GitPython, 105 | [https://github.com/gitpython-developers/GitPython](https://github.com/gitpython-developers/GitPython), 106 | BSD 3-Clause 107 | -------------------------------------------------------------------------------- /4-motokimura/code/scripts/test/postprocess.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | source activate solaris 4 | 5 | TEST_ROOT=$1 # path/to/SN7_buildings/test/ 6 | OUTPUT_CSV_PATH=$2 # path/to/solution.csv 7 | 8 | source /work/settings.sh 9 | 10 | echo '' 11 | echo 'ensembling model predictions...' 12 | /work/tools/ensemble_models.py \ 13 | INPUT.TEST_DIR ${TEST_ROOT} \ 14 | PREDICTION_ROOT ${PREDICTION_ROOT} \ 15 | ENSEMBLED_PREDICTION_ROOT ${ENSEMBLED_PREDICTION_ROOT} \ 16 | ENSEMBLE_EXP_IDS "[200, 201, 202, 203, 204, 500, 501, 502, 503, 504]" 17 | 18 | echo '' 19 | echo 'refining predicted masks...' 20 | /work/tools/refine_pred_mask.py \ 21 | INPUT.TEST_DIR ${TEST_ROOT} \ 22 | ENSEMBLED_PREDICTION_ROOT ${ENSEMBLED_PREDICTION_ROOT} \ 23 | REFINED_PREDICTION_ROOT ${REFINED_PREDICTION_ROOT} \ 24 | ENSEMBLE_EXP_IDS "[200, 201, 202, 203, 204, 500, 501, 502, 503, 504]" 25 | 26 | echo '' 27 | echo 'generating polygons...' 28 | /work/tools/pred_mask_to_poly.py \ 29 | REFINED_PREDICTION_ROOT ${REFINED_PREDICTION_ROOT} \ 30 | POLY_ROOT ${POLY_ROOT} \ 31 | ENSEMBLE_EXP_IDS "[200, 201, 202, 203, 204, 500, 501, 502, 503, 504]" 32 | 33 | echo '' 34 | echo 'tracking polygons...' 35 | /work/tools/track_polys.py \ 36 | INPUT.TEST_DIR ${TEST_ROOT} \ 37 | POLY_ROOT ${POLY_ROOT} \ 38 | TRACKED_POLY_ROOT ${TRACKED_POLY_ROOT} \ 39 | SOLUTION_OUTPUT_PATH ${OUTPUT_CSV_PATH} \ 40 | ENSEMBLE_EXP_IDS "[200, 201, 202, 203, 204, 500, 501, 502, 503, 504]" 41 | -------------------------------------------------------------------------------- /4-motokimura/code/scripts/test/test_cnns.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | source activate solaris 4 | 5 | TEST_ROOT=$1 # path/to/SN7_buildings/test/ 6 | 7 | source /work/settings.sh 8 | 9 | # predict with trained models 10 | TEST_ARGS="\ 11 | --exp_log_dir ${LOG_ROOT} \ 12 | --model_weight_dir ${WEIGHT_ROOT} \ 13 | PREDICTION_ROOT ${PREDICTION_ROOT} \ 14 | INPUT.TEST_DIR ${TEST_ROOT} \ 15 | " 16 | 17 | mkdir -p ${TEST_STDOUT_ROOT} 18 | 19 | echo '' 20 | echo 'predicting... (1/3)' 21 | echo 'this will take ~10 min' 22 | echo 'you can check progress from '${TEST_STDOUT_ROOT}'/*.out' 23 | 24 | nohup env CUDA_VISIBLE_DEVICES=0 /work/tools/test_spacenet7_model.py \ 25 | --exp_id 200 \ 26 | ${TEST_ARGS} \ 27 | > ${TEST_STDOUT_ROOT}/exp_0200.out 2>&1 & 28 | 29 | nohup env CUDA_VISIBLE_DEVICES=1 /work/tools/test_spacenet7_model.py \ 30 | --exp_id 201 \ 31 | ${TEST_ARGS} \ 32 | > ${TEST_STDOUT_ROOT}/exp_0201.out 2>&1 & 33 | 34 | nohup env CUDA_VISIBLE_DEVICES=2 /work/tools/test_spacenet7_model.py \ 35 | --exp_id 202 \ 36 | ${TEST_ARGS} \ 37 | > ${TEST_STDOUT_ROOT}/exp_0202.out 2>&1 & 38 | 39 | nohup env CUDA_VISIBLE_DEVICES=3 /work/tools/test_spacenet7_model.py \ 40 | --exp_id 203 \ 41 | ${TEST_ARGS} \ 42 | > ${TEST_STDOUT_ROOT}/exp_0203.out 2>&1 & 43 | 44 | wait 45 | 46 | echo '' 47 | echo 'predicting... (2/3)' 48 | echo 'this will take ~15 min' 49 | echo 'you can check progress from '${TEST_STDOUT_ROOT}'/*.out' 50 | 51 | nohup env CUDA_VISIBLE_DEVICES=0 /work/tools/test_spacenet7_model.py \ 52 | --exp_id 204 \ 53 | ${TEST_ARGS} \ 54 | > ${TEST_STDOUT_ROOT}/exp_0204.out 2>&1 & 55 | 56 | nohup env CUDA_VISIBLE_DEVICES=1 /work/tools/test_spacenet7_model.py \ 57 | --exp_id 500 \ 58 | ${TEST_ARGS} \ 59 | > ${TEST_STDOUT_ROOT}/exp_0500.out 2>&1 & 60 | 61 | nohup env CUDA_VISIBLE_DEVICES=2 /work/tools/test_spacenet7_model.py \ 62 | --exp_id 501 \ 63 | ${TEST_ARGS} \ 64 | > ${TEST_STDOUT_ROOT}/exp_0501.out 2>&1 & 65 | 66 | nohup env CUDA_VISIBLE_DEVICES=3 /work/tools/test_spacenet7_model.py \ 67 | --exp_id 502 \ 68 | ${TEST_ARGS} \ 69 | > ${TEST_STDOUT_ROOT}/exp_0502.out 2>&1 & 70 | 71 | wait 72 | 73 | echo '' 74 | echo 'predicting... (3/3)' 75 | echo 'this will take ~15 min' 76 | echo 'you can check progress from '${TEST_STDOUT_ROOT}'/*.out' 77 | 78 | nohup env CUDA_VISIBLE_DEVICES=0 /work/tools/test_spacenet7_model.py \ 79 | --exp_id 503 \ 80 | ${TEST_ARGS} \ 81 | > ${TEST_STDOUT_ROOT}/exp_0503.out 2>&1 & 82 | 83 | nohup env CUDA_VISIBLE_DEVICES=1 /work/tools/test_spacenet7_model.py \ 84 | --exp_id 504 \ 85 | ${TEST_ARGS} \ 86 | > ${TEST_STDOUT_ROOT}/exp_0504.out 2>&1 & 87 | 88 | wait 89 | 90 | echo 'done predicting all models!' 91 | -------------------------------------------------------------------------------- /4-motokimura/code/scripts/train/preprocess.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | source activate solaris 4 | 5 | TRAIN_ROOT=$1 # path/to/SN7_buildings/train/ 6 | 7 | source /work/settings.sh 8 | 9 | echo '' 10 | echo 'generating building masks...' 11 | /work/tools/geojson_to_mask.py \ 12 | --train_dir ${TRAIN_ROOT} \ 13 | --out_dir ${BUILDING_MASK_ROOT} 14 | 15 | echo '' 16 | echo 'splitting dataset...' 17 | /work/tools/split_dataset.py \ 18 | --train_dir ${TRAIN_ROOT} \ 19 | --mask_dir ${BUILDING_MASK_ROOT} \ 20 | --out_dir ${TRAIN_VAL_SPLIT_ROOT} 21 | -------------------------------------------------------------------------------- /4-motokimura/code/scripts/train/train_cnns.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | source activate solaris 4 | 5 | source /work/settings.sh 6 | 7 | TRAIN_ARGS="\ 8 | INPUT.TRAIN_VAL_SPLIT_DIR ${TRAIN_VAL_SPLIT_ROOT} \ 9 | LOG_ROOT ${LOG_ROOT} \ 10 | WEIGHT_ROOT ${WEIGHT_ROOT} \ 11 | SAVE_CHECKPOINTS False \ 12 | DUMP_GIT_INFO False \ 13 | EVAL.VAL_INTERVAL_EPOCH 3 \ 14 | " 15 | # comment out the line below for debugging 16 | #TRAIN_ARGS=${TRAIN_ARGS}" SOLVER.EPOCHS 10" 17 | 18 | mkdir -p ${TRAIN_STDOUT_ROOT} 19 | 20 | CONFIG_020x='/work/configs/unet_timm-efficientnet-b3_scale-3.0_v_02.yml' # exp_200~204 (effnet-b3, scale=3, focal) 21 | CONFIG_050x='/work/configs/unet_timm-efficientnet-b3_scale-4.0_v_01.yml' # exp_500~504 (effnet-b3, scale=4, bce) 22 | 23 | echo '' 24 | echo 'training... (1/3)' 25 | echo 'this will take ~8 hours' 26 | echo 'you can check progress from '${TRAIN_STDOUT_ROOT}'/*.out' 27 | 28 | ## 200 29 | nohup env CUDA_VISIBLE_DEVICES=0 /work/tools/train_spacenet7_model.py \ 30 | --config ${CONFIG_020x} \ 31 | ${TRAIN_ARGS} \ 32 | INPUT.TRAIN_VAL_SPLIT_ID 0 \ 33 | EXP_ID 200 \ 34 | > ${TRAIN_STDOUT_ROOT}/exp_0200.out 2>&1 & 35 | 36 | ## 201 37 | nohup env CUDA_VISIBLE_DEVICES=1 /work/tools/train_spacenet7_model.py \ 38 | --config ${CONFIG_020x} \ 39 | ${TRAIN_ARGS} \ 40 | INPUT.TRAIN_VAL_SPLIT_ID 1 \ 41 | EXP_ID 201 \ 42 | > ${TRAIN_STDOUT_ROOT}/exp_0201.out 2>&1 & 43 | 44 | ## 202 45 | nohup env CUDA_VISIBLE_DEVICES=2 /work/tools/train_spacenet7_model.py \ 46 | --config ${CONFIG_020x} \ 47 | ${TRAIN_ARGS} \ 48 | INPUT.TRAIN_VAL_SPLIT_ID 2 \ 49 | EXP_ID 202 \ 50 | > ${TRAIN_STDOUT_ROOT}/exp_0202.out 2>&1 & 51 | 52 | ## 203 53 | nohup env CUDA_VISIBLE_DEVICES=3 /work/tools/train_spacenet7_model.py \ 54 | --config ${CONFIG_020x} \ 55 | ${TRAIN_ARGS} \ 56 | INPUT.TRAIN_VAL_SPLIT_ID 3 \ 57 | EXP_ID 203 \ 58 | > ${TRAIN_STDOUT_ROOT}/exp_0203.out 2>&1 & 59 | 60 | wait 61 | 62 | echo '' 63 | echo 'training... (2/3)' 64 | echo 'this will take ~11 hours' 65 | echo 'you can check progress from '${TRAIN_STDOUT_ROOT}'/*.out' 66 | 67 | ## 204 68 | nohup env CUDA_VISIBLE_DEVICES=0 /work/tools/train_spacenet7_model.py \ 69 | --config ${CONFIG_020x} \ 70 | ${TRAIN_ARGS} \ 71 | INPUT.TRAIN_VAL_SPLIT_ID 4 \ 72 | EXP_ID 204 \ 73 | > ${TRAIN_STDOUT_ROOT}/exp_0204.out 2>&1 & 74 | 75 | ## 500 76 | nohup env CUDA_VISIBLE_DEVICES=1 /work/tools/train_spacenet7_model.py \ 77 | --config ${CONFIG_050x} \ 78 | ${TRAIN_ARGS} \ 79 | INPUT.TRAIN_VAL_SPLIT_ID 0 \ 80 | EXP_ID 500 \ 81 | > ${TRAIN_STDOUT_ROOT}/exp_0500.out 2>&1 & 82 | 83 | ## 501 84 | nohup env CUDA_VISIBLE_DEVICES=2 /work/tools/train_spacenet7_model.py \ 85 | --config ${CONFIG_050x} \ 86 | ${TRAIN_ARGS} \ 87 | INPUT.TRAIN_VAL_SPLIT_ID 1 \ 88 | EXP_ID 501 \ 89 | > ${TRAIN_STDOUT_ROOT}/exp_0501.out 2>&1 & 90 | 91 | ## 502 92 | nohup env CUDA_VISIBLE_DEVICES=3 /work/tools/train_spacenet7_model.py \ 93 | --config ${CONFIG_050x} \ 94 | ${TRAIN_ARGS} \ 95 | INPUT.TRAIN_VAL_SPLIT_ID 2 \ 96 | EXP_ID 502 \ 97 | > ${TRAIN_STDOUT_ROOT}/exp_0502.out 2>&1 & 98 | 99 | wait 100 | 101 | echo '' 102 | echo 'training... (3/3)' 103 | echo 'this will take ~11 hours' 104 | echo 'you can check progress from '${TRAIN_STDOUT_ROOT}'/*.out' 105 | 106 | ## 503 107 | nohup env CUDA_VISIBLE_DEVICES=0 /work/tools/train_spacenet7_model.py \ 108 | --config ${CONFIG_050x} \ 109 | ${TRAIN_ARGS} \ 110 | INPUT.TRAIN_VAL_SPLIT_ID 3 \ 111 | EXP_ID 503 \ 112 | > ${TRAIN_STDOUT_ROOT}/exp_0503.out 2>&1 & 113 | 114 | ## 504 115 | nohup env CUDA_VISIBLE_DEVICES=1 /work/tools/train_spacenet7_model.py \ 116 | --config ${CONFIG_050x} \ 117 | ${TRAIN_ARGS} \ 118 | INPUT.TRAIN_VAL_SPLIT_ID 4 \ 119 | EXP_ID 504 \ 120 | > ${TRAIN_STDOUT_ROOT}/exp_0504.out 2>&1 & 121 | 122 | wait 123 | 124 | echo 'done training all models!' 125 | -------------------------------------------------------------------------------- /4-motokimura/code/settings.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # paths to model related files 4 | MODEL_ROOT=/work/models 5 | 6 | LOG_ROOT=${MODEL_ROOT}/logs 7 | WEIGHT_ROOT=${MODEL_ROOT}/weights 8 | 9 | # path to artifact files 10 | ARTIFACT_ROOT=/wdata 11 | 12 | BUILDING_MASK_ROOT=${ARTIFACT_ROOT}/building_masks 13 | TRAIN_VAL_SPLIT_ROOT=${ARTIFACT_ROOT}/split 14 | 15 | PREDICTION_ROOT=${ARTIFACT_ROOT}/predictions 16 | ENSEMBLED_PREDICTION_ROOT=${ARTIFACT_ROOT}/ensembled_predictions 17 | REFINED_PREDICTION_ROOT=${ARTIFACT_ROOT}/refined_predictions 18 | POLY_ROOT=${ARTIFACT_ROOT}/polygons 19 | TRACKED_POLY_ROOT=${ARTIFACT_ROOT}/tracked_polygons 20 | 21 | # path to stdout files 22 | TRAIN_STDOUT_ROOT=${ARTIFACT_ROOT}/stdout/train 23 | TEST_STDOUT_ROOT=${ARTIFACT_ROOT}/stdout/test 24 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/4-motokimura/code/spacenet7_model/__init__.py -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/configs/__init__.py: -------------------------------------------------------------------------------- 1 | from .load_config import load_config 2 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/configs/load_config.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os.path 3 | 4 | from ..utils import config_filename, experiment_subdir, weight_best_filename 5 | from .defaults import get_default_config 6 | 7 | 8 | def _load_previous_experiment(config, exp_id, exp_log_dir, model_weight_dir): 9 | """[summary] 10 | 11 | Args: 12 | config ([type]): [description] 13 | exp_id ([type]): [description] 14 | exp_log_dir ([type]): [description] 15 | model_weight_dir ([type]): [description] 16 | 17 | Returns: 18 | [type]: [description] 19 | """ 20 | exp_log_dir = exp_log_dir if exp_log_dir else config.LOG_ROOT 21 | model_weight_dir = model_weight_dir if model_weight_dir else config.WEIGHT_ROOT 22 | exp_subdir = experiment_subdir(exp_id) 23 | 24 | # overwrite config 25 | config_path = os.path.join(exp_log_dir, exp_subdir, config_filename()) 26 | config.merge_from_file(config_path) 27 | 28 | # overwrite weight path 29 | weight_path = os.path.join(model_weight_dir, exp_subdir, 30 | weight_best_filename()) 31 | config.MODEL.WEIGHT = weight_path 32 | 33 | return config 34 | 35 | 36 | def get_config_with_previous_experiment(exp_id=-1, 37 | exp_log_dir=None, 38 | model_weight_dir=None): 39 | """[summary] 40 | 41 | Args: 42 | exp_id (int, optional): [description]. Defaults to -1. 43 | exp_log_dir ([type], optional): [description]. Defaults to None. 44 | model_weight_dir ([type], optional): [description]. Defaults to None. 45 | 46 | Returns: 47 | [type]: [description] 48 | """ 49 | config = get_default_config() 50 | if exp_id >= 0: 51 | # overwrite hyper parameters with the ones used by previous experiment 52 | config = _load_previous_experiment(config, exp_id, exp_log_dir, 53 | model_weight_dir) 54 | else: 55 | # if exp_id < 0, return default config 56 | pass 57 | return config 58 | 59 | 60 | def load_config(): 61 | """[summary] 62 | 63 | Returns: 64 | [type]: [description] 65 | """ 66 | parser = argparse.ArgumentParser() 67 | parser.add_argument('--config', help='path to YAML config file', type=str) 68 | parser.add_argument('--exp_id', 69 | help='id of the previous experiment to load', 70 | type=int, 71 | default=-1) 72 | parser.add_argument( 73 | '--exp_log_dir', 74 | help='directory in which the previous experiment config is saved', 75 | type=str) 76 | parser.add_argument( 77 | '--model_weight_dir', 78 | help='directory in which the previous experiment model weight is saved', 79 | type=str) 80 | parser.add_argument('opts', 81 | default=None, 82 | help='parameter name and value pairs', 83 | nargs=argparse.REMAINDER) 84 | args = parser.parse_args() 85 | 86 | config = get_config_with_previous_experiment( 87 | exp_id=args.exp_id, 88 | exp_log_dir=args.exp_log_dir, 89 | model_weight_dir=args.model_weight_dir) 90 | if args.config: 91 | # overwrite hyper parameters with the ones given by the YAML file. 92 | config.merge_from_file(args.config) 93 | if args.opts: 94 | # overwrite hyper parameters with the ones given by command line args. 95 | config.merge_from_list(args.opts) 96 | config.freeze() 97 | 98 | return config 99 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/evaluations/__init__.py: -------------------------------------------------------------------------------- 1 | import segmentation_models_pytorch as smp 2 | 3 | 4 | def _iou_metrics(config, thresh=0.5): 5 | """[summary] 6 | 7 | Args: 8 | config ([type]): [description] 9 | thresh (float, optional): [description]. Defaults to 0.5. 10 | 11 | Returns: 12 | [type]: [description] 13 | """ 14 | metrics = [] 15 | 16 | # mean iou of all classes 17 | metric = smp.utils.metrics.IoU(threshold=thresh) 18 | metric.__name__ = 'iou/all' 19 | metrics.append(metric) 20 | 21 | # iou of each category 22 | classes = config.INPUT.CLASSES 23 | for target_class in classes: 24 | other_classes = classes.copy() 25 | other_classes.remove(target_class) 26 | ignore_channels = [classes.index(c) for c in other_classes] 27 | 28 | metric = smp.utils.metrics.IoU(threshold=thresh, 29 | ignore_channels=ignore_channels) 30 | metric.__name__ = f'iou/{target_class}' 31 | 32 | metrics.append(metric) 33 | 34 | return metrics 35 | 36 | 37 | def get_metrics(config): 38 | """[summary] 39 | 40 | Args: 41 | config ([type]): [description] 42 | """ 43 | def _get_metrics(config, metric_name): 44 | """[summary] 45 | 46 | Args: 47 | config ([type]): [description] 48 | metric_name ([type]): [description] 49 | 50 | Raises: 51 | ValueError: [description] 52 | 53 | Returns: 54 | [type]: [description] 55 | """ 56 | # TODO: support multiple metrics 57 | if metric_name == 'iou': 58 | return _iou_metrics(config) 59 | else: 60 | raise ValueError() 61 | 62 | metrics = [] 63 | for metric_name in config.EVAL.METRICS: 64 | metrics.extend(_get_metrics(config, metric_name)) 65 | 66 | return metrics 67 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/models/__init__.py: -------------------------------------------------------------------------------- 1 | import segmentation_models_pytorch as smp 2 | import torch 3 | 4 | 5 | def get_model(config): 6 | """[summary] 7 | 8 | Args: 9 | config ([type]): [description] 10 | 11 | Raises: 12 | ValueError: [description] 13 | 14 | Returns: 15 | [type]: [description] 16 | """ 17 | arch = config.MODEL.ARCHITECTURE 18 | backbone = config.MODEL.BACKBONE 19 | encoder_weights = config.MODEL.ENCODER_PRETRAINED_FROM 20 | n_classes = len(config.INPUT.CLASSES) 21 | activation = config.MODEL.ACTIVATION 22 | 23 | # compute model input channels 24 | base_channels = config.MODEL.IN_CHANNELS 25 | in_channels = base_channels 26 | if config.INPUT.CONCAT_PREV_FRAME: 27 | in_channels += base_channels 28 | if config.INPUT.CONCAT_NEXT_FRAME: 29 | in_channels += base_channels 30 | 31 | # unet specific 32 | decoder_attention_type = 'scse' if config.MODEL.UNET_ENABLE_DECODER_SCSE else None 33 | 34 | if arch == 'unet': 35 | model = smp.Unet(encoder_name=backbone, 36 | encoder_weights=encoder_weights, 37 | decoder_channels=config.MODEL.UNET_DECODER_CHANNELS, 38 | decoder_attention_type=decoder_attention_type, 39 | in_channels=in_channels, 40 | classes=n_classes, 41 | activation=activation) 42 | elif arch == 'fpn': 43 | model = smp.FPN(encoder_name=backbone, 44 | encoder_weights=encoder_weights, 45 | decoder_dropout=config.MODEL.FPN_DECODER_DROPOUT, 46 | in_channels=in_channels, 47 | classes=n_classes, 48 | activation=activation) 49 | elif arch == 'pan': 50 | model = smp.PAN(encoder_name=backbone, 51 | encoder_weights=encoder_weights, 52 | in_channels=in_channels, 53 | classes=n_classes, 54 | activation=activation) 55 | elif arch == 'pspnet': 56 | model = smp.PSPNet(encoder_name=backbone, 57 | encoder_weights=encoder_weights, 58 | psp_dropout=config.MODEL.PSPNET_DROPOUT, 59 | in_channels=in_channels, 60 | classes=n_classes, 61 | activation=activation) 62 | elif arch == 'deeplabv3': 63 | model = smp.DeepLabV3(encoder_name=backbone, 64 | encoder_weights=encoder_weights, 65 | in_channels=in_channels, 66 | classes=n_classes, 67 | activation=activation) 68 | elif arch == 'linknet': 69 | model = smp.Linknet(encoder_name=backbone, 70 | encoder_weights=encoder_weights, 71 | in_channels=in_channels, 72 | classes=n_classes, 73 | activation=activation) 74 | else: 75 | raise ValueError() 76 | 77 | model = torch.nn.DataParallel(model) 78 | 79 | if config.MODEL.WEIGHT and config.MODEL.WEIGHT != 'none': 80 | # load weight from file 81 | model.load_state_dict( 82 | torch.load(config.MODEL.WEIGHT, map_location=torch.device('cpu'))) 83 | 84 | model = model.to(config.MODEL.DEVICE) 85 | return model 86 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/solvers/__init__.py: -------------------------------------------------------------------------------- 1 | import segmentation_models_pytorch as smp 2 | import torch.optim 3 | 4 | from .losses import BinaryFocalLoss, CombinedLoss 5 | 6 | 7 | def get_optimizer(config, model): 8 | """[summary] 9 | 10 | Args: 11 | config ([type]): [description] 12 | model ([type]): [description] 13 | 14 | Raises: 15 | ValueError: [description] 16 | 17 | Returns: 18 | [type]: [description] 19 | """ 20 | optimizer_name = config.SOLVER.OPTIMIZER 21 | if optimizer_name == 'adam': 22 | return torch.optim.Adam(model.parameters(), 23 | lr=config.SOLVER.LR, 24 | weight_decay=config.SOLVER.WEIGHT_DECAY) 25 | elif optimizer_name == 'adamw': 26 | return torch.optim.AdamW(model.parameters(), 27 | lr=config.SOLVER.LR, 28 | weight_decay=config.SOLVER.WEIGHT_DECAY) 29 | else: 30 | raise ValueError() 31 | 32 | 33 | def get_lr_scheduler(config, optimizer): 34 | """[summary] 35 | 36 | Args: 37 | config ([type]): [description] 38 | optimizer ([type]): [description] 39 | 40 | Raises: 41 | ValueError: [description] 42 | 43 | Returns: 44 | [type]: [description] 45 | """ 46 | scheduler_name = config.SOLVER.LR_SCHEDULER 47 | if scheduler_name == 'multistep': 48 | return torch.optim.lr_scheduler.MultiStepLR( 49 | optimizer, 50 | milestones=config.SOLVER.LR_MULTISTEP_MILESTONES, 51 | gamma=config.SOLVER.LR_MULTISTEP_GAMMA) 52 | elif scheduler_name == 'annealing': 53 | return torch.optim.lr_scheduler.CosineAnnealingLR( 54 | optimizer, 55 | T_max=config.SOLVER.LR_ANNEALING_T_MAX, 56 | eta_min=config.SOLVER.LR_ANNEALING_ETA_MIN, 57 | ) 58 | else: 59 | raise ValueError() 60 | 61 | 62 | def get_loss(config): 63 | """[summary] 64 | 65 | Args: 66 | config ([type]): [description] 67 | """ 68 | def _get_loss(config, loss_name): 69 | if loss_name == 'bce': 70 | return smp.utils.losses.BCELoss() 71 | elif loss_name == 'dice': 72 | return smp.utils.losses.DiceLoss() 73 | elif loss_name == 'focal': 74 | return BinaryFocalLoss(gamma=config.SOLVER.FOCAL_LOSS_GAMMA) 75 | else: 76 | raise ValueError() 77 | 78 | loss_modules = [] 79 | for loss_name in config.SOLVER.LOSSES: 80 | loss_modules.append(_get_loss(config, loss_name)) 81 | 82 | return CombinedLoss(loss_modules, config.SOLVER.LOSS_WEIGHTS) 83 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/solvers/losses.py: -------------------------------------------------------------------------------- 1 | import segmentation_models_pytorch as smp 2 | import torch 3 | 4 | 5 | class CombinedLoss(smp.utils.base.Loss): 6 | """[summary] 7 | 8 | Args: 9 | smp ([type]): [description] 10 | 11 | Returns: 12 | [type]: [description] 13 | """ 14 | __name__ = 'loss' 15 | 16 | def __init__(self, loss_modules, loss_weights, **kwargs): 17 | assert len(loss_modules) == len(loss_weights) 18 | 19 | super().__init__(**kwargs) 20 | 21 | self._loss_modules = loss_modules 22 | self._loss_weights = loss_weights 23 | 24 | def forward(self, y_pr, y_gt): 25 | """ 26 | """ 27 | losses = self._loss_modules 28 | weights = self._loss_weights 29 | 30 | loss = losses[0](y_pr, y_gt) * weights[0] 31 | if len(losses) == 1: 32 | return loss 33 | 34 | for i in range(1, len(losses)): 35 | loss += losses[i](y_pr, y_gt) * weights[i] 36 | return loss 37 | 38 | 39 | class BinaryFocalLoss(smp.utils.base.Loss): 40 | """[summary] 41 | 42 | Args: 43 | smp ([type]): [description] 44 | 45 | Returns: 46 | [type]: [description] 47 | """ 48 | 49 | # references: 50 | # https://github.com/SpaceNetChallenge/SpaceNet_Off_Nadir_Solutions/blob/812f151d244565f29987ebec7683ef42622ae16e/cannab/losses.py#L259 51 | def __init__(self, gamma=2.0, **kwargs): 52 | super().__init__(**kwargs) 53 | 54 | self._gamma = gamma 55 | 56 | def forward(self, y_pr, y_gt): 57 | """ 58 | """ 59 | eps = 1e-8 60 | y_gt = torch.clamp(y_gt, eps, 1 - eps) 61 | y_pr = torch.clamp(y_pr, eps, 1 - eps) 62 | 63 | pt = (1 - y_gt) * (1 - y_pr) + y_gt * y_pr 64 | loss = -(1 - pt)**self._gamma * torch.log(pt) 65 | return loss.mean() 66 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/transforms/__init__.py: -------------------------------------------------------------------------------- 1 | from .augmentations import get_spacenet7_augmentation 2 | from .preprocesses import get_spacenet7_preprocess 3 | 4 | 5 | def get_preprocess(config, is_test): 6 | """[summary] 7 | 8 | Args: 9 | config ([type]): [description] 10 | is_test (bool): [description] 11 | 12 | Returns: 13 | [type]: [description] 14 | """ 15 | return get_spacenet7_preprocess(config, is_test) 16 | 17 | 18 | def get_augmentation(config, 19 | is_train, 20 | tta_resize_wh=None, 21 | tta_hflip=False, 22 | tta_vflip=False): 23 | """[summary] 24 | 25 | Args: 26 | config ([type]): [description] 27 | is_train (bool): [description] 28 | tta_resize_wh ([type], optional): [description]. Defaults to None. 29 | tta_hflip (bool, optional): [description]. Defaults to False. 30 | tta_vflip (bool, optional): [description]. Defaults to False. 31 | 32 | Returns: 33 | [type]: [description] 34 | """ 35 | return get_spacenet7_augmentation(config, 36 | is_train, 37 | tta_resize_wh=tta_resize_wh, 38 | tta_hflip=tta_hflip, 39 | tta_vflip=tta_vflip) 40 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/transforms/augmentations.py: -------------------------------------------------------------------------------- 1 | import functools 2 | import random 3 | 4 | import numpy as np 5 | 6 | import albumentations as albu 7 | 8 | 9 | def get_spacenet7_augmentation(config, 10 | is_train, 11 | tta_resize_wh=None, 12 | tta_hflip=False, 13 | tta_vflip=False): 14 | """[summary] 15 | 16 | Args: 17 | config ([type]): [description] 18 | is_train (bool): [description] 19 | tta_resize_wh ([type], optional): [description]. Defaults to None. 20 | tta_hflip (bool, optional): [description]. Defaults to False. 21 | tta_vflip (bool, optional): [description]. Defaults to False. 22 | 23 | Returns: 24 | [type]: [description] 25 | """ 26 | size_scale = config.TRANSFORM.SIZE_SCALE 27 | 28 | if is_train: 29 | assert tta_resize_wh is None 30 | assert not tta_hflip 31 | assert not tta_vflip 32 | 33 | # size after cropping 34 | base_width = config.TRANSFORM.TRAIN_RANDOM_CROP_SIZE[0] 35 | base_height = config.TRANSFORM.TRAIN_RANDOM_CROP_SIZE[1] 36 | 37 | augmentation = [ 38 | # random flip 39 | albu.HorizontalFlip(p=config.TRANSFORM.TRAIN_HORIZONTAL_FLIP_PROB), 40 | albu.VerticalFlip(p=config.TRANSFORM.TRAIN_VERTICAL_FLIP_PROB), 41 | # random rotate 42 | albu.ShiftScaleRotate( 43 | scale_limit=0.0, 44 | rotate_limit=config.TRANSFORM.TRAIN_RANDOM_ROTATE_DEG, 45 | shift_limit=0.0, 46 | p=config.TRANSFORM.TRAIN_RANDOM_ROTATE_PROB, 47 | border_mode=0), 48 | # random crop 49 | albu.RandomCrop(width=base_width, 50 | height=base_height, 51 | always_apply=True), 52 | # random brightness 53 | albu.Lambda(image=functools.partial( 54 | _random_brightness, 55 | brightness_std=config.TRANSFORM.TRAIN_RANDOM_BRIGHTNESS_STD, 56 | p=config.TRANSFORM.TRAIN_RANDOM_BRIGHTNESS_PROB)), 57 | ] 58 | 59 | resize_width = int(size_scale * base_width) 60 | resize_height = int(size_scale * base_height) 61 | 62 | else: 63 | # size after padding 64 | base_width = config.TRANSFORM.TEST_SIZE[0] 65 | base_height = config.TRANSFORM.TEST_SIZE[1] 66 | 67 | augmentation = [ 68 | # padding 69 | albu.PadIfNeeded(min_width=base_width, 70 | min_height=base_height, 71 | always_apply=True, 72 | border_mode=0), 73 | ] 74 | 75 | # tta flipping 76 | if tta_hflip: 77 | augmentation.append(albu.HorizontalFlip(always_apply=True)) 78 | if tta_vflip: 79 | augmentation.append(albu.VerticalFlip(always_apply=True)) 80 | 81 | # tta size jitter 82 | if tta_resize_wh is None: 83 | tta_width, tta_height = base_width, base_height 84 | else: 85 | tta_width, tta_height = tta_resize_wh 86 | resize_width = int(size_scale * tta_width) 87 | resize_height = int(size_scale * tta_height) 88 | 89 | if (base_width != resize_width) or (base_height != resize_height): 90 | # append resizing 91 | augmentation.append( 92 | albu.Resize(width=resize_width, 93 | height=resize_height, 94 | always_apply=True)) 95 | 96 | return albu.Compose(augmentation) 97 | 98 | 99 | def _random_brightness(image, brightness_std, p=1.0, **kwargs): 100 | """[summary] 101 | 102 | Args: 103 | image ([type]): [description] 104 | brightness_std ([type]): [description] 105 | p (float, optional): [description]. Defaults to 1.0. 106 | 107 | Returns: 108 | [type]: [description] 109 | """ 110 | if brightness_std <= 0: 111 | return image 112 | 113 | if random.random() >= p: 114 | return image 115 | 116 | gauss = np.random.normal(0, brightness_std) 117 | brightness_noise = gauss * image 118 | noised = image + brightness_noise 119 | 120 | return noised 121 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/transforms/preprocesses.py: -------------------------------------------------------------------------------- 1 | import functools 2 | 3 | import numpy as np 4 | 5 | import albumentations as albu 6 | import torch 7 | 8 | 9 | def get_spacenet7_preprocess(config, is_test): 10 | """[summary] 11 | 12 | Args: 13 | config ([type]): [description] 14 | is_test (bool): [description] 15 | 16 | Returns: 17 | [type]: [description] 18 | """ 19 | in_channels = config.MODEL.IN_CHANNELS 20 | assert in_channels in [3, 4] 21 | 22 | if in_channels == 3: 23 | mean = np.array([0.485, 0.456, 0.406 24 | ]) * 255 # imagenet mean in RGB order 25 | std = np.array([0.229, 0.224, 0.225 26 | ]) * 255 # imagenet std in RGB order 27 | else: 28 | mean = np.array([0.485, 0.456, 0.406, 0.5 29 | ]) * 255 # imagenet mean in RGB order + alpha mean 30 | std = np.array([0.229, 0.224, 0.225, 0.5 31 | ]) * 255 # imagenet std in RGB order + aplha std 32 | 33 | # tile mean and std if concat previous/next frames 34 | N_tile = 1 35 | if config.INPUT.CONCAT_PREV_FRAME: 36 | N_tile += 1 37 | if config.INPUT.CONCAT_NEXT_FRAME: 38 | N_tile += 1 39 | mean = np.tile(mean, N_tile) 40 | std = np.tile(std, N_tile) 41 | 42 | mean = mean[np.newaxis, np.newaxis, :] 43 | std = std[np.newaxis, np.newaxis, :] 44 | 45 | if is_test: 46 | to_tensor = albu.Lambda(image=functools.partial(_to_tensor)) 47 | else: 48 | to_tensor = albu.Lambda(image=functools.partial(_to_tensor), 49 | mask=functools.partial(_to_tensor)) 50 | 51 | preprocess = [ 52 | albu.Lambda( 53 | image=functools.partial(_normalize_image, mean=mean, std=std)), 54 | to_tensor, 55 | ] 56 | return albu.Compose(preprocess) 57 | 58 | 59 | def _normalize_image(image, mean, std, **kwargs): 60 | """[summary] 61 | 62 | Args: 63 | image ([type]): [description] 64 | mean ([type]): [description] 65 | std ([type]): [description] 66 | 67 | Returns: 68 | [type]: [description] 69 | """ 70 | normalized = image.astype('float32') 71 | normalized = (image - mean) / std 72 | return normalized 73 | 74 | 75 | def _to_tensor(x, **kwargs): 76 | """[summary] 77 | 78 | Args: 79 | x ([type]): [description] 80 | 81 | Returns: 82 | [type]: [description] 83 | """ 84 | x = x.transpose(2, 0, 1).astype('float32') 85 | return torch.from_numpy(x) 86 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/utils/__init__.py: -------------------------------------------------------------------------------- 1 | from .checkpoint import * 2 | from .utils import * 3 | -------------------------------------------------------------------------------- /4-motokimura/code/spacenet7_model/utils/checkpoint.py: -------------------------------------------------------------------------------- 1 | import os.path 2 | 3 | import torch 4 | 5 | 6 | def checkpoint_epoch_filename(epoch): 7 | """[summary] 8 | 9 | Args: 10 | epoch ([type]): [description] 11 | 12 | Returns: 13 | [type]: [description] 14 | """ 15 | assert 0 <= epoch <= 9999 16 | return f'checkpoint_{epoch:04d}.ckpt' 17 | 18 | 19 | def checkpoint_latest_filename(): 20 | """[summary] 21 | 22 | Returns: 23 | [type]: [description] 24 | """ 25 | return 'checkpoint_latest.ckpt' 26 | 27 | 28 | def save_checkpoint(checkpoint_path, model, optimizer, lr_scheduler, epoch, 29 | best_score): 30 | """[summary] 31 | 32 | Args: 33 | checkpoint_path ([type]): [description] 34 | model ([type]): [description] 35 | optimizer ([type]): [description] 36 | lr_scheduler ([type]): [description] 37 | epoch ([type]): [description] 38 | best_score ([type]): [description] 39 | """ 40 | checkpoint = { 41 | 'model': model.state_dict(), 42 | 'optimizer': optimizer.state_dict(), 43 | 'lr_scheduler': lr_scheduler.state_dict(), 44 | 'epoch': epoch, 45 | 'best_score': best_score 46 | } 47 | torch.save(checkpoint, checkpoint_path) 48 | 49 | 50 | def load_checkpoint(checkpoint_path, model, optimizer, lr_scheduler, epoch, 51 | best_score): 52 | """[summary] 53 | 54 | Args: 55 | checkpoint_path ([type]): [description] 56 | model ([type]): [description] 57 | optimizer ([type]): [description] 58 | lr_scheduler ([type]): [description] 59 | epoch ([type]): [description] 60 | best_score ([type]): [description] 61 | 62 | Returns: 63 | [type]: [description] 64 | """ 65 | checkpoint = torch.load(checkpoint_path) 66 | model.load_state_dict(checkpoint['model']) 67 | optimizer.load_state_dict(checkpoint['optimizer']) 68 | lr_scheduler.load_state_dict(checkpoint['lr_scheduler']) 69 | epoch = checkpoint['epoch'] 70 | best_score = checkpoint['best_score'] 71 | 72 | return model, optimizer, lr_scheduler, epoch, best_score 73 | 74 | 75 | def checkpoint_exists(checkpoint_dir): 76 | """[summary] 77 | 78 | Args: 79 | checkpoint_dir ([type]): [description] 80 | 81 | Returns: 82 | [type]: [description] 83 | """ 84 | checkpoint_path = os.path.join(checkpoint_dir, 85 | checkpoint_latest_filename()) 86 | return os.path.exists(checkpoint_path) 87 | 88 | 89 | def load_latest_checkpoint(checkpoint_dir, model, optimizer, lr_scheduler, 90 | epoch, best_score): 91 | """[summary] 92 | 93 | Args: 94 | checkpoint_dir ([type]): [description] 95 | model ([type]): [description] 96 | optimizer ([type]): [description] 97 | lr_scheduler ([type]): [description] 98 | epoch ([type]): [description] 99 | best_score ([type]): [description] 100 | 101 | Returns: 102 | [type]: [description] 103 | """ 104 | if checkpoint_exists(checkpoint_dir): 105 | print('resume training from the last checkpoint!') 106 | model, optimizer, lr_scheduler, epoch, best_score = load_checkpoint( 107 | os.path.join(checkpoint_dir, checkpoint_latest_filename()), model, 108 | optimizer, lr_scheduler, epoch, best_score) 109 | 110 | return model, optimizer, lr_scheduler, epoch, best_score 111 | -------------------------------------------------------------------------------- /4-motokimura/code/static/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/4-motokimura/code/static/.gitkeep -------------------------------------------------------------------------------- /4-motokimura/code/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | START_TIME=$SECONDS 4 | 5 | source activate solaris 6 | 7 | TEST_ROOT=$1 # path/to/SN7_buildings/test/ 8 | OUTPUT_CSV_PATH=$2 # path/to/solution.csv 9 | 10 | source /work/settings.sh 11 | 12 | rm -rf ${PREDICTION_ROOT} ${ENSEMBLED_PREDICTION_ROOT} ${REFINED_PREDICTION_ROOT} ${POLY_ROOT} ${TRACKED_POLY_ROOT} ${TEST_STDOUT_ROOT} 13 | 14 | # predict with trained models 15 | /work/scripts/test/test_cnns.sh ${TEST_ROOT} 16 | 17 | # postprocess predictions 18 | /work/scripts/test/postprocess.sh ${TEST_ROOT} ${OUTPUT_CSV_PATH} 19 | 20 | ELAPSED_TIME=$(($SECONDS - $START_TIME)) 21 | echo 'Total time for testing: ' $(($ELAPSED_TIME / 60 + 1)) '[min]' -------------------------------------------------------------------------------- /4-motokimura/code/tools/_init_path.py: -------------------------------------------------------------------------------- 1 | import os.path 2 | import sys 3 | 4 | 5 | def init_path(): 6 | """[summary] 7 | """ 8 | this_dir = os.path.dirname(os.path.abspath(__file__)) 9 | proj_dir = os.path.abspath(os.path.join(this_dir, '..')) 10 | sys.path.append(proj_dir) 11 | 12 | 13 | init_path() 14 | -------------------------------------------------------------------------------- /4-motokimura/code/tools/ensemble_models.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import json 4 | import os.path 5 | import multiprocessing as mp 6 | import timeit 7 | 8 | import numpy as np 9 | 10 | import _init_path 11 | from skimage import io 12 | from spacenet7_model.configs import load_config 13 | from spacenet7_model.utils import (dump_prediction_to_png, ensemble_subdir, 14 | experiment_subdir, get_aoi_from_path, 15 | get_image_paths, load_prediction_from_png, 16 | map_wrapper, val_list_filename) 17 | from tqdm import tqdm 18 | 19 | 20 | def ensemble_preds(image_path, aoi, out_dir, weights, config): 21 | """[summary] 22 | 23 | Args: 24 | image_path ([type]): [description] 25 | aoi ([type]): [description] 26 | out_root ([type]): [description] 27 | weights ([type]): [description] 28 | config ([type]): [description] 29 | """ 30 | image_orig = io.imread(image_path) 31 | roi_mask = image_orig[:, :, 3] > 0 32 | h, w = roi_mask.shape 33 | ensembled_score = np.zeros(shape=[len(config.INPUT.CLASSES), h, w]) 34 | 35 | image_filename = os.path.basename(image_path) 36 | array_filename, _ = os.path.splitext(image_filename) 37 | array_filename = f'{array_filename}.png' 38 | 39 | for exp_id, weight in zip(config.ENSEMBLE_EXP_IDS, weights): 40 | exp_subdir = experiment_subdir(exp_id) 41 | score_array = load_prediction_from_png( 42 | os.path.join(config.PREDICTION_ROOT, exp_subdir, aoi, 43 | array_filename), 44 | n_channels=len(config.INPUT.CLASSES)) 45 | score_array[:, np.logical_not(roi_mask)] = 0 46 | assert score_array.min() >= 0 and score_array.max() <= 1 47 | ensembled_score += score_array * weight 48 | 49 | assert ensembled_score.min() >= 0 and ensembled_score.max() <= 1 50 | dump_prediction_to_png(os.path.join(out_dir, array_filename), 51 | ensembled_score) 52 | 53 | 54 | if __name__ == '__main__': 55 | t0 = timeit.default_timer() 56 | 57 | config = load_config() 58 | 59 | assert len(config.ENSEMBLE_EXP_IDS) >= 1 60 | N = len(config.ENSEMBLE_EXP_IDS) 61 | 62 | n_thread = config.ENSEMBLE_NUM_THREADS 63 | n_thread = n_thread if n_thread > 0 else mp.cpu_count() 64 | print(f'N_thread for multiprocessing: {n_thread}') 65 | 66 | # prepare ensemble weights 67 | if len(config.ENSEMBLE_WEIGHTS) == 0: 68 | weights = np.ones(shape=(N)) 69 | else: 70 | assert len(config.ENSEMBLE_WEIGHTS) == N 71 | weights = np.array(config.ENSEMBLE_WEIGHTS) 72 | weights = weights / weights.sum() 73 | 74 | # get full paths to image files 75 | if config.TEST_TO_VAL: 76 | # use val split for test. 77 | data_list_path = os.path.join( 78 | config.INPUT.TRAIN_VAL_SPLIT_DIR, 79 | val_list_filename(config.INPUT.TRAIN_VAL_SPLIT_ID)) 80 | with open(data_list_path) as f: 81 | data_list = json.load(f) 82 | image_paths = [data['image_masked'] for data in data_list] 83 | else: 84 | # use test data for test (default). 85 | image_paths = get_image_paths(config.INPUT.TEST_DIR) 86 | 87 | subdir = ensemble_subdir(config.ENSEMBLE_EXP_IDS) 88 | out_root = os.path.join(config.ENSEMBLED_PREDICTION_ROOT, subdir) 89 | os.makedirs(out_root, exist_ok=False) 90 | 91 | print('preparing input args...') 92 | input_args = [] 93 | for image_path in image_paths: 94 | aoi = get_aoi_from_path(image_path) 95 | # prepare aoi sub directory to output ensemble results 96 | out_dir = os.path.join(out_root, aoi) 97 | os.makedirs(out_dir, exist_ok=True) 98 | input_args.append( 99 | [ensemble_preds, image_path, aoi, out_dir, weights, config]) 100 | 101 | print('running multiprocessing...') 102 | with mp.Pool(processes=n_thread) as pool: 103 | with tqdm(total=len(input_args)) as t: 104 | for _ in pool.imap_unordered(map_wrapper, input_args): 105 | t.update(1) 106 | 107 | elapsed = timeit.default_timer() - t0 108 | print('Time: {:.3f} min'.format(elapsed / 60.0)) 109 | -------------------------------------------------------------------------------- /4-motokimura/code/tools/filter_small_polygons.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import argparse 3 | import os 4 | import timeit 5 | from glob import glob 6 | 7 | import geopandas as gpd 8 | from shapely.geometry import Polygon 9 | from tqdm import tqdm 10 | 11 | import _init_path 12 | from spacenet7_model.utils import get_subdirs, save_empty_geojson 13 | 14 | 15 | def parse_args(): 16 | """[summary] 17 | 18 | Returns: 19 | [type]: [description] 20 | """ 21 | parser = argparse.ArgumentParser() 22 | parser.add_argument('--train_dir', 23 | help='directory containing spacenet7 train dataset', 24 | default='/data/spacenet7/spacenet7/train/') 25 | parser.add_argument('--out_dir', 26 | help='directory to output mask images', 27 | default='/data/spacenet7/labels_pix_filtered/') 28 | parser.add_argument('--min_area', 29 | help='min area in pixels', 30 | type=float, 31 | default=4.0) 32 | return parser.parse_args() 33 | 34 | 35 | if __name__ == '__main__': 36 | t0 = timeit.default_timer() 37 | 38 | args = parse_args() 39 | 40 | os.makedirs(args.out_dir, exist_ok=False) 41 | 42 | aois = get_subdirs(args.train_dir) 43 | for i, aoi in enumerate(aois): 44 | print(f'processing {aoi} ({i + 1}/{len(aois)}) ...') 45 | 46 | json_paths = glob( 47 | os.path.join(args.train_dir, aoi, 'labels_match_pix', '*.geojson')) 48 | json_paths.sort() 49 | 50 | out_dir = os.path.join(args.out_dir, aoi) 51 | os.makedirs(out_dir, exist_ok=False) 52 | 53 | for json_path in tqdm(json_paths): 54 | df = gpd.read_file(json_path) 55 | 56 | # add `area` column 57 | areas = [] 58 | for index, row in df.iterrows(): 59 | area = Polygon(row['geometry']).area 60 | areas.append(area) 61 | df['area'] = areas 62 | 63 | # filter out small polygons 64 | mask = df['area'] >= args.min_area 65 | df = df[mask] 66 | 67 | # dump 68 | output_path = os.path.join(out_dir, os.path.basename(json_path)) 69 | if len(df) > 0: 70 | df.to_file(output_path, driver='GeoJSON') 71 | else: 72 | print(f'warning: {output_path} is an empty geojson.') 73 | save_empty_geojson(output_path) 74 | 75 | elapsed = timeit.default_timer() - t0 76 | print('Time: {:.3f} min'.format(elapsed / 60.0)) 77 | -------------------------------------------------------------------------------- /4-motokimura/code/tools/split_dataset.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import json 5 | import os 6 | import random 7 | import timeit 8 | from glob import glob 9 | 10 | import numpy as np 11 | 12 | 13 | def parse_args(): 14 | """[summary] 15 | 16 | Returns: 17 | [type]: [description] 18 | """ 19 | parser = argparse.ArgumentParser() 20 | parser.add_argument('--train_dir', 21 | help='directory containing spacenet7 train dataset', 22 | default='/data/spacenet7/spacenet7/train/') 23 | parser.add_argument('--mask_dir', 24 | help='directory containing building mask image files', 25 | default='/data/spacenet7/building_masks/') 26 | parser.add_argument('--out_dir', 27 | help='output root directory', 28 | default='/data/spacenet7/split/') 29 | parser.add_argument('--split_num', 30 | help='number of split', 31 | type=int, 32 | default=5) 33 | return parser.parse_args() 34 | 35 | 36 | def dump_file_paths(aois, output_path, train_dir, mask_dir): 37 | """[summary] 38 | 39 | Args: 40 | aois ([type]): [description] 41 | output_path ([type]): [description] 42 | train_dir ([type]): [description] 43 | mask_dir ([type]): [description] 44 | """ 45 | 46 | results = [] 47 | 48 | for aoi in aois: 49 | image_paths = glob( 50 | os.path.join(train_dir, aoi, 'images_masked', '*.tif')) 51 | image_paths.sort() 52 | 53 | N = len(image_paths) 54 | for i in range(N): 55 | # get path to mask 56 | image_path = image_paths[i] 57 | filename = os.path.basename(image_path) 58 | mask_path = os.path.join(mask_dir, aoi, filename) 59 | assert os.path.exists(mask_path) 60 | 61 | # previous frame 62 | image_prev_path = image_paths[0] if i == 0 \ 63 | else image_paths[i - 1] 64 | 65 | # next frame 66 | image_next_path = image_paths[N - 1] if i == N - 1 \ 67 | else image_paths[i + 1] 68 | 69 | result = {} 70 | result['image_masked'] = image_path 71 | result['building_mask'] = mask_path 72 | result['image_masked_prev'] = image_prev_path 73 | result['image_masked_next'] = image_next_path 74 | results.append(result) 75 | 76 | with open(output_path, 'w') as f: 77 | json.dump(results, 78 | f, 79 | ensure_ascii=False, 80 | indent=4, 81 | sort_keys=False, 82 | separators=(',', ': ')) 83 | 84 | 85 | if __name__ == '__main__': 86 | t0 = timeit.default_timer() 87 | 88 | args = parse_args() 89 | 90 | os.makedirs(args.out_dir) 91 | 92 | aois = sorted([ 93 | d for d in os.listdir(args.train_dir) 94 | if os.path.isdir(os.path.join(args.train_dir, d)) 95 | ]) 96 | 97 | random.seed(777) 98 | random.shuffle(aois) 99 | 100 | # split aois into train and val 101 | n = args.split_num 102 | aois_divided = np.array([aois[i::n] for i in range(n)]) 103 | 104 | for val_idx in range(n): 105 | # dump file paths for val split 106 | val_aois = aois_divided[val_idx] 107 | 108 | dump_file_paths(val_aois, 109 | os.path.join(args.out_dir, f'val_{val_idx}.json'), 110 | args.train_dir, args.mask_dir) 111 | 112 | # dump file paths for train split 113 | train_mask = np.ones(n, dtype=bool) 114 | train_mask[val_idx] = False 115 | train_aois = aois_divided[train_mask] 116 | train_aois = np.concatenate(train_aois, axis=0).tolist() 117 | 118 | dump_file_paths(train_aois, 119 | os.path.join(args.out_dir, f'train_{val_idx}.json'), 120 | args.train_dir, args.mask_dir) 121 | 122 | elapsed = timeit.default_timer() - t0 123 | print('Time: {:.3f} min'.format(elapsed / 60.0)) 124 | -------------------------------------------------------------------------------- /4-motokimura/code/tools/split_dataset_random.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import json 5 | import os 6 | import random 7 | import timeit 8 | from glob import glob 9 | 10 | import numpy as np 11 | 12 | import _init_path 13 | from spacenet7_model.utils import get_image_paths, get_aoi_from_path 14 | 15 | 16 | def parse_args(): 17 | """[summary] 18 | 19 | Returns: 20 | [type]: [description] 21 | """ 22 | parser = argparse.ArgumentParser() 23 | parser.add_argument('--train_dir', 24 | help='directory containing spacenet7 train dataset', 25 | default='/data/spacenet7/spacenet7/train/') 26 | parser.add_argument('--mask_dir', 27 | help='directory containing building mask image files', 28 | default='/data/spacenet7/building_masks/') 29 | parser.add_argument('--out_dir', 30 | help='output root directory', 31 | default='/data/spacenet7/split_random/') 32 | parser.add_argument('--split_num', 33 | help='number of split', 34 | type=int, 35 | default=5) 36 | return parser.parse_args() 37 | 38 | 39 | def dump_file_paths(image_paths, output_path, mask_dir): 40 | """[summary] 41 | 42 | Args: 43 | image_paths ([type]): [description] 44 | output_path ([type]): [description] 45 | mask_dir ([type]): [description] 46 | """ 47 | 48 | results = [] 49 | 50 | for image_path in image_paths: 51 | filename = os.path.basename(image_path) 52 | aoi = get_aoi_from_path(image_path) 53 | mask_path = os.path.join(mask_dir, aoi, filename) 54 | assert os.path.exists(mask_path) 55 | 56 | result = {} 57 | result['image_masked'] = image_path 58 | result['building_mask'] = mask_path 59 | results.append(result) 60 | 61 | with open(output_path, 'w') as f: 62 | json.dump(results, 63 | f, 64 | ensure_ascii=False, 65 | indent=4, 66 | sort_keys=False, 67 | separators=(',', ': ')) 68 | 69 | 70 | if __name__ == '__main__': 71 | t0 = timeit.default_timer() 72 | 73 | args = parse_args() 74 | 75 | os.makedirs(args.out_dir) 76 | 77 | image_paths = get_image_paths(args.train_dir) 78 | image_paths.sort() 79 | 80 | random.seed(777) 81 | random.shuffle(image_paths) 82 | 83 | # split aois into train and val 84 | n = args.split_num 85 | image_paths_divided = np.array([image_paths[i::n] for i in range(n)]) 86 | 87 | for val_idx in range(n): 88 | # dump file paths for val split 89 | val_image_paths = image_paths_divided[val_idx] 90 | 91 | dump_file_paths(val_image_paths, 92 | os.path.join(args.out_dir, f'val_{val_idx}.json'), 93 | args.mask_dir) 94 | 95 | # dump file paths for train split 96 | train_mask = np.ones(n, dtype=bool) 97 | train_mask[val_idx] = False 98 | train_image_paths = image_paths_divided[train_mask] 99 | train_image_paths = np.concatenate(train_image_paths, axis=0).tolist() 100 | 101 | dump_file_paths(train_image_paths, 102 | os.path.join(args.out_dir, f'train_{val_idx}.json'), 103 | args.mask_dir) 104 | 105 | elapsed = timeit.default_timer() - t0 106 | print('Time: {:.3f} min'.format(elapsed / 60.0)) 107 | -------------------------------------------------------------------------------- /4-motokimura/code/train.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | START_TIME=$SECONDS 4 | 5 | source activate solaris 6 | 7 | TRAIN_ROOT=$1 # path/to/SN7_buildings/train/ 8 | 9 | source /work/settings.sh 10 | 11 | # removing motokimura's home build models here! 12 | rm -rf ${MODEL_ROOT}/* 13 | 14 | # preprocess dataset 15 | /work/scripts/train/preprocess.sh ${TRAIN_ROOT} 16 | 17 | # train CNN models 18 | /work/scripts/train/train_cnns.sh 19 | 20 | ELAPSED_TIME=$(($SECONDS - $START_TIME)) 21 | echo 'Total time for testing: ' $(($ELAPSED_TIME / 60 + 1)) '[min]' 22 | -------------------------------------------------------------------------------- /4-motokimura/detection_pipeline_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/4-motokimura/detection_pipeline_01.png -------------------------------------------------------------------------------- /4-motokimura/weights/README.txt: -------------------------------------------------------------------------------- 1 | See s3://spacenet-dataset/spacenet-model-weights/spacenet-7/ -------------------------------------------------------------------------------- /5-MaksimovKA/MaksimovKA Solution Description.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/5-MaksimovKA/MaksimovKA Solution Description.docx -------------------------------------------------------------------------------- /5-MaksimovKA/code/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM catalystteam/catalyst:20.04 2 | ENV DEBIAN_FRONTEND=noninteractive 3 | RUN apt-get update 4 | RUN apt-get install -y wget htop mc 5 | 6 | RUN conda install -y gdal 7 | 8 | COPY ./requirements.txt /tmp 9 | RUN pip install -r /tmp/requirements.txt 10 | 11 | RUN mkdir /project 12 | COPY ./ /project/ 13 | 14 | RUN chmod +x /project/train.sh 15 | RUN chmod +x /project/test.sh 16 | WORKDIR /project 17 | 18 | ENV PYTHONPATH "${PYTHONPATH}:/project" 19 | ENV TORCH_HOME="/wdata/pretrained_models/" -------------------------------------------------------------------------------- /5-MaksimovKA/code/callbacks.py: -------------------------------------------------------------------------------- 1 | from catalyst.core.callbacks import MetricCallback 2 | import torch 3 | 4 | from catalyst.utils import get_activation_fn 5 | 6 | def dice( 7 | outputs: torch.Tensor, 8 | targets: torch.Tensor, 9 | eps: float = 1e-7, 10 | threshold: float = 0.5, 11 | activation: str = "Sigmoid" 12 | ): 13 | """ 14 | Computes the dice metric 15 | Args: 16 | outputs (list): A list of predicted elements 17 | targets (list): A list of elements that are to be predicted 18 | eps (float): epsilon 19 | threshold (float): threshold for outputs binarization 20 | activation (str): An torch.nn activation applied to the outputs. 21 | Must be one of ["none", "Sigmoid", "Softmax2d"] 22 | Returns: 23 | double: Dice score 24 | """ 25 | outputs = outputs[:, 0, ...] 26 | targets = targets[:, 0, ...] 27 | 28 | activation_fn = get_activation_fn(activation) 29 | outputs = activation_fn(outputs) 30 | 31 | if threshold is not None: 32 | outputs = (outputs > threshold).float() 33 | 34 | intersection = torch.sum(targets * outputs) 35 | union = torch.sum(targets) + torch.sum(outputs) 36 | # this looks a bit awkward but `eps * (union == 0)` term 37 | # makes sure that if I and U are both 0, than Dice == 1 38 | # and if U != 0 and I == 0 the eps term in numerator is zeroed out 39 | # i.e. (0 + eps) / (U - 0 + eps) doesn't happen 40 | dice = 2 * (intersection + eps * (union == 0)) / (union + eps) 41 | 42 | return dice 43 | 44 | class DiceCallback(MetricCallback): 45 | """ 46 | Dice metric callback. 47 | """ 48 | def __init__( 49 | self, 50 | input_key: str = "targets", 51 | output_key: str = "logits", 52 | prefix: str = "dice", 53 | eps: float = 1e-7, 54 | threshold: float = None, 55 | activation: str = "Sigmoid" 56 | ): 57 | """ 58 | Args: 59 | input_key (str): input key to use for dice calculation; 60 | specifies our `y_true`. 61 | output_key (str): output key to use for dice calculation; 62 | specifies our `y_pred`. 63 | """ 64 | super().__init__( 65 | prefix=prefix, 66 | metric_fn=dice, 67 | input_key=input_key, 68 | output_key=output_key, 69 | eps=eps, 70 | threshold=threshold, 71 | activation=activation 72 | ) 73 | -------------------------------------------------------------------------------- /5-MaksimovKA/code/config.py: -------------------------------------------------------------------------------- 1 | import albumentations as albu 2 | import segmentation_models_pytorch as smp 3 | 4 | logs_path = '/wdata/segmentation_logs/' 5 | folds_file = '/wdata/folds.csv' 6 | load_from = '/wdata/segmentation_logs/fold_1_siamse-senet154/checkpoints/best.pth' 7 | multiplier = 5 8 | 9 | main_metric = 'dice' 10 | minimize_metric = False 11 | device = 'cuda' 12 | val_fold = 1 13 | folds_to_use = (2, 3, 4, 5, 6, 7, 8) 14 | n_classes = 3 15 | input_channels = 3 16 | crop_size = (320, 320) 17 | val_size = (1024, 1024) 18 | 19 | batch_size = 4 20 | num_workers = 4 21 | val_batch_size = 1 22 | 23 | shuffle = True 24 | lr = 1e-4 25 | momentum = 0.0 26 | decay = 0.0 27 | loss = 'focal_dice' 28 | optimizer = 'adam_gcc' 29 | fp16 = False 30 | 31 | alias = 'fold_' 32 | model_name = 'siamse-senet154' 33 | scheduler = 'steps' 34 | steps = [15, 25] 35 | step_gamma = 0.25 36 | augs_p = 0.5 37 | best_models_count = 5 38 | epochs = 30 39 | weights = 'imagenet' 40 | limit_files = None 41 | 42 | preprocessing_fn = smp.encoders.get_preprocessing_fn('senet154', weights) 43 | 44 | train_augs = albu.Compose([albu.OneOf([albu.RandomCrop(crop_size[0], crop_size[1], p=1.0) 45 | ], p=1.0), 46 | albu.Flip(p=augs_p), 47 | albu.RandomBrightnessContrast(brightness_limit=0.1, contrast_limit=0.1, p=augs_p), 48 | ], p=augs_p) 49 | 50 | valid_augs = albu.Compose([albu.PadIfNeeded(min_height=val_size[0], min_width=val_size[1], p=1.0)]) -------------------------------------------------------------------------------- /5-MaksimovKA/code/create_folds.py: -------------------------------------------------------------------------------- 1 | import os 2 | import fire 3 | import pandas as pd 4 | import numpy as np 5 | from sklearn.model_selection import KFold 6 | 7 | 8 | def create_folds(images_path='/data/SN7_buildings/train/', 9 | seed=769, 10 | n_folds=8, 11 | out_file='/wdata/folds.csv', 12 | masks_path='/wdata/train_masks/'): 13 | aois = [el for el in os.listdir(images_path) if os.path.isdir(os.path.join(images_path, el))] 14 | all_ids = np.array(sorted(aois)) 15 | sub_targets = [] 16 | kf = KFold(n_splits=n_folds, random_state=seed, shuffle=True) 17 | for i, (_, evaluate_index) in enumerate(kf.split(all_ids)): 18 | # print(i) 19 | ids = all_ids[evaluate_index] 20 | print(i, len(ids), ids) 21 | for _id in ids: 22 | id_path = os.path.join(images_path, _id, 'images_masked') 23 | files = os.listdir(id_path) 24 | masks = [os.path.join(masks_path, _id, el) for el in files] 25 | files = [os.path.join(id_path, el) for el in files] 26 | 27 | target_df = {'image_path': [], 'mask_path': [], 'fold_number': []} 28 | target_df['image_path'] += files 29 | target_df['fold_number'] += [i + 1 for el in files] 30 | target_df['mask_path'] += masks 31 | 32 | target_df = pd.DataFrame(target_df) 33 | target_df = target_df[['image_path', 'mask_path', 'fold_number']] 34 | 35 | sub_targets.append(target_df) 36 | target_df = pd.concat(sub_targets) 37 | target_df.to_csv(out_file, index=False) 38 | print(target_df.head()) 39 | 40 | 41 | if __name__ == '__main__': 42 | fire.Fire(create_folds) 43 | -------------------------------------------------------------------------------- /5-MaksimovKA/code/create_masks.py: -------------------------------------------------------------------------------- 1 | import shutil 2 | import os 3 | import fire 4 | import numpy as np 5 | import cv2 6 | import geopandas as gpd 7 | import skimage.io 8 | from tqdm import tqdm 9 | from scipy import ndimage as ndi 10 | from skimage import measure 11 | from skimage.morphology import dilation, square, watershed 12 | from scipy.ndimage import binary_erosion 13 | from multiprocessing.pool import Pool 14 | from multiprocessing import cpu_count 15 | from rasterio import features 16 | from scipy.ndimage import binary_dilation 17 | 18 | 19 | def create_separation(labels): 20 | tmp = dilation(labels > 0, square(3)) 21 | tmp2 = watershed(tmp, labels, mask=tmp, watershed_line=True) > 0 22 | tmp = tmp ^ tmp2 23 | tmp = dilation(tmp, square(3)) 24 | 25 | msk1 = np.zeros_like(labels, dtype='bool') 26 | 27 | for y0 in range(labels.shape[0]): 28 | for x0 in range(labels.shape[1]): 29 | if not tmp[y0, x0]: 30 | continue 31 | sz = 1 32 | uniq = np.unique(labels[max(0, y0 - sz):min(labels.shape[0], y0 + sz + 1), 33 | max(0, x0 - sz):min(labels.shape[1], x0 + sz + 1)]) 34 | if len(uniq[uniq > 0]) > 1: 35 | msk1[y0, x0] = True 36 | return msk1 37 | 38 | 39 | def mask_fro_id(param): 40 | _id, labels_path, rasters_path, result_path = param 41 | label_path = os.path.join(labels_path, _id + '_Buildings.geojson') 42 | raster_path = os.path.join(rasters_path, _id + '.tif') 43 | geoms = gpd.read_file(label_path)['geometry'].tolist() 44 | image = cv2.imread(raster_path) 45 | h, w, c = image.shape 46 | 47 | buildings = np.zeros((h, w), dtype=np.int64) 48 | outer_contur = np.zeros((h, w), dtype=np.int64) 49 | inter_contur = np.zeros((h, w), dtype=np.int64) 50 | contour_size = 1 51 | 52 | for i in range(len(geoms)): 53 | mask = features.rasterize([(geoms[i], 1)], out_shape=(h, w)) 54 | buildings += mask 55 | dilated = binary_dilation(mask, iterations=contour_size) 56 | countour_mask = dilated ^ mask 57 | outer_contur += countour_mask 58 | 59 | eroded = binary_erosion(mask, iterations=contour_size) 60 | countour_mask = eroded ^ mask 61 | inter_contur += countour_mask 62 | 63 | outer_contur = (outer_contur > 0).astype(np.uint8) 64 | inter_contur = (inter_contur > 0).astype(np.uint8) 65 | buildings = (buildings > 0).astype(np.uint8) 66 | buildings[outer_contur == 1] = 0 67 | 68 | labels = ndi.label(buildings, output=np.uint32)[0] 69 | separation = create_separation(labels) 70 | separation = separation > 0 71 | separation = separation.astype(np.uint8) 72 | 73 | result = np.zeros((h, w, 3), dtype=np.uint8) 74 | result[:, :, 0] = buildings * 255 75 | result[:, :, 1] = inter_contur * 255 76 | result[:, :, 2] = separation * 255 77 | out_path = os.path.join(result_path, _id + '.tif') 78 | skimage.io.imsave(out_path, result, plugin='tifffile') 79 | 80 | 81 | def create_masks(data_root_path='/data/SN7_buildings/train/', 82 | result_path='/wdata/train_masks/'): 83 | 84 | if os.path.exists(result_path): 85 | shutil.rmtree(result_path) 86 | os.mkdir(result_path) 87 | ids = os.listdir(data_root_path) 88 | all_params = [] 89 | for _id in tqdm(ids[:]): 90 | id_path = os.path.join(data_root_path, _id) 91 | if not os.path.isdir(id_path): 92 | continue 93 | sub_res_path = os.path.join(result_path, _id) 94 | os.mkdir(sub_res_path) 95 | labels_path = os.path.join(id_path, 'labels_match_pix') 96 | rasters_path = os.path.join(id_path, 'images') 97 | 98 | files = sorted(os.listdir(labels_path)) 99 | files = [el for el in files if 'UDM' not in el] 100 | files = ['_'.join(el.split('.')[0] .split('_')[:-1]) for el in files] 101 | params = [(el, labels_path, rasters_path, sub_res_path) for el in files] 102 | all_params += params 103 | 104 | n_cpus = cpu_count() 105 | pool = Pool(n_cpus) 106 | for _ in tqdm(pool.imap_unordered(mask_fro_id, all_params), total=len(all_params)): 107 | pass 108 | 109 | 110 | if __name__ == '__main__': 111 | fire.Fire(create_masks) 112 | -------------------------------------------------------------------------------- /5-MaksimovKA/code/flags.txt: -------------------------------------------------------------------------------- 1 | --ipc=host --shm-size 16G -------------------------------------------------------------------------------- /5-MaksimovKA/code/losses.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch import Tensor 3 | from torch.nn.modules.loss import _Loss 4 | 5 | 6 | def get_loss(loss_name='focal_dice'): 7 | if loss_name == 'focal_dice': 8 | return FocalDiceLoss() 9 | 10 | 11 | class FocalDiceLoss(torch.nn.Module): 12 | def __init__(self, coef_focal=1.0, coef_dice=1.0, weights=(1.0, 0.0, 1.0, 1.0, 0.0, 1.0)): 13 | super().__init__() 14 | 15 | self.dice_loss = DiceLoss() 16 | self.focal_loss = FocalLoss() 17 | self.weights = weights 18 | 19 | self.coef_focal = coef_focal 20 | self.coef_dice = coef_dice 21 | 22 | def forward(self, outputs, targets): 23 | loss = 0.0 24 | # print(outputs.shape, targets.shape) 25 | for i in range(0, 3): 26 | dice = self.weights[i] * self.dice_loss(outputs[:, i, ...], targets[:, i, ...]) 27 | focal = self.weights[i] * self.focal_loss(outputs[:, i, ...], targets[:, i, ...]) 28 | loss += self.coef_dice * dice + self.coef_focal * focal 29 | 30 | for i in range(3, 6): 31 | dice = self.weights[i] * self.dice_loss(outputs[:, i, ...], targets[:, i, ...]) 32 | focal = self.weights[i] * self.focal_loss(outputs[:, i, ...], targets[:, i, ...]) 33 | loss += self.coef_dice * dice + self.coef_focal * focal 34 | 35 | return loss 36 | 37 | 38 | class DiceLoss(_Loss): 39 | 40 | def __init__(self, per_image=False): 41 | super(DiceLoss, self).__init__() 42 | self.per_image = per_image 43 | 44 | def forward(self, y_pred: Tensor, y_true: Tensor): 45 | """ 46 | :param y_pred: NxCxHxW 47 | :param y_true: NxCxHxW 48 | :return: scalar 49 | """ 50 | per_image = self.per_image 51 | y_pred = y_pred.sigmoid() 52 | 53 | batch_size = y_pred.size()[0] 54 | eps = 1e-5 55 | if not per_image: 56 | batch_size = 1 57 | 58 | dice_target = y_true.contiguous().view(batch_size, -1).float() 59 | dice_output = y_pred.contiguous().view(batch_size, -1) 60 | intersection = torch.sum(dice_output * dice_target, dim=1) 61 | union = torch.sum(dice_output, dim=1) + torch.sum(dice_target, dim=1) + eps 62 | loss = (1 - (2 * intersection + eps) / union).mean() 63 | 64 | return loss 65 | 66 | 67 | class FocalLoss(_Loss): 68 | 69 | def __init__(self, ignore_index=255, gamma=2): 70 | super(FocalLoss, self).__init__() 71 | self.gamma = gamma 72 | self.ignore_index = ignore_index 73 | 74 | def forward(self, y_pred: Tensor, y_true: Tensor): 75 | """ 76 | :param y_pred: NxCxHxW 77 | :param y_true: NxCxHxW 78 | :return: scalar 79 | """ 80 | y_pred = y_pred.sigmoid() 81 | gamma = self.gamma 82 | ignore_index = self.ignore_index 83 | 84 | outputs = y_pred.contiguous() 85 | targets = y_true.contiguous() 86 | eps = 1e-8 87 | non_ignored = targets.view(-1) != ignore_index 88 | targets = targets.view(-1)[non_ignored].float() 89 | outputs = outputs.contiguous().view(-1)[non_ignored] 90 | outputs = torch.clamp(outputs, eps, 1. - eps) 91 | targets = torch.clamp(targets, eps, 1. - eps) 92 | pt = (1 - targets) * (1 - outputs) + targets * outputs 93 | return (-(1. - pt) ** gamma * torch.log(pt)).mean() 94 | 95 | 96 | -------------------------------------------------------------------------------- /5-MaksimovKA/code/mean_folds.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | import shutil 4 | from tqdm import tqdm 5 | import numpy as np 6 | 7 | paths = sorted([os.path.join('/wdata/folds_predicts/', el) for el in sorted(os.listdir('/wdata/folds_predicts/'))]) 8 | print(paths) 9 | save_path = '/wdata/mix_predicts/' 10 | if os.path.exists(save_path): 11 | shutil.rmtree(save_path) 12 | os.mkdir(save_path) 13 | 14 | aois = os.listdir(paths[0]) 15 | 16 | for aoi in tqdm(aois): 17 | aoi_save_path = os.path.join(save_path, aoi) 18 | if not os.path.exists(aoi_save_path): 19 | os.mkdir(aoi_save_path) 20 | files = os.listdir(os.path.join(paths[0], aoi)) 21 | for _file in files: 22 | for mask_i, model_path in enumerate(paths): 23 | 24 | mask_path = os.path.join(model_path, aoi, _file) 25 | #print(mask_path) 26 | if mask_i == 0: 27 | pred_data = cv2.imread(mask_path) / 255.0 28 | else: 29 | pred_data += cv2.imread(mask_path) / 255.0 30 | pred_data = pred_data / len(paths) 31 | data = (pred_data * 255).astype(np.uint8) 32 | file_name = os.path.join(save_path, aoi, _file) 33 | cv2.imwrite(file_name, data) -------------------------------------------------------------------------------- /5-MaksimovKA/code/optimizers.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from Adam import Adam_GCC 3 | 4 | 5 | def get_optimizer(optimizer_name, model, lr, momentum, decay): 6 | if optimizer_name == 'adam': 7 | 8 | optimizer = torch.optim.Adam(model.parameters(), 9 | lr, 10 | weight_decay=decay) 11 | 12 | elif optimizer_name == 'sgd': 13 | optimizer = torch.optim.SGD(model.parameters(), 14 | lr, 15 | momentum=momentum, 16 | weight_decay=decay) 17 | elif optimizer_name == 'adam_gcc': 18 | print('ADAM_GCC optimizer') 19 | optimizer = Adam_GCC(model.parameters(), 20 | lr, 21 | weight_decay=decay) 22 | 23 | else: 24 | optimizer = None 25 | return optimizer 26 | -------------------------------------------------------------------------------- /5-MaksimovKA/code/requirements.txt: -------------------------------------------------------------------------------- 1 | fire==0.2.1 2 | scikit-learn==0.22.2 3 | pandas==1.0.3 4 | numpy==1.18.2 5 | scikit-image==0.16.2 6 | pretrainedmodels==0.7.4 7 | segmentation_models_pytorch==0.1.0 8 | albumentations==0.4.5 9 | shapely==1.7.0 10 | wget==3.2 11 | gdown==3.10.3 12 | rasterio==1.1.3 13 | pytorch-toolbelt==0.3.1 14 | geopandas==0.8.1 15 | tensorflow==1.15.0 16 | tensorboard==1.15.0 -------------------------------------------------------------------------------- /5-MaksimovKA/code/submit.py: -------------------------------------------------------------------------------- 1 | import os 2 | import tqdm 3 | import glob 4 | import fiona 5 | import geopandas as gpd 6 | from fire import Fire 7 | 8 | 9 | def sn7_convert_geojsons_to_csv(json_dirs, output_csv_path, population='proposal'): 10 | ''' 11 | Convert jsons to csv 12 | Population is either "ground" or "proposal" 13 | ''' 14 | 15 | first_file = True # switch that will be turned off once we process the first file 16 | for json_dir in tqdm.tqdm(json_dirs[:]): 17 | json_files = sorted(glob.glob(os.path.join(json_dir, '*.geojson'))) 18 | for json_file in tqdm.tqdm(json_files): 19 | try: 20 | df = gpd.read_file(json_file) 21 | except (fiona.errors.DriverError): 22 | message = '! Invalid dataframe for %s' % json_file 23 | print(message) 24 | continue 25 | # raise Exception(message) 26 | if population == 'ground': 27 | file_name_col = df.image_fname.apply(lambda x: os.path.splitext(x)[0]) 28 | elif population == 'proposal': 29 | file_name_col = os.path.splitext(os.path.basename(json_file))[0] 30 | else: 31 | raise Exception('! Invalid population') 32 | 33 | if len(df) == 0: 34 | message = '! Empty dataframe for %s' % json_file 35 | print(message) 36 | # raise Exception(message) 37 | df = gpd.GeoDataFrame({ 38 | 'filename': file_name_col, 39 | 'id': 0, 40 | 'geometry': "POLYGON EMPTY", 41 | }) 42 | else: 43 | try: 44 | df = gpd.GeoDataFrame({ 45 | 'filename': file_name_col, 46 | 'id': df.Id.astype(int), 47 | 'geometry': df.geometry, 48 | }) 49 | except: 50 | print(df) 51 | if first_file: 52 | net_df = df 53 | first_file = False 54 | else: 55 | net_df = net_df.append(df) 56 | 57 | net_df.to_csv(output_csv_path, index=False) 58 | return net_df 59 | 60 | def make_submit(out_file='/wdata/solution.csv'): 61 | pred_top_dir = '/wdata/' 62 | 63 | # out_dir_csv = os.path.join(pred_top_dir, 'csvs') 64 | # os.makedirs(out_dir_csv, exist_ok=True) 65 | # prop_file = os.path.join(out_dir_csv, 'solution.csv') 66 | prop_file = out_file 67 | aoi_dirs = sorted([os.path.join(pred_top_dir, 'pred_jsons_match', aoi) \ 68 | for aoi in os.listdir(os.path.join(pred_top_dir, 'pred_jsons_match')) \ 69 | if os.path.isdir(os.path.join(pred_top_dir, 'pred_jsons_match', aoi))]) 70 | print("aoi_dirs:", aoi_dirs) 71 | 72 | # Execute 73 | if os.path.exists(prop_file): 74 | os.remove(prop_file) 75 | net_df = sn7_convert_geojsons_to_csv(aoi_dirs, prop_file, 'proposal') 76 | 77 | print("prop_file:", prop_file) 78 | 79 | if __name__ == '__main__': 80 | Fire(make_submit) -------------------------------------------------------------------------------- /5-MaksimovKA/code/test.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | ARG1=${1:-/data/SN7_buildings/test_public/} 3 | ARG2=${2:-/wdata/solution.csv} 4 | 5 | mkdir -p /wdata/segmentation_logs/ /wdata/folds_predicts/ /wdata/pretrained_models/checkpoints/ 6 | 7 | if [ "$(ls -A /wdata/segmentation_logs/)" ]; then 8 | echo "trained weights available" 9 | else 10 | echo "loading pretrained weights" 11 | 12 | mkdir -p /wdata/segmentation_logs/fold_1_siamse-senet154/checkpoints/ 13 | gdown https://drive.google.com/uc?id=1WXnq7biVcRoYbbKKj2o0huti3rJ6JlKs -O /wdata/segmentation_logs/fold_1_siamse-senet154/checkpoints/best.pth 14 | fi 15 | 16 | gdown https://drive.google.com/uc?id=1WVQWAwCFgkwmSwkxp-kLiS89IdFrh-T8 -O /wdata/pretrained_models/checkpoints/senet154-c7b49a05.pth 17 | 18 | python3 /project/predict.py --config_path /project/config.py --gpu '"0"' --test_images $ARG1 --workers 1 --batch_size 1 \ 19 | & python3 /project/predict2.py --config_path /project/config.py --gpu '"1"' --test_images $ARG1 --workers 1 --batch_size 1 \ 20 | & python3 /project/predict3.py --config_path /project/config.py --gpu '"2"' --test_images $ARG1 --workers 1 --batch_size 1 \ 21 | & python3 /project/predict4.py --config_path /project/config.py --gpu '"3"' --test_images $ARG1 --workers 1 --batch_size 1 & wait 22 | 23 | python3 /project/mean_folds.py 24 | python3 /project/masks_to_jsons.py 25 | python3 /project/map_jsons.py 26 | python3 /project/submit.py --out_file $ARG2 -------------------------------------------------------------------------------- /5-MaksimovKA/code/train.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | ARG1=${1:-/data/SN7_buildings/train/} 3 | rm -rf /wdata/* 4 | # make directories 5 | mkdir -p /wdata/train_masks/ /wdata/pretrained_models/checkpoints/ /wdata/segmentation_logs/ /wdata/final_models/ 6 | 7 | # create masks and folds 8 | echo "creating masks" 9 | python3 /project/create_masks.py --data_root_path $ARG1 10 | echo "creating folds" 11 | python3 /project/create_folds.py --images_path $ARG1 12 | 13 | # load pretrained 14 | echo "loading pretrained weights" 15 | gdown https://drive.google.com/uc?id=1WVQWAwCFgkwmSwkxp-kLiS89IdFrh-T8 -O /wdata/pretrained_models/checkpoints/senet154-c7b49a05.pth 16 | echo "training" 17 | 18 | python3 /project/train.py --data_path $ARG1 --config_path /project/config.py --gpu '"0"' -------------------------------------------------------------------------------- /5-MaksimovKA/code/tta.py: -------------------------------------------------------------------------------- 1 | import torch 2 | 3 | 4 | def torch_flipud(x): 5 | """ 6 | Flip image tensor vertically 7 | :param x: 8 | :return: 9 | """ 10 | return x.flip(2) 11 | 12 | 13 | def torch_fliplr(x): 14 | """ 15 | Flip image tensor horizontally 16 | :param x: 17 | :return: 18 | """ 19 | return x.flip(3) 20 | 21 | def flip_image2mask(model, image): 22 | """Test-time augmentation for image segmentation that averages predictions 23 | for input image and vertically flipped one. 24 | For segmentation we need to reverse the transformation after making a prediction 25 | on augmented input. 26 | :param model: Model to use for making predictions. 27 | :param image: Model input. 28 | :return: Arithmetically averaged predictions 29 | """ 30 | output = (torch.sigmoid(model(image)) + 31 | torch.sigmoid(torch_fliplr(model(torch_fliplr(image)))) + 32 | torch.sigmoid(torch_flipud(model(torch_flipud(image)))) 33 | ) 34 | one_over_3 = float(1.0 / 3.0) 35 | return output * one_over_3 -------------------------------------------------------------------------------- /5-MaksimovKA/weights/README.txt: -------------------------------------------------------------------------------- 1 | See s3://spacenet-dataset/spacenet-model-weights/spacenet-7/ -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

2 | SpaceNet LLC 3 |

4 |

The SpaceNet 7 Multi-temporal Urban Development Challenge

5 |

Winning Solutions

6 |
7 | 8 | ## Summary 9 | 10 | The five subdirectories in this repository comprise the code for the winning solutions of SpaceNet 7 hosted by TopCoder. Each subdirectory contains the competitors' written descriptions of their solution to the challenge. See the blog post on CosmiQ Works' blog [The DownlinQ](?) for an additional summary. Baseline code can be found [here](https://github.com/CosmiQ/CosmiQ_SN7_Baseline). 11 | 12 | Data is hosted on aws at: 13 | ``` 14 | s3://spacenet-dataset/spacenet/SN7_buildings/ 15 | ``` 16 | 17 | Winning model weights are hosted at: 18 | ``` 19 | s3://spacenet-dataset/spacenet-model-weights/spacenet-7/ 20 | ``` 21 | 22 | The winning solutions all use Docker, and assume SpaceNet 7 data is mounted in the `/data/` directory. 23 | 24 | Performance of the algorithms on the SpaceNet 7 final test set is shown below: 25 | 26 | ![alt text](_figs/table1.png) 27 | 28 | ![alt text](_figs/scot_rate_plot.png) 29 | 30 | --------- 31 | 32 | Questions about SpaceNet? Check out our website at [https://spacenet.ai](https://spacenet.ai). 33 | 34 | 35 | 36 | -------------------------------------------------------------------------------- /_figs/scot_rate_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/_figs/scot_rate_plot.png -------------------------------------------------------------------------------- /_figs/sn_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/_figs/sn_logo.png -------------------------------------------------------------------------------- /_figs/table1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SpaceNetChallenge/SpaceNet7_Multi-Temporal_Solutions/ff1241f01e391248a9d6111a60c97973fc5a9643/_figs/table1.png --------------------------------------------------------------------------------