├── .gitignore ├── Dockerfile ├── README.md ├── build.sh ├── crowdai.json ├── crowdai_helpers.py ├── debug.sh ├── environ.sh.example ├── run.py ├── run.sh └── submit_helper.sh /.gitignore: -------------------------------------------------------------------------------- 1 | environ.sh 2 | tagPusher.sh 3 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04 2 | 3 | RUN apt-get update && apt-get install -y \ 4 | build-essential \ 5 | bzip2 \ 6 | cmake \ 7 | curl \ 8 | git \ 9 | g++ \ 10 | libboost-all-dev \ 11 | pkg-config \ 12 | rsync \ 13 | software-properties-common \ 14 | sudo \ 15 | tar \ 16 | timidity \ 17 | unzip \ 18 | wget \ 19 | locales \ 20 | zlib1g-dev \ 21 | python3-dev \ 22 | python3 \ 23 | python3-pip \ 24 | python3-tk \ 25 | libjpeg-dev \ 26 | libpng-dev 27 | 28 | # Python3 29 | RUN pip3 install pip --upgrade 30 | RUN pip3 install cython crowdai_api timeout_decorator \ 31 | numpy \ 32 | matplotlib \ 33 | crowdai-repo2docker 34 | RUN pip3 install git+https://github.com/crowdai/coco.git#subdirectory=PythonAPI 35 | RUN pip3 install tensorflow-gpu 36 | 37 | # Unicode support: 38 | RUN locale-gen en_US.UTF-8 39 | ENV LANG en_US.UTF-8 40 | ENV LANGUAGE en_US:en 41 | ENV LC_ALL en_US.UTF-8 42 | 43 | # Enables X11 sharing and creates user home directory 44 | ENV USER_NAME crowdai 45 | ENV HOME_DIR /home/$USER_NAME 46 | # 47 | # Replace HOST_UID/HOST_GUID with your user / group id (needed for X11) 48 | ENV HOST_UID 1000 49 | ENV HOST_GID 1000 50 | 51 | RUN export uid=${HOST_UID} gid=${HOST_GID} && \ 52 | mkdir -p ${HOME_DIR} && \ 53 | echo "$USER_NAME:x:${uid}:${gid}:$USER_NAME,,,:$HOME_DIR:/bin/bash" >> /etc/passwd && \ 54 | echo "$USER_NAME:x:${uid}:" >> /etc/group && \ 55 | echo "$USER_NAME ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/$USER_NAME && \ 56 | chmod 0440 /etc/sudoers.d/$USER_NAME && \ 57 | chown ${uid}:${gid} -R ${HOME_DIR} 58 | 59 | USER ${USER_NAME} 60 | WORKDIR ${HOME_DIR} 61 | 62 | COPY . . 63 | 64 | RUN sudo chown ${HOST_UID}:${HOST_GID} -R * 65 | RUN sudo chmod 775 -R * 66 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # mapping-challenge-starter-kit (Round-2) 2 | ![CrowdAI-Logo](https://github.com/crowdAI/crowdai/raw/master/app/assets/images/misc/crowdai-logo-smile.svg?sanitize=true) 3 | 4 | [![gitter-badge](https://badges.gitter.im/crowdAI/crowdai-mapping-challenge.png)](https://gitter.im/crowdAI/crowdai-mapping-challenge) 5 | 6 | This is a set of instructions to make a submission to the **Round 2** of [crowdAI Mapping Challenge](https://www.crowdai.org/challenges/mapping-challenge). 7 | In Round 2, Participants are required to submit their code as repositories with [Dockerfiles](https://docs.docker.com/engine/reference/builder/), so that we can build and execute their code as containers. The submitted code will be used to make predictions for an arbitrary number of test images to evaluate its performance. 8 | 9 | The code can be added as **private repository** on [gitlab.crowdai.org](https://gitlab.crowdai.org). And each **tag push** to the repository will be considered as a submission. After every tag push, please refer to the "Issues" section of your repository for the evaluation logs related to your submission. You can find more details about the submission process in the following sections. 10 | 11 | The submitted code will be used to build a docker image (as described in [build.sh](build.sh)), which will be orchestrated by our evaluation interface (similar to) as described in [debug.sh](debug.sh). Your code is supposed to accept the path to a folder of images, and write the predicted annotations to a specified file path. 12 | **Note** : Your code is expected to only do the inference (and not the training). 13 | 14 | # Contents 15 | * [Prerequisites](#prerequisites-dependencies) 16 | * [Setup](#setup) 17 | * [Build Image Locally](#build-image-locally) 18 | * [Debug Locally](#debug-locally) 19 | * [Changes to your Code](#changes-to-your-code) 20 | * [Important Concepts](#important-concepts) 21 | - [Repository Structure](#repository-structure) 22 | - crowdai.json 23 | - Dockerfile 24 | - [Submission](#submission) 25 | * [Hardware Limits](#hardware-limits) 26 | * [Submission TL;DR](#submission-tldr) 27 | 28 | # Prerequisites Dependencies 29 | * **docker-ce**: Install instructions [here](https://docs.docker.com/install/) 30 | * **nvidia-docker**: Install instructions [here](https://github.com/nvidia/nvidia-docker/wiki/Installation-(version-2.0)) 31 | 32 | # Setup 33 | ``` 34 | # Clone Repository 35 | git clone git@github.com:crowdAI/mapping-challenge-round2-starter-kit.git 36 | cd mapping-challenge-round2-starter-kit 37 | cp environ.sh.example environ.sh 38 | # Edit `environ.sh` to point to your local test_images directory 39 | source environ.sh 40 | ``` 41 | 42 | # Build Image Locally 43 | ``` 44 | cd mapping-challenge-round2-starter-kit 45 | source environ.sh 46 | ./build.sh 47 | ``` 48 | 49 | # Debug Locally 50 | ``` 51 | cd mapping-challenge-round2-starter-kit 52 | # Remember to edit `environ.sh` to point to your local test_images directory 53 | source environ.sh 54 | ./debug.sh 55 | # This should write the final predictions to /tmp/output.json 56 | ``` 57 | ## Changes-to-your-code 58 | The entry point for your code in your container has to be : `/home/crowdai/run.sh`, 59 | and you will have access to two Environment Variables : 60 | * `CROWDAI_TEST_IMAGES_PATH` : Which holds the path of the folder containing all the test images 61 | * ` CROWDAI_PREDICTIONS_OUTPUT_PATH` : Path where you are supposed to write your final predictions output (as a JSON). The predictions have to be a list of annotations, the same as what was used in Round 1. An example structure can be found [here](https://github.com/crowdAI/mapping-challenge-starter-kit/blob/master/Random%20Submission.ipynb#Submission-Format) 62 | 63 | * The submitted container, needs to communicate with the rest of the evaluation interface by 64 | using a simple api, that is abstracted by `crowdai_helpers.py`. Please do not make any changes to 65 | the communication protocol. [run.py](run.py) has a simple example implementation of a random submission. 66 | 67 | # Important Concepts 68 | 69 | ## Repository Structure 70 | * `crowdai.json` 71 | Each repository should have a `crowdai.json` with the following content : 72 | ```json 73 | { 74 | "challenge_id" : "crowdAIMappingChallenge", 75 | "grader_id" : "crowdAIMappingChallenge", 76 | "authors" : ["your-crowdai-username"], 77 | "description" : "sample description", 78 | "license" : "MIT", 79 | "gpu":false 80 | } 81 | ``` 82 | This is used to map your submission to the said challenge. 83 | 84 | Please specify if your code will a GPU or not for the evaluation of your model. If you specify `true` for the GPU, a **NVIDIA Titan-X (Maxwell Series) GPU** will be provided and used for the evaluation. 85 | 86 | * `Dockerfile` 87 | Each repository should have a valid Dockerfile which should build an image for your code. 88 | The evaluator starts the container from your image by running the entrypoint : 89 | ``` 90 | /home/crowdai/run.sh 91 | ``` 92 | Please refer to the [sample Dockerfile in this repository](Dockerfile) for an example on how to 93 | create the correct directory structure inside your image. 94 | 95 | * The submitted container, needs to communicate with the rest of the evaluation interface by 96 | using a simple api, that is abstracted by `crowdai_helpers.py`. Please do not make any changes to 97 | the communication protocol. [run.py](run.py) has a simple example implementation of a random submission. 98 | 99 | 100 | # Submission 101 | To make a submission, you will have to create a private repository on [https://gitlab.crowdai.org](https://gitlab.crowdai.org). 102 | 103 | You will have to add your SSH Keys to your GitLab account by following the instructions [here](https://docs.gitlab.com/ee/gitlab-basics/create-your-ssh-keys.html). 104 | If you do not have SSH Keys, you will first need to [generate one](https://docs.gitlab.com/ee/ssh/README.html#generating-a-new-ssh-key-pair). 105 | 106 | Then you can create a submission by making a *tag push* to your repository on [https://gitlab.crowdai.org](https://gitlab.crowdai.org). **Any tag push to your private repository is considered as a submission** 107 | Then you can add the correct git remote by doing : 108 | 109 | ``` 110 | cd mapping-challenge-round2-starter-kit 111 | # Add crowdAI git remote endpoint 112 | git remote add crowdai git@gitlab.crowdai.org:/mapping-challenge-round2-starter-kit.git 113 | git push crowdai master 114 | # Update Author 115 | sed -i 's/spMohanty//g' crowdai.json 116 | git commit -am "update crowdai.json" 117 | 118 | # Create a tag for your submission and push 119 | git tag -am "v0.1" v0.1 120 | git push crowdai v0.1 121 | ``` 122 | You now should be able to see the details of your submission at : 123 | [gitlab.crowdai.org//mapping-challenge-round-2/issues](gitlab.crowdai.org//mapping-challenge-round-2/issues) 124 | 125 | **NOTE**: Remember to update your username in the link above :wink: 126 | 127 | **Best of Luck** 128 | 129 | # Hardware-Limits 130 | Your code will be allowed to use a maximum of : 131 | * 1 cpu core 132 | * upto 8GB of Memory 133 | * upto 10GB of Disk Space 134 | * (optional) 1 NVIDIA Titan X (Maxwell Series) GPU 135 | 136 | **Note** Your evaluation will have a total timeout of 5 hours. 137 | 138 | # Submission TL;DR 139 | **Note**: This section assumes, that you have setup your SSH keys on [https://gitlab.crowdai.org](https://gitlab.crowdai.org) by following the instructions [here](https://docs.gitlab.com/ee/gitlab-basics/create-your-ssh-keys.html). 140 | ``` 141 | # Clone Repository 142 | git clone git@github.com:crowdAI/mapping-challenge-round2-starter-kit.git 143 | cd mapping-challenge-round2-starter-kit 144 | 145 | # Add crowdAI git remote endpoint 146 | git remote add crowdai git@gitlab.crowdai.org:/mapping-challenge-round2-starter-kit.git 147 | git push crowdai master 148 | 149 | # Update Author 150 | sed -i 's/spMohanty//g' crowdai.json 151 | git commit -am "Update crowdai.json" 152 | 153 | # Create a tag for your submission and push 154 | git tag -am "version 0.1" v0.1 155 | git push crowdai v0.1 156 | echo "Check the status of your submission at : 'https://gitlab.crowdai.org//mapping-challenge-round2-starter-kit/issues" 157 | ``` 158 | 159 | # Author 160 | Sharada Mohanty 161 | -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ARGS=$1 4 | 5 | docker build -t ${IMAGE_NAME} . 6 | 7 | if [ "$ARGS" = "push" ]; then 8 | docker push ${IMAGE_NAME} 9 | fi 10 | -------------------------------------------------------------------------------- /crowdai.json: -------------------------------------------------------------------------------- 1 | { 2 | "challenge_id" : "crowdAIMappingChallenge", 3 | "grader_id" : "crowdAIMappingChallenge", 4 | "authors" : ["spMohanty"], 5 | "description" : "Random Submission for CrowdAI Mapping Challenge Round-2", 6 | "license" : "MIT", 7 | "gpu": false 8 | } 9 | -------------------------------------------------------------------------------- /crowdai_helpers.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import crowdai_api 3 | import os 4 | 5 | ######################################################################## 6 | # Instatiate Event Notifier 7 | ######################################################################## 8 | crowdai_events = crowdai_api.events.CrowdAIEvents() 9 | 10 | 11 | def execution_start(): 12 | ######################################################################## 13 | # Register Evaluation Start event 14 | ######################################################################## 15 | crowdai_events.register_event( 16 | event_type=crowdai_events.CROWDAI_EVENT_INFO, 17 | message="execution_started", 18 | payload={ #Arbitrary Payload 19 | "event_type": "mapping_challenge:execution_started" 20 | } 21 | ) 22 | 23 | 24 | def execution_progress(progress_payload): 25 | image_ids = progress_payload["image_ids"] 26 | ######################################################################## 27 | # Register Evaluation Progress event 28 | ######################################################################## 29 | crowdai_events.register_event( 30 | event_type=crowdai_events.CROWDAI_EVENT_INFO, 31 | message="execution_progress", 32 | payload={ #Arbitrary Payload 33 | "event_type": "mapping_challenge:execution_progress", 34 | "image_ids" : image_ids 35 | } 36 | ) 37 | 38 | def execution_success(payload): 39 | predictions_output_path = payload["predictions_output_path"] 40 | ######################################################################## 41 | # Register Evaluation Complete event 42 | ######################################################################## 43 | expected_output_path = os.getenv("CROWDAI_PREDICTIONS_OUTPUT_PATH", False) 44 | if expected_output_path != predictions_output_path: 45 | raise Exception("Please write the output to the path specified in the environment variable : CROWDAI_PREDICTIONS_OUTPUT_PATH instead of {}".format(predictions_output_path)) 46 | 47 | crowdai_events.register_event( 48 | event_type=crowdai_events.CROWDAI_EVENT_SUCCESS, 49 | message="execution_success", 50 | payload={ #Arbitrary Payload 51 | "event_type": "mapping_challenge:execution_success", 52 | "predictions_output_path" : predictions_output_path 53 | }, 54 | blocking=True 55 | ) 56 | 57 | def execution_error(error): 58 | ######################################################################## 59 | # Register Evaluation Complete event 60 | ######################################################################## 61 | crowdai_events.register_event( 62 | event_type=crowdai_events.CROWDAI_EVENT_ERROR, 63 | message="execution_error", 64 | payload={ #Arbitrary Payload 65 | "event_type": "mapping_challenge:execution_error", 66 | "error" : error 67 | }, 68 | blocking=True 69 | ) 70 | -------------------------------------------------------------------------------- /debug.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | source environ.sh 4 | ./build.sh 5 | 6 | # If the output file already exists, delete it 7 | if [ -e "/tmp/output.json" ]; then 8 | rm -rf /tmp/output.json 9 | fi 10 | 11 | # Run the build docker container 12 | # - Mount the test images from the host to /test_images inside the container 13 | # - Assign the correct environment variables 14 | nvidia-docker run -it \ 15 | --net=host \ 16 | -v $CROWDAI_TEST_IMAGES_PATH:/test_images \ 17 | -v /tmp:/tmp_host \ 18 | -e CROWDAI_TEST_IMAGES_PATH="/test_images" \ 19 | -e CROWDAI_PREDICTIONS_OUTPUT_PATH="/tmp_host/output.json" \ 20 | $IMAGE_NAME \ 21 | /home/crowdai/run.sh 22 | 23 | echo "Final Predictions at /tmp/output.json" 24 | echo "Filesize (bytes) of output : `cat /tmp/output.json | wc -c `" 25 | -------------------------------------------------------------------------------- /environ.sh.example: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | export IMAGE_NAME="my_crowdai_submission" 4 | export CROWDAI_TEST_IMAGES_PATH="" 5 | export CROWDAI_PREDICTIONS_OUTPUT_PATH="/tmp/output.json" 6 | -------------------------------------------------------------------------------- /run.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import random 3 | import json 4 | import numpy as np 5 | import argparse 6 | import base64 7 | 8 | import crowdai_helpers 9 | import time 10 | import traceback 11 | 12 | import glob 13 | import os 14 | import json 15 | 16 | 17 | """ 18 | Expected ENVIRONMENT Variables 19 | 20 | * CROWDAI_TEST_IMAGES_PATH : abs path to folder containing all the test images 21 | * CROWDAI_PREDICTIONS_OUTPUT_PATH : path where you are supposed to write the output predictions.json 22 | """ 23 | 24 | # Configuration Variables 25 | IMAGE_WIDTH = 300 26 | IMAGE_HEIGHT = 300 27 | padding = 50 28 | SEGMENTATION_LENGTH = 10 29 | MAX_NUMBER_OF_ANNOTATIONS = 10 30 | 31 | # Helper functions 32 | def bounding_box_from_points(points): 33 | """ 34 | This function only supports the `poly` format. 35 | """ 36 | points = np.array(points).flatten() 37 | even_locations = np.arange(points.shape[0]/2) * 2 38 | odd_locations = even_locations + 1 39 | X = np.take(points, even_locations.tolist()) 40 | Y = np.take(points, odd_locations.tolist()) 41 | bbox = [X.min(), Y.min(), X.max()-X.min(), Y.max()-Y.min()] 42 | bbox = [int(b) for b in bbox] 43 | return bbox 44 | 45 | def single_segmentation(image_width, image_height, number_of_points=10): 46 | points = [] 47 | for k in range(number_of_points): 48 | # Choose a random x-coordinate 49 | random_x = int(random.randint(0, image_width)) 50 | # Choose a random y-coordinate 51 | random_y = int(random.randint(0, image_height)) 52 | #Flatly append them to the list of points 53 | points.append(random_x) 54 | points.append(random_y) 55 | return [points] 56 | 57 | def single_annotation(image_id, number_of_points=10): 58 | _result = {} 59 | _result["image_id"] = image_id 60 | _result["category_id"] = 100 # as 100 is the category_id of Building 61 | _result["score"] = np.random.rand() # a random score between 0 and 1 62 | 63 | _result["segmentation"] = single_segmentation(IMAGE_WIDTH, IMAGE_HEIGHT, number_of_points=number_of_points) 64 | _result["bbox"] = bounding_box_from_points(_result["segmentation"]) 65 | return _result 66 | 67 | def gather_images(test_images_path): 68 | images = glob.glob(os.path.join( 69 | test_images_path, "*.jpg" 70 | )) 71 | return images 72 | 73 | def gather_image_ids(test_images_path): 74 | images = gather_images(test_images_path) 75 | filenames = [os.path.basename(image_path).replace(".jpg","") for image_path in images] 76 | image_ids = [int(x) for x in filenames] 77 | return image_ids 78 | 79 | def get_image_path(image_id): 80 | test_images_path = os.getenv("CROWDAI_TEST_IMAGES_PATH", False) 81 | return os.path.join(test_images_path, str(image_id).zfill(12)) 82 | 83 | def gather_input_output_path(): 84 | test_images_path = os.getenv("CROWDAI_TEST_IMAGES_PATH", False) 85 | assert test_images_path != False, "Please provide the path to the test images using the environment variable : CROWDAI_TEST_IMAGES_PATH" 86 | 87 | predictions_output_path = os.getenv("CROWDAI_PREDICTIONS_OUTPUT_PATH", False) 88 | assert predictions_output_path != False, "Please provide the output path (for writing the predictions.json) using the environment variable : CROWDAI_PREDICTIONS_OUTPUT_PATH" 89 | 90 | return test_images_path, predictions_output_path 91 | 92 | def run(): 93 | ######################################################################## 94 | # Register Prediction Start 95 | ######################################################################## 96 | crowdai_helpers.execution_start() 97 | 98 | ######################################################################## 99 | # Gather Input and Output paths from environment variables 100 | ######################################################################## 101 | test_images_path, predictions_output_path = gather_input_output_path() 102 | 103 | ######################################################################## 104 | # Gather Image IDS 105 | ######################################################################## 106 | image_ids = gather_image_ids(test_images_path) 107 | 108 | ######################################################################## 109 | # Generate Predictions 110 | ######################################################################## 111 | predictions = [] 112 | for image_id in image_ids: 113 | number_of_annotations = random.randint(0, MAX_NUMBER_OF_ANNOTATIONS) 114 | for _idx in range(number_of_annotations): 115 | _annotation = single_annotation(image_id) 116 | predictions.append(_annotation) 117 | ######################################################################## 118 | # Register Prediction 119 | # 120 | # Note, this prediction register is not a requirement. It is used to 121 | # provide you feedback of how far are you in the overall evaluation. 122 | # In the absence of it, the evaluation will still work, but you 123 | # will see progress of the evaluation as 0 until it is complete 124 | # 125 | # Here you simply announce that you completed processing a set of 126 | # image-ids 127 | ######################################################################## 128 | crowdai_helpers.execution_progress({ 129 | "image_ids" : [image_id] 130 | }) 131 | 132 | 133 | # Write output 134 | fp = open(predictions_output_path, "w") 135 | fp.write(json.dumps(predictions)) 136 | fp.close() 137 | ######################################################################## 138 | # Register Prediction Complete 139 | ######################################################################## 140 | crowdai_helpers.execution_success({ 141 | "predictions_output_path" : predictions_output_path 142 | }) 143 | 144 | 145 | if __name__ == "__main__": 146 | try: 147 | run() 148 | except Exception as e: 149 | error = traceback.format_exc() 150 | print(error) 151 | crowdai_helpers.execution_error(error) 152 | -------------------------------------------------------------------------------- /run.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | python3 run.py 4 | -------------------------------------------------------------------------------- /submit_helper.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | 4 | echo "SUBMISSION : $k" 5 | 6 | RANDOM_STRING=`echo $RANDOM | tr '[0-9]' '[a-zA-Z]'` 7 | 8 | echo $RANDOM_STRING >> something1 9 | git add . 10 | git commit -am "Added random string" 11 | git tag -am "Adding tag $RANDOM_STRING " $RANDOM_STRING 12 | git push crowdai $RANDOM_STRING 13 | --------------------------------------------------------------------------------