├── .gitignore ├── .gitmodules └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.pth -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "detectron.jittor"] 2 | path = detectron.jittor 3 | url = git@github.com:li-xl/detectron.jittor.git 4 | [submodule "Yolact.jittor"] 5 | path = Yolact.jittor 6 | url = git@github.com:li-xl/Yolact.jittor.git 7 | [submodule "Pose2Seg.jittor"] 8 | path = Pose2Seg.jittor 9 | url = git@github.com:li-xl/Pose2Seg.jittor.git 10 | [submodule "ViT.jittor"] 11 | path = ViT.jittor 12 | url = git@github.com:li-xl/ViT.jittor.git 13 | [submodule "yolo.jittor"] 14 | path = yolo.jittor 15 | url = https://github.com/li-xl/yolo.jittor 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Instance Segmentation Model Zoo 2 | 3 | 实例分割作为计算机视觉的一项基本任务，在方方面面发挥着巨大的作用，本教程使用*Jittor*框架实现了实例分割模型库 4 | 5 | 使用以下方式获取最新的代码 6 | ```shell 7 | git clone --recurse-submodules https://github.com/Jittor/InstanceSegmentation-jittor 8 | cd InstanceSegmentation-jittor 9 | git submodule update --remote 10 | ``` 11 | 或者 12 | ```shell 13 | git clone https://github.com/Jittor/InstanceSegmentation-jittor 14 | cd InstanceSegmentation-jittor 15 | git submodule init 16 | git submodule update --remote 17 | ``` 18 | 19 | 本仓库为各个不同的实例分割和检测模型的合并仓库,不同的模型链接如下： 20 | 21 | Detectron.jittor: [https://github.com/li-xl/detectron.jittor](https://github.com/li-xl/detectron.jittor) 22 | 23 | Yolact.jittor: [https://github.com/li-xl/Yolact.jittor](https://github.com/li-xl/Yolact.jittor) 24 | 25 | Pose2Seg.jittor: [https://github.com/li-xl/Pose2Seg.jittor](https://github.com/li-xl/Pose2Seg.jittor) 26 | 27 | yolo.jittor: [https://github.com/li-xl/yolo.jittor](https://github.com/li-xl/yolo.jittor) 28 | 29 | ViT.jittor: [https://github.com/li-xl/ViT.jittor](https://github.com/li-xl/ViT.jittor) 30 | 31 | ### 1 数据集 32 | 33 | #### 1.1 COCO数据集下载 34 | 35 | ```bash 36 | sudo wget -c http://images.cocodataset.org/zips/train2017.zip 37 | sudo wget -c http://images.cocodataset.org/zips/val2017.zip 38 | sudo wget -c http://images.cocodataset.org/zips/test2017.zip 39 | sudo wget -c http://images.cocodataset.org/annotations/annotations_trainval2017.zip 40 | sudo wget -c http://images.cocodataset.org/annotations/stuff_annotations_trainval2017.zip 41 | ``` 42 | 43 | 将压缩包进行解压 44 | 45 | ```bash 46 | unzip train2017.zip 47 | unzip val2017.zip 48 | unzip test2017.zip 49 | unzip annotations_trainval2017.zip 50 | unzip stuff_annotations_trainval2017.zip 51 | ``` 52 | 53 | 数据文件夹如下： 54 | 55 | ``` 56 | data 57 | ├── coco2017 58 | │ ├── annotations 59 | │ │ ├── instances_train2017.json 60 | │ │ ├── instances_val2017.json 61 | │ ├── images 62 | | │ ├── train2017 63 | │ │ │ ├── ####.jpg 64 | │ │ ├── val2017 65 | │ │ │ ├── ####.jpg 66 | 67 | ``` 68 | 69 | #### 1.2 数据集路径配置 70 | 71 | **Pose2Seg**: 72 | 73 | 下载转换好的pose2seg.json: 74 | 75 | - [COCOPersons Train Annotation (person_keypoints_train2017_pose2seg.json) [166MB]](https://github.com/liruilong940607/Pose2Seg/releases/download/data/person_keypoints_train2017_pose2seg.json) 76 | - [COCOPersons Val Annotation (person_keypoints_val2017_pose2seg.json) [7MB]](https://github.com/liruilong940607/Pose2Seg/releases/download/data/person_keypoints_val2017_pose2seg.json) 77 | 78 | 下载OCHuman: 79 | 80 | - [images [667MB] & annotations](https://cg.cs.tsinghua.edu.cn/dataset/form.html?dataset=ochuman) 81 | 82 | 数据格式： 83 | 84 | ``` 85 | data 86 | ├── coco2017 87 | │ ├── annotations 88 | │ │ ├── person_keypoints_train2017_pose2seg.json 89 | │ │ ├── person_keypoints_val2017_pose2seg.json 90 | │ ├── train2017 91 | │ │ ├── ####.jpg 92 | │ ├── val2017 93 | │ │ ├── ####.jpg 94 | ├── OCHuman 95 | │ ├── annotations 96 | │ │ ├── ochuman_coco_format_test_range_0.00_1.00.json 97 | │ │ ├── ochuman_coco_format_val_range_0.00_1.00.json 98 | │ ├── images 99 | │ │ ├── ####.jpg 100 | ``` 101 | 102 | 配置train.py和test.py中Dataset的文件路径 103 | 104 | ```python 105 | class Dataset(): 106 | def __init__(self): 107 | ImageRoot = './data/coco2017/train2017' 108 | AnnoFile = './data/coco2017/annotations/person_keypoints_train2017_pose2seg.json' 109 | ``` 110 | 111 | ```python 112 | if dataset == 'OCHumanVal': 113 | ImageRoot = './data/OCHuman/images' 114 | AnnoFile = './data/OCHuman/annotations/ochuman_coco_format_val_range_0.00_1.00.json' 115 | elif dataset == 'OCHumanTest': 116 | ImageRoot = './data/OCHuman/images' 117 | AnnoFile = './data/OCHuman/annotations/ochuman_coco_format_test_range_0.00_1.00.json' 118 | elif dataset == 'cocoVal': 119 | ImageRoot = './data/coco2017/val2017' 120 | AnnoFile = './data/coco2017/annotations/person_keypoints_val2017_pose2seg.json' 121 | ``` 122 | 123 | **Yolact:** 124 | 125 | 修改data/config.py的数据集路径为本地路径 126 | 127 | ```python 128 | coco2017_dataset = dataset_base.copy({ 129 | 'name': 'COCO 2017', 130 | 131 | 'train_info': './data/coco/annotations/instances_train2017.json', 132 | 'valid_info': './data/coco/annotations/instances_val2017.json', 133 | 134 | 'label_map': COCO_LABEL_MAP 135 | }) 136 | ``` 137 | 138 | **Detectron:** 139 | 140 | 配置detectron/config/paths_catalog.py 141 | 142 | ```python 143 | class DatasetCatalog(object): 144 | DATA_DIR = "datasets" 145 | DATASETS = { 146 | "coco_2017_train": { 147 | "img_dir": "coco/train2017", 148 | "ann_file": "coco/annotations/instances_train2017.json" 149 | }, 150 | "coco_2017_val": { 151 | "img_dir": "coco/val2017", 152 | "ann_file": "coco/annotations/instances_val2017.json" 153 | }, 154 | "coco_2014_train": { 155 | "img_dir": "coco/train2014", 156 | "ann_file": "coco/annotations/instances_train2014.json" 157 | }, 158 | "coco_2014_val": { 159 | "img_dir": "coco/val2014", 160 | "ann_file": "coco/annotations/instances_val2014.json" 161 | }, 162 | "coco_2014_minival": { 163 | "img_dir": "coco/val2014", 164 | "ann_file": "coco/annotations/instances_minival2014.json" 165 | }, 166 | "coco_2014_valminusminival": { 167 | "img_dir": "coco/val2014", 168 | "ann_file": "coco/annotations/instances_valminusminival2014.json" 169 | }, 170 | ``` 171 | 172 | 173 | 174 | ### 2 使用教程 175 | 176 | #### 2.1 安装Jittor 177 | 178 | ```bash 179 | sudo apt install python3.7-dev libomp-dev 180 | sudo python3.7 -m pip install git+https://github.com/Jittor/jittor.git 181 | ``` 182 | 183 | 184 | 185 | #### 2.2 使用Pose2Seg人体分割模型 186 | 187 | 下载pretrained模型：[here](https://drive.google.com/file/d/193i8b40MJFxawcJoNLq1sG0vhAeLoVJG/view?usp=sharing) 188 | 189 | **Train:** 190 | 191 | ```bash 192 | python train.py 193 | ``` 194 | 195 | **Test:** 196 | 197 | ```python 198 | python test.py --weights last.pkl --coco --OCHuman 199 | ``` 200 | 201 | 202 | 203 | #### 2.3 使用Yolact实时分割模型 204 | 205 | Pretrained模型 (来源：https://github.com/dbolya/yolact)： 206 | 207 | Here are YOLACT models (released on April 5th, 2019) along with their FPS on a Titan Xp and mAP on `test-dev`: 208 | 209 | | Image Size | Backbone | FPS | mAP | Weights | | 210 | | ---------- | ------------- | ---- | ---- | ------------------------------------------------------------ | ------------------------------------------------------------ | 211 | | 550 | Resnet50-FPN | 42.5 | 28.2 | [yolact_resnet50_54_800000.pth](https://drive.google.com/file/d/1yp7ZbbDwvMiFJEq4ptVKTYTI2VeRDXl0/view?usp=sharing) | [Mirror](https://ucdavis365-my.sharepoint.com/:u:/g/personal/yongjaelee_ucdavis_edu/EUVpxoSXaqNIlssoLKOEoCcB1m0RpzGq_Khp5n1VX3zcUw) | 212 | | 550 | Darknet53-FPN | 40.0 | 28.7 | [yolact_darknet53_54_800000.pth](https://drive.google.com/file/d/1dukLrTzZQEuhzitGkHaGjphlmRJOjVnP/view?usp=sharing) | [Mirror](https://ucdavis365-my.sharepoint.com/:u:/g/personal/yongjaelee_ucdavis_edu/ERrao26c8llJn25dIyZPhwMBxUp2GdZTKIMUQA3t0djHLw) | 213 | | 550 | Resnet101-FPN | 33.5 | 29.8 | [yolact_base_54_800000.pth](https://drive.google.com/file/d/1UYy3dMapbH1BnmtZU4WH1zbYgOzzHHf_/view?usp=sharing) | [Mirror](https://ucdavis365-my.sharepoint.com/:u:/g/personal/yongjaelee_ucdavis_edu/EYRWxBEoKU9DiblrWx2M89MBGFkVVB_drlRd_v5sdT3Hgg) | 214 | | 700 | Resnet101-FPN | 23.6 | 31.2 | [yolact_im700_54_800000.pth](https://drive.google.com/file/d/1lE4Lz5p25teiXV-6HdTiOJSnS7u7GBzg/view?usp=sharing) | [Mirror](https://ucdavis365-my.sharepoint.com/:u:/g/personal/yongjaelee_ucdavis_edu/Eagg5RSc5hFEhp7sPtvLNyoBjhlf2feog7t8OQzHKKphjw) | 215 | 216 | YOLACT++ models (released on December 16th, 2019): 217 | 218 | | Image Size | Backbone | FPS | mAP | Weights | | 219 | | ---------- | ------------- | ---- | ---- | ------------------------------------------------------------ | ------------------------------------------------------------ | 220 | | 550 | Resnet50-FPN | 33.5 | 34.1 | [yolact_plus_resnet50_54_800000.pth](https://drive.google.com/file/d/1ZPu1YR2UzGHQD0o1rEqy-j5bmEm3lbyP/view?usp=sharing) | [Mirror](https://ucdavis365-my.sharepoint.com/:u:/g/personal/yongjaelee_ucdavis_edu/EcJAtMiEFlhAnVsDf00yWRIBUC4m8iE9NEEiV05XwtEoGw) | 221 | | 550 | Resnet101-FPN | 27.3 | 34.6 | [yolact_plus_base_54_800000.pth](https://drive.google.com/file/d/15id0Qq5eqRbkD-N3ZjDZXdCvRyIaHpFB/view?usp=sharing) | [Mirror](https://ucdavis365-my.sharepoint.com/:u:/g/personal/yongjaelee_ucdavis_edu/EVQ62sF0SrJPrl_68onyHF8BpG7c05A8PavV4a849sZgEA) | 222 | 223 | **Train:** 224 | 225 | ```bash 226 | # Trains using the base config with a batch size of 8 (the default). 227 | python train.py --config=yolact_base_config 228 | 229 | # Trains yolact_base_config with a batch_size of 5. For the 550px models, 1 batch takes up around 1.5 gigs of VRAM, so specify accordingly. 230 | python train.py --config=yolact_base_config --batch_size=5 231 | 232 | # Resume training yolact_base with a specific weight file and start from the iteration specified in the weight file's name. 233 | python train.py --config=yolact_base_config --resume=weights/yolact_base_10_32100.pth --start_iter=-1 234 | 235 | # Use the help option to see a description of all available command line arguments 236 | python train.py --help 237 | ``` 238 | 239 | **Test:** 240 | 241 | ```bash 242 | # Display qualitative results on the specified image. 243 | python eval.py --trained_model=weights/yolact_base_54_800000.pth --score_threshold=0.15 --top_k=15 --image=my_image.png 244 | 245 | # Process an image and save it to another file. 246 | python eval.py --trained_model=weights/yolact_base_54_800000.pth --score_threshold=0.15 --top_k=15 --image=input_image.png:output_image.png 247 | 248 | # Process a whole folder of images. 249 | python eval.py --trained_model=weights/yolact_base_54_800000.pth --score_threshold=0.15 --top_k=15 --images=path/to/input/folder:path/to/output/folder 250 | ``` 251 | 252 | #### 2.4 使用Detectron库 253 | 254 | **安装：** 255 | 256 | ```bash 257 | cd detectron.jittor 258 | python setup.py install 259 | ``` 260 | 261 | **配置config （参考configs的样例）**： 262 | 263 | ```yaml 264 | MODEL: 265 | META_ARCHITECTURE: "GeneralizedRCNN" 266 | WEIGHT: "https://download.pytorch.org/models/maskrcnn/e2e_mask_rcnn_R_50_C4_1x.pth" 267 | RPN: 268 | PRE_NMS_TOP_N_TEST: 6000 269 | POST_NMS_TOP_N_TEST: 1000 270 | ROI_MASK_HEAD: 271 | PREDICTOR: "MaskRCNNC4Predictor" 272 | SHARE_BOX_FEATURE_EXTRACTOR: True 273 | MASK_ON: True 274 | DATASETS: 275 | TRAIN: ("coco_2014_train", "coco_2014_valminusminival") 276 | TEST: ("coco_2014_minival",) 277 | SOLVER: 278 | BASE_LR: 0.01 279 | WEIGHT_DECAY: 0.0001 280 | STEPS: (120000, 160000) 281 | MAX_ITER: 180000 282 | IMS_PER_BATCH: 8 283 | 284 | ``` 285 | 286 | **使用：** 287 | 288 | ```python 289 | #coding=utf-8 290 | import requests 291 | from io import BytesIO 292 | from PIL import Image 293 | import cv2 294 | import numpy as np 295 | import jittor as jt 296 | from detectron.config import cfg 297 | from predictor import COCODemo 298 | 299 | def load(url): 300 | """ 301 | Given an url of an image, downloads the image and 302 | returns a PIL image 303 | """ 304 | response = requests.get(url) 305 | pil_image = Image.open(BytesIO(response.content)).convert("RGB") 306 | # convert to BGR format 307 | image = np.array(pil_image)[:, :, [2, 1, 0]] 308 | return image 309 | 310 | # turn on cuda 311 | jt.flags.use_cuda = 1 312 | 313 | # set config 314 | config_file = '../configs/maskrcnn_benchmark/e2e_mask_rcnn_R_50_FPN_1x.yaml' 315 | # update the config options with the config file 316 | cfg.merge_from_file(config_file) 317 | #cfg.MODEL.WEIGHT = "weight/maskrcnn_r50.pth" 318 | 319 | #set predictor 320 | coco_demo = COCODemo( 321 | cfg, 322 | min_image_size=800, 323 | confidence_threshold=0.5, 324 | ) 325 | 326 | #load image 327 | pil_image = Image.open('test.jpg').convert("RGB") 328 | image = np.array(pil_image)[:, :, [2, 1, 0]] 329 | 330 | # compute predictions 331 | predictions = coco_demo.run_on_opencv_image(image) 332 | 333 | # save result 334 | cv2.imwrite('predicton.jpg',predictions) 335 | ``` 336 | 337 | **Train:** 338 | 339 | ```python 340 | python tools/train_net.py --config-file /path/your/configfile 341 | ``` 342 | 343 | **Test:** 344 | 345 | ``` 346 | python tools/test_net.py --config-file /path/your/configfile 347 | ``` 348 | 349 | 350 | 351 | ### 3 参考 352 | 353 | 1. https://github.com/liruilong940607/Pose2Seg 354 | 2. https://arxiv.org/abs/1803.10683 355 | 3. https://github.com/dbolya/yolact 356 | 4. https://arxiv.org/abs/1904.02689 357 | 5. https://arxiv.org/abs/1912.06218 358 | 6. https://github.com/facebookresearch/maskrcnn-benchmark --------------------------------------------------------------------------------