├── README.md └── img ├── panoptic_segmentation_overview.png ├── panoptic_segmentation_overview2.png ├── panoptic_structure.png ├── pc_metric.png └── pq_metric.png /README.md: -------------------------------------------------------------------------------- 1 | # Awesome-Panoptic-Segmentation [![Awesome](https://awesome.re/badge.svg)](https://awesome.re) 2 | This repo is a collection of the challenging panoptic segmentation, including papers, codes, and benchmark results, etc. 3 | 4 | ## Outline 5 | * [Panoptic Segmentation](#panoptic-segmentation) 6 | * [Datasets](#datasets) 7 | * [Evaluation](#evaluation) 8 | * [Benchmark Results](#benchmark-results) 9 | * [Papers](#papers) 10 | * [Tutorials](#tutorials) 11 | * [Blogs](#blogs) 12 | 13 | 14 | ## Panoptic Segmentation 15 |
16 | Summarize in one sentence : Panoptic Segmentation proposes to solve the semantic segmentation(*Stuff*) and instance segmentation(*Thing*) in a unified and general manner. 17 | 18 | 19 | ### Structure Overview 20 |
21 | 22 | from [UPSNet](https://arxiv.org/pdf/1901.03784.pdf). 23 | 24 | 25 | ## Datasets 26 | Generally, the datasets which contains both semantic and instance annotations can be used to solve the challenging *panoptic* task. 27 | 28 | * [COCO-Panoptic](http://cocodataset.org/) 29 | * [Cityscapes](https://www.cityscapes-dataset.com/) 30 | * [Mapillary Vistas](https://blog.mapillary.com/product/2017/05/03/mapillary-vistas-dataset.html) 31 | * [ADE20K](http://groups.csail.mit.edu/vision/datasets/ADE20K/) 32 | * [IDD20K](http://idd.insaan.iiit.ac.in/) 33 | 34 | 35 | ## Evaluation 36 | ### Metrics 37 | * ``PQ`` are the standard metrics described in [Panoptic Segmentation](https://arxiv.org/pdf/1801.00868.pdf). 38 |
39 | 40 | * ``PC`` are the standard metrics described in [DeeperLab](https://arxiv.org/pdf/1902.05093). 41 |
42 | 43 | ### Evaluation Code 44 | * [cocodataset/panopticapi](https://github.com/cocodataset/panopticapi) 45 | * [mcordts/cityscapesScripts](https://github.com/mcordts/cityscapesScripts) 46 | 47 | ### Competition 48 | * [AutoNUE 2019 Panoptic Segmentation Challenge (ICCV 2019 Workshop, Closed)](https://cvit.iiit.ac.in/autonue2019/challenge/) 49 | * [COCO 2019 Panoptic Segmentation Task (ICCV 2019 Workshop, Closed)](http://cocodataset.org/#panoptic-2019) 50 | * [Mapillary 2019 Panoptic Segmentation Task (ICCV 2019 Workshop, Closed)](https://research.mapillary.com/eccv18/#panoptic) 51 | * [Cityscapes Panoptic Semantic Labeling Task (Open)](https://www.cityscapes-dataset.com/benchmarks/#panoptic-scene-labeling-task) 52 | * [COCO 2018 Panoptic Segmentation Task (ECCV 2018 Workshop, Closed)](http://cocodataset.org/index.htm#panoptic-2018) 53 | * [Mapillary Vistas 2018 Panoptic Segmentation Task (ECCV 2018 Workshop, Closed)](https://research.mapillary.com/eccv18/#panoptic) 54 | 55 | 56 | ## Benchmark Results 57 | ### COCO `val` Benchmark 58 | | Method | Backbone | PQ | PQ-Thing | PQ-Stuff | SQ | RQ | mIoU | AP-Mask | PC | e2e | 59 | | :----------: | :-----------: | :-----------: | :-----------: |:-----------: |:-----------: |:-----------: |:-----------: |:-----------: | :-----------: | :-----------: | 60 | | SOGNet | ResNet-50 | 43.7 | 50.6 | 33.2 | 78.7 | 53.5 | 54.56 | 34.2 | - | :white_check_mark: | 61 | | UPSNet | ResNet-50 | 42.5 | 48.6 | 33.4 | - | - | 54.3 | 34.3 | - | :white_check_mark: | 62 | | OANet | ResNet-101 | 41.3 | 50.4 | 27.7 | - | - | - | - | - | :white_check_mark: | 63 | | OCFusion | ResNet-50 | 41.0 | 49.0 | 29.0 | 77.1 | 50.6 | - | - | - | :white_check_mark: | 64 | | Panoptic FPN | ResNet-101 | 40.9 | 48.3 | 29.7 | - | - | - | - | - | :white_check_mark: | 65 | | AUNet | ResNet-50 | 39.6 | 49.1 | 25.2 | - | - | 45.1 | 34.7 | - | :white_check_mark: | 66 | | AdaptIS | ResNet-101 | 37.0 | 41.8 | 29.9 | - | - | - | - | - | :white_check_mark: | 67 | | DeeperLab | Xception-71 | 34.3 | 37.5 | 29.6 | 77.1 | 43.1 | - | - | 56.8 | :white_check_mark: | 68 | 69 | ### Cityscapes `val`Benchmark 70 | | Method | Backbone | PQ | PQ-Thing | PQ-Stuff | SQ | RQ | mIoU | AP-Mask | PC | e2e | 71 | | :----------: | :-----------: | :-----------: | :-----------: |:-----------: |:-----------: |:-----------: |:-----------: |:-----------: | :-----------: | :-----------: | 72 | | Panoptic(Merge) | - | 61.2 | 66.4 | 54.0 | 80.9 | 74.4 | - | - | - | :negative_squared_cross_mark: | 73 | | AdaptIS | ResNet-101 | 60.6 | 58.7 | 64.4 | - | - | 79.2 | 36.3 | - | :white_check_mark: | 74 | | SOGNet | ResNet-50 | 60.0 | 56.7 | 62.5 | - | - | - | - | - | :white_check_mark: | 75 | | Seamless | ResNet-50 | 59.8 | 53.4 | 64.5 | - | - | 75.4 | 31.9 | - | :white_check_mark: | 76 | | UPSNet | ResNet-50 | 59.3 | 54.6 | 62.7 | 79.7 | 73.0 | 75.2 | 33.3 | - | :white_check_mark: | 77 | | TASCNet | ResNet-101 | 59.2 | 56 | 61.5 | - | - | 77.8 | 37.6 | - | :white_check_mark: | 78 | | AUNet | ResNet-101 | 59.0 | 54.8 | 62.1 | - | - | 75.6 | 34.4 | - | :white_check_mark: | 79 | | Panoptic FPN | ResNet-101 | 58.1 | 52.0 | 62.5 | - | - | 75.7 |33.0 | - | :white_check_mark: | 80 | | DeeperLab | Xception-71 | 56.5 | - | - | - | - | - | - | 75.6 | :white_check_mark: | 81 | 82 | ### Mapillary `val` Benchmark 83 | | Method | Backbone | PQ | PQ-Thing | PQ-Stuff | SQ | RQ | mIoU | AP-Mask | PC | e2e | 84 | | :----------: | :-----------: | :-----------: | :-----------: |:-----------: |:-----------: |:-----------: |:-----------: |:-----------: | :-----------: | :-----------: | 85 | | Panoptic(Merge) | - | 38.3 | 41.8 | 35.7 | 73.6 | 47.7 | - | - | - | :negative_squared_cross_mark: | 86 | | Seamless | ResNet-50 | 37.2 | 33.2 | 42.5 | - | - | 50.2 | 16.3 | - | :white_check_mark: | 87 | | AdaptIS | ResNet-101 | 33.4 | 28.3 | 40.3 | - | - | - | - | - | :white_check_mark: | 88 | | TASCNet | ResNet-101 | 32.6 | 31.3 | 34.4 | - | - | 35.0 | 18.5 | - | :white_check_mark: | 89 | | DeeperLab | Xception-71 | 32.0 | - | - | - | - | - | - | 55.3 | :white_check_mark: | 90 | 91 | 92 | ## Papers 93 | ### AAAI2020 94 | * **SOGNet:** Yibo Yang, Hongyang Li, Xia Li, Qijie Zhao, Jianlong Wu, Zhouchen Lin.
"SOGNet: Scene Overlap Graph Network for Panoptic Segmentation." AAAI (2020). [[paper](https://arxiv.org/pdf/1911.07527.pdf)] 95 | 96 | ### ICCV2019 97 | * **AdaptIS:** Konstantin Sofiiuk, Olga Barinova, Anton Konushin.
"AdaptIS: Adaptive Instance Selection Network." ICCV (2019). [[paper](https://arxiv.org/pdf/1909.07829.pdf)] 98 | 99 | * Cheng-Yang Fu, Tamara L. Berg, Alexander C. Berg.
"IMP: Instance Mask Projection for High Accuracy Semantic Segmentation of Things." ICCV (2019). [[paper](https://arxiv.org/pdf/1906.06597.pdf)] 100 | 101 | * Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, Liang-Chieh Chen.
"Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation Bowen." ICCVW (2019). [[paper](https://arxiv.org/pdf/1911.10194.pdf)] 102 | 103 | ### CVPR2019 104 | * **Panoptic Segmentation:** Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollár.
"Panoptic Segmentation." CVPR (2019). [[paper](https://arxiv.org/pdf/1801.00868.pdf)] 105 | 106 | * **Panoptic FPN:** Alexander Kirillov, Ross Girshick, Kaiming He, Piotr Dollár.
"Panoptic Feature Pyramid Networks." CVPR (2019 **oral**). [[paper](https://arxiv.org/pdf/1901.02446.pdf)] [[unofficial code](https://github.com/Angzz/panoptic-fpn-gluon)][[detectron2]](https://github.com/facebookresearch/detectron2) 107 | 108 | * **AUNet:** Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, Xingang Wang.
"Attention-guided Unified Network for Panoptic Segmentation." CVPR (2019). [[paper](https://arxiv.org/pdf/1812.03904.pdf)] 109 | 110 | * **UPSNet:** Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun.
"UPSNet: A Unified Panoptic Segmentation Network." CVPR (2019 **oral**). [[paper](https://arxiv.org/pdf/1901.03784.pdf)] [[code](https://github.com/uber-research/UPSNet)] 111 | 112 | * **DeeperLab:** Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, Liang-Chieh Chen.
"DeeperLab: Single-Shot Image Parser." CVPR (2019 **oral**). [[paper](https://arxiv.org/pdf/1902.05093)] [[project](http://deeperlab.mit.edu)] [[code](https://github.com/tensorflow/models/tree/master/research/deeplab/evaluation)] 113 | 114 | * **OANet:** Huanyu Liu, Chao Peng, Changqian Yu, Jingbo Wang, Xu Liu, Gang Yu, Wei Jiang.
"An End-to-End Network for Panoptic Segmentation." CVPR (2019). [[paper](https://arxiv.org/pdf/1903.05027.pdf)] 115 | 116 | * Eirikur Agustsson, Jasper R. R. Uijlings, Vittorio Ferrari 117 | .
"Interactive Full Image Segmentation by Considering All Regions Jointly." CVPR (2019). [[paper](https://arxiv.org/pdf/1812.01888.pdf)] 118 | 119 | * **Seamless:** Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, Peter Kontschieder.
"Seamless Scene Segmentation." CVPR (2019) (Extended Version). [[paper](https://arxiv.org/pdf/1905.01220.pdf)][[code](https://github.com/mapillary/seamseg)] 120 | 121 | ### ECCV2018 122 | * Qizhu Li, Anurag Arnab, Philip H.S. Torr.
"Weakly- and Semi-Supervised Panoptic Segmentation." ECCV (2018). [[paper](https://arxiv.org/pdf/1812.01192.pdf)] [[code](https://github.com/qizhuli/Weakly-Supervised-Panoptic-Segmentation)] 123 | 124 | ### ArXiv 125 | * Rohit Mohan, Abhinav Valada.
126 | "EfficientPS: Efficient Panoptic Segmentation." arXiv (2020). [[paper]](https://arxiv.org/abs/2004.02307) 127 | 128 | * Rui Hou, Jie Li, Arjun Bhargava, Allan Raventos, Vitor Guizilini, Chao Fang, Jerome Lynch, Adrien Gaidon.
129 | "Real-Time Panoptic Segmentation from Dense Detections." arXiv (2019). [[paper]](https://arxiv.org/pdf/1912.01202.pdf) 130 | 131 | * Mark Weber, Jonathon Luiten, Bastian Leibe.
"Single-Shot Panoptic Segmentation." arXiv (2019). [[paper](https://arxiv.org/pdf/1911.00764.pdf)] 132 | 133 | * Qiang Chen, Anda Cheng, Xiangyu He, Peisong Wang, Jian Cheng.
"SpatialFlow: Bridging All Tasks for Panoptic Segmentation." arXiv (2019). [[paper](https://arxiv.org/pdf/1910.08787.pdf)] 134 | 135 | * Sagi Eppel, Alan Aspuru-Guzik.
"Generator evaluator-selector net: a modular approach for panoptic segmentation." arXiv (2019). [[paper](https://arxiv.org/pdf/1908.09108.pdf)] 136 | 137 | * Jasper R. R. Uijlings, Mykhaylo Andriluka, Vittorio Ferrari.
"Panoptic Image Annotation with a Collaborative Assistant." arXiv (2019). [[paper](https://arxiv.org/pdf/1906.06798.pdf)] 138 | 139 | * **OCFusion:** Justin Lazarow, Kwonjoon Lee, Zhuowen Tu.
"Learning Instance Occlusion for Panoptic Segmentation." arXiv (2019). [[paper](https://arxiv.org/pdf/1906.05896.pdf)] 140 | 141 | * **PEN:** Yuan Hu, Yingtian Zou, Jiashi Feng.
"Panoptic Edge Detection." arXiv (2019). [[paper](https://arxiv.org/pdf/1906.00590.pdf)] 142 | 143 | * **TASCNet:** Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, Adrien Gaidon.
"Learning to Fuse Things and Stuff." arXiv (2018). [[paper](https://arxiv.org/pdf/1812.01192.pdf)] 144 | 145 | * Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.
"Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network." arXiv (2018). [[paper](https://arxiv.org/pdf/1809.02110.pdf)] 146 | 147 | * Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.
"Single Network Panoptic Segmentation for Street Scene Understanding." arXiv (2019). [[paper](https://arxiv.org/pdf/1902.02678.pdf)] 148 | 149 | * David Owen, Ping-Lin Chang.
"Detecting Reflections by Combining Semantic and Instance Segmentation." arXiv (2019). [[paper](https://arxiv.org/pdf/1904.13273.pdf)] 150 | 151 | * Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji.
"PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things." arXiv (2019, IROS). [[paper](https://arxiv.org/pdf/1903.01177.pdf)] 152 | 153 | 154 | ## Tutorials 155 | * CVPR 2019 Tutorial on Visual Recognition and Beyond. [[slides](https://www.dropbox.com/s/t6tg87t78pdq6v3/cvpr19_tutorial_alexander_kirillov.pdf?dl=0)] [[homepage](http://feichtenhofer.github.io/cvpr2019-recognition-tutorial/?nsukey=sJf%2BXalFUZ1SdTHfNF9ApK0yHb3RiOUTjCdoDI0FTj2gtGZgjyITEf3MIwlgv1CWJywF4qeEOFiUd14dVkeQjn61Yh4mOoqDVb%2Ff4BDiWtBZCNZzozDG5ryVLAM4y8kHxz2NXKdlyjgF2BwgPUMBLs4RrvMNRpgQl8PZ9KPBmhwEXq71r6E4dCCvEFCdio1Lj3aQc%2FoyG%2FdLIN3tBmSjPQ%3D%3D)] 156 | * COCO 2017 Workshop. [[slides](http://presentations.cocodataset.org/COCO17-Stuff-FAIR.pdf)] 157 | 158 | 159 | ## Blogs 160 | * Megvii(Face++) Detection Team. [[zhihu]](https://zhuanlan.zhihu.com/p/59141570) 161 | 162 | -------------------------------------------------------------------------------- /img/panoptic_segmentation_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Angzz/awesome-panoptic-segmentation/159c3f97be86f0a223ce089db0fb6cf9515ce314/img/panoptic_segmentation_overview.png -------------------------------------------------------------------------------- /img/panoptic_segmentation_overview2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Angzz/awesome-panoptic-segmentation/159c3f97be86f0a223ce089db0fb6cf9515ce314/img/panoptic_segmentation_overview2.png -------------------------------------------------------------------------------- /img/panoptic_structure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Angzz/awesome-panoptic-segmentation/159c3f97be86f0a223ce089db0fb6cf9515ce314/img/panoptic_structure.png -------------------------------------------------------------------------------- /img/pc_metric.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Angzz/awesome-panoptic-segmentation/159c3f97be86f0a223ce089db0fb6cf9515ce314/img/pc_metric.png -------------------------------------------------------------------------------- /img/pq_metric.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Angzz/awesome-panoptic-segmentation/159c3f97be86f0a223ce089db0fb6cf9515ce314/img/pq_metric.png --------------------------------------------------------------------------------