├── .gitmodules ├── LICENSE ├── README.md └── img.png /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "coperception"] 2 | path = coperception 3 | url = https://github.com/coperception/coperception 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Automation and Intelligence for Civil Engineering (AI4CE) Lab @ NYU 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DiscoNet: Learning Distilled Collaboration Graph for Multi-Agent Perception [NeurIPS 2021] 2 | 3 | [Yiming Li](https://scholar.google.com/citations?user=i_aajNoAAAAJ), [Shunli Ren](https://github.com/ShunliRen), [Pengxiang Wu](https://scholar.google.com/citations?user=MXLs7GcAAAAJ&hl=en), [Siheng Chen](https://scholar.google.com/citations?user=W_Q33RMAAAAJ&hl=en), [Chen Feng](https://scholar.google.com/citations?user=YeG8ZM0AAAAJ), [Wenjun Zhang](https://www.researchgate.net/profile/Wenjun-Zhang-29) 4 | 5 | **"Learn a digraph with matrix-valued edge weight for multi-agent perception."** 6 | 7 | [![Documentation Status](https://readthedocs.org/projects/coperception/badge/?version=latest)](https://coperception.readthedocs.io/en/latest/?badge=latest) 8 | ![PyTorch](https://img.shields.io/badge/PyTorch-%23EE4C2C.svg?logo=PyTorch&logoColor=white) 9 | [![GitLab issues total](https://badgen.net/github/issues/ai4ce/DiscoNet)](https://gitlab.com/ai4ce/V2X-Sim/issues) 10 | [![GitHub stars](https://img.shields.io/github/stars/ai4ce/DiscoNet.svg?style=social&label=Star&maxAge=2592000)](https://GitHub.com/ai4ce/V2X-Sim/stargazers/) 11 | 12 |

13 | 14 | ## News 15 | **[2022-07]** We updated the codebase to [coperception library](https://github.com/coperception/coperception) and dataset to [V2X-Sim 2.0](https://ai4ce.github.io/V2X-Sim). New tasks including segmentation and tracking are included. 16 | 17 | **[2021-11]** Our paper is available on [arxiv](https://arxiv.org/pdf/2111.00643.pdf). 18 | 19 | **[2021-10]** Our dataset **V2X-Sim 1.0** is available [here](https://ai4ce.github.io/V2X-Sim/). 20 | 21 | **[2021-09]** 🔥 DiscoNet is accepted at **NeurIPS 2021**. 22 | 23 | ## Abstract 24 | To promote better performance-bandwidth trade-off for multi-agent perception, we propose a novel distilled collaboration graph (DiscoGraph) to model trainable, pose-aware, and adaptive collaboration among agents. Our key novelties lie in two aspects. First, we propose a teacher-student framework to train DiscoGraph via knowledge distillation. The teacher model employs an early collaboration with holistic-view inputs; the student model is based on intermediate collaboration with single-view inputs. Our framework trains DiscoGraph by constraining post-collaboration feature maps in the student model to match the correspondences in the teacher model. Second, we propose a matrix-valued edge weight in DiscoGraph. In such a matrix, each element reflects the inter-agent attention at a specific spatial region, allowing an agent to adaptively highlight the informative regions. During inference, we only need to use the student model named as the distilled collaboration network (DiscoNet). Attributed to the teacher-student framework, multiple agents with the shared DiscoNet could collaboratively approach the performance of a hypothetical teacher model with a holistic view. Our approach is validated on V2X-Sim 1.0, a large-scale multi-agent perception dataset that we synthesized using CARLA and SUMO co-simulation. Our quantitative and qualitative experiments in multi-agent 3D object detection show that DiscoNet could not only achieve a better performance-bandwidth trade-off than the state-of-the-art collaborative perception methods, but also bring more straightforward design rationale. Our code is available on https://github.com/ai4ce/DiscoNet. 25 | 26 | ## Getting started 27 | Please refer to the docs website of `coperception` for detailed documentations: https://coperception.readthedocs.io/en/latest/ 28 | Installation: 29 | - [Installation documentations](https://coperception.readthedocs.io/en/latest/getting_started/installation/) 30 | 31 | Download dataset: 32 | - Original dataset (you are going to parse this dataset yourself with `create_data.py` scripts for specific tasks): [Google Drive (US)](https://drive.google.com/drive/folders/1nVmY7g_kprOX-I0Bqsiz6-zdJM-UXFXa) 33 | - Parsed datasets for detection and segmentation tasks and model checkpoints: [Google Drive (US)](https://drive.google.com/drive/folders/1NMag-yZSflhNw4y22i8CHTX5l8KDXnNd?usp=sharing) 34 | 35 | How to create datasets & run tasks: 36 | - [Detection](https://coperception.readthedocs.io/en/latest/tools/det/) 37 | - [Segmentation](https://coperception.readthedocs.io/en/latest/tools/seg/) 38 | - [Tracking](https://coperception.readthedocs.io/en/latest/tools/track/) 39 | 40 | Example arguments to run DiscoNet: 41 | (assume dataset created properly according to the documentations) 42 | - Detection / segmentation: 43 | Inside coperception codebase: 44 | ``` 45 | cd tools/det 46 | ``` 47 | or 48 | ``` 49 | cd tools/seg 50 | ``` 51 | 52 | Training: 53 | ```bash 54 | python train_codet.py \ 55 | --data /path/to/training/dataset \ 56 | --com disco \ 57 | --log --batch 4 \ 58 | --kd_flag 1 \ 59 | --resume_teacher /path/to/teacher/checkpoint.pth \ 60 | --auto_resume_path logs \ 61 | --logpath logs \ 62 | --nepoch 100 \ 63 | -- rsu [0/1] 64 | ``` 65 | 66 | Testing: 67 | ```bash 68 | python test_codet.py \ 69 | --data /path/to/testing/dataset \ 70 | --com disco \ 71 | --resume /path/to/teacher/checkpoint.pth \ 72 | --tracking \ 73 | --logpath logs \ 74 | --visualization 1 \ 75 | --rsu 1 76 | ``` 77 | 78 | - Tracking: 79 | Inside coperception codebase: 80 | ``` 81 | cd tools/track 82 | ``` 83 | Run tracker: 84 | ```bash 85 | make sort \ 86 | mode=disco/[no_rsu/with_rsu] \ 87 | split [test/val] \ 88 | from_agent 1 \ 89 | to_agent 6 \ 90 | det_logs_path /path/to/detection/checkpoints.pth 91 | ``` 92 | See performance: 93 | ```bash 94 | make eval 95 | mode=disco \ 96 | rsu=[0/1] \ 97 | from_agent=1 \ 98 | to_agent=6 \ 99 | split=test 100 | ``` 101 | 102 | ## Acknowledgment 103 | This project is not possible without the following great codebases. 104 | * [**MotionNet**](https://github.com/pxiangwu/MotionNet) 105 | * [**mmdetection**](https://github.com/open-mmlab/mmdetection) 106 | * [**nuSenes-devkit**](https://github.com/nutonomy/nuscenes-devkit) 107 | * [**when2com**](https://github.com/GT-RIPL/MultiAgentPerception) 108 | * [**V2X-Sim**](https://github.com/ai4ce/V2X-Sim) 109 | 110 | ## Citation 111 | If you find V2X-Sim 1.0 or DiscoNet useful in your research, please cite our paper. 112 | ``` 113 | @InProceedings{Li_2021_NeurIPS, 114 |     title = {Learning Distilled Collaboration Graph for Multi-Agent Perception}, 115 |     author = {Li, Yiming and Ren, Shunli and Wu, Pengxiang and Chen, Siheng and Feng, Chen and Zhang, Wenjun}, 116 |     booktitle = {Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021)}, 117 |     year = {2021} 118 | } 119 | 120 | @article{Li_2021_RAL, 121 | title = {V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving}, 122 | author = {Li, Yiming and Ma, Dekun and An, Ziyan and Wang, Zixun and Zhong, Yiqi and Chen, Siheng and Feng, Chen}, 123 | journal = {IEEE Robotics and Automation Letters}, 124 | year = {2022} 125 | } 126 | ``` 127 | -------------------------------------------------------------------------------- /img.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ai4ce/DiscoNet/9338e670eab9ba1f24944dc34f6f29e636b0ac72/img.png --------------------------------------------------------------------------------