├── LICENSE ├── README.md ├── Task1 ├── README.md ├── Task1_COVID_82_datasplit.pkl └── Task1_COVID_datasplit.pkl ├── Task2 ├── README.md ├── Task200_StructSegLung_datasplit.pkl ├── Task201_NSCLCLung_datasplit.pkl ├── Task202_MSD_LungTumor_datasplit.pkl ├── Task203_StructSegTumor_datasplit.pkl └── Task204_NSCLCPE_datasplit.pkl ├── Task3 ├── LungPseudoLabels │ ├── README.md │ ├── Task300-Infer-COVID-19-20-Training.zip │ └── Task300-Infer-COVID-19-20-Validation.zip ├── README.md ├── Task300_StructSegLung_datasplit.pkl ├── Task301_NSCLCLung_datasplit.pkl ├── Task302_MSD_LungTumor_datasplit.pkl ├── Task303_StructSegTumor_datasplit.pkl └── Task304_NSCLCPE_datasplit.pkl └── utils ├── COVID-19-Seg-Evaluation.py ├── ImageExamples.png ├── README.md └── SurfaceDice.py /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

Towards Efficient COVID-19 CT Annotation: A Benchmark for Lung and Infection Segmentation

2 | 3 | - [Task 1: Learning with limited annotations](#1) 4 | - [Task 2: Learning to segment COVID-19 CT scans from non-COVID-19 CT scans](#2) 5 | - [Task 3: Learning with both COVID-19 and non-COVID-19 CT scans](#3) 6 | 7 | 8 | ## Motivation 9 | 10 | Tremendous [studies](https://github.com/HzFu/COVID19_imaging_AI_paper_list#technical_CT) show that deep learning methods have potential for providing accurate and quantitative assessment of COVID-19 infection in CT scans if hundreds of well-labeled training cases are available. However, manual delineation of lung and infection is time-consuming and labor-intensive. Thus, we set up this benchmark to explore annotation-efficient methods for COVID-19 CT scans segmentation. In particular, we focus on learning to segment left lung, right lung and infection using 11 | 12 | - pure but limited COVID-19 CT scans; 13 | 14 | - existing labeled lung CT dataset from other non-COVID-19 lung diseases; 15 | - heterogeneous datasets include both COVID-19 and non-COVID-19 CT scans. 16 | 17 | **Ultimate goal: training a model on limited data that can generalize on infinite data!** 18 | 19 | ``` 20 | @article{MP-COVID-19-SegBenchmark, 21 | title={Towards Data-Efficient Learning: A Benchmark for COVID-19 CT Lung and Infection Segmentation}, 22 | author = {Ma, Jun and Wang, Yixin and An, Xingle and Ge, Cheng and Yu, Ziqi and Chen, Jianan and Zhu, Qiongjie and Dong, Guoqiang and He, Jian and He, Zhiqiang and Cao, Tianjia and Zhu, Yuntao and Nie, Ziwei and Yang, Xiaoping}, 23 | journal = {Medical Physics}, 24 | volume = {48}, 25 | number = {3}, 26 | pages = {1197-1210}, 27 | doi = {https://doi.org/10.1002/mp.14676}, 28 | year = {2021} 29 | } 30 | ``` 31 | 32 | ## Datasets 33 | 34 | | Download Dataset | Description | License | 35 | | ------------------------------------------------------------ | :----------------------------------------------------------- | ------- | 36 | | [StructSeg 2019](https://structseg2019.grand-challenge.org/) | 50 lung CT scans; Annotations include left lung, right lung, spinal cord, esophagus, heart, trachea and gross target volume of lung cancer. |Hold by the [challenge organizers](https://structseg2019.grand-challenge.org/Download/) | 37 | | [NSCLC](https://wiki.cancerimagingarchive.net/display/DOI/Thoracic+Volume+and+Pleural+Effusion+Segmentations+in+Diseased+Lungs+for+Benchmarking+Chest+CT+Processing+Pipelines#7c5a8c0c0cef44e488b824bd7de60428) | 402 lung CT scans; Annotations include left lung, right lung and pleural effusion (78 cases). |CC BY-NC | 38 | | [MSD Lung Tumor](http://medicaldecathlon.com/) | 63 lung CT scans; Annotations include lung cancer. |CC BY-SA | 39 | | [COVID-19-CT-Seg](https://zenodo.org/record/3757476#.Xpz8OcgzZPY) | 20 lung CT scans; Annotations include left lung, right lung and infections. |CC BY-NC-SA | 40 | |[MosMed](https://www.medrxiv.org/content/10.1101/2020.05.20.20100362v1)|50 labelled COVID-19 CT scans; Annotations include infections.|CC BY-NC-ND| 41 | 42 | ![Examples](https://github.com/JunMa11/COVID-19-CT-Seg-Benchmark/blob/master/utils/ImageExamples.png) 43 | 44 | ## Segmentation Task 1: Learning with limited annotations
45 | 46 | > This task is based on the COVID-19-CT-Seg dataset with 20 cases. Three subtasks are to segment lung, infection or both of them. For each task, 5-fold cross-validation results should be reported. 47 | > It should be noted that each fold only has 4 training cases, and remained 16 cases are used for testing. In other words, this is a few-shot or zero-shot segmentation task. 48 | > Dataset split file and quantitative results of U-Net baseline are presented in Task1 folder. 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 |
SubtaskTraining and TestingTesting
Lung5-fold cross validation
4 cases (20% for training)
16 cases (80% for testing)		MosMed(50)
Infection
Lung and infection
69 | 70 | ## Segmentation Task 2: Learning to segment COVID-19 CT scans from non-COVID-19 CT scans
71 | 72 | > This task is to segment lung and infection in COVID-19 CT scans. The main difficulty is that the training set and testing set differ in data distribution. Although all the datasets are lung CT, they vary in lesion types (i.e., cancer, pleural effusion, and COVID-19), patient cohorts and imaging scanners. 73 | 74 | > It should be noted that labeled COVID-19 CT scans are not allowed to be used during training. The following table presents the details of training, validation, and testing set. Name (Num.) denotes the dataset name and the number of cases in this dataset, e.g., StructSeg Lung (40) denotes that 40 cases in StructSeg dataset are used for training. 75 | 76 | > Dataset split file and quantitative results of U-Net baseline are presented in Task2 folder. 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 |
SubtaskTrainingIn-domain Testing(Unseen)Testing 1(Unseen)Testing 2
LungStructSeg Lung (40)
NSCLC Lung (322)		StructSeg Lung (10)
NSCLC Lung (80)		COVID-19-CT-Seg
Lung (20)		-
InfectionMSD Lung Tumor (51)
StructSeg Gross Target (40)
NSCLC Plcural Effusion (62)		MSD Lung Tumor (12)
StructSeg Gross Target (10)
NSCLC Plcural Effusion (16)		COVID-19-CT-Seg
Infection(20)		MosMed(50)
103 | 104 | 105 | 106 | ## Segmentation Task 3: Learning with both COVID-19 and non-COVID-19 CT scans
107 | 108 | > This task is also to segment lung and infection in COVID-19 CT scans, but a limited labeled COVID-19 CT scans are allowed to be used during training. For each subtask, 5-fold cross-validation results should be reported. 109 | 110 | > Dataset split file and quantitative results of U-Net baseline will be presented in Task3 folder. 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 |
Subtask
Training
Validation
Testing 1
Testing 2
Lung
StructSeg Lung (40)
NSCLC Lung (322)
COVID-19-CT-Seg Lung(4)
StructSeg Lung (10)
NSCLC Lung (80)
COVID-19-CT-Seg Lung(16)
-
Infection
MSD Lung Tumor (51)
StructSeg Gross Target (40)
NSCLC Plcural Effusion (62)
COVID-19-CT-Seg Infection(4)
MSD Lung Tumor (12)
StructSeg Gross Target (10)
NSCLC Plcural Effusion (16)
COVID-19-CT-Seg Infection(16)
MosMed(50)
139 | 140 | 141 | 142 | ## Guidelines 143 | 144 | - We hope these tasks can serve as a benchmark for novel annotation-efficient segmentation methods of COVID-19 CT scans. Both semi-automatic (e.g., level set, graph cut...) and fully automatic methods (e.g., CNNs...) are welcome. 145 | - Evaluation metrics are Dice similarity coefficient (DSC) and normalized surface Dice (NSD), and the python implementations are [here](http://medicaldecathlon.com/files/Surface_distance_based_measures.ipynb). 146 | - In [COVID-19-CT-Seg](https://zenodo.org/record/3757476#.Xpz8OcgzZPY) dataset, the last 10 cases from Radiopaedia have been adjusted to lung window [-1250,250], and then normalized to [0,255], we recommend to adust the first 10 cases from Coronacases with the same method. 147 | - Nifty format of the NSCLC dataset can be downloaded [here (pw:1qop)](https://pan.baidu.com/s/1K7iGRIX8lOiaaTbhBJi7Vw). It should be noted that all the copyrights belong to the original dataset contributors, and please also [cite the corresponding publications](https://wiki.cancerimagingarchive.net/display/DOI/Thoracic+Volume+and+Pleural+Effusion+Segmentations+in+Diseased+Lungs+for+Benchmarking+Chest+CT+Processing+Pipelines#4dc5f53338634b35a3500cbed18472e0) if you use this dataset. 148 | - 2D/3D U-Net baselines are based on [nnU-Net](https://github.com/MIC-DKFZ/nnUNet). 100 pretrained baseline models and corresponding segmentation results are available: [3D U-Net](http://doi.org/10.5281/zenodo.3789644) and [2D U-Net](http://doi.org/10.5281/zenodo.3870441). 149 | > [Baidu Net Disk mirror](https://pan.baidu.com/s/1t-Y-twHSrCiDRZKt_r2m5A) (pw: t5mj) 150 | 151 | 152 | 153 | 154 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 | 288 | 289 | 290 | 291 | 292 | 293 | 294 | 295 | 296 | 297 | 298 | 299 | 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 316 | 317 | 318 | 319 | 320 | 321 | 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | 335 | 336 | 337 | 338 | 339 | 340 | 341 | 342 | 343 | 344 | 345 | 346 | 347 | 348 | 349 | 350 | 351 | 352 | 353 | 354 | 355 | 356 | 357 | 358 | 359 | 360 | 361 | 362 | 363 | 364 | 365 |
3D

U-Net
155 |
Subtask
156 |
Left Lung
Right Lung
Infection(COVID-19-CT-Seg)
Infection(MosMed)
DSC
NSD
DSC
NSD
DSC
NSD
DSC
NSD
Task1-Separate
85.8±10.5
71.2±13.8
87.9±9.3
74.8±11.9
67.3±22.3
70.0±24.4
58.8±20.6
66.4±20.3
Task1-Union
64.6±26.4
51.1±23.4
75.0±16.8
57.7±17.4
61.0±26.2
61.8±27.4
48.2±22.1
41.4±19.1
Task2-MSD----
25.2±27.4
26.0±28.5
16.2±23.2
17.5±23.4
Task2-StructSeg
92.2±19.7
82.0±15.7
95.5±7.2
84.2±11.6
6.0±12.75.5±10.7
2.6±9.5
3.3±9.9
Task2-NSCLC
57.5±21.5
46.9±17.0
72.2±15.3
51.7±16.8
0.4±0.93.7±4.8
0.0±0.0
0.5±1.4
Task3-MSD 96.5±2.887.9±7.996.9±2.288.5±7.1
62.3±25.7
61.3±27.6
39.2±30.6
41.3±30.5
Task3-StructSeg 97.3±2.190.6±6.297.7±2.191.4±6.1
64.2±24.5
63.3±25.7
44.3±25.3
49.1±25.8
Task3-NSCLC 93.5±5.4
76.9±13.3
94.0±5.3
77.2±14.1
60.2±25.4
58.5±26.7
30.1±26.7
33.4±27.1
2D

U-Net
261 |
Subtask
262 |
Left Lung
Right Lung
Infection(COVID-19-CT-Seg)
Infection(MosMed)
DSC
NSD
DSC
NSD
DSC
NSD
DSC
NSD
Task1-Separate
95.1±7.9
84.6±12.7
95.6±7.4
85.5±12.8
60.9±24.5
61.5±27.0
53.7±21.4
61.5±21.2
Task1-Union
87.3±15.8
70.5±18.7
89.4±12.8
71.0±17.8
57.7±26.3
57.2±29.0
52.2±21.6
46.2±18.3
Task2-MSD----
7.9±11.5
12.9±15.3
7.6±15.8
9.9±17.1
Task2-StructSeg
46.3±47.6
28.4±31.7
45.3±46.7
28.0±31.3
0.2±0.80.6±1.6
1.9±10.1
2.2±10.0
Task2-NSCLC
47.3±48.6
37.9±40.1
47.6±48.9
38.0±40.2
1.2±2.97.3±9.7
0.0±0.0
1.0±1.9
Task3-MSD 96.9±4.989.8±9.197.1±4.989.8±9.1
51.2±26.8
52.7±27.4
24.1±23.5
29.0±24.5
Task3-StructSeg 96.3±7.688.7±10.896.7±7.089.0±11.6
57.4±26.6
57.3±28.4
48.2±23.1
55.0±23.6
Task3-NSCLC 92.5±17.3
82.5±18.6
93.3±15.9
82.9±18.6
52.5±29.6
52.6±30.3
31.7±24.6
38.9±25.9
366 | 367 | 368 | - **How to reproduce the baseline results?** 369 | 370 | > Step 1. Install the nnU-Net following the official [guidance](https://github.com/MIC-DKFZ/nnUNet). 371 | 372 | > Step 2. Download the [3D](http://doi.org/10.5281/zenodo.3789644) or [2D](http://doi.org/10.5281/zenodo.3870441) trained models and put them into your model folder. 373 | 374 | > Step 3. Run the [inference code](https://github.com/MIC-DKFZ/nnUNet/blob/master/readme.md#run-inference). 375 | 376 | 377 | - [Github mirror](https://github.com/JunMa11/COVID-19-CT-Seg-Benchmark); [Gitee mirror](https://gitee.com/junma11/COVID-19-CT-Seg-Benchmark). 378 | 379 | ## Update 380 | - 2020.12: A large COVID-19 CT dataset with 632 patients is available at [The Cancer Imaging Archive](https://wiki.cancerimagingarchive.net/display/Public/CT+Images+in+COVID-19) 381 | - 2020.06.14: Introducing [MosMed COVID-19 dataset](https://www.medrxiv.org/content/10.1101/2020.05.20.20100362v1) as an independent testing set for each task and reporting corresponding results. 382 | 383 | > Due to the license limitation, we can not directly share this dataset, pleanse download it from the [official homepage](https://mosmed.ai/en/). 384 | 385 | - 2020.06.30: Lung annotations of MSD dataset. [Baidu NetDisk](https://pan.baidu.com/s/1A1pTzgBcqrDFW_gdefspxA) (pw: q2qv) 386 | 387 | ## TODO 388 | 389 | - [x] Provide pretrained [3D U-Net models](http://doi.org/10.5281/zenodo.3789644) by 5.6. 390 | 391 | - [x] Provide pretrained [2D U-Net models](http://doi.org/10.5281/zenodo.3870441) by 5.31. 392 | 393 | - [x] Provide lung annotations of MSD dataset by 6.30. 394 | 395 | 396 | 397 | ## Acknowledgements 398 | 399 | We thank all the organizers of MICCAI 2018 Medical Segmentation Decathlon, MICCAI 2019 Automatic Structure Segmentation for Radiotherapy Planning Challenge, [the Coronacases Initiative](https://coronacases.org ) and [Radiopaedia](https://radiopaedia.org/articles/covid-19-3) for the publicly available lung CT dataset. 400 | We also thank [Joseph Paul Cohen](https://github.com/ieee8023/covid-chestxray-dataset) for providing convenient download [link](https://academictorrents.com/details/136ffddd0959108becb2b3a86630bec049fcb0ff) of 20 COVID-19 CT scans. 401 | We also thank all the contributor of [NSCLC](https://wiki.cancerimagingarchive.net/display/DOI/Thoracic+Volume+and+Pleural+Effusion+Segmentations+in+Diseased+Lungs+for+Benchmarking+Chest+CT+Processing+Pipelines#7c5a8c0c0cef44e488b824bd7de60428) and [COVID-19-Seg-CT](https://zenodo.org/record/3757476#.XqU5iGgzZPY) dataset for providing annotations of lung, pleural effusion and COVID-19 infection. 402 | We also thank the organizers of [TMI Special Issue on Annotation-Efficient Deep Learning for Medical Imaging](http://www.embs.org/wp-content/uploads/2020/04/Special_Issue_CFP_DL4MI.pdf) because we get lots of insights from the call for papers when designing these segmentation tasks. We also thank the contributors of these great COVID-19 related resources: [COVID19_imaging_AI_paper_list](https://github.com/HzFu/COVID19_imaging_AI_paper_list) and [MedSeg](http://medicalsegmentation.com/covid19/). Last but not least, we thank Chen Chen, Xin Yang, and Yao Zhang for their important feedback on this benchmark. 403 | -------------------------------------------------------------------------------- /Task1/README.md: -------------------------------------------------------------------------------- 1 | # Task 1: Learning with limited annotations (20% training, 80% testing) 2 | 3 | ## 3D U-Net baselines for individual lung and infection segmentation 4 | 5 | 6 | 7 | 11 | 12 | 15 | 16 | 23 | 24 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 71 | 72 | 73 | 74 | 75 | 76 | 77 |
Subtask
8 |
Lung 9 |
Infection 10 |
Left Lung 13 |
Right Lung 14 |
DSC 17 |
NSD 18 |
DSC 19 |
NSD 20 |
DSC 21 |
NSD 22 |
Fold0 25 |
0.8488±0.08242
0.6869±0.1329
0.8521±0.1299
0.7055±0.1578
0.6808±0.2049
0.7088±0.2130
Fold1 34 |
0.8028±0.1454
0.6182±0.1510
0.8388±0.09582
0.6825±0.09003
0.7132±0.2053
0.7182±0.2296
Fold2 43 |
0.8714±0.1213
0.7434±0.1595
0.9034±0.08237
0.7845±0.1195
0.6618±0.2168
0.7171±0.2415
Fold3 52 |
0.8844±0.07027
0.7518±0.08787
0.8986±0.06260
0.7845±0.07952
0.6813±0.231
0.7084±0.2706
Fold4 61 |
0.8833±0.7597
0.7583±0.1104
0.9022±0.06963
0.7831±0.1020
0.6267±0.2689
0.6493±0.2823
Avg 70 |
0.8582±0.1052
0.7117±0.1384
0.8790±0.09315
0.7480±0.1191
0.6728±0.2227
0.7004±0.2437
78 | 79 | 80 | ## 3D U-Net baselines for joint lung and infection segmentation 81 | 82 | 83 | 84 | 87 | 88 | 92 | 93 | 100 | 101 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 148 | 149 | 150 | 151 | 152 | 153 | 154 |
Subtask
85 |
Lung and Infection 86 |
Left Lung 89 |
Right Lung 90 |
Infection 91 |
DSC 94 |
NSD 95 |
DSC 96 |
NSD 97 |
DSC 98 |
NSD 99 |
Fold0 102 |
0.5376±0.284
0.3905±0.1832
0.6547±0.1936
0.4736±0.1426
0.6543±0.2388
0.6815±0.232
Fold1 111 |
0.4031±0.1871
0.2753±0.1198
0.6014±0.1112
0.4171±0.0994
0.6471±0.2183
0.6055±0.2511
Fold2 120 |
0.8032±0.1875
0.6679±0.1884
0.8521±0.1243
0.6862±0.1506
0.6069±0.276
0.6245±0.289
Fold3 129 |
0.7965±0.1360
0.6543±0.1442
0.8401±0.0977
0.6770±0.1304
0.6198±0.2787
0.6532±0.2891
Fold4 138 |
0.7242±0.2109
0.5855±0.2081
0.8086±0.1344
0.6340±0.1586
0.5138±0.3015
0.5186±0.3101
Avg 147 |
0.6544±0.2556
0.5163±0.2291
0.7527±0.1678
0.5789±0.1746
0.6078±0.2628
0.6159±0.2748
155 | 156 | 157 | 158 | ## 3D U-Net baselines for infection segmentation, and joint lung and infection segmentation (80% training, 20% testing) 159 | 160 | 161 | 162 | 166 | 167 | 171 | 172 | 181 | 182 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 |
Subtask
163 |
Infection 164 |
Lung and Infection 165 |
Left Lung 168 |
Right Lung 169 |
Infection 170 |
DSC 173 |
NSD 174 |
DSC 175 |
NSD 176 |
DSC 177 |
NSD 178 |
DSC 179 |
NSD 180 |
Fold0 183 |
0.8019±0.0603
0.8599±0.09182
0.9207±0.08202
0.8028±0.1474
0.9355±0.05415
0.8018±0.1208
0.7919±0.06362
0.8471±0.09475
Fold1 194 |
0.7807±0.05981
0.8358±0.07287
0.9387±0.08439
0.8419±0.1574
0.9411±0.06709
0.8192±0.1406
0.8044±0.04674
0.8581±0.05861
Fold2 205 |
0.8289±0.06594
0.8602±0.04390
0.9362±0.02470
0.7909±0.04488
0.9452±0.01671
0.8016±0.03551
0.8106±0.07865
0.8286±0.1029
Fold3 216 |
0.8155±0.08941
0.7971±0.1656
0.7995±0.2199
0.6781±0.2224
0.8168±0.2201
0.6834±0.2137
0.8013±0.09603
0.7883±0.1687
Fold4 227 |
0.6575±0.4195
0.6886±0.4232
0.9604±0.02932
0.8583±0.05223
0.9694±0.01188
0.8556±0.009157
0.6533±0.4286
0.6852±0.4376
Avg 238 |
0.7769±0.1868
0.8083±0.1985
0.9111±0.1162
0.7944±0.1418
0.9216±0.1092
0.7923±0.1280
0.7723±0.1902
0.8015±0.2062
248 | -------------------------------------------------------------------------------- /Task1/Task1_COVID_82_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task1/Task1_COVID_82_datasplit.pkl -------------------------------------------------------------------------------- /Task1/Task1_COVID_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task1/Task1_COVID_datasplit.pkl -------------------------------------------------------------------------------- /Task2/README.md: -------------------------------------------------------------------------------- 1 | # Task 2: Learning to segment COVID-19 CT scans from non-COVID-19 CT scans 2 | 3 | 4 | ## 3D U-Net baselines for lung segmentation 5 | 6 | 7 | 8 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 |
Subtask
9 |
Validation Set 10 |
Testing Set 11 |
Left Lung
Right Lung
Left Lung
Right Lung
DSC
NSD
DSC
NSD
DSC
NSD
DSC
NSD
StructSeg Lung
0.9642±0.01362
0.7464±0.0913
0.9730±0.0026
0.7434±0.07216
0.9215±0.1965
0.8202±0.1573
0.9554±0.07216
0.8419±0.1159
NSCLC Lung
0.9530±0.04923
0.8017±0.08274
0.9536±0.1086
0.8066±0.1067
0.5751±0.2149
0.4687±0.1693
0.7219±0.1533
0.5171±0.1676
51 | 52 | 53 | ## 3D U-Net baselines for infection segmentation 54 | 55 | 56 | 57 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 |
Subtask
58 |
Validation Set
Testing Set
DSC
NSD
DSC
NSD
MSD Lung Tumor
0.6720±0.2708
0.7708±0.3139
0.2517±0.2741
0.2595±0.2851
StructSeg Gross Target
0.7132±0.2957
0.7032±0.2948
0.05998±0.1270
0.0550±0.1070
NSCLC-PE
0.6435±0.1549
0.7368±0.1293
0.003850±0.008947
0.03687±0.04834
89 | -------------------------------------------------------------------------------- /Task2/Task200_StructSegLung_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task2/Task200_StructSegLung_datasplit.pkl -------------------------------------------------------------------------------- /Task2/Task201_NSCLCLung_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task2/Task201_NSCLCLung_datasplit.pkl -------------------------------------------------------------------------------- /Task2/Task202_MSD_LungTumor_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task2/Task202_MSD_LungTumor_datasplit.pkl -------------------------------------------------------------------------------- /Task2/Task203_StructSegTumor_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task2/Task203_StructSegTumor_datasplit.pkl -------------------------------------------------------------------------------- /Task2/Task204_NSCLCPE_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task2/Task204_NSCLCPE_datasplit.pkl -------------------------------------------------------------------------------- /Task3/LungPseudoLabels/README.md: -------------------------------------------------------------------------------- 1 | We provide lung pseudo labels for public COVID-19 CT datasets, which are generated by the pretrained models. 2 | 3 | ### COVID-19 Lung CT Lesion Segmentation Challenge - 2020 [(COVID-19-20)](https://covid-segmentation.grand-challenge.org/COVID-19-20/) 4 | 5 | - `Task300-Infer-COVID-19-20-Training.zip`: lung pseudo labels of the training set 6 | - `Task300-Infer-COVID-19-20-Validation.zip`: lung pseudo labels of the validation set 7 | 8 | **Please keep in mind that these labels are pseudo labels rather than ground truth.** Thus, the product between the images and the pseudo labels is not the complete lung tissue. 9 | One possible usage of the pseudo labels is to extract the bounding box information of the lungs. 10 | 11 | 12 | -------------------------------------------------------------------------------- /Task3/LungPseudoLabels/Task300-Infer-COVID-19-20-Training.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task3/LungPseudoLabels/Task300-Infer-COVID-19-20-Training.zip -------------------------------------------------------------------------------- /Task3/LungPseudoLabels/Task300-Infer-COVID-19-20-Validation.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task3/LungPseudoLabels/Task300-Infer-COVID-19-20-Validation.zip -------------------------------------------------------------------------------- /Task3/README.md: -------------------------------------------------------------------------------- 1 | # Task 3: Learning with both COVID-19 and non-COVID-19 CT scans 2 | 3 | 4 | ## 3D U-Net baselines for lung segmentation 5 | 6 | 7 | 11 | 12 | 17 | 18 | 27 | 28 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 |
Subtask
8 |
Validation Set 9 |
Testing Set 10 |
Left Lung 13 |
Right Lung 14 |
Left Lung 15 |
Right Lung 16 |
DSC 19 |
NSD 20 |
DSC 21 |
NSD 22 |
DSC 23 |
NSD 24 |
DSC 25 |
NSD 26 |
StructSeg Lung
29 |
Fold0 30 |
0.9632±0.01222
0.7392±0.08513
0.9718±0.003526
0.7393±0.07017
0.9737±0.01928
0.9033±0.05893
0.9760±0.02040
0.9082±0.06067
Fold1 41 |
0.9629±0.01212
0.7370±0.08380
0.9714±0.003654
0.7342±0.07034
0.9768±0.01287
0.9103±0.05301
0.9799±0.01138
0.9178±0.04883
Fold2 52 |
0.9635±0.01254
0.7426±0.08482
0.9719±0.003474
0.7399±0.06888
0.9681±0.03147
0.8937±0.08818
0.9761±0.02844
0.9094±0.07768
Fold3 63 |
0.9631±0.01206
0.7388±0.08532
0.9719±0.003377
0.7394±0.06996
0.9693±0.02504
0.9066±0.05615
0.9725±0.02522
0.9128±0.06349
Fold4 74 |
0.9628±0.01294
0.7382±0.08891
0.9717±0.003885
0.7383±0.07256
0.9777±0.01294
0.9159±0.05313
0.9804±0.01332
0.9199±0.05670
Avg 85 |
0.9631±0.01187
0.7392±0.08207
0.9717±0.003443
0.7382±0.06749
0.9731±0.02136
0.9060±0.06212
0.9770±0.02050
0.9136±0.06078
NSCLC Lung
96 |
Fold0 97 |
0.9574±0.04580
0.8120±0.07485
0.9547±0.1087
0.8097±0.1089
0.9272±0.06327
0.7535±0.1448
0.9297±0.06976
0.8526±0.1603
Fold1 108 |
0.9536±0.05027
0.8053±0.08703
0.9516±0.1108
0.8047±0.1141
0.9223±0.07158
0.7357±0.1749
0.9427±0.03816
0.7668±0.1431
Fold2 119 |
0.9535±0.04857
0.7971±0.08188
0.9523±0.1088
0.7998±0.1125
0.9409±0.04120
0.7739±0.1199
0.9379±0.05813
0.7555±0.1623
Fold3 130 |
0.9537±0.04944
0.7975±0.07700
0.9484±0.1115
0.7950±0.1163
0.9357±0.05080
0.7786±0.1162
0.9358±0.05910
0.7820±0.1325
Fold4 141 |
0.9569±0.04607
0.8114±0.07851
0.9550±0.1086
0.8091±0.1095
0.9476±0.03763
0.8053±0.1049
0.9512±0.03294
0.8052±0.1113
Avg 152 |
0.9550±0.04785
0.8047±0.07983
0.9524±0.1092
0.8036±0.1117
0.9347±0.0538
0.7694±0.1332
0.9395±0.05260
0.7724±0.1408
162 | 163 | ## 3D U-Net baselines for infection segmentation 164 | 165 | 166 | 167 | 171 | 172 | 177 | 178 | 181 | 182 | 183 | 184 | 185 | 186 | 188 | 189 | 190 | 191 | 192 | 193 | 195 | 196 | 197 | 198 | 199 | 200 | 202 | 203 | 204 | 205 | 206 | 207 | 209 | 210 | 211 | 212 | 213 | 214 | 216 | 217 | 218 | 219 | 220 | 221 | 224 | 225 | 226 | 227 | 228 | 229 | 231 | 232 | 233 | 234 | 235 | 236 | 238 | 239 | 240 | 241 | 242 | 243 | 245 | 246 | 247 | 248 | 249 | 250 | 252 | 253 | 254 | 255 | 256 | 257 | 259 | 260 | 261 | 262 | 263 | 264 | 267 | 268 | 269 | 270 | 271 | 272 | 274 | 275 | 276 | 277 | 278 | 279 | 281 | 282 | 283 | 284 | 285 | 286 | 288 | 289 | 290 | 291 | 292 | 293 | 295 | 296 | 297 | 298 | 299 | 300 | 302 | 303 | 304 | 305 | 306 |
Subtask
168 |
Validation Set 169 |
Testing Set 170 |
DSC 173 |
NSD 174 |
DSC 175 |
NSD 176 |
MSD Lung Tumor
179 |
Fold0 180 |
0.6718±0.2665
0.7814±0.3082
0.6803±0.2246
0.6656±0.2374
Fold1 187 |
0.6625±0.2613
0.7686±0.2963
0.6704±0.2199
0.6511±0.2585
Fold2 194 |
0.6714±0.2537
0.7744±0.2780
0.6299±0.2789
0.6441±0.2869
Fold3 201 |
0.6392±0.2656
0.7376±0.3106
0.6171±0.2451
0.5975±0.2850
Fold4 208 |
0.6801±0.2591
0.7877±0.2988
0.5189±0.3061
0.5057±0.3081
Avg 215 |
0.6650±0.2527
0.7699±0.2888
0.6233±0.2570
0.6128±0.2755
StructSeg Gross Target
222 |
Fold0 223 |
0.7823±0.1413
0.7542±0.1753
0.6929±0.2046
0.6795±0.2177
Fold1 230 |
0.7859±0.1403
00.7605±0.1806
0.6830±0.2240
0.6478±0.2627
Fold2 237 |
0.7702±0.1373
0.7297±0.1701
0.6358±0.2535
0.6612±0.2550
Fold3 244 |
0.7875±0.1360
0.7597±0.1754
0.6702±0.2406
0.6639±0.2521
Fold4 251 |
0.7747±0.1365
0.7463±0.1838
0.5257±0.2874
0.5117±0.2854
Avg 258 |
0.7801±0.1327
0.7501±0.1701
0.6415±0.2452
0.6328±0.2565
NSCLC-PE
265 |
Fold0 266 |
0.6548±0.1544
0.7428±0.1319
0.6921±0.2068
0.6648±0.2235
Fold1 273 |
0.6470±0.1544
0.7384±0.1426
0.5969±0.2272
0.5579±0.2546
Fold2 280 |
0.6551±0.1517
0.7480±0.1368
0.6158±0.2824
0.6171±0.2913
Fold3 287 |
0.6468±0.1596
0.7398±0.1361
0.6267±0.2565
0.6201±0.2773
Fold4 294 |
0.6523±0.1561
0.7472±0.1301
0.4765±0.2723
0.4651±0.2698
Avg 301 |
0.6512±0.1518
0.7433±0.1319
0.6016±0.2542
0.5850±0.2670
307 | -------------------------------------------------------------------------------- /Task3/Task300_StructSegLung_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task3/Task300_StructSegLung_datasplit.pkl -------------------------------------------------------------------------------- /Task3/Task301_NSCLCLung_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task3/Task301_NSCLCLung_datasplit.pkl -------------------------------------------------------------------------------- /Task3/Task302_MSD_LungTumor_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task3/Task302_MSD_LungTumor_datasplit.pkl -------------------------------------------------------------------------------- /Task3/Task303_StructSegTumor_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task3/Task303_StructSegTumor_datasplit.pkl -------------------------------------------------------------------------------- /Task3/Task304_NSCLCPE_datasplit.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/Task3/Task304_NSCLCPE_datasplit.pkl -------------------------------------------------------------------------------- /utils/COVID-19-Seg-Evaluation.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Mon Apr 13 19:43:47 2020 4 | 5 | @author: JUN MA 6 | """ 7 | 8 | import numpy as np 9 | import nibabel as nb 10 | import os 11 | from collections import OrderedDict 12 | import pandas as pd 13 | from SurfaceDice import compute_surface_distances, compute_surface_dice_at_tolerance, compute_dice_coefficient 14 | join = os.path.join 15 | 16 | 17 | seg_path = 'path to segmentation' 18 | gt_path = 'path to ground truth' 19 | save_path = 'path to save segmentation metrics' 20 | save_name = 'Task_num_SegMetric.csv' 21 | 22 | filenames = os.listdir(seg_path) 23 | filenames.sort() 24 | num_labels = np.max(nb.load(join(gt_path, filenames[0])).get_fdata()) 25 | 26 | seg_metrics = OrderedDict() 27 | seg_metrics['Name'] = list() 28 | 29 | if num_labels==1: 30 | seg_metrics['LesionDSC'] = list() 31 | seg_metrics['LesionNSD-3mm'] = list() 32 | 33 | elif num_labels==2: 34 | seg_metrics['L-lungDSC'] = list() 35 | seg_metrics['L-lung-1mm'] = list() 36 | 37 | seg_metrics['R-lungDSC'] = list() 38 | seg_metrics['R-lung-1mm'] = list() 39 | else: 40 | seg_metrics['L-lungDSC'] = list() 41 | seg_metrics['L-lung-1mm'] = list() 42 | 43 | seg_metrics['R-lungDSC'] = list() 44 | seg_metrics['R-lung-1mm'] = list() 45 | 46 | seg_metrics['LesionDSC'] = list() 47 | seg_metrics['LesionNSD-3mm'] = list() 48 | 49 | 50 | for name in filenames: 51 | seg_metrics['Name'].append(name) 52 | # load grond truth and segmentation 53 | gt_nii = nb.load(join(gt_path, name)) 54 | case_spacing = gt_nii.header.get_zooms() 55 | gt_data = np.uint8(gt_nii.get_fdata()) 56 | seg_data = nb.load(join(seg_path, name)).get_fdata() 57 | 58 | labels = np.unique(gt_data)[1:] 59 | labels = labels.tolist() 60 | 61 | if num_labels==1: # Lesion 62 | surface_distances = compute_surface_distances(gt_data==1, seg_data==1, case_spacing) 63 | seg_metrics['LesionDSC'].append( compute_dice_coefficient(gt_data==1, seg_data==1)) 64 | seg_metrics['LesionNSD-3mm'].append(compute_surface_dice_at_tolerance(surface_distances, 3)) 65 | 66 | elif num_labels==2: # left and right lung 67 | surface_distances = compute_surface_distances(gt_data==1, seg_data==1, case_spacing) 68 | seg_metrics['L-lungDSC'].append(compute_dice_coefficient(gt_data==1, seg_data==1)) 69 | seg_metrics['L-lung-1mm'].append(compute_surface_dice_at_tolerance(surface_distances, 1)) 70 | 71 | surface_distances = compute_surface_distances(gt_data==2, seg_data==2, case_spacing) 72 | seg_metrics['R-lungDSC'].append(compute_dice_coefficient(gt_data==2, seg_data==2)) 73 | seg_metrics['R-lung-1mm'].append(compute_surface_dice_at_tolerance(surface_distances, 1)) 74 | 75 | else: # left lung, right lung and infections 76 | surface_distances = compute_surface_distances(gt_data==1, seg_data==1, case_spacing) 77 | seg_metrics['L-lungDSC'].append( compute_dice_coefficient(gt_data==1, seg_data==1)) 78 | seg_metrics['L-lung-1mm'].append( compute_surface_dice_at_tolerance(surface_distances, 1) ) 79 | 80 | surface_distances = compute_surface_distances(gt_data==2, seg_data==2, case_spacing) 81 | seg_metrics['R-lungDSC'].append( compute_dice_coefficient(gt_data==2, seg_data==2)) 82 | seg_metrics['R-lung-1mm'].append( compute_surface_dice_at_tolerance(surface_distances, 1)) 83 | 84 | surface_distances = compute_surface_distances(gt_data==3, seg_data==3, case_spacing) 85 | seg_metrics['LesionDSC'].append(compute_dice_coefficient(gt_data==3, seg_data==3)) 86 | seg_metrics['LesionNSD-3mm'].append( compute_surface_dice_at_tolerance(surface_distances, 3)) 87 | 88 | dataframe = pd.DataFrame(seg_metrics) 89 | dataframe.to_csv(join(save_path, save_name), index=False) 90 | 91 | 92 | -------------------------------------------------------------------------------- /utils/ImageExamples.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JunMa11/COVID-19-CT-Seg-Benchmark/b87e0544c08f3dff3513fc488d0624b372fecc1d/utils/ImageExamples.png -------------------------------------------------------------------------------- /utils/README.md: -------------------------------------------------------------------------------- 1 | TBA 2 | -------------------------------------------------------------------------------- /utils/SurfaceDice.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Code for computing surface Dice 4 | copy from http://medicaldecathlon.com/files/Surface_distance_based_measures.ipynb 5 | """ 6 | 7 | import numpy as np 8 | import scipy.ndimage 9 | 10 | # neighbour_code_to_normals is a lookup table. 11 | # For every binary neighbour code 12 | # (2x2x2 neighbourhood = 8 neighbours = 8 bits = 256 codes) 13 | # it contains the surface normals of the triangles (called "surfel" for 14 | # "surface element" in the following). The length of the normal 15 | # vector encodes the surfel area. 16 | # 17 | # created by compute_surface_area_lookup_table.ipynb using the 18 | # marching_cube algorithm, see e.g. https://en.wikipedia.org/wiki/Marching_cubes 19 | # 20 | neighbour_code_to_normals = [ 21 | [[0,0,0]], 22 | [[0.125,0.125,0.125]], 23 | [[-0.125,-0.125,0.125]], 24 | [[-0.25,-0.25,0.0],[0.25,0.25,-0.0]], 25 | [[0.125,-0.125,0.125]], 26 | [[-0.25,-0.0,-0.25],[0.25,0.0,0.25]], 27 | [[0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 28 | [[0.5,0.0,-0.0],[0.25,0.25,0.25],[0.125,0.125,0.125]], 29 | [[-0.125,0.125,0.125]], 30 | [[0.125,0.125,0.125],[-0.125,0.125,0.125]], 31 | [[-0.25,0.0,0.25],[-0.25,0.0,0.25]], 32 | [[0.5,0.0,0.0],[-0.25,-0.25,0.25],[-0.125,-0.125,0.125]], 33 | [[0.25,-0.25,0.0],[0.25,-0.25,0.0]], 34 | [[0.5,0.0,0.0],[0.25,-0.25,0.25],[-0.125,0.125,-0.125]], 35 | [[-0.5,0.0,0.0],[-0.25,0.25,0.25],[-0.125,0.125,0.125]], 36 | [[0.5,0.0,0.0],[0.5,0.0,0.0]], 37 | [[0.125,-0.125,-0.125]], 38 | [[0.0,-0.25,-0.25],[0.0,0.25,0.25]], 39 | [[-0.125,-0.125,0.125],[0.125,-0.125,-0.125]], 40 | [[0.0,-0.5,0.0],[0.25,0.25,0.25],[0.125,0.125,0.125]], 41 | [[0.125,-0.125,0.125],[0.125,-0.125,-0.125]], 42 | [[0.0,0.0,-0.5],[0.25,0.25,0.25],[-0.125,-0.125,-0.125]], 43 | [[-0.125,-0.125,0.125],[0.125,-0.125,0.125],[0.125,-0.125,-0.125]], 44 | [[-0.125,-0.125,-0.125],[-0.25,-0.25,-0.25],[0.25,0.25,0.25],[0.125,0.125,0.125]], 45 | [[-0.125,0.125,0.125],[0.125,-0.125,-0.125]], 46 | [[0.0,-0.25,-0.25],[0.0,0.25,0.25],[-0.125,0.125,0.125]], 47 | [[-0.25,0.0,0.25],[-0.25,0.0,0.25],[0.125,-0.125,-0.125]], 48 | [[0.125,0.125,0.125],[0.375,0.375,0.375],[0.0,-0.25,0.25],[-0.25,0.0,0.25]], 49 | [[0.125,-0.125,-0.125],[0.25,-0.25,0.0],[0.25,-0.25,0.0]], 50 | [[0.375,0.375,0.375],[0.0,0.25,-0.25],[-0.125,-0.125,-0.125],[-0.25,0.25,0.0]], 51 | [[-0.5,0.0,0.0],[-0.125,-0.125,-0.125],[-0.25,-0.25,-0.25],[0.125,0.125,0.125]], 52 | [[-0.5,0.0,0.0],[-0.125,-0.125,-0.125],[-0.25,-0.25,-0.25]], 53 | [[0.125,-0.125,0.125]], 54 | [[0.125,0.125,0.125],[0.125,-0.125,0.125]], 55 | [[0.0,-0.25,0.25],[0.0,0.25,-0.25]], 56 | [[0.0,-0.5,0.0],[0.125,0.125,-0.125],[0.25,0.25,-0.25]], 57 | [[0.125,-0.125,0.125],[0.125,-0.125,0.125]], 58 | [[0.125,-0.125,0.125],[-0.25,-0.0,-0.25],[0.25,0.0,0.25]], 59 | [[0.0,-0.25,0.25],[0.0,0.25,-0.25],[0.125,-0.125,0.125]], 60 | [[-0.375,-0.375,0.375],[-0.0,0.25,0.25],[0.125,0.125,-0.125],[-0.25,-0.0,-0.25]], 61 | [[-0.125,0.125,0.125],[0.125,-0.125,0.125]], 62 | [[0.125,0.125,0.125],[0.125,-0.125,0.125],[-0.125,0.125,0.125]], 63 | [[-0.0,0.0,0.5],[-0.25,-0.25,0.25],[-0.125,-0.125,0.125]], 64 | [[0.25,0.25,-0.25],[0.25,0.25,-0.25],[0.125,0.125,-0.125],[-0.125,-0.125,0.125]], 65 | [[0.125,-0.125,0.125],[0.25,-0.25,0.0],[0.25,-0.25,0.0]], 66 | [[0.5,0.0,0.0],[0.25,-0.25,0.25],[-0.125,0.125,-0.125],[0.125,-0.125,0.125]], 67 | [[0.0,0.25,-0.25],[0.375,-0.375,-0.375],[-0.125,0.125,0.125],[0.25,0.25,0.0]], 68 | [[-0.5,0.0,0.0],[-0.25,-0.25,0.25],[-0.125,-0.125,0.125]], 69 | [[0.25,-0.25,0.0],[-0.25,0.25,0.0]], 70 | [[0.0,0.5,0.0],[-0.25,0.25,0.25],[0.125,-0.125,-0.125]], 71 | [[0.0,0.5,0.0],[0.125,-0.125,0.125],[-0.25,0.25,-0.25]], 72 | [[0.0,0.5,0.0],[0.0,-0.5,0.0]], 73 | [[0.25,-0.25,0.0],[-0.25,0.25,0.0],[0.125,-0.125,0.125]], 74 | [[-0.375,-0.375,-0.375],[-0.25,0.0,0.25],[-0.125,-0.125,-0.125],[-0.25,0.25,0.0]], 75 | [[0.125,0.125,0.125],[0.0,-0.5,0.0],[-0.25,-0.25,-0.25],[-0.125,-0.125,-0.125]], 76 | [[0.0,-0.5,0.0],[-0.25,-0.25,-0.25],[-0.125,-0.125,-0.125]], 77 | [[-0.125,0.125,0.125],[0.25,-0.25,0.0],[-0.25,0.25,0.0]], 78 | [[0.0,0.5,0.0],[0.25,0.25,-0.25],[-0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 79 | [[-0.375,0.375,-0.375],[-0.25,-0.25,0.0],[-0.125,0.125,-0.125],[-0.25,0.0,0.25]], 80 | [[0.0,0.5,0.0],[0.25,0.25,-0.25],[-0.125,-0.125,0.125]], 81 | [[0.25,-0.25,0.0],[-0.25,0.25,0.0],[0.25,-0.25,0.0],[0.25,-0.25,0.0]], 82 | [[-0.25,-0.25,0.0],[-0.25,-0.25,0.0],[-0.125,-0.125,0.125]], 83 | [[0.125,0.125,0.125],[-0.25,-0.25,0.0],[-0.25,-0.25,0.0]], 84 | [[-0.25,-0.25,0.0],[-0.25,-0.25,0.0]], 85 | [[-0.125,-0.125,0.125]], 86 | [[0.125,0.125,0.125],[-0.125,-0.125,0.125]], 87 | [[-0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 88 | [[-0.125,-0.125,0.125],[-0.25,-0.25,0.0],[0.25,0.25,-0.0]], 89 | [[0.0,-0.25,0.25],[0.0,-0.25,0.25]], 90 | [[0.0,0.0,0.5],[0.25,-0.25,0.25],[0.125,-0.125,0.125]], 91 | [[0.0,-0.25,0.25],[0.0,-0.25,0.25],[-0.125,-0.125,0.125]], 92 | [[0.375,-0.375,0.375],[0.0,-0.25,-0.25],[-0.125,0.125,-0.125],[0.25,0.25,0.0]], 93 | [[-0.125,-0.125,0.125],[-0.125,0.125,0.125]], 94 | [[0.125,0.125,0.125],[-0.125,-0.125,0.125],[-0.125,0.125,0.125]], 95 | [[-0.125,-0.125,0.125],[-0.25,0.0,0.25],[-0.25,0.0,0.25]], 96 | [[0.5,0.0,0.0],[-0.25,-0.25,0.25],[-0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 97 | [[-0.0,0.5,0.0],[-0.25,0.25,-0.25],[0.125,-0.125,0.125]], 98 | [[-0.25,0.25,-0.25],[-0.25,0.25,-0.25],[-0.125,0.125,-0.125],[-0.125,0.125,-0.125]], 99 | [[-0.25,0.0,-0.25],[0.375,-0.375,-0.375],[0.0,0.25,-0.25],[-0.125,0.125,0.125]], 100 | [[0.5,0.0,0.0],[-0.25,0.25,-0.25],[0.125,-0.125,0.125]], 101 | [[-0.25,0.0,0.25],[0.25,0.0,-0.25]], 102 | [[-0.0,0.0,0.5],[-0.25,0.25,0.25],[-0.125,0.125,0.125]], 103 | [[-0.125,-0.125,0.125],[-0.25,0.0,0.25],[0.25,0.0,-0.25]], 104 | [[-0.25,-0.0,-0.25],[-0.375,0.375,0.375],[-0.25,-0.25,0.0],[-0.125,0.125,0.125]], 105 | [[0.0,0.0,-0.5],[0.25,0.25,-0.25],[-0.125,-0.125,0.125]], 106 | [[-0.0,0.0,0.5],[0.0,0.0,0.5]], 107 | [[0.125,0.125,0.125],[0.125,0.125,0.125],[0.25,0.25,0.25],[0.0,0.0,0.5]], 108 | [[0.125,0.125,0.125],[0.25,0.25,0.25],[0.0,0.0,0.5]], 109 | [[-0.25,0.0,0.25],[0.25,0.0,-0.25],[-0.125,0.125,0.125]], 110 | [[-0.0,0.0,0.5],[0.25,-0.25,0.25],[0.125,-0.125,0.125],[0.125,-0.125,0.125]], 111 | [[-0.25,0.0,0.25],[-0.25,0.0,0.25],[-0.25,0.0,0.25],[0.25,0.0,-0.25]], 112 | [[0.125,-0.125,0.125],[0.25,0.0,0.25],[0.25,0.0,0.25]], 113 | [[0.25,0.0,0.25],[-0.375,-0.375,0.375],[-0.25,0.25,0.0],[-0.125,-0.125,0.125]], 114 | [[-0.0,0.0,0.5],[0.25,-0.25,0.25],[0.125,-0.125,0.125]], 115 | [[0.125,0.125,0.125],[0.25,0.0,0.25],[0.25,0.0,0.25]], 116 | [[0.25,0.0,0.25],[0.25,0.0,0.25]], 117 | [[-0.125,-0.125,0.125],[0.125,-0.125,0.125]], 118 | [[0.125,0.125,0.125],[-0.125,-0.125,0.125],[0.125,-0.125,0.125]], 119 | [[-0.125,-0.125,0.125],[0.0,-0.25,0.25],[0.0,0.25,-0.25]], 120 | [[0.0,-0.5,0.0],[0.125,0.125,-0.125],[0.25,0.25,-0.25],[-0.125,-0.125,0.125]], 121 | [[0.0,-0.25,0.25],[0.0,-0.25,0.25],[0.125,-0.125,0.125]], 122 | [[0.0,0.0,0.5],[0.25,-0.25,0.25],[0.125,-0.125,0.125],[0.125,-0.125,0.125]], 123 | [[0.0,-0.25,0.25],[0.0,-0.25,0.25],[0.0,-0.25,0.25],[0.0,0.25,-0.25]], 124 | [[0.0,0.25,0.25],[0.0,0.25,0.25],[0.125,-0.125,-0.125]], 125 | [[-0.125,0.125,0.125],[0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 126 | [[-0.125,0.125,0.125],[0.125,-0.125,0.125],[-0.125,-0.125,0.125],[0.125,0.125,0.125]], 127 | [[-0.0,0.0,0.5],[-0.25,-0.25,0.25],[-0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 128 | [[0.125,0.125,0.125],[0.125,-0.125,0.125],[0.125,-0.125,-0.125]], 129 | [[-0.0,0.5,0.0],[-0.25,0.25,-0.25],[0.125,-0.125,0.125],[0.125,-0.125,0.125]], 130 | [[0.125,0.125,0.125],[-0.125,-0.125,0.125],[0.125,-0.125,-0.125]], 131 | [[0.0,-0.25,-0.25],[0.0,0.25,0.25],[0.125,0.125,0.125]], 132 | [[0.125,0.125,0.125],[0.125,-0.125,-0.125]], 133 | [[0.5,0.0,-0.0],[0.25,-0.25,-0.25],[0.125,-0.125,-0.125]], 134 | [[-0.25,0.25,0.25],[-0.125,0.125,0.125],[-0.25,0.25,0.25],[0.125,-0.125,-0.125]], 135 | [[0.375,-0.375,0.375],[0.0,0.25,0.25],[-0.125,0.125,-0.125],[-0.25,0.0,0.25]], 136 | [[0.0,-0.5,0.0],[-0.25,0.25,0.25],[-0.125,0.125,0.125]], 137 | [[-0.375,-0.375,0.375],[0.25,-0.25,0.0],[0.0,0.25,0.25],[-0.125,-0.125,0.125]], 138 | [[-0.125,0.125,0.125],[-0.25,0.25,0.25],[0.0,0.0,0.5]], 139 | [[0.125,0.125,0.125],[0.0,0.25,0.25],[0.0,0.25,0.25]], 140 | [[0.0,0.25,0.25],[0.0,0.25,0.25]], 141 | [[0.5,0.0,-0.0],[0.25,0.25,0.25],[0.125,0.125,0.125],[0.125,0.125,0.125]], 142 | [[0.125,-0.125,0.125],[-0.125,-0.125,0.125],[0.125,0.125,0.125]], 143 | [[-0.25,-0.0,-0.25],[0.25,0.0,0.25],[0.125,0.125,0.125]], 144 | [[0.125,0.125,0.125],[0.125,-0.125,0.125]], 145 | [[-0.25,-0.25,0.0],[0.25,0.25,-0.0],[0.125,0.125,0.125]], 146 | [[0.125,0.125,0.125],[-0.125,-0.125,0.125]], 147 | [[0.125,0.125,0.125],[0.125,0.125,0.125]], 148 | [[0.125,0.125,0.125]], 149 | [[0.125,0.125,0.125]], 150 | [[0.125,0.125,0.125],[0.125,0.125,0.125]], 151 | [[0.125,0.125,0.125],[-0.125,-0.125,0.125]], 152 | [[-0.25,-0.25,0.0],[0.25,0.25,-0.0],[0.125,0.125,0.125]], 153 | [[0.125,0.125,0.125],[0.125,-0.125,0.125]], 154 | [[-0.25,-0.0,-0.25],[0.25,0.0,0.25],[0.125,0.125,0.125]], 155 | [[0.125,-0.125,0.125],[-0.125,-0.125,0.125],[0.125,0.125,0.125]], 156 | [[0.5,0.0,-0.0],[0.25,0.25,0.25],[0.125,0.125,0.125],[0.125,0.125,0.125]], 157 | [[0.0,0.25,0.25],[0.0,0.25,0.25]], 158 | [[0.125,0.125,0.125],[0.0,0.25,0.25],[0.0,0.25,0.25]], 159 | [[-0.125,0.125,0.125],[-0.25,0.25,0.25],[0.0,0.0,0.5]], 160 | [[-0.375,-0.375,0.375],[0.25,-0.25,0.0],[0.0,0.25,0.25],[-0.125,-0.125,0.125]], 161 | [[0.0,-0.5,0.0],[-0.25,0.25,0.25],[-0.125,0.125,0.125]], 162 | [[0.375,-0.375,0.375],[0.0,0.25,0.25],[-0.125,0.125,-0.125],[-0.25,0.0,0.25]], 163 | [[-0.25,0.25,0.25],[-0.125,0.125,0.125],[-0.25,0.25,0.25],[0.125,-0.125,-0.125]], 164 | [[0.5,0.0,-0.0],[0.25,-0.25,-0.25],[0.125,-0.125,-0.125]], 165 | [[0.125,0.125,0.125],[0.125,-0.125,-0.125]], 166 | [[0.0,-0.25,-0.25],[0.0,0.25,0.25],[0.125,0.125,0.125]], 167 | [[0.125,0.125,0.125],[-0.125,-0.125,0.125],[0.125,-0.125,-0.125]], 168 | [[-0.0,0.5,0.0],[-0.25,0.25,-0.25],[0.125,-0.125,0.125],[0.125,-0.125,0.125]], 169 | [[0.125,0.125,0.125],[0.125,-0.125,0.125],[0.125,-0.125,-0.125]], 170 | [[-0.0,0.0,0.5],[-0.25,-0.25,0.25],[-0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 171 | [[-0.125,0.125,0.125],[0.125,-0.125,0.125],[-0.125,-0.125,0.125],[0.125,0.125,0.125]], 172 | [[-0.125,0.125,0.125],[0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 173 | [[0.0,0.25,0.25],[0.0,0.25,0.25],[0.125,-0.125,-0.125]], 174 | [[0.0,-0.25,-0.25],[0.0,0.25,0.25],[0.0,0.25,0.25],[0.0,0.25,0.25]], 175 | [[0.0,0.0,0.5],[0.25,-0.25,0.25],[0.125,-0.125,0.125],[0.125,-0.125,0.125]], 176 | [[0.0,-0.25,0.25],[0.0,-0.25,0.25],[0.125,-0.125,0.125]], 177 | [[0.0,-0.5,0.0],[0.125,0.125,-0.125],[0.25,0.25,-0.25],[-0.125,-0.125,0.125]], 178 | [[-0.125,-0.125,0.125],[0.0,-0.25,0.25],[0.0,0.25,-0.25]], 179 | [[0.125,0.125,0.125],[-0.125,-0.125,0.125],[0.125,-0.125,0.125]], 180 | [[-0.125,-0.125,0.125],[0.125,-0.125,0.125]], 181 | [[0.25,0.0,0.25],[0.25,0.0,0.25]], 182 | [[0.125,0.125,0.125],[0.25,0.0,0.25],[0.25,0.0,0.25]], 183 | [[-0.0,0.0,0.5],[0.25,-0.25,0.25],[0.125,-0.125,0.125]], 184 | [[0.25,0.0,0.25],[-0.375,-0.375,0.375],[-0.25,0.25,0.0],[-0.125,-0.125,0.125]], 185 | [[0.125,-0.125,0.125],[0.25,0.0,0.25],[0.25,0.0,0.25]], 186 | [[-0.25,-0.0,-0.25],[0.25,0.0,0.25],[0.25,0.0,0.25],[0.25,0.0,0.25]], 187 | [[-0.0,0.0,0.5],[0.25,-0.25,0.25],[0.125,-0.125,0.125],[0.125,-0.125,0.125]], 188 | [[-0.25,0.0,0.25],[0.25,0.0,-0.25],[-0.125,0.125,0.125]], 189 | [[0.125,0.125,0.125],[0.25,0.25,0.25],[0.0,0.0,0.5]], 190 | [[0.125,0.125,0.125],[0.125,0.125,0.125],[0.25,0.25,0.25],[0.0,0.0,0.5]], 191 | [[-0.0,0.0,0.5],[0.0,0.0,0.5]], 192 | [[0.0,0.0,-0.5],[0.25,0.25,-0.25],[-0.125,-0.125,0.125]], 193 | [[-0.25,-0.0,-0.25],[-0.375,0.375,0.375],[-0.25,-0.25,0.0],[-0.125,0.125,0.125]], 194 | [[-0.125,-0.125,0.125],[-0.25,0.0,0.25],[0.25,0.0,-0.25]], 195 | [[-0.0,0.0,0.5],[-0.25,0.25,0.25],[-0.125,0.125,0.125]], 196 | [[-0.25,0.0,0.25],[0.25,0.0,-0.25]], 197 | [[0.5,0.0,0.0],[-0.25,0.25,-0.25],[0.125,-0.125,0.125]], 198 | [[-0.25,0.0,-0.25],[0.375,-0.375,-0.375],[0.0,0.25,-0.25],[-0.125,0.125,0.125]], 199 | [[-0.25,0.25,-0.25],[-0.25,0.25,-0.25],[-0.125,0.125,-0.125],[-0.125,0.125,-0.125]], 200 | [[-0.0,0.5,0.0],[-0.25,0.25,-0.25],[0.125,-0.125,0.125]], 201 | [[0.5,0.0,0.0],[-0.25,-0.25,0.25],[-0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 202 | [[-0.125,-0.125,0.125],[-0.25,0.0,0.25],[-0.25,0.0,0.25]], 203 | [[0.125,0.125,0.125],[-0.125,-0.125,0.125],[-0.125,0.125,0.125]], 204 | [[-0.125,-0.125,0.125],[-0.125,0.125,0.125]], 205 | [[0.375,-0.375,0.375],[0.0,-0.25,-0.25],[-0.125,0.125,-0.125],[0.25,0.25,0.0]], 206 | [[0.0,-0.25,0.25],[0.0,-0.25,0.25],[-0.125,-0.125,0.125]], 207 | [[0.0,0.0,0.5],[0.25,-0.25,0.25],[0.125,-0.125,0.125]], 208 | [[0.0,-0.25,0.25],[0.0,-0.25,0.25]], 209 | [[-0.125,-0.125,0.125],[-0.25,-0.25,0.0],[0.25,0.25,-0.0]], 210 | [[-0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 211 | [[0.125,0.125,0.125],[-0.125,-0.125,0.125]], 212 | [[-0.125,-0.125,0.125]], 213 | [[-0.25,-0.25,0.0],[-0.25,-0.25,0.0]], 214 | [[0.125,0.125,0.125],[-0.25,-0.25,0.0],[-0.25,-0.25,0.0]], 215 | [[-0.25,-0.25,0.0],[-0.25,-0.25,0.0],[-0.125,-0.125,0.125]], 216 | [[-0.25,-0.25,0.0],[-0.25,-0.25,0.0],[-0.25,-0.25,0.0],[0.25,0.25,-0.0]], 217 | [[0.0,0.5,0.0],[0.25,0.25,-0.25],[-0.125,-0.125,0.125]], 218 | [[-0.375,0.375,-0.375],[-0.25,-0.25,0.0],[-0.125,0.125,-0.125],[-0.25,0.0,0.25]], 219 | [[0.0,0.5,0.0],[0.25,0.25,-0.25],[-0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 220 | [[-0.125,0.125,0.125],[0.25,-0.25,0.0],[-0.25,0.25,0.0]], 221 | [[0.0,-0.5,0.0],[-0.25,-0.25,-0.25],[-0.125,-0.125,-0.125]], 222 | [[0.125,0.125,0.125],[0.0,-0.5,0.0],[-0.25,-0.25,-0.25],[-0.125,-0.125,-0.125]], 223 | [[-0.375,-0.375,-0.375],[-0.25,0.0,0.25],[-0.125,-0.125,-0.125],[-0.25,0.25,0.0]], 224 | [[0.25,-0.25,0.0],[-0.25,0.25,0.0],[0.125,-0.125,0.125]], 225 | [[0.0,0.5,0.0],[0.0,-0.5,0.0]], 226 | [[0.0,0.5,0.0],[0.125,-0.125,0.125],[-0.25,0.25,-0.25]], 227 | [[0.0,0.5,0.0],[-0.25,0.25,0.25],[0.125,-0.125,-0.125]], 228 | [[0.25,-0.25,0.0],[-0.25,0.25,0.0]], 229 | [[-0.5,0.0,0.0],[-0.25,-0.25,0.25],[-0.125,-0.125,0.125]], 230 | [[0.0,0.25,-0.25],[0.375,-0.375,-0.375],[-0.125,0.125,0.125],[0.25,0.25,0.0]], 231 | [[0.5,0.0,0.0],[0.25,-0.25,0.25],[-0.125,0.125,-0.125],[0.125,-0.125,0.125]], 232 | [[0.125,-0.125,0.125],[0.25,-0.25,0.0],[0.25,-0.25,0.0]], 233 | [[0.25,0.25,-0.25],[0.25,0.25,-0.25],[0.125,0.125,-0.125],[-0.125,-0.125,0.125]], 234 | [[-0.0,0.0,0.5],[-0.25,-0.25,0.25],[-0.125,-0.125,0.125]], 235 | [[0.125,0.125,0.125],[0.125,-0.125,0.125],[-0.125,0.125,0.125]], 236 | [[-0.125,0.125,0.125],[0.125,-0.125,0.125]], 237 | [[-0.375,-0.375,0.375],[-0.0,0.25,0.25],[0.125,0.125,-0.125],[-0.25,-0.0,-0.25]], 238 | [[0.0,-0.25,0.25],[0.0,0.25,-0.25],[0.125,-0.125,0.125]], 239 | [[0.125,-0.125,0.125],[-0.25,-0.0,-0.25],[0.25,0.0,0.25]], 240 | [[0.125,-0.125,0.125],[0.125,-0.125,0.125]], 241 | [[0.0,-0.5,0.0],[0.125,0.125,-0.125],[0.25,0.25,-0.25]], 242 | [[0.0,-0.25,0.25],[0.0,0.25,-0.25]], 243 | [[0.125,0.125,0.125],[0.125,-0.125,0.125]], 244 | [[0.125,-0.125,0.125]], 245 | [[-0.5,0.0,0.0],[-0.125,-0.125,-0.125],[-0.25,-0.25,-0.25]], 246 | [[-0.5,0.0,0.0],[-0.125,-0.125,-0.125],[-0.25,-0.25,-0.25],[0.125,0.125,0.125]], 247 | [[0.375,0.375,0.375],[0.0,0.25,-0.25],[-0.125,-0.125,-0.125],[-0.25,0.25,0.0]], 248 | [[0.125,-0.125,-0.125],[0.25,-0.25,0.0],[0.25,-0.25,0.0]], 249 | [[0.125,0.125,0.125],[0.375,0.375,0.375],[0.0,-0.25,0.25],[-0.25,0.0,0.25]], 250 | [[-0.25,0.0,0.25],[-0.25,0.0,0.25],[0.125,-0.125,-0.125]], 251 | [[0.0,-0.25,-0.25],[0.0,0.25,0.25],[-0.125,0.125,0.125]], 252 | [[-0.125,0.125,0.125],[0.125,-0.125,-0.125]], 253 | [[-0.125,-0.125,-0.125],[-0.25,-0.25,-0.25],[0.25,0.25,0.25],[0.125,0.125,0.125]], 254 | [[-0.125,-0.125,0.125],[0.125,-0.125,0.125],[0.125,-0.125,-0.125]], 255 | [[0.0,0.0,-0.5],[0.25,0.25,0.25],[-0.125,-0.125,-0.125]], 256 | [[0.125,-0.125,0.125],[0.125,-0.125,-0.125]], 257 | [[0.0,-0.5,0.0],[0.25,0.25,0.25],[0.125,0.125,0.125]], 258 | [[-0.125,-0.125,0.125],[0.125,-0.125,-0.125]], 259 | [[0.0,-0.25,-0.25],[0.0,0.25,0.25]], 260 | [[0.125,-0.125,-0.125]], 261 | [[0.5,0.0,0.0],[0.5,0.0,0.0]], 262 | [[-0.5,0.0,0.0],[-0.25,0.25,0.25],[-0.125,0.125,0.125]], 263 | [[0.5,0.0,0.0],[0.25,-0.25,0.25],[-0.125,0.125,-0.125]], 264 | [[0.25,-0.25,0.0],[0.25,-0.25,0.0]], 265 | [[0.5,0.0,0.0],[-0.25,-0.25,0.25],[-0.125,-0.125,0.125]], 266 | [[-0.25,0.0,0.25],[-0.25,0.0,0.25]], 267 | [[0.125,0.125,0.125],[-0.125,0.125,0.125]], 268 | [[-0.125,0.125,0.125]], 269 | [[0.5,0.0,-0.0],[0.25,0.25,0.25],[0.125,0.125,0.125]], 270 | [[0.125,-0.125,0.125],[-0.125,-0.125,0.125]], 271 | [[-0.25,-0.0,-0.25],[0.25,0.0,0.25]], 272 | [[0.125,-0.125,0.125]], 273 | [[-0.25,-0.25,0.0],[0.25,0.25,-0.0]], 274 | [[-0.125,-0.125,0.125]], 275 | [[0.125,0.125,0.125]], 276 | [[0,0,0]]] 277 | 278 | 279 | def compute_surface_distances(mask_gt, mask_pred, spacing_mm): 280 | """Compute closest distances from all surface points to the other surface. 281 | 282 | Finds all surface elements "surfels" in the ground truth mask `mask_gt` and 283 | the predicted mask `mask_pred`, computes their area in mm^2 and the distance 284 | to the closest point on the other surface. It returns two sorted lists of 285 | distances together with the corresponding surfel areas. If one of the masks 286 | is empty, the corresponding lists are empty and all distances in the other 287 | list are `inf` 288 | 289 | Args: 290 | mask_gt: 3-dim Numpy array of type bool. The ground truth mask. 291 | mask_pred: 3-dim Numpy array of type bool. The predicted mask. 292 | spacing_mm: 3-element list-like structure. Voxel spacing in x0, x1 and x2 293 | direction 294 | 295 | Returns: 296 | A dict with 297 | "distances_gt_to_pred": 1-dim numpy array of type float. The distances in mm 298 | from all ground truth surface elements to the predicted surface, 299 | sorted from smallest to largest 300 | "distances_pred_to_gt": 1-dim numpy array of type float. The distances in mm 301 | from all predicted surface elements to the ground truth surface, 302 | sorted from smallest to largest 303 | "surfel_areas_gt": 1-dim numpy array of type float. The area in mm^2 of 304 | the ground truth surface elements in the same order as 305 | distances_gt_to_pred 306 | "surfel_areas_pred": 1-dim numpy array of type float. The area in mm^2 of 307 | the predicted surface elements in the same order as 308 | distances_pred_to_gt 309 | 310 | """ 311 | 312 | # compute the area for all 256 possible surface elements 313 | # (given a 2x2x2 neighbourhood) according to the spacing_mm 314 | neighbour_code_to_surface_area = np.zeros([256]) 315 | for code in range(256): 316 | normals = np.array(neighbour_code_to_normals[code]) 317 | sum_area = 0 318 | for normal_idx in range(normals.shape[0]): 319 | # normal vector 320 | n = np.zeros([3]) 321 | n[0] = normals[normal_idx,0] * spacing_mm[1] * spacing_mm[2] 322 | n[1] = normals[normal_idx,1] * spacing_mm[0] * spacing_mm[2] 323 | n[2] = normals[normal_idx,2] * spacing_mm[0] * spacing_mm[1] 324 | area = np.linalg.norm(n) 325 | sum_area += area 326 | neighbour_code_to_surface_area[code] = sum_area 327 | 328 | # compute the bounding box of the masks to trim 329 | # the volume to the smallest possible processing subvolume 330 | mask_all = mask_gt | mask_pred 331 | bbox_min = np.zeros(3, np.int64) 332 | bbox_max = np.zeros(3, np.int64) 333 | 334 | # max projection to the x0-axis 335 | proj_0 = np.max(np.max(mask_all, axis=2), axis=1) 336 | idx_nonzero_0 = np.nonzero(proj_0)[0] 337 | if len(idx_nonzero_0) == 0: 338 | return {"distances_gt_to_pred": np.array([]), 339 | "distances_pred_to_gt": np.array([]), 340 | "surfel_areas_gt": np.array([]), 341 | "surfel_areas_pred": np.array([])} 342 | 343 | bbox_min[0] = np.min(idx_nonzero_0) 344 | bbox_max[0] = np.max(idx_nonzero_0) 345 | 346 | # max projection to the x1-axis 347 | proj_1 = np.max(np.max(mask_all, axis=2), axis=0) 348 | idx_nonzero_1 = np.nonzero(proj_1)[0] 349 | bbox_min[1] = np.min(idx_nonzero_1) 350 | bbox_max[1] = np.max(idx_nonzero_1) 351 | 352 | # max projection to the x2-axis 353 | proj_2 = np.max(np.max(mask_all, axis=1), axis=0) 354 | idx_nonzero_2 = np.nonzero(proj_2)[0] 355 | bbox_min[2] = np.min(idx_nonzero_2) 356 | bbox_max[2] = np.max(idx_nonzero_2) 357 | 358 | # print("bounding box min = {}".format(bbox_min)) 359 | # print("bounding box max = {}".format(bbox_max)) 360 | 361 | # crop the processing subvolume. 362 | # we need to zeropad the cropped region with 1 voxel at the lower, 363 | # the right and the back side. This is required to obtain the "full" 364 | # convolution result with the 2x2x2 kernel 365 | cropmask_gt = np.zeros((bbox_max - bbox_min)+2, np.uint8) 366 | cropmask_pred = np.zeros((bbox_max - bbox_min)+2, np.uint8) 367 | 368 | cropmask_gt[0:-1, 0:-1, 0:-1] = mask_gt[bbox_min[0]:bbox_max[0]+1, 369 | bbox_min[1]:bbox_max[1]+1, 370 | bbox_min[2]:bbox_max[2]+1] 371 | 372 | cropmask_pred[0:-1, 0:-1, 0:-1] = mask_pred[bbox_min[0]:bbox_max[0]+1, 373 | bbox_min[1]:bbox_max[1]+1, 374 | bbox_min[2]:bbox_max[2]+1] 375 | 376 | # compute the neighbour code (local binary pattern) for each voxel 377 | # the resultsing arrays are spacially shifted by minus half a voxel in each axis. 378 | # i.e. the points are located at the corners of the original voxels 379 | kernel = np.array([[[128,64], 380 | [32,16]], 381 | [[8,4], 382 | [2,1]]]) 383 | neighbour_code_map_gt = scipy.ndimage.filters.correlate(cropmask_gt.astype(np.uint8), kernel, mode="constant", cval=0) 384 | neighbour_code_map_pred = scipy.ndimage.filters.correlate(cropmask_pred.astype(np.uint8), kernel, mode="constant", cval=0) 385 | 386 | # create masks with the surface voxels 387 | borders_gt = ((neighbour_code_map_gt != 0) & (neighbour_code_map_gt != 255)) 388 | borders_pred = ((neighbour_code_map_pred != 0) & (neighbour_code_map_pred != 255)) 389 | 390 | # compute the distance transform (closest distance of each voxel to the surface voxels) 391 | if borders_gt.any(): 392 | distmap_gt = scipy.ndimage.morphology.distance_transform_edt(~borders_gt, sampling=spacing_mm) 393 | else: 394 | distmap_gt = np.Inf * np.ones(borders_gt.shape) 395 | 396 | if borders_pred.any(): 397 | distmap_pred = scipy.ndimage.morphology.distance_transform_edt(~borders_pred, sampling=spacing_mm) 398 | else: 399 | distmap_pred = np.Inf * np.ones(borders_pred.shape) 400 | 401 | # compute the area of each surface element 402 | surface_area_map_gt = neighbour_code_to_surface_area[neighbour_code_map_gt] 403 | surface_area_map_pred = neighbour_code_to_surface_area[neighbour_code_map_pred] 404 | 405 | # create a list of all surface elements with distance and area 406 | distances_gt_to_pred = distmap_pred[borders_gt] 407 | distances_pred_to_gt = distmap_gt[borders_pred] 408 | surfel_areas_gt = surface_area_map_gt[borders_gt] 409 | surfel_areas_pred = surface_area_map_pred[borders_pred] 410 | 411 | # sort them by distance 412 | if distances_gt_to_pred.shape != (0,): 413 | sorted_surfels_gt = np.array(sorted(zip(distances_gt_to_pred, surfel_areas_gt))) 414 | distances_gt_to_pred = sorted_surfels_gt[:,0] 415 | surfel_areas_gt = sorted_surfels_gt[:,1] 416 | 417 | if distances_pred_to_gt.shape != (0,): 418 | sorted_surfels_pred = np.array(sorted(zip(distances_pred_to_gt, surfel_areas_pred))) 419 | distances_pred_to_gt = sorted_surfels_pred[:,0] 420 | surfel_areas_pred = sorted_surfels_pred[:,1] 421 | 422 | 423 | return {"distances_gt_to_pred": distances_gt_to_pred, 424 | "distances_pred_to_gt": distances_pred_to_gt, 425 | "surfel_areas_gt": surfel_areas_gt, 426 | "surfel_areas_pred": surfel_areas_pred} 427 | 428 | 429 | def compute_average_surface_distance(surface_distances): 430 | distances_gt_to_pred = surface_distances["distances_gt_to_pred"] 431 | distances_pred_to_gt = surface_distances["distances_pred_to_gt"] 432 | surfel_areas_gt = surface_distances["surfel_areas_gt"] 433 | surfel_areas_pred = surface_distances["surfel_areas_pred"] 434 | average_distance_gt_to_pred = np.sum( distances_gt_to_pred * surfel_areas_gt) / np.sum(surfel_areas_gt) 435 | average_distance_pred_to_gt = np.sum( distances_pred_to_gt * surfel_areas_pred) / np.sum(surfel_areas_pred) 436 | return (average_distance_gt_to_pred, average_distance_pred_to_gt) 437 | 438 | def compute_robust_hausdorff(surface_distances, percent): 439 | distances_gt_to_pred = surface_distances["distances_gt_to_pred"] 440 | distances_pred_to_gt = surface_distances["distances_pred_to_gt"] 441 | surfel_areas_gt = surface_distances["surfel_areas_gt"] 442 | surfel_areas_pred = surface_distances["surfel_areas_pred"] 443 | if len(distances_gt_to_pred) > 0: 444 | surfel_areas_cum_gt = np.cumsum(surfel_areas_gt) / np.sum(surfel_areas_gt) 445 | idx = np.searchsorted(surfel_areas_cum_gt, percent/100.0) 446 | perc_distance_gt_to_pred = distances_gt_to_pred[min(idx, len(distances_gt_to_pred)-1)] 447 | else: 448 | perc_distance_gt_to_pred = np.Inf 449 | 450 | if len(distances_pred_to_gt) > 0: 451 | surfel_areas_cum_pred = np.cumsum(surfel_areas_pred) / np.sum(surfel_areas_pred) 452 | idx = np.searchsorted(surfel_areas_cum_pred, percent/100.0) 453 | perc_distance_pred_to_gt = distances_pred_to_gt[min(idx, len(distances_pred_to_gt)-1)] 454 | else: 455 | perc_distance_pred_to_gt = np.Inf 456 | 457 | return max( perc_distance_gt_to_pred, perc_distance_pred_to_gt) 458 | 459 | def compute_surface_overlap_at_tolerance(surface_distances, tolerance_mm): 460 | distances_gt_to_pred = surface_distances["distances_gt_to_pred"] 461 | distances_pred_to_gt = surface_distances["distances_pred_to_gt"] 462 | surfel_areas_gt = surface_distances["surfel_areas_gt"] 463 | surfel_areas_pred = surface_distances["surfel_areas_pred"] 464 | rel_overlap_gt = np.sum(surfel_areas_gt[distances_gt_to_pred <= tolerance_mm]) / np.sum(surfel_areas_gt) 465 | rel_overlap_pred = np.sum(surfel_areas_pred[distances_pred_to_gt <= tolerance_mm]) / np.sum(surfel_areas_pred) 466 | return (rel_overlap_gt, rel_overlap_pred) 467 | 468 | def compute_surface_dice_at_tolerance(surface_distances, tolerance_mm): 469 | distances_gt_to_pred = surface_distances["distances_gt_to_pred"] 470 | distances_pred_to_gt = surface_distances["distances_pred_to_gt"] 471 | surfel_areas_gt = surface_distances["surfel_areas_gt"] 472 | surfel_areas_pred = surface_distances["surfel_areas_pred"] 473 | overlap_gt = np.sum(surfel_areas_gt[distances_gt_to_pred <= tolerance_mm]) 474 | overlap_pred = np.sum(surfel_areas_pred[distances_pred_to_gt <= tolerance_mm]) 475 | surface_dice = (overlap_gt + overlap_pred) / ( 476 | np.sum(surfel_areas_gt) + np.sum(surfel_areas_pred)) 477 | return surface_dice 478 | 479 | 480 | def compute_dice_coefficient(mask_gt, mask_pred): 481 | """Compute soerensen-dice coefficient. 482 | 483 | compute the soerensen-dice coefficient between the ground truth mask `mask_gt` 484 | and the predicted mask `mask_pred`. 485 | 486 | Args: 487 | mask_gt: 3-dim Numpy array of type bool. The ground truth mask. 488 | mask_pred: 3-dim Numpy array of type bool. The predicted mask. 489 | 490 | Returns: 491 | the dice coeffcient as float. If both masks are empty, the result is NaN 492 | """ 493 | volume_sum = mask_gt.sum() + mask_pred.sum() 494 | if volume_sum == 0: 495 | return np.NaN 496 | volume_intersect = (mask_gt & mask_pred).sum() 497 | return 2*volume_intersect / volume_sum 498 | 499 | 500 | #%% Some Simple Tests 501 | # single pixels, 2mm away 502 | mask_gt = np.zeros((128,128,128), np.uint8) 503 | mask_pred = np.zeros((128,128,128), np.uint8) 504 | mask_gt[50,60,70] = 1 505 | mask_pred[50,60,72] = 1 506 | surface_distances = compute_surface_distances(mask_gt, mask_pred, spacing_mm=(3,2,1)) 507 | print("surface dice at 1mm: {}".format(compute_surface_dice_at_tolerance(surface_distances, 1))) 508 | print("volumetric dice: {}".format(compute_dice_coefficient(mask_gt, mask_pred))) 509 | 510 | 511 | #%% two cubes. cube 1 is 100x100x100 mm^3 and cube 2 is 102x100x100 mm^3 512 | mask_gt = np.zeros((100,100,100), np.uint8) 513 | mask_pred = np.zeros((100,100,100), np.uint8) 514 | spacing_mm=(2,1,1) 515 | mask_gt[0:50, :, :] = 1 516 | mask_pred[0:51, :, :] = 1 517 | surface_distances = compute_surface_distances(mask_gt, mask_pred, spacing_mm) 518 | print("surface dice at 1mm: {}".format(compute_surface_dice_at_tolerance(surface_distances, 1))) 519 | print("volumetric dice: {}".format(compute_dice_coefficient(mask_gt, mask_pred))) 520 | print("") 521 | print("expected average_distance_gt_to_pred = 1./6 * 2mm = {}mm".format(1./6 * 2)) 522 | print("expected volumetric dice: {}".format(2.*100*100*100 / (100*100*100 + 102*100*100) )) 523 | 524 | 525 | 526 | #%% test empty mask in prediction 527 | mask_gt = np.zeros((128,128,128), np.uint8) 528 | mask_pred = np.zeros((128,128,128), np.uint8) 529 | mask_gt[50,60,70] = 1 530 | #mask_pred[50,60,72] = 1 531 | surface_distances = compute_surface_distances(mask_gt, mask_pred, spacing_mm=(3,2,1)) 532 | print("average surface distance: {} mm".format(compute_average_surface_distance(surface_distances))) 533 | print("hausdorff (100%): {} mm".format(compute_robust_hausdorff(surface_distances, 100))) 534 | print("hausdorff (95%): {} mm".format(compute_robust_hausdorff(surface_distances, 95))) 535 | print("surface overlap at 1mm: {}".format(compute_surface_overlap_at_tolerance(surface_distances, 1))) 536 | print("surface dice at 1mm: {}".format(compute_surface_dice_at_tolerance(surface_distances, 1))) 537 | print("volumetric dice: {}".format(compute_dice_coefficient(mask_gt, mask_pred))) 538 | 539 | 540 | #%% test empty mask in ground truth 541 | mask_gt = np.zeros((128,128,128), np.uint8) 542 | mask_pred = np.zeros((128,128,128), np.uint8) 543 | #mask_gt[50,60,70] = 1 544 | mask_pred[50,60,72] = 1 545 | surface_distances = compute_surface_distances(mask_gt, mask_pred, spacing_mm=(3,2,1)) 546 | print("average surface distance: {} mm".format(compute_average_surface_distance(surface_distances))) 547 | print("hausdorff (100%): {} mm".format(compute_robust_hausdorff(surface_distances, 100))) 548 | print("hausdorff (95%): {} mm".format(compute_robust_hausdorff(surface_distances, 95))) 549 | print("surface overlap at 1mm: {}".format(compute_surface_overlap_at_tolerance(surface_distances, 1))) 550 | print("surface dice at 1mm: {}".format(compute_surface_dice_at_tolerance(surface_distances, 1))) 551 | print("volumetric dice: {}".format(compute_dice_coefficient(mask_gt, mask_pred))) 552 | 553 | 554 | #%% test both masks empty 555 | mask_gt = np.zeros((128,128,128), np.uint8) 556 | mask_pred = np.zeros((128,128,128), np.uint8) 557 | #mask_gt[50,60,70] = 1 558 | #mask_pred[50,60,72] = 1 559 | surface_distances = compute_surface_distances(mask_gt, mask_pred, spacing_mm=(3,2,1)) 560 | print("average surface distance: {} mm".format(compute_average_surface_distance(surface_distances))) 561 | print("hausdorff (100%): {} mm".format(compute_robust_hausdorff(surface_distances, 100))) 562 | print("hausdorff (95%): {} mm".format(compute_robust_hausdorff(surface_distances, 95))) 563 | print("surface overlap at 1mm: {}".format(compute_surface_overlap_at_tolerance(surface_distances, 1))) 564 | print("surface dice at 1mm: {}".format(compute_surface_dice_at_tolerance(surface_distances, 1))) 565 | print("volumetric dice: {}".format(compute_dice_coefficient(mask_gt, mask_pred))) 566 | 567 | 568 | 569 | 570 | 571 | 572 | 573 | 574 | 575 | 576 | 577 | --------------------------------------------------------------------------------