├── groundtruths ├── 00006.txt ├── 00007.txt ├── 00001.txt ├── 00002.txt ├── 00004.txt ├── 00005.txt └── 00003.txt ├── detections ├── 00007.txt ├── 00001.txt ├── 00002.txt ├── 00006.txt ├── 00004.txt ├── 00005.txt └── 00003.txt ├── samples ├── sample_2 │ ├── groundtruths │ │ ├── 00001.txt │ │ ├── 00002.txt │ │ ├── 00004.txt │ │ ├── 00005.txt │ │ ├── 00006.txt │ │ ├── 00007.txt │ │ └── 00003.txt │ ├── detections │ │ ├── 00007.txt │ │ ├── 00001.txt │ │ ├── 00002.txt │ │ ├── 00006.txt │ │ ├── 00004.txt │ │ ├── 00005.txt │ │ └── 00003.txt │ ├── _init_paths.py │ ├── sample_2.py │ └── README.md └── sample_1 │ ├── images │ ├── 000001.jpg │ ├── 000002.jpg │ ├── 000003.jpg │ ├── detections │ │ ├── 000001.png │ │ ├── 000002.png │ │ └── 000003.png │ └── groundtruths │ │ ├── 000001.jpg │ │ ├── 000002.jpg │ │ └── 000003.jpg │ ├── _init_paths.py │ ├── sample_1.py │ └── README.md ├── groundtruths_rel ├── 00004.txt ├── 00006.txt ├── 00002.txt ├── 00001.txt ├── 00005.txt ├── 00007.txt └── 00003.txt ├── detections_rel ├── 00007.txt ├── 00006.txt ├── 00001.txt ├── 00002.txt ├── 00005.txt ├── 00004.txt └── 00003.txt ├── aux_images ├── iou.png ├── table_1_v2.png ├── table_2_v2.png ├── samples_1_v2.png ├── 11-pointInterpolation.png ├── interpolated_precision_v2.png ├── interpolated_precision-AUC_v2.png └── precision_recall_example_1_v2.png ├── results ├── person.png ├── person_11-pointInterpolation.png └── results.txt ├── lib ├── __init__.py ├── BoundingBoxes.py ├── utils.py ├── BoundingBox.py └── Evaluator.py ├── LICENSE ├── _init_paths.py ├── .gitignore ├── pascalvoc.py └── README.md /groundtruths/00006.txt: -------------------------------------------------------------------------------- 1 | person 36 89 52 76 2 | person 62 58 44 67 3 | -------------------------------------------------------------------------------- /groundtruths/00007.txt: -------------------------------------------------------------------------------- 1 | person 28 31 55 63 2 | person 58 67 50 58 3 | -------------------------------------------------------------------------------- /groundtruths/00001.txt: -------------------------------------------------------------------------------- 1 | person 25 16 38 56 2 | person 129 123 41 62 3 | -------------------------------------------------------------------------------- /groundtruths/00002.txt: -------------------------------------------------------------------------------- 1 | person 123 11 43 55 2 | person 38 132 59 45 3 | -------------------------------------------------------------------------------- /groundtruths/00004.txt: -------------------------------------------------------------------------------- 1 | person 53 42 40 52 2 | person 154 43 31 34 3 | -------------------------------------------------------------------------------- /groundtruths/00005.txt: -------------------------------------------------------------------------------- 1 | person 59 31 44 51 2 | person 48 128 34 52 3 | -------------------------------------------------------------------------------- /detections/00007.txt: -------------------------------------------------------------------------------- 1 | person .48 16 20 101 88 2 | person .95 33 116 37 49 3 | -------------------------------------------------------------------------------- /samples/sample_2/groundtruths/00001.txt: -------------------------------------------------------------------------------- 1 | object 25 16 38 56 2 | object 129 123 41 62 3 | -------------------------------------------------------------------------------- /samples/sample_2/groundtruths/00002.txt: -------------------------------------------------------------------------------- 1 | object 123 11 43 55 2 | object 38 132 59 45 3 | -------------------------------------------------------------------------------- /samples/sample_2/groundtruths/00004.txt: -------------------------------------------------------------------------------- 1 | object 53 42 40 52 2 | object 154 43 31 34 3 | -------------------------------------------------------------------------------- /samples/sample_2/groundtruths/00005.txt: -------------------------------------------------------------------------------- 1 | object 59 31 44 51 2 | object 48 128 34 52 3 | -------------------------------------------------------------------------------- /samples/sample_2/groundtruths/00006.txt: -------------------------------------------------------------------------------- 1 | object 36 89 52 76 2 | object 62 58 44 67 3 | -------------------------------------------------------------------------------- /samples/sample_2/groundtruths/00007.txt: -------------------------------------------------------------------------------- 1 | object 28 31 55 63 2 | object 58 67 50 58 3 | -------------------------------------------------------------------------------- /groundtruths/00003.txt: -------------------------------------------------------------------------------- 1 | person 16 14 35 48 2 | person 123 30 49 44 3 | person 99 139 47 47 4 | -------------------------------------------------------------------------------- /groundtruths_rel/00004.txt: -------------------------------------------------------------------------------- 1 | person 0.365 0.34 0.2 0.26 2 | person 0.8475 0.3 0.155 0.17 3 | -------------------------------------------------------------------------------- /groundtruths_rel/00006.txt: -------------------------------------------------------------------------------- 1 | person 0.31 0.635 0.26 0.38 2 | person 0.42 0.4575 0.22 0.335 3 | -------------------------------------------------------------------------------- /samples/sample_2/detections/00007.txt: -------------------------------------------------------------------------------- 1 | object .48 16 20 101 88 2 | object .95 33 116 37 49 3 | -------------------------------------------------------------------------------- /detections/00001.txt: -------------------------------------------------------------------------------- 1 | person .88 5 67 31 48 2 | person .70 119 111 40 67 3 | person .80 124 9 49 67 4 | -------------------------------------------------------------------------------- /detections/00002.txt: -------------------------------------------------------------------------------- 1 | person .71 64 111 64 58 2 | person .54 26 140 60 47 3 | person .74 19 18 43 35 4 | -------------------------------------------------------------------------------- /detections/00006.txt: -------------------------------------------------------------------------------- 1 | person .45 43 48 74 38 2 | person .84 17 155 29 35 3 | person .43 95 110 25 42 4 | -------------------------------------------------------------------------------- /detections_rel/00007.txt: -------------------------------------------------------------------------------- 1 | person .48 0.3325 0.32 0.505 0.44 2 | person .95 0.2575 0.7025 0.185 0.245 3 | -------------------------------------------------------------------------------- /groundtruths_rel/00002.txt: -------------------------------------------------------------------------------- 1 | person 0.7225 0.1925 0.215 0.275 2 | person 0.3375 0.7725 0.295 0.225 3 | -------------------------------------------------------------------------------- /aux_images/iou.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/aux_images/iou.png -------------------------------------------------------------------------------- /groundtruths_rel/00001.txt: -------------------------------------------------------------------------------- 1 | person 0.22 0.22 0.19 0.28 2 | person 0.7475 0.77 0.20500000000000002 0.31 3 | -------------------------------------------------------------------------------- /groundtruths_rel/00005.txt: -------------------------------------------------------------------------------- 1 | person 0.405 0.28250000000000003 0.22 0.255 2 | person 0.325 0.77 0.17 0.26 3 | -------------------------------------------------------------------------------- /results/person.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/results/person.png -------------------------------------------------------------------------------- /groundtruths_rel/00007.txt: -------------------------------------------------------------------------------- 1 | person 0.2775 0.3125 0.275 0.315 2 | person 0.41500000000000004 0.48 0.25 0.29 3 | -------------------------------------------------------------------------------- /samples/sample_2/groundtruths/00003.txt: -------------------------------------------------------------------------------- 1 | object 16 14 35 48 2 | object 123 30 49 44 3 | object 99 139 47 47 4 | -------------------------------------------------------------------------------- /aux_images/table_1_v2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/aux_images/table_1_v2.png -------------------------------------------------------------------------------- /aux_images/table_2_v2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/aux_images/table_2_v2.png -------------------------------------------------------------------------------- /aux_images/samples_1_v2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/aux_images/samples_1_v2.png -------------------------------------------------------------------------------- /samples/sample_2/detections/00001.txt: -------------------------------------------------------------------------------- 1 | object .88 5 67 31 48 2 | object .70 119 111 40 67 3 | object .80 124 9 49 67 4 | -------------------------------------------------------------------------------- /samples/sample_2/detections/00002.txt: -------------------------------------------------------------------------------- 1 | object .71 64 111 64 58 2 | object .54 26 140 60 47 3 | object .74 19 18 43 35 4 | -------------------------------------------------------------------------------- /samples/sample_2/detections/00006.txt: -------------------------------------------------------------------------------- 1 | object .45 43 48 74 38 2 | object .84 17 155 29 35 3 | object .43 95 110 25 42 4 | -------------------------------------------------------------------------------- /detections/00004.txt: -------------------------------------------------------------------------------- 1 | person .35 83 28 28 26 2 | person .78 28 68 42 67 3 | person .45 87 89 25 39 4 | person .14 10 155 60 26 5 | -------------------------------------------------------------------------------- /detections/00005.txt: -------------------------------------------------------------------------------- 1 | person .62 50 38 28 46 2 | person .44 95 11 53 28 3 | person .95 29 131 72 29 4 | person .23 29 163 72 29 5 | -------------------------------------------------------------------------------- /samples/sample_1/images/000001.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/samples/sample_1/images/000001.jpg -------------------------------------------------------------------------------- /samples/sample_1/images/000002.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/samples/sample_1/images/000002.jpg -------------------------------------------------------------------------------- /samples/sample_1/images/000003.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/samples/sample_1/images/000003.jpg -------------------------------------------------------------------------------- /aux_images/11-pointInterpolation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/aux_images/11-pointInterpolation.png -------------------------------------------------------------------------------- /aux_images/interpolated_precision_v2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/aux_images/interpolated_precision_v2.png -------------------------------------------------------------------------------- /results/person_11-pointInterpolation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/results/person_11-pointInterpolation.png -------------------------------------------------------------------------------- /samples/sample_2/detections/00004.txt: -------------------------------------------------------------------------------- 1 | object .35 83 28 28 26 2 | object .78 28 68 42 67 3 | object .45 87 89 25 39 4 | object .14 10 155 60 26 5 | -------------------------------------------------------------------------------- /samples/sample_2/detections/00005.txt: -------------------------------------------------------------------------------- 1 | object .62 50 38 28 46 2 | object .44 95 11 53 28 3 | object .95 29 131 72 29 4 | object .23 29 163 72 29 5 | -------------------------------------------------------------------------------- /detections_rel/00006.txt: -------------------------------------------------------------------------------- 1 | person .45 0.4 0.335 0.37 0.19 2 | person .84 0.1575 0.8625 0.145 0.17500000000000002 3 | person .43 0.5375 0.655 0.125 0.21 4 | -------------------------------------------------------------------------------- /aux_images/interpolated_precision-AUC_v2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/aux_images/interpolated_precision-AUC_v2.png -------------------------------------------------------------------------------- /aux_images/precision_recall_example_1_v2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/aux_images/precision_recall_example_1_v2.png -------------------------------------------------------------------------------- /samples/sample_1/images/detections/000001.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/samples/sample_1/images/detections/000001.png -------------------------------------------------------------------------------- /samples/sample_1/images/detections/000002.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/samples/sample_1/images/detections/000002.png -------------------------------------------------------------------------------- /samples/sample_1/images/detections/000003.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/samples/sample_1/images/detections/000003.png -------------------------------------------------------------------------------- /detections/00003.txt: -------------------------------------------------------------------------------- 1 | person .18 109 15 77 39 2 | person .67 86 63 46 45 3 | person .38 160 62 36 53 4 | person .91 105 131 47 47 5 | person .44 18 148 40 44 6 | -------------------------------------------------------------------------------- /samples/sample_1/images/groundtruths/000001.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/samples/sample_1/images/groundtruths/000001.jpg -------------------------------------------------------------------------------- /samples/sample_1/images/groundtruths/000002.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/samples/sample_1/images/groundtruths/000002.jpg -------------------------------------------------------------------------------- /samples/sample_1/images/groundtruths/000003.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Adamdad/Object-Detection-Metrics/HEAD/samples/sample_1/images/groundtruths/000003.jpg -------------------------------------------------------------------------------- /detections_rel/00001.txt: -------------------------------------------------------------------------------- 1 | person .88 0.10250000000000001 0.455 0.155 0.24 2 | person .70 0.6950000000000001 0.7225 0.2 0.335 3 | person .80 0.7425 0.2125 0.245 0.335 4 | -------------------------------------------------------------------------------- /groundtruths_rel/00003.txt: -------------------------------------------------------------------------------- 1 | person 0.1675 0.19 0.17500000000000002 0.24 2 | person 0.7375 0.26 0.245 0.22 3 | person 0.6125 0.8125 0.23500000000000001 0.23500000000000001 4 | -------------------------------------------------------------------------------- /samples/sample_2/detections/00003.txt: -------------------------------------------------------------------------------- 1 | object .18 109 15 77 39 2 | object .67 86 63 46 45 3 | object .38 160 62 36 53 4 | object .91 105 131 47 47 5 | object .44 18 148 40 44 6 | -------------------------------------------------------------------------------- /detections_rel/00002.txt: -------------------------------------------------------------------------------- 1 | person .71 0.48 0.7000000000000001 0.32 0.29 2 | person .54 0.28 0.8175 0.3 0.23500000000000001 3 | person .74 0.2025 0.1775 0.215 0.17500000000000002 4 | -------------------------------------------------------------------------------- /detections_rel/00005.txt: -------------------------------------------------------------------------------- 1 | person .62 0.32 0.305 0.14 0.23 2 | person .44 0.6075 0.125 0.265 0.14 3 | person .95 0.325 0.7275 0.36 0.145 4 | person .23 0.325 0.8875000000000001 0.36 0.145 5 | -------------------------------------------------------------------------------- /detections_rel/00004.txt: -------------------------------------------------------------------------------- 1 | person .35 0.485 0.20500000000000002 0.14 0.13 2 | person .78 0.245 0.5075000000000001 0.21 0.335 3 | person .45 0.4975 0.5425 0.125 0.195 4 | person .14 0.2 0.84 0.3 0.13 5 | -------------------------------------------------------------------------------- /detections_rel/00003.txt: -------------------------------------------------------------------------------- 1 | person .18 0.7375 0.17250000000000001 0.385 0.195 2 | person .67 0.545 0.4275 0.23 0.225 3 | person .38 0.89 0.4425 0.18 0.265 4 | person .91 0.6425 0.7725 0.23500000000000001 0.23500000000000001 5 | person .44 0.19 0.85 0.2 0.22 6 | -------------------------------------------------------------------------------- /lib/__init__.py: -------------------------------------------------------------------------------- 1 | ########################################################################################### 2 | # Developed by: Rafael Padilla # 3 | # SMT - Signal Multimedia and Telecommunications Lab # 4 | # COPPE - Universidade Federal do Rio de Janeiro # 5 | # Last modification: May 24th 2018 # 6 | ########################################################################################### 7 | -------------------------------------------------------------------------------- /results/results.txt: -------------------------------------------------------------------------------- 1 | Object Detection Metrics 2 | https://github.com/rafaelpadilla/Object-Detection-Metrics 3 | 4 | 5 | Average Precision (AP), Precision and Recall per class: 6 | 7 | Class: person 8 | AP: 24.57% 9 | Precision: ['1.00', '0.50', '0.67', '0.50', '0.40', '0.33', '0.29', '0.25', '0.22', '0.30', '0.27', '0.33', '0.38', '0.43', '0.40', '0.38', '0.35', '0.33', '0.32', '0.30', '0.29', '0.27', '0.30', '0.29'] 10 | Recall: ['0.07', '0.07', '0.13', '0.13', '0.13', '0.13', '0.13', '0.13', '0.13', '0.20', '0.20', '0.27', '0.33', '0.40', '0.40', '0.40', '0.40', '0.40', '0.40', '0.40', '0.40', '0.40', '0.47', '0.47'] 11 | 12 | 13 | mAP: 24.57% -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Rafael Padilla 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /_init_paths.py: -------------------------------------------------------------------------------- 1 | ########################################################################################### 2 | # # 3 | # Set up paths for the Object Detection Metrics # 4 | # # 5 | # Developed by: Rafael Padilla (rafael.padilla@smt.ufrj.br) # 6 | # SMT - Signal Multimedia and Telecommunications Lab # 7 | # COPPE - Universidade Federal do Rio de Janeiro # 8 | # Last modification: May 24th 2018 # 9 | ########################################################################################### 10 | 11 | import sys 12 | import os 13 | 14 | 15 | def add_path(path): 16 | if path not in sys.path: 17 | sys.path.insert(0, path) 18 | 19 | 20 | currentPath = os.path.dirname(os.path.realpath(__file__)) 21 | 22 | # Add lib to PYTHONPATH 23 | libPath = os.path.join(currentPath, 'lib') 24 | add_path(libPath) 25 | -------------------------------------------------------------------------------- /samples/sample_1/_init_paths.py: -------------------------------------------------------------------------------- 1 | ########################################################################################### 2 | # # 3 | # Set up paths for the Object Detection Metrics # 4 | # # 5 | # Developed by: Rafael Padilla (rafael.padilla@smt.ufrj.br) # 6 | # SMT - Signal Multimedia and Telecommunications Lab # 7 | # COPPE - Universidade Federal do Rio de Janeiro # 8 | # Last modification: May 24th 2018 # 9 | ########################################################################################### 10 | 11 | import sys 12 | import os 13 | 14 | 15 | def add_path(path): 16 | if path not in sys.path: 17 | sys.path.insert(0, path) 18 | 19 | 20 | currentPath = os.path.dirname(os.path.realpath(__file__)) 21 | 22 | # Add lib to PYTHONPATH 23 | libPath = os.path.join(currentPath, '..', '..', 'lib') 24 | add_path(libPath) 25 | -------------------------------------------------------------------------------- /samples/sample_2/_init_paths.py: -------------------------------------------------------------------------------- 1 | ########################################################################################### 2 | # # 3 | # Set up paths for the Object Detection Metrics # 4 | # # 5 | # Developed by: Rafael Padilla (rafael.padilla@smt.ufrj.br) # 6 | # SMT - Signal Multimedia and Telecommunications Lab # 7 | # COPPE - Universidade Federal do Rio de Janeiro # 8 | # Last modification: May 24th 2018 # 9 | ########################################################################################### 10 | 11 | import sys 12 | import os 13 | 14 | 15 | def add_path(path): 16 | if path not in sys.path: 17 | sys.path.insert(0, path) 18 | 19 | 20 | currentPath = os.path.dirname(os.path.realpath(__file__)) 21 | 22 | # Add lib to PYTHONPATH 23 | libPath = os.path.join(currentPath, '..', '..', 'lib') 24 | add_path(libPath) 25 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | ### Python ### 5 | # Byte-compiled / optimized / DLL files 6 | __pycache__/ 7 | *.py[cod] 8 | *$py.class 9 | 10 | # C extensions 11 | *.so 12 | 13 | # Distribution / packaging 14 | .Python 15 | build/ 16 | develop-eggs/ 17 | dist/ 18 | downloads/ 19 | eggs/ 20 | .eggs/ 21 | lib64/ 22 | parts/ 23 | sdist/ 24 | var/ 25 | wheels/ 26 | *.egg-info/ 27 | .installed.cfg 28 | *.egg 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | .pytest_cache/ 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | .hypothesis/ 51 | 52 | # Translations 53 | *.mo 54 | *.pot 55 | 56 | # Flask stuff: 57 | instance/ 58 | .webassets-cache 59 | 60 | # Scrapy stuff: 61 | .scrapy 62 | 63 | # Sphinx documentation 64 | docs/_build/ 65 | 66 | # PyBuilder 67 | target/ 68 | 69 | # Jupyter Notebook 70 | .ipynb_checkpoints 71 | 72 | # pyenv 73 | .python-version 74 | 75 | # celery beat schedule file 76 | celerybeat-schedule.* 77 | 78 | # SageMath parsed files 79 | *.sage.py 80 | 81 | # Environments 82 | .env 83 | .venv 84 | env/ 85 | venv/ 86 | ENV/ 87 | env.bak/ 88 | venv.bak/ 89 | 90 | # Spyder project settings 91 | .spyderproject 92 | .spyproject 93 | 94 | # Rope project settings 95 | .ropeproject 96 | 97 | # mkdocs documentation 98 | /site 99 | 100 | # mypy 101 | .mypy_cache/ 102 | 103 | ### VisualStudioCode ### 104 | .vscode/ 105 | .vscode/* 106 | !.vscode/settings.json 107 | !.vscode/tasks.json 108 | !.vscode/launch.json 109 | !.vscode/extensions.json 110 | .history 111 | 112 | # My stuff 113 | ToDo.txt 114 | test.py 115 | references/ 116 | aux_images/older_version/ 117 | -------------------------------------------------------------------------------- /lib/BoundingBoxes.py: -------------------------------------------------------------------------------- 1 | from BoundingBox import * 2 | from utils import * 3 | 4 | 5 | class BoundingBoxes: 6 | def __init__(self): 7 | self._boundingBoxes = [] 8 | 9 | def addBoundingBox(self, bb): 10 | self._boundingBoxes.append(bb) 11 | 12 | def removeBoundingBox(self, _boundingBox): 13 | for d in self._boundingBoxes: 14 | if BoundingBox.compare(d, _boundingBox): 15 | del self._boundingBoxes[d] 16 | return 17 | 18 | def removeAllBoundingBoxes(self): 19 | self._boundingBoxes = [] 20 | 21 | def getBoundingBoxes(self): 22 | return self._boundingBoxes 23 | 24 | def getBoundingBoxByClass(self, classId): 25 | boundingBoxes = [] 26 | for d in self._boundingBoxes: 27 | if d.getClassId() == classId: # get only specified bounding box type 28 | boundingBoxes.append(d) 29 | return boundingBoxes 30 | 31 | def getClasses(self): 32 | classes = [] 33 | for d in self._boundingBoxes: 34 | c = d.getClassId() 35 | if c not in classes: 36 | classes.append(c) 37 | return classes 38 | 39 | def getBoundingBoxesByType(self, bbType): 40 | # get only specified bb type 41 | return [d for d in self._boundingBoxes if d.getBBType() == bbType] 42 | 43 | def getBoundingBoxesByImageName(self, imageName): 44 | # get only specified bb type 45 | return [d for d in self._boundingBoxes if d.getImageName() == imageName] 46 | 47 | def count(self, bbType=None): 48 | if bbType is None: # Return all bounding boxes 49 | return len(self._boundingBoxes) 50 | count = 0 51 | for d in self._boundingBoxes: 52 | if d.getBBType() == bbType: # get only specified bb type 53 | count += 1 54 | return count 55 | 56 | def clone(self): 57 | newBoundingBoxes = BoundingBoxes() 58 | for d in self._boundingBoxes: 59 | det = BoundingBox.clone(d) 60 | newBoundingBoxes.addBoundingBox(det) 61 | return newBoundingBoxes 62 | 63 | def drawAllBoundingBoxes(self, image, imageName): 64 | bbxes = self.getBoundingBoxesByImageName(imageName) 65 | for bb in bbxes: 66 | if bb.getBBType() == BBType.GroundTruth: # if ground truth 67 | image = add_bb_into_image(image, bb, color=(0, 255, 0)) # green 68 | else: # if detection 69 | image = add_bb_into_image(image, bb, color=(255, 0, 0)) # red 70 | return image 71 | 72 | # def drawAllBoundingBoxes(self, image): 73 | # for gt in self.getBoundingBoxesByType(BBType.GroundTruth): 74 | # image = add_bb_into_image(image, gt ,color=(0,255,0)) 75 | # for det in self.getBoundingBoxesByType(BBType.Detected): 76 | # image = add_bb_into_image(image, det ,color=(255,0,0)) 77 | # return image 78 | -------------------------------------------------------------------------------- /lib/utils.py: -------------------------------------------------------------------------------- 1 | from enum import Enum 2 | 3 | import cv2 4 | 5 | 6 | class MethodAveragePrecision(Enum): 7 | """ 8 | Class representing if the coordinates are relative to the 9 | image size or are absolute values. 10 | 11 | Developed by: Rafael Padilla 12 | Last modification: Apr 28 2018 13 | """ 14 | EveryPointInterpolation = 1 15 | ElevenPointInterpolation = 2 16 | 17 | 18 | class CoordinatesType(Enum): 19 | """ 20 | Class representing if the coordinates are relative to the 21 | image size or are absolute values. 22 | 23 | Developed by: Rafael Padilla 24 | Last modification: Apr 28 2018 25 | """ 26 | Relative = 1 27 | Absolute = 2 28 | 29 | 30 | class BBType(Enum): 31 | """ 32 | Class representing if the bounding box is groundtruth or not. 33 | 34 | Developed by: Rafael Padilla 35 | Last modification: May 24 2018 36 | """ 37 | GroundTruth = 1 38 | Detected = 2 39 | 40 | 41 | class BBFormat(Enum): 42 | """ 43 | Class representing the format of a bounding box. 44 | It can be (X,Y,width,height) => XYWH 45 | or (X1,Y1,X2,Y2) => XYX2Y2 46 | 47 | Developed by: Rafael Padilla 48 | Last modification: May 24 2018 49 | """ 50 | XYWH = 1 51 | XYX2Y2 = 2 52 | 53 | 54 | # size => (width, height) of the image 55 | # box => (X1, X2, Y1, Y2) of the bounding box 56 | def convertToRelativeValues(size, box): 57 | dw = 1. / (size[0]) 58 | dh = 1. / (size[1]) 59 | cx = (box[1] + box[0]) / 2.0 60 | cy = (box[3] + box[2]) / 2.0 61 | w = box[1] - box[0] 62 | h = box[3] - box[2] 63 | x = cx * dw 64 | y = cy * dh 65 | w = w * dw 66 | h = h * dh 67 | # x,y => (bounding_box_center)/width_of_the_image 68 | # w => bounding_box_width / width_of_the_image 69 | # h => bounding_box_height / height_of_the_image 70 | return (x, y, w, h) 71 | 72 | 73 | # size => (width, height) of the image 74 | # box => (centerX, centerY, w, h) of the bounding box relative to the image 75 | def convertToAbsoluteValues(size, box): 76 | # w_box = round(size[0] * box[2]) 77 | # h_box = round(size[1] * box[3]) 78 | xIn = round(((2 * float(box[0]) - float(box[2])) * size[0] / 2)) 79 | yIn = round(((2 * float(box[1]) - float(box[3])) * size[1] / 2)) 80 | xEnd = xIn + round(float(box[2]) * size[0]) 81 | yEnd = yIn + round(float(box[3]) * size[1]) 82 | if xIn < 0: 83 | xIn = 0 84 | if yIn < 0: 85 | yIn = 0 86 | if xEnd >= size[0]: 87 | xEnd = size[0] - 1 88 | if yEnd >= size[1]: 89 | yEnd = size[1] - 1 90 | return (xIn, yIn, xEnd, yEnd) 91 | 92 | 93 | def add_bb_into_image(image, bb, color=(255, 0, 0), thickness=2, label=None): 94 | r = int(color[0]) 95 | g = int(color[1]) 96 | b = int(color[2]) 97 | 98 | font = cv2.FONT_HERSHEY_SIMPLEX 99 | fontScale = 0.5 100 | fontThickness = 1 101 | 102 | x1, y1, x2, y2 = bb.getAbsoluteBoundingBox(BBFormat.XYX2Y2) 103 | x1 = int(x1) 104 | y1 = int(y1) 105 | x2 = int(x2) 106 | y2 = int(y2) 107 | cv2.rectangle(image, (x1, y1), (x2, y2), (b, g, r), thickness) 108 | # Add label 109 | if label is not None: 110 | # Get size of the text box 111 | (tw, th) = cv2.getTextSize(label, font, fontScale, fontThickness)[0] 112 | # Top-left coord of the textbox 113 | (xin_bb, yin_bb) = (x1 + thickness, y1 - th + int(12.5 * fontScale)) 114 | # Checking position of the text top-left (outside or inside the bb) 115 | if yin_bb - th <= 0: # if outside the image 116 | yin_bb = y1 + th # put it inside the bb 117 | r_Xin = x1 - int(thickness / 2) 118 | r_Yin = y1 - th - int(thickness / 2) 119 | # Draw filled rectangle to put the text in it 120 | cv2.rectangle(image, (r_Xin, r_Yin - thickness), 121 | (r_Xin + tw + thickness * 3, r_Yin + th + int(12.5 * fontScale)), (b, g, r), 122 | -1) 123 | cv2.putText(image, label, (xin_bb, yin_bb), font, fontScale, (0, 0, 0), fontThickness, 124 | cv2.LINE_AA) 125 | return image 126 | -------------------------------------------------------------------------------- /samples/sample_1/sample_1.py: -------------------------------------------------------------------------------- 1 | ########################################################################################### 2 | # # 3 | # This sample demonstrates: # 4 | # * How to create your own bounding boxes (detections and ground truth) manually; # 5 | # * Fill the object of the class BoundingBoxes with your bounding boxes; # 6 | # * Create images with detections and ground truth; # 7 | # # 8 | # Developed by: Rafael Padilla (rafael.padilla@smt.ufrj.br) # 9 | # SMT - Signal Multimedia and Telecommunications Lab # 10 | # COPPE - Universidade Federal do Rio de Janeiro # 11 | # Last modification: May 24th 2018 # 12 | ########################################################################################### 13 | 14 | import _init_paths 15 | import cv2 16 | import os 17 | from utils import * 18 | from BoundingBox import BoundingBox 19 | from BoundingBoxes import BoundingBoxes 20 | 21 | ########################### 22 | # Defining bounding boxes # 23 | ########################### 24 | # Ground truth bounding boxes of 000001.jpg 25 | gt_boundingBox_1 = BoundingBox( 26 | imageName='000001', 27 | classId='dog', 28 | x=0.34419263456090654, 29 | y=0.611, 30 | w=0.4164305949008499, 31 | h=0.262, 32 | typeCoordinates=CoordinatesType.Relative, 33 | bbType=BBType.GroundTruth, 34 | format=BBFormat.XYWH, 35 | imgSize=(353, 500)) 36 | gt_boundingBox_2 = BoundingBox( 37 | imageName='000001', 38 | classId='person', 39 | x=0.509915014164306, 40 | y=0.51, 41 | w=0.9745042492917847, 42 | h=0.972, 43 | typeCoordinates=CoordinatesType.Relative, 44 | bbType=BBType.GroundTruth, 45 | format=BBFormat.XYWH, 46 | imgSize=(353, 500)) 47 | # Ground truth bounding boxes of 000002.jpg 48 | gt_boundingBox_3 = BoundingBox( 49 | imageName='000002', 50 | classId='train', 51 | x=0.5164179104477612, 52 | y=0.501, 53 | w=0.20298507462686569, 54 | h=0.202, 55 | typeCoordinates=CoordinatesType.Relative, 56 | bbType=BBType.GroundTruth, 57 | format=BBFormat.XYWH, 58 | imgSize=(335, 500)) 59 | # Ground truth bounding boxes of 000003.jpg 60 | gt_boundingBox_4 = BoundingBox( 61 | imageName='000003', 62 | classId='bench', 63 | x=0.338, 64 | y=0.4666666666666667, 65 | w=0.184, 66 | h=0.10666666666666666, 67 | typeCoordinates=CoordinatesType.Relative, 68 | bbType=BBType.GroundTruth, 69 | format=BBFormat.XYWH, 70 | imgSize=(500, 375)) 71 | gt_boundingBox_5 = BoundingBox( 72 | imageName='000003', 73 | classId='bench', 74 | x=0.546, 75 | y=0.48133333333333334, 76 | w=0.136, 77 | h=0.13066666666666665, 78 | typeCoordinates=CoordinatesType.Relative, 79 | bbType=BBType.GroundTruth, 80 | format=BBFormat.XYWH, 81 | imgSize=(500, 375)) 82 | # Detected bounding boxes of 000001.jpg 83 | detected_boundingBox_1 = BoundingBox( 84 | imageName='000001', 85 | classId='person', 86 | classConfidence=0.893202, 87 | x=52, 88 | y=4, 89 | w=352, 90 | h=442, 91 | typeCoordinates=CoordinatesType.Absolute, 92 | bbType=BBType.Detected, 93 | format=BBFormat.XYX2Y2, 94 | imgSize=(353, 500)) 95 | # Detected bounding boxes of 000002.jpg 96 | detected_boundingBox_2 = BoundingBox( 97 | imageName='000002', 98 | classId='train', 99 | classConfidence=0.863700, 100 | x=140, 101 | y=195, 102 | w=209, 103 | h=293, 104 | typeCoordinates=CoordinatesType.Absolute, 105 | bbType=BBType.Detected, 106 | format=BBFormat.XYX2Y2, 107 | imgSize=(335, 500)) 108 | # Detected bounding boxes of 000003.jpg 109 | detected_boundingBox_3 = BoundingBox( 110 | imageName='000003', 111 | classId='bench', 112 | classConfidence=0.278000, 113 | x=388, 114 | y=288, 115 | w=493, 116 | h=331, 117 | typeCoordinates=CoordinatesType.Absolute, 118 | bbType=BBType.Detected, 119 | format=BBFormat.XYX2Y2, 120 | imgSize=(500, 375)) 121 | # Creating the object of the class BoundingBoxes 122 | myBoundingBoxes = BoundingBoxes() 123 | # Add all bounding boxes to the BoundingBoxes object: 124 | myBoundingBoxes.addBoundingBox(gt_boundingBox_1) 125 | myBoundingBoxes.addBoundingBox(gt_boundingBox_2) 126 | myBoundingBoxes.addBoundingBox(gt_boundingBox_3) 127 | myBoundingBoxes.addBoundingBox(gt_boundingBox_4) 128 | myBoundingBoxes.addBoundingBox(gt_boundingBox_5) 129 | myBoundingBoxes.addBoundingBox(detected_boundingBox_1) 130 | myBoundingBoxes.addBoundingBox(detected_boundingBox_2) 131 | myBoundingBoxes.addBoundingBox(detected_boundingBox_3) 132 | 133 | ################### 134 | # Creating images # 135 | ################### 136 | currentPath = os.path.dirname(os.path.realpath(__file__)) 137 | gtImages = ['000001', '000002', '000003'] 138 | for imageName in gtImages: 139 | im = cv2.imread(os.path.join(currentPath, 'images', 'groundtruths', imageName) + '.jpg') 140 | # Add bounding boxes 141 | im = myBoundingBoxes.drawAllBoundingBoxes(im, imageName) 142 | # cv2.imshow(imageName+'.jpg', im) 143 | # cv2.waitKey(0) 144 | cv2.imwrite(os.path.join(currentPath, 'images', imageName + '.jpg'), im) 145 | print('Image %s created successfully!' % imageName) 146 | -------------------------------------------------------------------------------- /lib/BoundingBox.py: -------------------------------------------------------------------------------- 1 | from utils import * 2 | 3 | 4 | class BoundingBox: 5 | def __init__(self, 6 | imageName, 7 | classId, 8 | x, 9 | y, 10 | w, 11 | h, 12 | typeCoordinates=CoordinatesType.Absolute, 13 | imgSize=None, 14 | bbType=BBType.GroundTruth, 15 | classConfidence=None, 16 | format=BBFormat.XYWH): 17 | """Constructor. 18 | Args: 19 | imageName: String representing the image name. 20 | classId: String value representing class id. 21 | x: Float value representing the X upper-left coordinate of the bounding box. 22 | y: Float value representing the Y upper-left coordinate of the bounding box. 23 | w: Float value representing the width bounding box. 24 | h: Float value representing the height bounding box. 25 | typeCoordinates: (optional) Enum (Relative or Absolute) represents if the bounding box 26 | coordinates (x,y,w,h) are absolute or relative to size of the image. Default:'Absolute'. 27 | imgSize: (optional) 2D vector (width, height)=>(int, int) represents the size of the 28 | image of the bounding box. If typeCoordinates is 'Relative', imgSize is required. 29 | bbType: (optional) Enum (Groundtruth or Detection) identifies if the bounding box 30 | represents a ground truth or a detection. If it is a detection, the classConfidence has 31 | to be informed. 32 | classConfidence: (optional) Float value representing the confidence of the detected 33 | class. If detectionType is Detection, classConfidence needs to be informed. 34 | format: (optional) Enum (BBFormat.XYWH or BBFormat.XYX2Y2) indicating the format of the 35 | coordinates of the bounding boxes. BBFormat.XYWH: 36 | BBFormat.XYX2Y2: . 37 | """ 38 | self._imageName = imageName 39 | self._typeCoordinates = typeCoordinates 40 | if typeCoordinates == CoordinatesType.Relative and imgSize is None: 41 | raise IOError( 42 | 'Parameter \'imgSize\' is required. It is necessary to inform the image size.') 43 | if bbType == BBType.Detected and classConfidence is None: 44 | raise IOError( 45 | 'For bbType=\'Detection\', it is necessary to inform the classConfidence value.') 46 | # if classConfidence != None and (classConfidence < 0 or classConfidence > 1): 47 | # raise IOError('classConfidence value must be a real value between 0 and 1. Value: %f' % 48 | # classConfidence) 49 | 50 | self._classConfidence = classConfidence 51 | self._bbType = bbType 52 | self._classId = classId 53 | self._format = format 54 | 55 | # If relative coordinates, convert to absolute values 56 | # For relative coords: (x,y,w,h)=(X_center/img_width , Y_center/img_height) 57 | if (typeCoordinates == CoordinatesType.Relative): 58 | (self._x, self._y, self._w, self._h) = convertToAbsoluteValues(imgSize, (x, y, w, h)) 59 | self._width_img = imgSize[0] 60 | self._height_img = imgSize[1] 61 | if format == BBFormat.XYWH: 62 | self._x2 = self._w 63 | self._y2 = self._h 64 | self._w = self._x2 - self._x 65 | self._h = self._y2 - self._y 66 | else: 67 | raise IOError( 68 | 'For relative coordinates, the format must be XYWH (x,y,width,height)') 69 | # For absolute coords: (x,y,w,h)=real bb coords 70 | else: 71 | self._x = x 72 | self._y = y 73 | if format == BBFormat.XYWH: 74 | self._w = w 75 | self._h = h 76 | self._x2 = self._x + self._w 77 | self._y2 = self._y + self._h 78 | else: # format == BBFormat.XYX2Y2: . 79 | self._x2 = w 80 | self._y2 = h 81 | self._w = self._x2 - self._x 82 | self._h = self._y2 - self._y 83 | if imgSize is None: 84 | self._width_img = None 85 | self._height_img = None 86 | else: 87 | self._width_img = imgSize[0] 88 | self._height_img = imgSize[1] 89 | 90 | def getAbsoluteBoundingBox(self, format=BBFormat.XYWH): 91 | if format == BBFormat.XYWH: 92 | return (self._x, self._y, self._w, self._h) 93 | elif format == BBFormat.XYX2Y2: 94 | return (self._x, self._y, self._x2, self._y2) 95 | 96 | def getRelativeBoundingBox(self, imgSize=None): 97 | if imgSize is None and self._width_img is None and self._height_img is None: 98 | raise IOError( 99 | 'Parameter \'imgSize\' is required. It is necessary to inform the image size.') 100 | if imgSize is None: 101 | return convertToRelativeValues((imgSize[0], imgSize[1]), 102 | (self._x, self._y, self._w, self._h)) 103 | else: 104 | return convertToRelativeValues((self._width_img, self._height_img), 105 | (self._x, self._y, self._w, self._h)) 106 | 107 | def getImageName(self): 108 | return self._imageName 109 | 110 | def getConfidence(self): 111 | return self._classConfidence 112 | 113 | def getFormat(self): 114 | return self._format 115 | 116 | def getClassId(self): 117 | return self._classId 118 | 119 | def getImageSize(self): 120 | return (self._width_img, self._height_img) 121 | 122 | def getCoordinatesType(self): 123 | return self._typeCoordinates 124 | 125 | def getBBType(self): 126 | return self._bbType 127 | 128 | @staticmethod 129 | def compare(det1, det2): 130 | det1BB = det1.getAbsoluteBoundingBox() 131 | det1ImgSize = det1.getImageSize() 132 | det2BB = det2.getAbsoluteBoundingBox() 133 | det2ImgSize = det2.getImageSize() 134 | 135 | if det1.getClassId() == det2.getClassId() and \ 136 | det1.classConfidence == det2.classConfidenc() and \ 137 | det1BB[0] == det2BB[0] and \ 138 | det1BB[1] == det2BB[1] and \ 139 | det1BB[2] == det2BB[2] and \ 140 | det1BB[3] == det2BB[3] and \ 141 | det1ImgSize[0] == det1ImgSize[0] and \ 142 | det2ImgSize[1] == det2ImgSize[1]: 143 | return True 144 | return False 145 | 146 | @staticmethod 147 | def clone(boundingBox): 148 | absBB = boundingBox.getAbsoluteBoundingBox(format=BBFormat.XYWH) 149 | # return (self._x,self._y,self._x2,self._y2) 150 | newBoundingBox = BoundingBox( 151 | boundingBox.getImageName(), 152 | boundingBox.getClassId(), 153 | absBB[0], 154 | absBB[1], 155 | absBB[2], 156 | absBB[3], 157 | typeCoordinates=boundingBox.getCoordinatesType(), 158 | imgSize=boundingBox.getImageSize(), 159 | bbType=boundingBox.getBBType(), 160 | classConfidence=boundingBox.getConfidence(), 161 | format=BBFormat.XYWH) 162 | return newBoundingBox 163 | -------------------------------------------------------------------------------- /samples/sample_2/sample_2.py: -------------------------------------------------------------------------------- 1 | ########################################################################################### 2 | # # 3 | # This sample shows how to evaluate object detections applying the following metrics: # 4 | # * Precision x Recall curve ----> used by VOC PASCAL 2012) # 5 | # * Average Precision (AP) ----> used by VOC PASCAL 2012) # 6 | # # 7 | # Developed by: Rafael Padilla (rafael.padilla@smt.ufrj.br) # 8 | # SMT - Signal Multimedia and Telecommunications Lab # 9 | # COPPE - Universidade Federal do Rio de Janeiro # 10 | # Last modification: May 24th 2018 # 11 | ########################################################################################### 12 | 13 | import _init_paths 14 | from BoundingBox import BoundingBox 15 | from BoundingBoxes import BoundingBoxes 16 | from Evaluator import * 17 | from utils import * 18 | 19 | 20 | def getBoundingBoxes(): 21 | """Read txt files containing bounding boxes (ground truth and detections).""" 22 | allBoundingBoxes = BoundingBoxes() 23 | import glob 24 | import os 25 | # Read ground truths 26 | currentPath = os.path.dirname(os.path.abspath(__file__)) 27 | folderGT = os.path.join(currentPath, 'groundtruths') 28 | os.chdir(folderGT) 29 | files = glob.glob("*.txt") 30 | files.sort() 31 | # Class representing bounding boxes (ground truths and detections) 32 | allBoundingBoxes = BoundingBoxes() 33 | # Read GT detections from txt file 34 | # Each line of the files in the groundtruths folder represents a ground truth bounding box 35 | # (bounding boxes that a detector should detect) 36 | # Each value of each line is "class_id, x, y, width, height" respectively 37 | # Class_id represents the class of the bounding box 38 | # x, y represents the most top-left coordinates of the bounding box 39 | # x2, y2 represents the most bottom-right coordinates of the bounding box 40 | for f in files: 41 | nameOfImage = f.replace(".txt", "") 42 | fh1 = open(f, "r") 43 | for line in fh1: 44 | line = line.replace("\n", "") 45 | if line.replace(' ', '') == '': 46 | continue 47 | splitLine = line.split(" ") 48 | idClass = splitLine[0] # class 49 | x = float(splitLine[1]) # confidence 50 | y = float(splitLine[2]) 51 | w = float(splitLine[3]) 52 | h = float(splitLine[4]) 53 | bb = BoundingBox( 54 | nameOfImage, 55 | idClass, 56 | x, 57 | y, 58 | w, 59 | h, 60 | CoordinatesType.Absolute, (200, 200), 61 | BBType.GroundTruth, 62 | format=BBFormat.XYWH) 63 | allBoundingBoxes.addBoundingBox(bb) 64 | fh1.close() 65 | # Read detections 66 | folderDet = os.path.join(currentPath, 'detections') 67 | os.chdir(folderDet) 68 | files = glob.glob("*.txt") 69 | files.sort() 70 | # Read detections from txt file 71 | # Each line of the files in the detections folder represents a detected bounding box. 72 | # Each value of each line is "class_id, confidence, x, y, width, height" respectively 73 | # Class_id represents the class of the detected bounding box 74 | # Confidence represents confidence (from 0 to 1) that this detection belongs to the class_id. 75 | # x, y represents the most top-left coordinates of the bounding box 76 | # x2, y2 represents the most bottom-right coordinates of the bounding box 77 | for f in files: 78 | # nameOfImage = f.replace("_det.txt","") 79 | nameOfImage = f.replace(".txt", "") 80 | # Read detections from txt file 81 | fh1 = open(f, "r") 82 | for line in fh1: 83 | line = line.replace("\n", "") 84 | if line.replace(' ', '') == '': 85 | continue 86 | splitLine = line.split(" ") 87 | idClass = splitLine[0] # class 88 | confidence = float(splitLine[1]) # confidence 89 | x = float(splitLine[2]) 90 | y = float(splitLine[3]) 91 | w = float(splitLine[4]) 92 | h = float(splitLine[5]) 93 | bb = BoundingBox( 94 | nameOfImage, 95 | idClass, 96 | x, 97 | y, 98 | w, 99 | h, 100 | CoordinatesType.Absolute, (200, 200), 101 | BBType.Detected, 102 | confidence, 103 | format=BBFormat.XYWH) 104 | allBoundingBoxes.addBoundingBox(bb) 105 | fh1.close() 106 | return allBoundingBoxes 107 | 108 | 109 | def createImages(dictGroundTruth, dictDetected): 110 | """Create representative images with bounding boxes.""" 111 | import numpy as np 112 | import cv2 113 | # Define image size 114 | width = 200 115 | height = 200 116 | # Loop through the dictionary with ground truth detections 117 | for key in dictGroundTruth: 118 | image = np.zeros((height, width, 3), np.uint8) 119 | gt_boundingboxes = dictGroundTruth[key] 120 | image = gt_boundingboxes.drawAllBoundingBoxes(image) 121 | detection_boundingboxes = dictDetected[key] 122 | image = detection_boundingboxes.drawAllBoundingBoxes(image) 123 | # Show detection and its GT 124 | cv2.imshow(key, image) 125 | cv2.waitKey() 126 | 127 | 128 | # Read txt files containing bounding boxes (ground truth and detections) 129 | boundingboxes = getBoundingBoxes() 130 | # Uncomment the line below to generate images based on the bounding boxes 131 | # createImages(dictGroundTruth, dictDetected) 132 | # Create an evaluator object in order to obtain the metrics 133 | evaluator = Evaluator() 134 | ############################################################## 135 | # VOC PASCAL Metrics 136 | ############################################################## 137 | # Plot Precision x Recall curve 138 | evaluator.PlotPrecisionRecallCurve( 139 | 'object', # Class to show 140 | boundingboxes, # Object containing all bounding boxes (ground truths and detections) 141 | IOUThreshold=0.3, # IOU threshold 142 | method=MethodAveragePrecision.EveryPointInterpolation, # As the official matlab code 143 | showAP=True, # Show Average Precision in the title of the plot 144 | showInterpolatedPrecision=True) # Don't plot the interpolated precision curve 145 | # Get metrics with PASCAL VOC metrics 146 | metricsPerClass = evaluator.GetPascalVOCMetrics( 147 | boundingboxes, # Object containing all bounding boxes (ground truths and detections) 148 | IOUThreshold=0.3, # IOU threshold 149 | method=MethodAveragePrecision.EveryPointInterpolation) # As the official matlab code 150 | print("Average precision values per class:\n") 151 | # Loop through classes to obtain their metrics 152 | for mc in metricsPerClass: 153 | # Get metric values per each class 154 | c = mc['class'] 155 | precision = mc['precision'] 156 | recall = mc['recall'] 157 | average_precision = mc['AP'] 158 | ipre = mc['interpolated precision'] 159 | irec = mc['interpolated recall'] 160 | # Print AP per class 161 | print('%s: %f' % (c, average_precision)) 162 | -------------------------------------------------------------------------------- /samples/sample_2/README.md: -------------------------------------------------------------------------------- 1 | # Sample 2 2 | 3 | This sample was created for those who want to understand more about the metric functions of this project. If you just want to evaluate your detections dealing with a high level interface, just check the instructions [here](https://github.com/rafaelpadilla/Object-Detection-Metrics/blob/master/README.md#how-to-use-this-project). 4 | 5 | In order to reproduce the results of this code **using the high level interface**, just navigate to the folder where the ```pascalvoc.py``` is and run the following command: ```python pascalvoc.py -t 0.3``` 6 | 7 | or if you want to be more complete: ```python pascalvoc.py -gt groundtruths/ -det detections/ -t 0.3``` 8 | 9 | or if you want to use [relative coordinates](https://github.com/rafaelpadilla/Object-Detection-Metrics/tree/master/detections_rel) (like retrieved results from YOLO), you need to inform that the detected coordinates are relative, specify the image size: 10 | ```python pascalvoc.py -gt groundtruths/ -det detections_rel/ -detcoords rel -imgsize 200,200 -t 0.3``` 11 | 12 | Or if you want to play a little bit with this project, follow the steps below: 13 | 14 | ### Evaluation Metrics 15 | 16 | First we need to represent each bounding box with the class `BoundingBox`. The function `getBoundingBoxes` reads .txt files containing the coordinates of the [detected](https://github.com/rafaelpadilla/Object-Detection-Metrics/tree/master/samples/sample_2/detections) and [ground truth](https://github.com/rafaelpadilla/Object-Detection-Metrics/tree/master/samples/sample_2/groundtruths) bounding boxes and creates a `BoundingBox` object for each of them. Then, it gathers all boxes in the `BoundingBoxes` object and returns it: 17 | 18 | ```python 19 | def getBoundingBoxes(): 20 | """Read txt files containing bounding boxes (ground truth and detections).""" 21 | allBoundingBoxes = BoundingBoxes() 22 | import glob 23 | import os 24 | # Read ground truths 25 | currentPath = os.path.dirname(os.path.abspath(__file__)) 26 | folderGT = os.path.join(currentPath,'groundtruths') 27 | os.chdir(folderGT) 28 | files = glob.glob("*.txt") 29 | files.sort() 30 | # Class representing bounding boxes (ground truths and detections) 31 | allBoundingBoxes = BoundingBoxes() 32 | # Read GT detections from txt file 33 | # Each line of the files in the groundtruths folder represents a ground truth bounding box (bounding boxes that a detector should detect) 34 | # Each value of each line is "class_id, x, y, width, height" respectively 35 | # Class_id represents the class of the bounding box 36 | # x, y represents the most top-left coordinates of the bounding box 37 | # x2, y2 represents the most bottom-right coordinates of the bounding box 38 | for f in files: 39 | nameOfImage = f.replace(".txt","") 40 | fh1 = open(f, "r") 41 | for line in fh1: 42 | line = line.replace("\n","") 43 | if line.replace(' ','') == '': 44 | continue 45 | splitLine = line.split(" ") 46 | idClass = splitLine[0] #class 47 | x = float(splitLine[1]) #confidence 48 | y = float(splitLine[2]) 49 | w = float(splitLine[3]) 50 | h = float(splitLine[4]) 51 | bb = BoundingBox(nameOfImage,idClass,x,y,w,h,CoordinatesType.Absolute, (200,200), BBType.GroundTruth, format=BBFormat.XYWH) 52 | allBoundingBoxes.addBoundingBox(bb) 53 | fh1.close() 54 | # Read detections 55 | folderDet = os.path.join(currentPath,'detections') 56 | os.chdir(folderDet) 57 | files = glob.glob("*.txt") 58 | files.sort() 59 | # Read detections from txt file 60 | # Each line of the files in the detections folder represents a detected bounding box. 61 | # Each value of each line is "class_id, confidence, x, y, width, height" respectively 62 | # Class_id represents the class of the detected bounding box 63 | # Confidence represents the confidence (from 0 to 1) that this detection belongs to the class_id. 64 | # x, y represents the most top-left coordinates of the bounding box 65 | # x2, y2 represents the most bottom-right coordinates of the bounding box 66 | for f in files: 67 | # nameOfImage = f.replace("_det.txt","") 68 | nameOfImage = f.replace(".txt","") 69 | # Read detections from txt file 70 | fh1 = open(f, "r") 71 | for line in fh1: 72 | line = line.replace("\n","") 73 | if line.replace(' ','') == '': 74 | continue 75 | splitLine = line.split(" ") 76 | idClass = splitLine[0] #class 77 | confidence = float(splitLine[1]) #confidence 78 | x = float(splitLine[2]) 79 | y = float(splitLine[3]) 80 | w = float(splitLine[4]) 81 | h = float(splitLine[5]) 82 | bb = BoundingBox(nameOfImage, idClass,x,y,w,h,CoordinatesType.Absolute, (200,200), BBType.Detected, confidence, format=BBFormat.XYWH) 83 | allBoundingBoxes.addBoundingBox(bb) 84 | fh1.close() 85 | return allBoundingBoxes 86 | 87 | # Read txt files containing bounding boxes (ground truth and detections) 88 | boundingboxes = getBoundingBoxes() 89 | ``` 90 | 91 | Note that the text files contain one bounding box per line in the format: 92 | 93 | **\ \ \ \ \**: For ground truth files. 94 | 95 | **\ \ \ \ \ \**: For detection files. 96 | 97 | The next step is to create an `Evaluator` object that provides us the metrics: 98 | 99 | ```python 100 | # Create an evaluator object in order to obtain the metrics 101 | evaluator = Evaluator() 102 | ``` 103 | 104 | With the ```evaluator``` object, you will have access to methods that retrieve the metrics: 105 | 106 | | Method | Description | Parameters | Returns | 107 | |------|-----------|----------|-------| 108 | | GetPascalVOCMetrics | Get the metrics used by the VOC Pascal 2012 challenge | `boundingboxes`: Object of the class `BoundingBoxes` representing ground truth and detected bounding boxes; `IOUThreshold`: IOU threshold indicating which detections will be considered TP or FP (default value = 0.5); | List of dictionaries. Each dictionary contains information and metrics of each class. The keys of each dictionary are: `dict['class']`: class representing the current dictionary; `dict['precision']`: array with the precision values; `dict['recall']`: array with the recall values; `dict['AP']`: **average precision**; `dict['interpolated precision']`: interpolated precision values; `dict['interpolated recall']`: interpolated recall values; `dict['total positives']`: total number of ground truth positives; `dict['total TP']`: total number of True Positive detections; `dict['total FP']`: total number of False Negative detections; | 109 | PlotPrecisionRecallCurve | Plot the Precision x Recall curve for a given class | `classId`: The class that will be plot; `boundingBoxes`: Object of the class `BoundingBoxes` representing ground truth and detected bounding boxes; `IOUThreshold`: IOU threshold indicating which detections will be considered TP or FP (default value = 0.5); `showAP`: if True, the average precision value will be shown in the title of the graph (default = False); `showInterpolatedPrecision`: if True, it will show in the plot the interpolated precision (default = False); `savePath`: if informed, the plot will be saved as an image in this path (ex: `/home/mywork/ap.png`) (default = None); `showGraphic`: if True, the plot will be shown (default = True) | The dictionary containing information and metric about the class. The keys of the dictionary are: `dict['class']`: class representing the current dictionary; `dict['precision']`: array with the precision values; `dict['recall']`: array with the recall values; `dict['AP']`: **average precision**; `dict['interpolated precision']`: interpolated precision values; `dict['interpolated recall']`: interpolated recall values; `dict['total positives']`: total number of ground truth positives; `dict['total TP']`: total number of True Positive detections; `dict['total FP']`: total number of False Negative detections | 110 | 111 | The snippet below is used to plot the Precision x Recall curve: 112 | 113 | ```python 114 | # Plot Precision x Recall curve 115 | evaluator.PlotPrecisionRecallCurve('object', # Class to show 116 | boundingboxes, # Object containing all bounding boxes (ground truths and detections) 117 | IOUThreshold=0.3, # IOU threshold 118 | showAP=True, # Show Average Precision in the title of the plot 119 | showInterpolatedPrecision=False) # Don't plot the interpolated precision curve 120 | ``` 121 | 122 | We can have access to the Average Precision value of each class using the method `GetPascalVocMetrics`: 123 | 124 | ```python 125 | metricsPerClass = evaluator.GetPascalVOCMetrics(boundingboxes, IOUThreshold=0.3) 126 | print("Average precision values per class:\n") 127 | # Loop through classes to obtain their metrics 128 | for mc in metricsPerClass: 129 | # Get metric values per each class 130 | c = mc['class'] 131 | precision = mc['precision'] 132 | recall = mc['recall'] 133 | average_precision = mc['AP'] 134 | ipre = mc['interpolated precision'] 135 | irec = mc['interpolated recall'] 136 | # Print AP per class 137 | print('%s: %f' % (c, average_precision)) 138 | ``` 139 | 140 | See [here](https://github.com/rafaelpadilla/Object-Detection-Metrics/blob/master/samples/sample_2/sample_2.py) the full code. 141 | -------------------------------------------------------------------------------- /samples/sample_1/README.md: -------------------------------------------------------------------------------- 1 | # Sample 1 2 | 3 | This sample was created to illustrate the usage of the classes **BoundingBox** and **BoundingBoxes**. Objects of the class `BoundingBox` are an abstraction of the detections or the ground truth boxes. The object of the class `BoundingBoxes` is used by evaluation methods and represents a collection of bounding boxes. 4 | 5 | The full code can be accessed [here](https://github.com/rafaelpadilla/Object-Detection-Metrics/blob/master/samples/sample_1/sample_1.py). 6 | 7 | If you just want to evaluate your detections dealing with a high level interface, just check the instructions [here](https://github.com/rafaelpadilla/Object-Detection-Metrics/blob/master/README.md#how-to-use-this-project). 8 | 9 | ### The code 10 | 11 | The classes BoudingBox and BoundingBoxes are in the `lib/` folder. The file `_init_paths.py` imports these contents into our example. The file `utils.py` contains basically enumerators and useful functions. The code below shows how to import them: 12 | 13 | ```python 14 | import _init_paths 15 | from utils import * 16 | from BoundingBox import BoundingBox 17 | from BoundingBoxes import BoundingBoxes 18 | ``` 19 | Don't forget to put the content of the folder `\lib` in the same folder of your code. 20 | 21 | All bounding boxes (detected and ground truth) are represented by objects of the class `BoundingBox`. Each bounding box is created using the constructor. Use the parameter `bbType` to identify if the box is a ground truth or a detected one. The parameter `imageName` determines the image that the box belongs to. All the parameters used to create the object are: 22 | 23 | * `imageName`: String representing the image name. 24 | * `classId`: String value representing class id (e.g. 'house', 'dog', 'person') 25 | * `x`: Float value representing the X upper-left coordinate of the bounding box. 26 | * `y`: Float value representing the Y upper-left coordinate of the bounding box. 27 | * `w`: Float value representing the width bounding box. It can also be used to represent the X lower-right coordinate of the bounding box. For that, use the parameter `format=BBFormat.XYX2Y2`. 28 | * `h`: Float value representing the height bounding box. It can also be used to represent the Y lower-right coordinate of the bounding box. For that, use the parameter `format=BBFormat.XYX2Y2`. 29 | * `typeCoordinates`: (optional) Enum (`CoordinatesType.Relative` or `CoordinatesType.Absolute`) representing if the bounding box coordinates (x,y,w,h) are absolute or relative to size of the image. Default: 'Absolute'. Some projects like YOLO identifies the detected bounding boxes as being relative to the image size, it may be useful for cases like that. Note that if the coordinate type is relative, the `imgSize` parameter is required. 30 | * `imgSize`: (optional) 2D vector (width, height)=>(int, int) representing the size of the image of the bounding box. If `typeCoordinates=CoordinatesType.Relative`, the parameter `imgSize` is required. 31 | * `bbType`: (optional) Enum (`bbType=BBType.Groundtruth` or `bbType=BBType.Detection`) identifies if the bounding box represents a ground truth or a detection. Not that if it is a detection, the classConfidence has to be informed. 32 | * `classConfidence`: (optional) Float value representing the confidence of the detected class. If detectionType is Detection, classConfidence needs to be informed. 33 | * `format`: (optional) Enum (`BBFormat.XYWH` or `BBFormat.XYX2Y2`) indicating the format of the coordinates of the bounding boxes. If `format=BBFormat.XYWH`, the parameters `x`,`y`,`w` and `h` are: \, \, \ and \ respectively. If `format=BBFormat.XYX2Y2`, the parameters `x`,`y`,`w` and `h` are: \, \, \ and \ respectively. 34 | 35 | **Attention**: The bounding boxes of the same image (detections or ground truth) must have have the same `imageName`. 36 | 37 | The snippet below shows the creation of bounding boxes of 3 different images (000001.jpg, 000002.jpg and 000003.jpg) containing 2, 1 and 1 ground truth objects to be detected respectively. There are 3 detected bounding boxes, one at each image. The images are available in the folder [sample_1/images/detections/](https://github.com/rafaelpadilla/Object-Detection-Metrics/tree/master/samples/sample_1/images/detections) and [sample_1/images/groundtruths/](https://github.com/rafaelpadilla/Object-Detection-Metrics/tree/master/samples/sample_1/images/groundtruths). 38 | 39 | ```python 40 | # Ground truth bounding boxes of 000001.jpg 41 | gt_boundingBox_1 = BoundingBox(imageName='000001', classId='dog', x=0.34419263456090654, y=0.611, 42 | w=0.4164305949008499, h=0.262, typeCoordinates=CoordinatesType.Relative, 43 | bbType=BBType.GroundTruth, format=BBFormat.XYWH, imgSize=(353,500)) 44 | gt_boundingBox_2 = BoundingBox(imageName='000001', classId='person', x=0.509915014164306, y=0.51, 45 | w=0.9745042492917847, h=0.972, typeCoordinates=CoordinatesType.Relative, 46 | bbType=BBType.GroundTruth, format=BBFormat.XYWH, imgSize=(353,500)) 47 | # Ground truth bounding boxes of 000002.jpg 48 | gt_boundingBox_3 = BoundingBox(imageName='000002', classId='train', x=0.5164179104477612, y=0.501, 49 | w=0.20298507462686569, h=0.202, typeCoordinates=CoordinatesType.Relative, 50 | bbType=BBType.GroundTruth, format=BBFormat.XYWH, imgSize=(335,500)) 51 | # Ground truth bounding boxes of 000003.jpg 52 | gt_boundingBox_4 = BoundingBox(imageName='000003', classId='bench', x=0.338, y=0.4666666666666667, 53 | w=0.184, h=0.10666666666666666, typeCoordinates=CoordinatesType.Relative, 54 | bbType=BBType.GroundTruth, format=BBFormat.XYWH, imgSize=(500,375)) 55 | gt_boundingBox_5 = BoundingBox(imageName='000003', classId='bench', x=0.546, y=0.48133333333333334, 56 | w=0.136, h=0.13066666666666665, typeCoordinates=CoordinatesType.Relative, 57 | bbType=BBType.GroundTruth, format=BBFormat.XYWH, imgSize=(500,375)) 58 | # Detected bounding boxes of 000001.jpg 59 | detected_boundingBox_1 = BoundingBox(imageName='000001', classId='person', classConfidence= 0.893202, 60 | x=52, y=4, w=352, h=442, typeCoordinates=CoordinatesType.Absolute, 61 | bbType=BBType.Detected, format=BBFormat.XYX2Y2, imgSize=(353,500)) 62 | # Detected bounding boxes of 000002.jpg 63 | detected_boundingBox_2 = BoundingBox(imageName='000002', classId='train', classConfidence=0.863700, 64 | x=140, y=195, w=209, h=293, typeCoordinates=CoordinatesType.Absolute, 65 | bbType=BBType.Detected, format=BBFormat.XYX2Y2, imgSize=(335,500)) 66 | # Detected bounding boxes of 000003.jpg 67 | detected_boundingBox_3 = BoundingBox(imageName='000003', classId='bench', classConfidence=0.278000, 68 | x=388, y=288, w=493, h=331, typeCoordinates=CoordinatesType.Absolute, 69 | bbType=BBType.Detected, format=BBFormat.XYX2Y2, imgSize=(500,375)) 70 | ``` 71 | 72 | The object `BoundingBoxes` represents a collection of the bounding boxes (ground truth and detected). Evaluation methods of the class `Evaluator` use the `BoundingBoxes` object to apply the metrics. The following code shows how to add the bounding boxes to the collection: 73 | 74 | ```python 75 | # Creating the object of the class BoundingBoxes 76 | myBoundingBoxes = BoundingBoxes() 77 | # Add all bounding boxes to the BoundingBoxes object: 78 | myBoundingBoxes.addBoundingBox(gt_boundingBox_1) 79 | myBoundingBoxes.addBoundingBox(gt_boundingBox_2) 80 | myBoundingBoxes.addBoundingBox(gt_boundingBox_3) 81 | myBoundingBoxes.addBoundingBox(gt_boundingBox_4) 82 | myBoundingBoxes.addBoundingBox(gt_boundingBox_5) 83 | myBoundingBoxes.addBoundingBox(detected_boundingBox_1) 84 | myBoundingBoxes.addBoundingBox(detected_boundingBox_2) 85 | myBoundingBoxes.addBoundingBox(detected_boundingBox_3) 86 | ``` 87 | 88 | You can use the method `drawAllBoundingBoxes(image, imageName)` to add ground truth bounding boxes (in green) and detected bounding boxes (in red) into your images: 89 | 90 | ```python 91 | import cv2 92 | import numpy as np 93 | import os 94 | currentPath = os.path.dirname(os.path.realpath(__file__)) 95 | gtImages = ['000001', '000002', '000003'] 96 | for imageName in gtImages: 97 | im = cv2.imread(os.path.join(currentPath,'images','groundtruths',imageName)+'.jpg') 98 | # Add bounding boxes 99 | im = myBoundingBoxes.drawAllBoundingBoxes(im, imageName) 100 | # Uncomment the lines below if you want to show the images 101 | #cv2.imshow(imageName+'.jpg', im) 102 | #cv2.waitKey(0) 103 | cv2.imwrite(os.path.join(currentPath,'images',imageName+'.jpg'),im) 104 | print('Image %s created successfully!' % imageName) 105 | ``` 106 | 107 | Results: 108 | 109 | 110 |

111 | /> 112 | /> 113 | /> 114 |

115 | 116 | **Of course you won't build your bounding boxes one by one as done in this example.** You should read your detections within a loop and create your bounding boxes inside of it. [Sample_2](https://github.com/rafaelpadilla/Object-Detection-Metrics/tree/master/samples/sample_2) demonstrates how to read detections from folders containing .txt files. 117 | -------------------------------------------------------------------------------- /pascalvoc.py: -------------------------------------------------------------------------------- 1 | ########################################################################################### 2 | # # 3 | # This sample shows how to evaluate object detections applying the following metrics: # 4 | # * Precision x Recall curve ----> used by VOC PASCAL 2012) # 5 | # * Average Precision (AP) ----> used by VOC PASCAL 2012) # 6 | # # 7 | # Developed by: Rafael Padilla (rafael.padilla@smt.ufrj.br) # 8 | # SMT - Signal Multimedia and Telecommunications Lab # 9 | # COPPE - Universidade Federal do Rio de Janeiro # 10 | # Last modification: Oct 9th 2018 # 11 | ########################################################################################### 12 | 13 | import _init_paths 14 | from BoundingBox import BoundingBox 15 | from BoundingBoxes import BoundingBoxes 16 | from Evaluator import * 17 | from utils import BBFormat 18 | import argparse 19 | # from argparse import RawTextHelpFormatter 20 | import sys 21 | import os 22 | import glob 23 | import shutil 24 | 25 | 26 | # Validate formats 27 | def ValidateFormats(argFormat, argName, errors): 28 | if argFormat == 'xywh': 29 | return BBFormat.XYWH 30 | elif argFormat == 'xyrb': 31 | return BBFormat.XYX2Y2 32 | elif argFormat is None: 33 | return BBFormat.XYWH # default when nothing is passed 34 | else: 35 | errors.append( 36 | 'argument %s: invalid value. It must be either \'xywh\' or \'xyrb\'' % argName) 37 | 38 | 39 | # Validate mandatory args 40 | def ValidateMandatoryArgs(arg, argName, errors): 41 | if arg is None: 42 | errors.append('argument %s: required argument' % argName) 43 | else: 44 | return True 45 | 46 | 47 | def ValidateImageSize(arg, argName, argInformed, errors): 48 | errorMsg = 'argument %s: required argument if %s is relative' % (argName, argInformed) 49 | ret = None 50 | if arg is None: 51 | errors.append(errorMsg) 52 | else: 53 | arg = arg.replace('(', '').replace(')', '') 54 | args = arg.split(',') 55 | if len(args) != 2: 56 | errors.append( 57 | '%s. It must be in the format \'width,height\' (e.g. \'600,400\')' % errorMsg) 58 | else: 59 | if not args[0].isdigit() or not args[1].isdigit(): 60 | errors.append( 61 | '%s. It must be in INdiaTEGER the format \'width,height\' (e.g. \'600,400\')' % 62 | errorMsg) 63 | else: 64 | ret = (int(args[0]), int(args[1])) 65 | return ret 66 | 67 | 68 | # Validate coordinate types 69 | def ValidateCoordinatesTypes(arg, argName, errors): 70 | if arg == 'abs': 71 | return CoordinatesType.Absolute 72 | elif arg == 'rel': 73 | return CoordinatesType.Relative 74 | elif arg is None: 75 | return CoordinatesType.Absolute # default when nothing is passed 76 | errors.append('argument %s: invalid value. It must be either \'rel\' or \'abs\'' % argName) 77 | 78 | 79 | def ValidatePaths(arg, nameArg, errors): 80 | if arg is None: 81 | errors.append('argument %s: invalid directory' % nameArg) 82 | elif os.path.isdir(arg) is False and os.path.isdir(os.path.join(currentPath, arg)) is False: 83 | errors.append('argument %s: directory does not exist \'%s\'' % (nameArg, arg)) 84 | # elif os.path.isdir(os.path.join(currentPath, arg)) is True: 85 | # arg = os.path.join(currentPath, arg) 86 | else: 87 | arg = os.path.join(currentPath, arg) 88 | return arg 89 | 90 | 91 | def getBoundingBoxes(directory, 92 | isGT, 93 | bbFormat, 94 | coordType, 95 | allBoundingBoxes=None, 96 | allClasses=None, 97 | imgSize=(0, 0)): 98 | """Read txt files containing bounding boxes (ground truth and detections).""" 99 | if allBoundingBoxes is None: 100 | allBoundingBoxes = BoundingBoxes() 101 | if allClasses is None: 102 | allClasses = [] 103 | # Read ground truths 104 | os.chdir(directory) 105 | files = glob.glob("*.txt") 106 | files.sort() 107 | # Read GT detections from txt file 108 | # Each line of the files in the groundtruths folder represents a ground truth bounding box 109 | # (bounding boxes that a detector should detect) 110 | # Each value of each line is "class_id, x, y, width, height" respectively 111 | # Class_id represents the class of the bounding box 112 | # x, y represents the most top-left coordinates of the bounding box 113 | # x2, y2 represents the most bottom-right coordinates of the bounding box 114 | for f in files: 115 | nameOfImage = f.replace(".txt", "") 116 | fh1 = open(f, "r") 117 | for line in fh1: 118 | line = line.replace("\n", "") 119 | if line.replace(' ', '') == '': 120 | continue 121 | splitLine = line.split(" ") 122 | if isGT: 123 | # idClass = int(splitLine[0]) #class 124 | idClass = (splitLine[0]) # class 125 | x = float(splitLine[1]) 126 | y = float(splitLine[2]) 127 | w = float(splitLine[3]) 128 | h = float(splitLine[4]) 129 | bb = BoundingBox( 130 | nameOfImage, 131 | idClass, 132 | x, 133 | y, 134 | w, 135 | h, 136 | coordType, 137 | imgSize, 138 | BBType.GroundTruth, 139 | format=bbFormat) 140 | else: 141 | # idClass = int(splitLine[0]) #class 142 | idClass = (splitLine[0]) # class 143 | confidence = float(splitLine[1]) 144 | x = float(splitLine[2]) 145 | y = float(splitLine[3]) 146 | w = float(splitLine[4]) 147 | h = float(splitLine[5]) 148 | bb = BoundingBox( 149 | nameOfImage, 150 | idClass, 151 | x, 152 | y, 153 | w, 154 | h, 155 | coordType, 156 | imgSize, 157 | BBType.Detected, 158 | confidence, 159 | format=bbFormat) 160 | allBoundingBoxes.addBoundingBox(bb) 161 | if idClass not in allClasses: 162 | allClasses.append(idClass) 163 | fh1.close() 164 | return allBoundingBoxes, allClasses 165 | 166 | # Get current path to set default folders 167 | currentPath = os.path.dirname(os.path.abspath(__file__)) 168 | 169 | VERSION = '0.1 (beta)' 170 | 171 | parser = argparse.ArgumentParser( 172 | prog='Object Detection Metrics - Pascal VOC', 173 | description='This project applies the most popular metrics used to evaluate object detection ' 174 | 'algorithms.\nThe current implemention runs the Pascal VOC metrics.\nFor further references, ' 175 | 'please check:\nhttps://github.com/rafaelpadilla/Object-Detection-Metrics', 176 | epilog="Developed by: Rafael Padilla (rafael.padilla@smt.ufrj.br)") 177 | # formatter_class=RawTextHelpFormatter) 178 | parser.add_argument('-v', '--version', action='version', version='%(prog)s ' + VERSION) 179 | # Positional arguments 180 | # Mandatory 181 | parser.add_argument( 182 | '-gt', 183 | '--gtfolder', 184 | dest='gtFolder', 185 | default=os.path.join(currentPath,'groundtruths'), 186 | metavar='', 187 | help='folder containing your ground truth bounding boxes') 188 | parser.add_argument( 189 | '-det', 190 | '--detfolder', 191 | dest='detFolder', 192 | default=os.path.join(currentPath,'detections'), 193 | metavar='', 194 | help='folder containing your detected bounding boxes') 195 | # Optional 196 | parser.add_argument( 197 | '-t', 198 | '--threshold', 199 | dest='iouThreshold', 200 | type=float, 201 | default=0.5, 202 | metavar='', 203 | help='IOU threshold. Default 0.5') 204 | parser.add_argument( 205 | '-gtformat', 206 | dest='gtFormat', 207 | metavar='', 208 | default='xywh', 209 | help='format of the coordinates of the ground truth bounding boxes: ' 210 | '(\'xywh\': )' 211 | ' or (\'xyrb\': )') 212 | parser.add_argument( 213 | '-detformat', 214 | dest='detFormat', 215 | metavar='', 216 | default='xywh', 217 | help='format of the coordinates of the detected bounding boxes ' 218 | '(\'xywh\': ) ' 219 | 'or (\'xyrb\': )') 220 | parser.add_argument( 221 | '-gtcoords', 222 | dest='gtCoordinates', 223 | default='abs', 224 | metavar='', 225 | help='reference of the ground truth bounding box coordinates: absolute ' 226 | 'values (\'abs\') or relative to its image size (\'rel\')') 227 | parser.add_argument( 228 | '-detcoords', 229 | default='abs', 230 | dest='detCoordinates', 231 | metavar='', 232 | help='reference of the ground truth bounding box coordinates: ' 233 | 'absolute values (\'abs\') or relative to its image size (\'rel\')') 234 | parser.add_argument( 235 | '-imgsize', 236 | dest='imgSize', 237 | metavar='', 238 | help='image size. Required if -gtcoords or -detcoords are \'rel\'') 239 | parser.add_argument( 240 | '-sp', '--savepath', dest='savePath', 241 | metavar='', help='folder where the plots are saved') 242 | parser.add_argument( 243 | '-np', 244 | '--noplot', 245 | dest='showPlot', 246 | action='store_false', 247 | help='no plot is shown during execution') 248 | args = parser.parse_args() 249 | 250 | iouThreshold = args.iouThreshold 251 | 252 | # Arguments validation 253 | errors = [] 254 | # Validate formats 255 | gtFormat = ValidateFormats(args.gtFormat, '-gtformat', errors) 256 | detFormat = ValidateFormats(args.detFormat, '-detformat', errors) 257 | # Groundtruth folder 258 | if ValidateMandatoryArgs(args.gtFolder, '-gt/--gtfolder', errors): 259 | gtFolder = ValidatePaths(args.gtFolder, '-gt/--gtfolder', errors) 260 | else: 261 | # errors.pop() 262 | gtFolder = os.path.join(currentPath, 'groundtruths') 263 | if os.path.isdir(gtFolder) is False: 264 | errors.append('folder %s not found' % gtFolder) 265 | # Coordinates types 266 | gtCoordType = ValidateCoordinatesTypes(args.gtCoordinates, '-gtCoordinates', errors) 267 | detCoordType = ValidateCoordinatesTypes(args.detCoordinates, '-detCoordinates', errors) 268 | imgSize = (0, 0) 269 | if gtCoordType == CoordinatesType.Relative: # Image size is required 270 | imgSize = ValidateImageSize(args.imgSize, '-imgsize', '-gtCoordinates', errors) 271 | if detCoordType == CoordinatesType.Relative: # Image size is required 272 | imgSize = ValidateImageSize(args.imgSize, '-imgsize', '-detCoordinates', errors) 273 | # Detection folder 274 | if ValidateMandatoryArgs(args.detFolder, '-det/--detfolder', errors): 275 | detFolder = ValidatePaths(args.detFolder, '-det/--detfolder', errors) 276 | else: 277 | # errors.pop() 278 | detFolder = os.path.join(currentPath, 'detections') 279 | if os.path.isdir(detFolder) is False: 280 | errors.append('folder %s not found' % detFolder) 281 | # Validate savePath 282 | if args.savePath is not None: 283 | savePath = ValidatePaths(args.savePath, '-sp/--savepath', errors) 284 | else: 285 | savePath = os.path.join(currentPath, 'results') 286 | # If error, show error messages 287 | if len(errors) is not 0: 288 | print("""usage: Object Detection Metrics [-h] [-v] [-gt] [-det] [-t] [-gtformat] 289 | [-detformat] [-save]""") 290 | print('Object Detection Metrics: error(s): ') 291 | [print(e) for e in errors] 292 | sys.exit() 293 | 294 | # Create directory to save results 295 | shutil.rmtree(savePath, ignore_errors=True) # Clear folder 296 | os.makedirs(savePath) 297 | # Show plot during execution 298 | showPlot = args.showPlot 299 | 300 | # print('iouThreshold= %f' % iouThreshold) 301 | # print('savePath = %s' % savePath) 302 | # print('gtFormat = %s' % gtFormat) 303 | # print('detFormat = %s' % detFormat) 304 | # print('gtFolder = %s' % gtFolder) 305 | # print('detFolder = %s' % detFolder) 306 | # print('gtCoordType = %s' % gtCoordType) 307 | # print('detCoordType = %s' % detCoordType) 308 | # print('showPlot %s' % showPlot) 309 | 310 | # Get groundtruth boxes 311 | allBoundingBoxes, allClasses = getBoundingBoxes( 312 | gtFolder, True, gtFormat, gtCoordType, imgSize=imgSize) 313 | # Get detected boxes 314 | allBoundingBoxes, allClasses = getBoundingBoxes( 315 | detFolder, False, detFormat, detCoordType, allBoundingBoxes, allClasses, imgSize=imgSize) 316 | allClasses.sort() 317 | 318 | f = open(os.path.join(savePath, 'results.txt'), 'w') 319 | f.write('Object Detection Metrics\n') 320 | f.write('https://github.com/rafaelpadilla/Object-Detection-Metrics\n\n\n') 321 | f.write('Average Precision (AP), Precision and Recall per class:') 322 | 323 | evaluator = Evaluator() 324 | acc_AP = 0 325 | validClasses = 0 326 | # for each class 327 | for c in allClasses: 328 | # Plot Precision x Recall curve 329 | metricsPerClass = evaluator.PlotPrecisionRecallCurve( 330 | c, # Class to show 331 | allBoundingBoxes, # Object containing all bounding boxes (ground truths and detections) 332 | IOUThreshold=iouThreshold, # IOU threshold 333 | showAP=True, # Show Average Precision in the title of the plot 334 | showInterpolatedPrecision=False, # Don't plot the interpolated precision curve 335 | savePath=os.path.join(savePath, c + '.png'), 336 | showGraphic=showPlot) 337 | # Get metric values per each class 338 | cl = metricsPerClass['class'] 339 | ap = metricsPerClass['AP'] 340 | precision = metricsPerClass['precision'] 341 | recall = metricsPerClass['recall'] 342 | totalPositives = metricsPerClass['total positives'] 343 | total_TP = metricsPerClass['total TP'] 344 | total_FP = metricsPerClass['total FP'] 345 | 346 | if totalPositives > 0: 347 | validClasses = validClasses + 1 348 | acc_AP = acc_AP + ap 349 | prec = ['%.2f' % p for p in precision] 350 | rec = ['%.2f' % r for r in recall] 351 | ap_str = "{0:.2f}%".format(ap * 100) 352 | # ap_str = str('%.2f' % ap) #AQUI 353 | print('AP: %s (%s)' % (ap_str, cl)) 354 | f.write('\n\nClass: %s' % cl) 355 | f.write('\nAP: %s' % ap_str) 356 | f.write('\nPrecision: %s' % prec) 357 | f.write('\nRecall: %s' % rec) 358 | 359 | mAP = acc_AP / validClasses 360 | mAP_str = "{0:.2f}%".format(mAP * 100) 361 | print('mAP: %s' % mAP_str) 362 | f.write('\n\n\nmAP: %s' % mAP_str) 363 | f.close() 364 | -------------------------------------------------------------------------------- /lib/Evaluator.py: -------------------------------------------------------------------------------- 1 | ########################################################################################### 2 | # # 3 | # Evaluator class: Implements the most popular metrics for object detection # 4 | # # 5 | # Developed by: Rafael Padilla (rafael.padilla@smt.ufrj.br) # 6 | # SMT - Signal Multimedia and Telecommunications Lab # 7 | # COPPE - Universidade Federal do Rio de Janeiro # 8 | # Last modification: Oct 9th 2018 # 9 | ########################################################################################### 10 | 11 | import sys 12 | from collections import Counter 13 | 14 | import matplotlib.pyplot as plt 15 | import numpy as np 16 | 17 | from BoundingBox import * 18 | from BoundingBoxes import * 19 | from utils import * 20 | 21 | 22 | class Evaluator: 23 | def GetPascalVOCMetrics(self, 24 | boundingboxes, 25 | IOUThreshold=0.5, 26 | method=MethodAveragePrecision.EveryPointInterpolation): 27 | """Get the metrics used by the VOC Pascal 2012 challenge. 28 | Get 29 | Args: 30 | boundingboxes: Object of the class BoundingBoxes representing ground truth and detected 31 | bounding boxes; 32 | IOUThreshold: IOU threshold indicating which detections will be considered TP or FP 33 | (default value = 0.5); 34 | method (default = EveryPointInterpolation): It can be calculated as the implementation 35 | in the official PASCAL VOC toolkit (EveryPointInterpolation), or applying the 11-point 36 | interpolatio as described in the paper "The PASCAL Visual Object Classes(VOC) Challenge" 37 | or EveryPointInterpolation" (ElevenPointInterpolation); 38 | Returns: 39 | A list of dictionaries. Each dictionary contains information and metrics of each class. 40 | The keys of each dictionary are: 41 | dict['class']: class representing the current dictionary; 42 | dict['precision']: array with the precision values; 43 | dict['recall']: array with the recall values; 44 | dict['AP']: average precision; 45 | dict['interpolated precision']: interpolated precision values; 46 | dict['interpolated recall']: interpolated recall values; 47 | dict['total positives']: total number of ground truth positives; 48 | dict['total TP']: total number of True Positive detections; 49 | dict['total FP']: total number of False Negative detections; 50 | """ 51 | ret = [] # list containing metrics (precision, recall, average precision) of each class 52 | # List with all ground truths (Ex: [imageName,class,confidence=1, (bb coordinates XYX2Y2)]) 53 | groundTruths = [] 54 | # List with all detections (Ex: [imageName,class,confidence,(bb coordinates XYX2Y2)]) 55 | detections = [] 56 | # Get all classes 57 | classes = [] 58 | # Loop through all bounding boxes and separate them into GTs and detections 59 | for bb in boundingboxes.getBoundingBoxes(): 60 | # [imageName, class, confidence, (bb coordinates XYX2Y2)] 61 | if bb.getBBType() == BBType.GroundTruth: 62 | groundTruths.append([ 63 | bb.getImageName(), 64 | bb.getClassId(), 1, 65 | bb.getAbsoluteBoundingBox(BBFormat.XYX2Y2) 66 | ]) 67 | else: 68 | detections.append([ 69 | bb.getImageName(), 70 | bb.getClassId(), 71 | bb.getConfidence(), 72 | bb.getAbsoluteBoundingBox(BBFormat.XYX2Y2) 73 | ]) 74 | # get class 75 | if bb.getClassId() not in classes: 76 | classes.append(bb.getClassId()) 77 | classes = sorted(classes) 78 | # Precision x Recall is obtained individually by each class 79 | # Loop through by classes 80 | for c in classes: 81 | # Get only detection of class c 82 | dects = [] 83 | [dects.append(d) for d in detections if d[1] == c] 84 | # Get only ground truths of class c 85 | gts = [] 86 | [gts.append(g) for g in groundTruths if g[1] == c] 87 | npos = len(gts) 88 | # sort detections by decreasing confidence 89 | dects = sorted(dects, key=lambda conf: conf[2], reverse=True) 90 | TP = np.zeros(len(dects)) 91 | FP = np.zeros(len(dects)) 92 | # create dictionary with amount of gts for each image 93 | det = Counter([cc[0] for cc in gts]) 94 | for key, val in det.items(): 95 | det[key] = np.zeros(val) 96 | # print("Evaluating class: %s (%d detections)" % (str(c), len(dects))) 97 | # Loop through detections 98 | for d in range(len(dects)): 99 | # print('dect %s => %s' % (dects[d][0], dects[d][3],)) 100 | # Find ground truth image 101 | gt = [gt for gt in gts if gt[0] == dects[d][0]] 102 | iouMax = sys.float_info.min 103 | for j in range(len(gt)): 104 | # print('Ground truth gt => %s' % (gt[j][3],)) 105 | iou = Evaluator.iou(dects[d][3], gt[j][3]) 106 | if iou > iouMax: 107 | iouMax = iou 108 | jmax = j 109 | # Assign detection as true positive/don't care/false positive 110 | if iouMax >= IOUThreshold: 111 | if det[dects[d][0]][jmax] == 0: 112 | TP[d] = 1 # count as true positive 113 | # print("TP") 114 | det[dects[d][0]][jmax] = 1 # flag as already 'seen' 115 | # - A detected "cat" is overlaped with a GT "cat" with IOU >= IOUThreshold. 116 | else: 117 | FP[d] = 1 # count as false positive 118 | # print("FP") 119 | # compute precision, recall and average precision 120 | acc_FP = np.cumsum(FP) 121 | acc_TP = np.cumsum(TP) 122 | rec = acc_TP / npos 123 | prec = np.divide(acc_TP, (acc_FP + acc_TP)) 124 | # Depending on the method, call the right implementation 125 | if method == MethodAveragePrecision.EveryPointInterpolation: 126 | [ap, mpre, mrec, ii] = Evaluator.CalculateAveragePrecision(rec, prec) 127 | else: 128 | [ap, mpre, mrec, _] = Evaluator.ElevenPointInterpolatedAP(rec, prec) 129 | # add class result in the dictionary to be returned 130 | r = { 131 | 'class': c, 132 | 'precision': prec, 133 | 'recall': rec, 134 | 'AP': ap, 135 | 'interpolated precision': mpre, 136 | 'interpolated recall': mrec, 137 | 'total positives': npos, 138 | 'total TP': np.sum(TP), 139 | 'total FP': np.sum(FP) 140 | } 141 | ret.append(r) 142 | return ret 143 | 144 | def PlotPrecisionRecallCurve(self, 145 | classId, 146 | boundingBoxes, 147 | IOUThreshold=0.5, 148 | method=MethodAveragePrecision.EveryPointInterpolation, 149 | showAP=False, 150 | showInterpolatedPrecision=False, 151 | savePath=None, 152 | showGraphic=True): 153 | """PlotPrecisionRecallCurve 154 | Plot the Precision x Recall curve for a given class. 155 | Args: 156 | classId: The class that will be plot; 157 | boundingBoxes: Object of the class BoundingBoxes representing ground truth and detected 158 | bounding boxes; 159 | IOUThreshold (optional): IOU threshold indicating which detections will be considered 160 | TP or FP (default value = 0.5); 161 | method (default = EveryPointInterpolation): It can be calculated as the implementation 162 | in the official PASCAL VOC toolkit (EveryPointInterpolation), or applying the 11-point 163 | interpolatio as described in the paper "The PASCAL Visual Object Classes(VOC) Challenge" 164 | or EveryPointInterpolation" (ElevenPointInterpolation). 165 | showAP (optional): if True, the average precision value will be shown in the title of 166 | the graph (default = False); 167 | showInterpolatedPrecision (optional): if True, it will show in the plot the interpolated 168 | precision (default = False); 169 | savePath (optional): if informed, the plot will be saved as an image in this path 170 | (ex: /home/mywork/ap.png) (default = None); 171 | showGraphic (optional): if True, the plot will be shown (default = True) 172 | Returns: 173 | A dictionary containing information and metric about the class. The keys of the 174 | dictionary are: 175 | dict['class']: class representing the current dictionary; 176 | dict['precision']: array with the precision values; 177 | dict['recall']: array with the recall values; 178 | dict['AP']: average precision; 179 | dict['interpolated precision']: interpolated precision values; 180 | dict['interpolated recall']: interpolated recall values; 181 | dict['total positives']: total number of ground truth positives; 182 | dict['total TP']: total number of True Positive detections; 183 | dict['total FP']: total number of False Negative detections; 184 | """ 185 | results = self.GetPascalVOCMetrics(boundingBoxes, IOUThreshold, method) 186 | result = None 187 | for res in results: 188 | if res['class'] == classId: 189 | result = res 190 | break 191 | if result is None: 192 | raise IOError('Error: Class %d could not be found.' % classId) 193 | 194 | precision = result['precision'] 195 | recall = result['recall'] 196 | average_precision = result['AP'] 197 | mpre = result['interpolated precision'] 198 | mrec = result['interpolated recall'] 199 | npos = result['total positives'] 200 | total_tp = result['total TP'] 201 | total_fp = result['total FP'] 202 | 203 | if showInterpolatedPrecision: 204 | if method == MethodAveragePrecision.EveryPointInterpolation: 205 | plt.plot(mrec, mpre, '--r', label='Interpolated precision (every point)') 206 | elif method == MethodAveragePrecision.ElevenPointInterpolation: 207 | # Uncomment the line below if you want to plot the area 208 | # plt.plot(mrec, mpre, 'or', label='11-point interpolated precision') 209 | # Remove duplicates, getting only the highest precision of each recall value 210 | nrec = [] 211 | nprec = [] 212 | for idx in range(len(mrec)): 213 | r = mrec[idx] 214 | if r not in nrec: 215 | idxEq = np.argwhere(mrec == r) 216 | nrec.append(r) 217 | nprec.append(max([mpre[int(id)] for id in idxEq])) 218 | plt.plot(nrec, nprec, 'or', label='11-point interpolated precision') 219 | plt.plot(recall, precision, label='Precision') 220 | plt.xlabel('recall') 221 | plt.ylabel('precision') 222 | if showAP: 223 | ap_str = "{0:.2f}%".format(average_precision * 100) 224 | plt.title('Precision x Recall curve \nClass: %s, AP: %s' % (str(classId), ap_str)) 225 | # plt.title('Precision x Recall curve \nClass: %s, AP: %.4f' % (str(classId), 226 | # average_precision)) 227 | else: 228 | plt.title('Precision x Recall curve \nClass: %d' % classId) 229 | plt.legend(shadow=True) 230 | plt.grid() 231 | ############################################################ 232 | # Uncomment the following block to create plot with points # 233 | ############################################################ 234 | # plt.plot(recall, precision, 'bo') 235 | # labels = ['R', 'Y', 'J', 'A', 'U', 'C', 'M', 'F', 'D', 'B', 'H', 'P', 'E', 'X', 'N', 'T', 236 | # 'K', 'Q', 'V', 'I', 'L', 'S', 'G', 'O'] 237 | # dicPosition = {} 238 | # dicPosition['left_zero'] = (-30,0) 239 | # dicPosition['left_zero_slight'] = (-30,-10) 240 | # dicPosition['right_zero'] = (30,0) 241 | # dicPosition['left_up'] = (-30,20) 242 | # dicPosition['left_down'] = (-30,-25) 243 | # dicPosition['right_up'] = (20,20) 244 | # dicPosition['right_down'] = (20,-20) 245 | # dicPosition['up_zero'] = (0,30) 246 | # dicPosition['up_right'] = (0,30) 247 | # dicPosition['left_zero_long'] = (-60,-2) 248 | # dicPosition['down_zero'] = (-2,-30) 249 | # vecPositions = [ 250 | # dicPosition['left_down'], 251 | # dicPosition['left_zero'], 252 | # dicPosition['right_zero'], 253 | # dicPosition['right_zero'], #'R', 'Y', 'J', 'A', 254 | # dicPosition['left_up'], 255 | # dicPosition['left_up'], 256 | # dicPosition['right_up'], 257 | # dicPosition['left_up'], # 'U', 'C', 'M', 'F', 258 | # dicPosition['left_zero'], 259 | # dicPosition['right_up'], 260 | # dicPosition['right_down'], 261 | # dicPosition['down_zero'], #'D', 'B', 'H', 'P' 262 | # dicPosition['left_up'], 263 | # dicPosition['up_zero'], 264 | # dicPosition['right_up'], 265 | # dicPosition['left_up'], # 'E', 'X', 'N', 'T', 266 | # dicPosition['left_zero'], 267 | # dicPosition['right_zero'], 268 | # dicPosition['left_zero_long'], 269 | # dicPosition['left_zero_slight'], # 'K', 'Q', 'V', 'I', 270 | # dicPosition['right_down'], 271 | # dicPosition['left_down'], 272 | # dicPosition['right_up'], 273 | # dicPosition['down_zero'] 274 | # ] # 'L', 'S', 'G', 'O' 275 | # for idx in range(len(labels)): 276 | # box = dict(boxstyle='round,pad=.5',facecolor='yellow',alpha=0.5) 277 | # plt.annotate(labels[idx], 278 | # xy=(recall[idx],precision[idx]), xycoords='data', 279 | # xytext=vecPositions[idx], textcoords='offset points', 280 | # arrowprops=dict(arrowstyle="->", connectionstyle="arc3"), 281 | # bbox=box) 282 | if savePath is not None: 283 | plt.savefig(savePath) 284 | if showGraphic is True: 285 | plt.show() 286 | # plt.waitforbuttonpress() 287 | ret = {} 288 | ret['class'] = classId 289 | ret['precision'] = precision 290 | ret['recall'] = recall 291 | ret['AP'] = average_precision 292 | ret['interpolated precision'] = mpre 293 | ret['interpolated recall'] = mrec 294 | ret['total positives'] = npos 295 | ret['total TP'] = total_tp 296 | ret['total FP'] = total_fp 297 | return ret 298 | 299 | @staticmethod 300 | def CalculateAveragePrecision(rec, prec): 301 | mrec = [] 302 | mrec.append(0) 303 | [mrec.append(e) for e in rec] 304 | mrec.append(1) 305 | mpre = [] 306 | mpre.append(0) 307 | [mpre.append(e) for e in prec] 308 | mpre.append(0) 309 | for i in range(len(mpre) - 1, 0, -1): 310 | mpre[i - 1] = max(mpre[i - 1], mpre[i]) 311 | ii = [] 312 | for i in range(len(mrec) - 1): 313 | if mrec[1:][i] != mrec[0:-1][i]: 314 | ii.append(i + 1) 315 | ap = 0 316 | for i in ii: 317 | ap = ap + np.sum((mrec[i] - mrec[i - 1]) * mpre[i]) 318 | # return [ap, mpre[1:len(mpre)-1], mrec[1:len(mpre)-1], ii] 319 | return [ap, mpre[0:len(mpre) - 1], mrec[0:len(mpre) - 1], ii] 320 | 321 | @staticmethod 322 | # 11-point interpolated average precision 323 | def ElevenPointInterpolatedAP(rec, prec): 324 | # def CalculateAveragePrecision2(rec, prec): 325 | mrec = [] 326 | # mrec.append(0) 327 | [mrec.append(e) for e in rec] 328 | # mrec.append(1) 329 | mpre = [] 330 | # mpre.append(0) 331 | [mpre.append(e) for e in prec] 332 | # mpre.append(0) 333 | recallValues = np.linspace(0, 1, 11) 334 | recallValues = list(recallValues[::-1]) 335 | rhoInterp = [] 336 | recallValid = [] 337 | # For each recallValues (0, 0.1, 0.2, ... , 1) 338 | for r in recallValues: 339 | # Obtain all recall values higher or equal than r 340 | argGreaterRecalls = np.argwhere(mrec[:-1] >= r) 341 | pmax = 0 342 | # If there are recalls above r 343 | if argGreaterRecalls.size != 0: 344 | pmax = max(mpre[argGreaterRecalls.min():]) 345 | recallValid.append(r) 346 | rhoInterp.append(pmax) 347 | # By definition AP = sum(max(precision whose recall is above r))/11 348 | ap = sum(rhoInterp) / 11 349 | # Generating values for the plot 350 | rvals = [] 351 | rvals.append(recallValid[0]) 352 | [rvals.append(e) for e in recallValid] 353 | rvals.append(0) 354 | pvals = [] 355 | pvals.append(0) 356 | [pvals.append(e) for e in rhoInterp] 357 | pvals.append(0) 358 | # rhoInterp = rhoInterp[::-1] 359 | cc = [] 360 | for i in range(len(rvals)): 361 | p = (rvals[i], pvals[i - 1]) 362 | if p not in cc: 363 | cc.append(p) 364 | p = (rvals[i], pvals[i]) 365 | if p not in cc: 366 | cc.append(p) 367 | recallValues = [i[0] for i in cc] 368 | rhoInterp = [i[1] for i in cc] 369 | return [ap, rhoInterp, recallValues, None] 370 | 371 | # For each detections, calculate IOU with reference 372 | @staticmethod 373 | def _getAllIOUs(reference, detections): 374 | ret = [] 375 | bbReference = reference.getAbsoluteBoundingBox(BBFormat.XYX2Y2) 376 | # img = np.zeros((200,200,3), np.uint8) 377 | for d in detections: 378 | bb = d.getAbsoluteBoundingBox(BBFormat.XYX2Y2) 379 | iou = Evaluator.iou(bbReference, bb) 380 | # Show blank image with the bounding boxes 381 | # img = add_bb_into_image(img, d, color=(255,0,0), thickness=2, label=None) 382 | # img = add_bb_into_image(img, reference, color=(0,255,0), thickness=2, label=None) 383 | ret.append((iou, reference, d)) # iou, reference, detection 384 | # cv2.imshow("comparing",img) 385 | # cv2.waitKey(0) 386 | # cv2.destroyWindow("comparing") 387 | return sorted(ret, key=lambda i: i[0], reverse=True) # sort by iou (from highest to lowest) 388 | 389 | @staticmethod 390 | def iou(boxA, boxB): 391 | # if boxes dont intersect 392 | if Evaluator._boxesIntersect(boxA, boxB) is False: 393 | return 0 394 | interArea = Evaluator._getIntersectionArea(boxA, boxB) 395 | union = Evaluator._getUnionAreas(boxA, boxB, interArea=interArea) 396 | # intersection over union 397 | iou = interArea / union 398 | assert iou >= 0 399 | return iou 400 | 401 | # boxA = (Ax1,Ay1,Ax2,Ay2) 402 | # boxB = (Bx1,By1,Bx2,By2) 403 | @staticmethod 404 | def _boxesIntersect(boxA, boxB): 405 | if boxA[0] > boxB[2]: 406 | return False # boxA is right of boxB 407 | if boxB[0] > boxA[2]: 408 | return False # boxA is left of boxB 409 | if boxA[3] < boxB[1]: 410 | return False # boxA is above boxB 411 | if boxA[1] > boxB[3]: 412 | return False # boxA is below boxB 413 | return True 414 | 415 | @staticmethod 416 | def _getIntersectionArea(boxA, boxB): 417 | xA = max(boxA[0], boxB[0]) 418 | yA = max(boxA[1], boxB[1]) 419 | xB = min(boxA[2], boxB[2]) 420 | yB = min(boxA[3], boxB[3]) 421 | # intersection area 422 | return (xB - xA + 1) * (yB - yA + 1) 423 | 424 | @staticmethod 425 | def _getUnionAreas(boxA, boxB, interArea=None): 426 | area_A = Evaluator._getArea(boxA) 427 | area_B = Evaluator._getArea(boxB) 428 | if interArea is None: 429 | interArea = Evaluator._getIntersectionArea(boxA, boxB) 430 | return float(area_A + area_B - interArea) 431 | 432 | @staticmethod 433 | def _getArea(box): 434 | return (box[2] - box[0] + 1) * (box[3] - box[1] + 1) 435 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

2 | 3 |

4 | 5 | # Metrics for object detection 6 | 7 | The motivation of this project is the lack of consensus used by different works and implementations concerning the **evaluation metrics of the object detection problem**. Although on-line competitions use their own metrics to evaluate the task of object detection, just some of them offer reference code snippets to calculate the accuracy of the detected objects. 8 | Researchers who want to evaluate their work using different datasets than those offered by the competitions, need to implement their own version of the metrics. Sometimes a wrong or different implementation can create different and biased results. Ideally, in order to have trustworthy benchmarking among different approaches, it is necessary to have a flexible implementation that can be used by everyone regardless the dataset used. 9 | 10 | **This project provides easy-to-use functions implementing the same metrics used by the the most popular competitions of object detection**. Our implementation does not require modifications of your detection model to complicated input formats, avoiding conversions to XML or JSON files. We simplified the input data (ground truth bounding boxes and detected bounding boxes) and gathered in a single project the main metrics used by the academia and challenges. Our implementation was carefully compared against the official implementations and our results are exactly the same. 11 | 12 | In the topics below you can find an overview of the most popular metrics used in different competitions and works, as well as samples showing how to use our code. 13 | 14 | ## Table of contents 15 | 16 | - [Motivation](#metrics-for-object-detection) 17 | - [Different competitions, different metrics](#different-competitions-different-metrics) 18 | - [Important definitions](#important-definitions) 19 | - [Metrics](#metrics) 20 | - [Precision x Recall curve](#precision-x-recall-curve) 21 | - [Average Precision](#average-precision) 22 | - [11-point interpolation](#11-point-interpolation) 23 | - [Interpolating all points](#interpolating-all-points) 24 | - [**How to use this project**](#how-to-use-this-project) 25 | - [References](#references) 26 | 27 | 28 | ## Different competitions, different metrics 29 | 30 | * **[PASCAL VOC Challenge](http://host.robots.ox.ac.uk/pascal/VOC/)** offers a Matlab script in order to evaluate the quality of the detected objects. Participants of the competition can use the provided Matlab script to measure the accuracy of their detections before submitting their results. The official documentation explaining their criteria for object detection metrics can be accessed [here](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/htmldoc/devkit_doc.html#SECTION00050000000000000000). The current metrics used by the current PASCAL VOC object detection challenge are the **Precision x Recall curve** and **Average Precision**. 31 | The PASCAL VOC Matlab evaluation code reads the ground truth bounding boxes from XML files, requiring changes in the code if you want to apply it to other datasets or to your speficic cases. Even though projects such as [Faster-RCNN](https://github.com/rbgirshick/py-faster-rcnn) implement PASCAL VOC evaluation metrics, it is also necessary to convert the detected bounding boxes into their specific format. [Tensorflow](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/evaluation_protocols.md) framework also has their PASCAL VOC metrics implementation. 32 | 33 | * **[COCO Detection Challenge](https://competitions.codalab.org/competitions/5181)** uses different metrics to evaluate the accuracy of object detection of different algorithms. [Here](http://cocodataset.org/#detection-eval) you can find a documentation explaining the 12 metrics used for characterizing the performance of an object detector on COCO. This competition offers Python and Matlab codes so users can verify their scores before submitting the results. It is also necessary to convert the results to a [format](http://cocodataset.org/#format-results) required by the competition. 34 | 35 | * **[Google Open Images Dataset V4 Competition](https://storage.googleapis.com/openimages/web/challenge.html)** also uses mean Average Precision (mAP) over the 500 classes to evaluate the object detection task. 36 | 37 | * **[ImageNet Object Localization Challenge](https://www.kaggle.com/c/imagenet-object-detection-challenge)** defines an error for each image considering the class and the overlapping region between ground truth and detected boxes. The total error is computed as the average of all min errors among all test dataset images. [Here](https://www.kaggle.com/c/imagenet-object-localization-challenge#evaluation) are more details about their evaluation method. 38 | 39 | ## Important definitions 40 | 41 | ### Intersection Over Union (IOU) 42 | 43 | Intersection Over Union (IOU) is measure based on Jaccard Index that evaluates the overlap between two bounding boxes. It requires a ground truth bounding box ![](http://latex.codecogs.com/gif.latex?B_%7Bgt%7D) and a predicted bounding box ![](http://latex.codecogs.com/gif.latex?B_p). By applying the IOU we can tell if a detection is valid (True Positive) or not (False Positive). 44 | IOU is given by the overlapping area between the predicted bounding box and the ground truth bounding box divided by the area of union between them:   45 | 46 |

47 | 48 |

49 | 50 | The image below illustrates the IOU between a ground truth bounding box (in green) and a detected bounding box (in red). 51 | 52 | 53 |

54 |

55 | 56 | ### True Positive, False Positive, False Negative and True Negative 57 | 58 | Some basic concepts used by the metrics: 59 | 60 | * **True Positive (TP)**: A correct detection. Detection with IOU ≥ _threshold_ 61 | * **False Positive (FP)**: A wrong detection. Detection with IOU < _threshold_ 62 | * **False Negative (FN)**: A ground truth not detected 63 | * **True Negative (TN)**: Does not apply. It would represent a corrected misdetection. In the object detection task there are many possible bounding boxes that should not be detected within an image. Thus, TN would be all possible bounding boxes that were corrrectly not detected (so many possible boxes within an image). That's why it is not used by the metrics. 64 | 65 | _threshold_: depending on the metric, it is usually set to 50%, 75% or 95%. 66 | 67 | ### Precision 68 | 69 | Precision is the ability of a model to identify **only** the relevant objects. It is the percentage of correct positive predictions and is given by: 70 | 71 |

72 | 73 |

74 | 75 | ### Recall 76 | 77 | Recall is the ability of a model to find all the relevant cases (all ground truth bounding boxes). It is the percentage of true positive detected among all relevant ground truths and is given by: 78 | 79 |

80 | 81 |

82 | 83 | ## Metrics 84 | 85 | In the topics below there are some comments on the most popular metrics used for object detection. 86 | 87 | ### Precision x Recall curve 88 | 89 | The Precision x Recall curve is a good way to evaluate the performance of an object detector as the confidence is changed by plotting a curve for each object class. An object detector of a particular class is considered good if its precision stays high as recall increases, which means that if you vary the confidence threshold, the precision and recall will still be high. Another way to identify a good object detector is to look for a detector that can identify only relevant objects (0 False Positives = high precision), finding all ground truth objects (0 False Negatives = high recall). 90 | 91 | A poor object detector needs to increase the number of detected objects (increasing False Positives = lower precision) in order to retrieve all ground truth objects (high recall). That's why the Precision x Recall curve usually starts with high precision values, decreasing as recall increases. You can see an example of the Prevision x Recall curve in the next topic (Average Precision). This kind of curve is used by the PASCAL VOC 2012 challenge and is available in our implementation. 92 | 93 | ### Average Precision 94 | 95 | Another way to compare the performance of object detectors is to calculate the area under the curve (AUC) of the Precision x Recall curve. As AP curves are often zigzag curves going up and down, comparing different curves (different detectors) in the same plot usually is not an easy task - because the curves tend to cross each other much frequently. That's why Average Precision (AP), a numerical metric, can also help us compare different detectors. In practice AP is the precision averaged across all recall values between 0 and 1. 96 | 97 | From 2010 on, the method of computing AP by the PASCAL VOC challenge has changed. Currently, **the interpolation performed by PASCAL VOC challenge uses all data points, rather than interpolating only 11 equally spaced points as stated in their [paper](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.157.5766&rep=rep1&type=pdf)**. As we want to reproduce their default implementation, our default code (as seen further) follows their most recent application (interpolating all data points). However, we also offer the 11-point interpolation approach. 98 | 99 | #### 11-point interpolation 100 | 101 | The 11-point interpolation tries to summarize the shape of the Precision x Recall curve by averaging the precision at a set of eleven equally spaced recall levels [0, 0.1, 0.2, ... , 1]: 102 | 103 |

104 | 105 |

106 | 107 | with 108 | 109 |

110 | 111 |

112 | 113 | where ![](http://latex.codecogs.com/gif.latex?%5Crho%5Cleft%20%28%20%5Ctilde%7Br%7D%20%5Cright%20%29) is the measured precision at recall ![](http://latex.codecogs.com/gif.latex?%5Ctilde%7Br%7D). 114 | 115 | Instead of using the precision observed at each point, the AP is obtained by interpolating the precision only at the 11 levels ![](http://latex.codecogs.com/gif.latex?r) taking the **maximum precision whose recall value is greater than ![](http://latex.codecogs.com/gif.latex?r)**. 116 | 117 | #### Interpolating all points 118 | 119 | Instead of interpolating only in the 11 equally spaced points, you could interpolate through all points in such way that: 120 | 121 |

122 | 123 |

124 | 125 | 126 | with 127 | 128 |

129 | 130 |

131 | 132 | 133 | where ![](http://latex.codecogs.com/gif.latex?%5Crho%5Cleft%20%28%20%5Ctilde%7Br%7D%20%5Cright%20%29) is the measured precision at recall ![](http://latex.codecogs.com/gif.latex?%5Ctilde%7Br%7D). 134 | 135 | In this case, instead of using the precision observed at only few points, the AP is now obtained by interpolating the precision at **each level**, ![](http://latex.codecogs.com/gif.latex?r) taking the **maximum precision whose recall value is greater or equal than ![](http://latex.codecogs.com/gif.latex?r+1)**. This way we calculate the estimated area under the curve. 136 | 137 | To make things more clear, we provided an example comparing both interpolations. 138 | 139 | 140 | #### An ilustrated example 141 | 142 | An example helps us understand better the concept of the interpolated average precision. Consider the detections below: 143 | 144 | 145 |

146 |

147 | 148 | There are 7 images with 15 ground truth objects representented by the green bounding boxes and 24 detected objects represented by the red bounding boxes. Each detected object has a confidence level and is identified by a letter (A,B,...,Y). 149 | 150 | The following table shows the bounding boxes with their corresponding confidences. The last column identifies the detections as TP or FP. In this example a TP is considered if IOU ![](http://latex.codecogs.com/gif.latex?%5Cgeq) 30%, otherwise it is a FP. By looking at the images above we can roughly tell if the detections are TP or FP. 151 | 152 | 153 |

154 |

155 | 156 | 184 | 185 | In some images there are more than one detection overlapping a ground truth (Images 2, 3, 4, 5, 6 and 7). For those cases the detection with the highest IOU is taken, discarding the other detections. This rule is applied by the PASCAL VOC 2012 metric: "e.g. 5 detections (TP) of a single object is counted as 1 correct detection and 4 false detections”. 186 | 187 | The Precision x Recall curve is plotted by calculating the precision and recall values of the accumulated TP or FP detections. For this, first we need to order the detections by their confidences, then we calculate the precision and recall for each accumulated detection as shown in the table below: 188 | 189 | 190 |

191 |

192 | 193 | 221 | 222 | Plotting the precision and recall values we have the following *Precision x Recall curve*: 223 | 224 | 225 |

226 | 227 |

228 | 229 | As mentioned before, there are two different ways to measure the interpolted average precision: **11-point interpolation** and **interpolating all points**. Below we make a comparisson between them: 230 | 231 | #### Calculating the 11-point interpolation 232 | 233 | The idea of the 11-point interpolated average precision is to average the precisions at a set of 11 recall levels (0,0.1,...,1). The interpolated precision values are obtained by taking the maximum precision whose recall value is greater than its current recall value as follows: 234 | 235 | 236 |

237 | 238 |

239 | 240 | By applying the 11-point interpolation, we have: 241 | 242 | ![](http://latex.codecogs.com/gif.latex?AP%20%3D%20%5Cfrac%7B1%7D%7B11%7D%5Csum_%7Br%5Cin%5C%7B0%2C0.1%2C...%2C1%5C%7D%7D%5Crho_%7B%5Ctext%7Binterp%7D%5Cleft%20%28r%5Cright%20%29%7D) 243 | ![](http://latex.codecogs.com/gif.latex?AP%20%3D%20%5Cfrac%7B1%7D%7B11%7D%20%5Cleft%20%28%201+0.6666+0.4285+0.4285+0.4285+0+0+0+0+0+0%20%5Cright%20%29) 244 | ![](http://latex.codecogs.com/gif.latex?AP%20%3D%2026.84%5C%25) 245 | 246 | 247 | #### Calculating the interpolation performed in all points 248 | 249 | By interpolating all points, the Average Precision (AP) can be interpreted as an approximated AUC of the Precision x Recall curve. The intention is to reduce the impact of the wiggles in the curve. By applying the equations presented before, we can obtain the areas as it will be demostrated here. We could also visually have the interpolated precision points by looking at the recalls starting from the highest (0.4666) to 0 (looking at the plot from right to left) and, as we decrease the recall, we collect the precision values that are the highest as shown in the image below: 250 | 251 | 252 |

253 | 254 |

255 | 256 | Looking at the plot above, we can divide the AUC into 4 areas (A1, A2, A3 and A4): 257 | 258 | 259 |

260 | 261 |

262 | 263 | Calculating the total area, we have the AP: 264 | 265 | ![](http://latex.codecogs.com/gif.latex?AP%20%3D%20A1%20+%20A2%20+%20A3%20+%20A4) 266 | 267 | ![](http://latex.codecogs.com/gif.latex?%5Ctext%7Bwith%3A%7D) 268 | ![](http://latex.codecogs.com/gif.latex?A1%20%3D%20%280.0666-0%29%5Ctimes1%20%3D%5Cmathbf%7B0.0666%7D) 269 | ![](http://latex.codecogs.com/gif.latex?A2%20%3D%20%280.1333-0.0666%29%5Ctimes0.6666%3D%5Cmathbf%7B0.04446222%7D) 270 | ![](http://latex.codecogs.com/gif.latex?A3%20%3D%20%280.4-0.1333%29%5Ctimes0.4285%20%3D%5Cmathbf%7B0.11428095%7D) 271 | ![](http://latex.codecogs.com/gif.latex?A4%20%3D%20%280.4666-0.4%29%5Ctimes0.3043%20%3D%5Cmathbf%7B0.02026638%7D) 272 | 273 | ![](http://latex.codecogs.com/gif.latex?AP%20%3D%200.0666+0.04446222+0.11428095+0.02026638) 274 | ![](http://latex.codecogs.com/gif.latex?AP%20%3D%200.24560955) 275 | ![](http://latex.codecogs.com/gif.latex?AP%20%3D%20%5Cmathbf%7B24.56%5C%25%7D) 276 | 277 | The results between the two different interpolation methods are a little different: 24.56% and 26.84% by the every point interpolation and the 11-point interpolation respectively. 278 | 279 | Our default implementation is the same as VOC PASCAL: every point interpolation. If you want to use the 11-point interpolation, change the functions that use the argument ```method=MethodAveragePrecision.EveryPointInterpolation``` to ```method=MethodAveragePrecision.ElevenPointInterpolation```. 280 | 281 | If you want to reproduce these results, see the **[Sample 2](https://github.com/rafaelpadilla/Object-Detection-Metrics/tree/master/samples/sample_2/)**. 282 | 283 | 284 | ## How to use this project 285 | 286 | This project was created to evaluate your detections in a very easy way. If you want to evaluate your algorithm with the most used object detection metrics, you are in the right place. 287 | 288 | [Sample_1](https://github.com/rafaelpadilla/Object-Detection-Metrics/tree/master/samples/sample_1) and [sample_2](https://github.com/rafaelpadilla/Object-Detection-Metrics/tree/master/samples/sample_2) are practical examples demonstrating how to access directly the core functions of this project, providing more flexibility on the usage of the metrics. But if you don't want to spend your time understanding our code, see the instructions below to easily evaluate your detections: 289 | 290 | Follow the steps below to start evaluating your detections: 291 | 292 | 1. [Create the ground truth files](#create-the-ground-truth-files) 293 | 2. [Create your detection files](#create-your-detection-files) 294 | 3. For **Pascal VOC metrics**, run the command: `python pascalvoc.py` 295 | If you want to reproduce the example above, run the command: `python pascalvoc.py -t 0.3` 296 | 4. (Optional) [You can use arguments to control the IOU threshold, bounding boxes format, etc.](#optional-arguments) 297 | 298 | ### Create the ground truth files 299 | 300 | - Create a separate ground truth text file for each image in the folder **groundtruths/**. 301 | - In these files each line should be in the format: ` `. 302 | - E.g. The ground truth bounding boxes of the image "2008_000034.jpg" are represented in the file "2008_000034.txt": 303 | ``` 304 | bottle 6 234 45 362 305 | person 1 156 103 336 306 | person 36 111 198 416 307 | person 91 42 338 500 308 | ``` 309 | 310 | If you prefer, you can also have your bounding boxes in the format: ` ` (see here [**\***](#asterisk) how to use it). In this case, your "2008_000034.txt" would be represented as: 311 | ``` 312 | bottle 6 234 39 128 313 | person 1 156 102 180 314 | person 36 111 162 305 315 | person 91 42 247 458 316 | ``` 317 | 318 | ### Create your detection files 319 | 320 | - Create a separate detection text file for each image in the folder **detections/**. 321 | - The names of the detection files must match their correspond ground truth (e.g. "detections/2008_000182.txt" represents the detections of the ground truth: "groundtruths/2008_000182.txt"). 322 | - In these files each line should be in the following format: ` ` (see here [**\***](#asterisk) how to use it). 323 | - E.g. "2008_000034.txt": 324 | ``` 325 | bottle 0.14981 80 1 295 500 326 | bus 0.12601 36 13 404 316 327 | horse 0.12526 430 117 500 307 328 | pottedplant 0.14585 212 78 292 118 329 | tvmonitor 0.070565 388 89 500 196 330 | ``` 331 | 332 | Also if you prefer, you could have your bounding boxes in the format: ` `. 333 | 334 | ### Optional arguments 335 | 336 | Optional arguments: 337 | 338 | | Argument                          | Description | Example | Default | 339 | |:-------------:|:-----------:|:-----------:|:-----------:| 340 | | `-h`,
`--help ` | show help message | `python pascalvoc.py -h` | | 341 | | `-v`,
`--version` | check version | `python pascalvoc.py -v` | | 342 | | `-gt`,
`--gtfolder` | folder that contains the ground truth bounding boxes files | `python pascalvoc.py -gt /home/whatever/my_groundtruths/` | `/Object-Detection-Metrics/groundtruths`| 343 | | `-det`,
`--detfolder` | folder that contains your detected bounding boxes files | `python pascalvoc.py -det /home/whatever/my_detections/` | `/Object-Detection-Metrics/detections/`| 344 | | `-t`,
`--threshold` | IOU thershold that tells if a detection is TP or FP | `python pascalvoc.py -t 0.75` | `0.50` | 345 | | `-gtformat` | format of the coordinates of the ground truth bounding boxes [**\***](#asterisk) | `python pascalvoc.py -gtformat xyrb` | `xywh` | 346 | | `-detformat` | format of the coordinates of the detected bounding boxes [**\***](#asterisk) | `python pascalvoc.py -detformat xyrb` | `xywh` | | 347 | | `-gtcoords` | reference of the ground truth bounding bounding box coordinates.
If the annotated coordinates are relative to the image size (as used in YOLO), set it to `rel`.
If the coordinates are absolute values, not depending to the image size, set it to `abs` | `python pascalvoc.py -gtcoords rel` | `abs` | 348 | | `-detcoords` | reference of the detected bounding bounding box coordinates.
If the coordinates are relative to the image size (as used in YOLO), set it to `rel`.
If the coordinates are absolute values, not depending to the image size, set it to `abs` | `python pascalvoc.py -detcoords rel` | `abs` | 349 | | `-imgsize ` | image size in the format `width,height` .
Required if `-gtcoords` or `-detcoords` is set to `rel` | `python pascalvoc.py -imgsize 600,400` | 350 | | `-sp`,
`--savepath` | folder where the plots are saved | `python pascalvoc.py -sp /home/whatever/my_results/` | `Object-Detection-Metrics/results/` | 351 | | `-np`,
`--noplot` | if present no plot is shown during execution | `python pascalvoc.py -np` | not presented.
Therefore, plots are shown | 352 | 353 | 354 | (**\***) set `-gtformat=xywh` and/or `-detformat=xywh` if format is ` `. Set to `-gtformat=xyrb` and/or `-detformat=xyrb` if format is ` `. 355 | 356 | ## References 357 | 358 | * The Relationship Between Precision-Recall and ROC Curves (Jesse Davis and Mark Goadrich) 359 | Department of Computer Sciences and Department of Biostatistics and Medical Informatics, University of 360 | Wisconsin 361 | http://pages.cs.wisc.edu/~jdavis/davisgoadrichcamera2.pdf 362 | 363 | * The PASCAL Visual Object Classes (VOC) Challenge 364 | http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.157.5766&rep=rep1&type=pdf 365 | 366 | * Evaluation of ranked retrieval results (Salton and Mcgill 1986) 367 | https://www.amazon.com/Introduction-Information-Retrieval-COMPUTER-SCIENCE/dp/0070544840 368 | https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html 369 | --------------------------------------------------------------------------------