├── .gitignore ├── CHANGELOG.md ├── README.md ├── annotation-tool ├── README.md ├── TaTo.py ├── config.json ├── config_distraction.json ├── config_drowsiness.json ├── config_gaze.json ├── config_hands.json ├── config_statics.json ├── setUp.py └── vcd4parser.py ├── docs ├── imgs │ ├── annotation_tool_info.png │ ├── block_annotation.png │ ├── colorized_depth.png │ ├── level_panel.png │ └── mobaxterm_config.png ├── issue_bug_template.md ├── issue_feature_template.md ├── readme-assets │ ├── cameras.png │ ├── dmdStructure.png │ ├── environments.png │ ├── gazeRegions.png │ ├── mosaic.png │ └── participants.png ├── setup_linux.md └── setup_windows.md └── exploreMaterial-tool ├── DExTool.py ├── README.md ├── Tutorial_DEx_(dataset_explorer_tool).ipynb ├── accessDMDAnn.py ├── config_DEx.json ├── group_split_material.py ├── statistics.py └── vcd4reader.py /.gitignore: -------------------------------------------------------------------------------- 1 | dmd/ 2 | dmd_rgb/ -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | All notable changes to the DMD repository will be documented in this file. 4 | 5 | The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 6 | and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 7 | 8 | ## [Unreleased] 9 | 10 | ## [1.0.0] - 2020-07-22 11 | 12 | ### Added 13 | 14 | - First version of annotation tool (TaTo). 15 | - New Readme files and steps to run the annotation tool. 16 | - The wiki includes the DMD file structure and annotation instructions for distraction related actions. 17 | 18 | [unreleased]: https://github.com/Vicomtech/DMD-Driver-Monitoring-Dataset/compare/v1.0.0...HEsAD 19 | [1.0.0]: https://github.com/Vicomtech/DMD-Driver-Monitoring-Dataset/release/tag/v1.0.0 -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Driver Monitoring Dataset (DMD) 2 | The [Driver Monitoring Dataset](http://dmd.vicomtech.org/) is the largest visual dataset for real driving actions, with footage from synchronized multiple cameras (body, face, hands) and multiple streams (RGB, Depth, IR) recorded in two scenarios (real car, driving simulator). Different annotated labels related to distraction, fatigue and gaze-head pose can be used to train Deep Learning models for Driver Monitor Systems. 3 | 4 | This project includes a tool to annotate the dataset, inspect the annotated data and export training sets. Output annotations are formatted using [OpenLABEL](https://www.asam.net/standards/detail/openlabel/) language [VCD (Video Content Description)](https://vcd.vicomtech.org/). 5 | 6 | ## Dataset details 7 | More details of the recording and video material of DMD can be found at the [official website](http://dmd.vicomtech.org/) 8 | 9 | In addition, this repository [wiki](https://github.com/Vicomtech/DMD-Driver-Monitoring-Dataset/wiki) has useful information about the DMD dataset and the annotation process. 10 | 11 | ## Available tools: 12 | - Temporal Annotation Tool (TaTo) - (more info [here](annotation-tool/README.md)) 13 | - Dataset Explorer Tool (DEx) - (more info [here](exploreMaterial-tool/README.md)) 14 | ### Annotation Instructions 15 | Depending the annotation problem, different annotation criteria should be defined to guarantee all the annotators produce the same output annotations. 16 | 17 | We have defined the following criteria to be used with tool to produce consistent annotations: 18 | 19 | - [DMD Distraction-related actions](https://github.com/Vicomtech/DMD-Driver-Monitoring-Dataset/wiki/DMD-distraction-related-action-annotation-criteria) annotation 20 | 21 | ## Known Issues 22 | - The version of OpenLABEL in the annotation files (OpenLabel) and in the tools in this repository has been updated to VCD>=5.0. Make sure you download the annotations files again and update the tools. 23 | - There was an error when uploading IR videos. They have to be .mp4 format, and they were uploaded as .avi. This is fixed now but requires the user to download them again. 24 | 25 | ## Credits 26 | Development of DMD was supported and funded by the European Commission (EC) Horizon 2020 programme (project [VI-DAS](http://www.vi-das.eu/), grant agreement 690772) 27 | 28 | Developed with :blue_heart: by: 29 | 30 | * Paola Cañas (pncanas@vicomtech.org) 31 | * Juan Diego Ortega (jdortega@vicomtech.org) 32 | 33 | Contributions of ideas and comments: Marcos Nieto, Mikel Garcia, Gonzalo Pierola, Itziar Sagastiberri, Itziar Urbieta, Eneritz Etxaniz, Orti Senderos. 34 | 35 | ## License 36 | Copyright :copyright: 2024 Vicomtech 37 | 38 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 39 | 40 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 41 | 42 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 43 | -------------------------------------------------------------------------------- /annotation-tool/README.md: -------------------------------------------------------------------------------- 1 | # Temporal Annotation Tool (TaTo) 2 | We have acquired a good amount of high quality and friendly driver’s material in the DMD (Driver Monitoring Dataset) with the purpose of developing computer vision algorithms for driver monitoring. But what would be a dataset without its corresponding annotations? 3 | 4 | We developed the TaTo tool to annotate temporal events and actions performed by the drivers in the video sequences. The tool was used to annotate distraction-related actions. However, through configuration, other labels can be annotated. 5 | 6 | ## Content 7 | - [Temporal Annotation Tool (TaTo)](#temporal-annotation-tool-tato) 8 | - [Content](#content) 9 | - [Setup and Launching](#setup-and-launching) 10 | - [TaTo characteristics](#tato-characteristics) 11 | - [Usage Instructions](#usage-instructions) 12 | - [General functionality](#general-functionality) 13 | - [TaTo Window description](#tato-window-description) 14 | - [Annotating with TaTo](#annotating-with-tato) 15 | - [Select the annotation level](#select-the-annotation-level) 16 | - [Special labels](#special-labels) 17 | - [Annotation Modes](#annotation-modes) 18 | - [Frame-by-frame annotation](#frame-by-frame-annotation) 19 | - [Block annotation](#block-annotation) 20 | - [Automatic annotation](#automatic-annotation) 21 | - [Keyboard Interaction](#keyboard-interaction) 22 | - [General keys](#general-keys) 23 | - [Video Navigation](#video-navigation) 24 | - [Playback keys](#playback-keys) 25 | - [Annotation keys](#annotation-keys) 26 | - [Saving annotations](#saving-annotations) 27 | - [Annotation criteria](#annotation-criteria) 28 | - [Changelog](#changelog) 29 | - [FAQs](#faqs) 30 | - [Known Issues](#known-issues) 31 | - [License](#license) 32 | 33 | ## Setup and Launching 34 | The TaTo tool has been tested using the following system configuration: 35 | 36 | **OS:** Ubuntu 18.04, Windows 10
37 | **Dependencies:** Python 3.8, OpenCV-Python 4.2.0, VCD>=6.0.3 38 | 39 | For a detailed description on how to configure the environment and launch the tool, check: [Linux](../docs/setup_linux.md) / [Windows](../docs/setup_windows.md) 40 | 41 | ## TaTo characteristics 42 | TaTo is a python keyboard-based software application to create and modify temporal annotations in [VCD 5.0. format](https://vcd.vicomtech.org/). It was planned to manage annotations from [DMD](http://dmd.vicomtech.org/) videos, but recently it was modified to be compatible with videos in general. It supports multi-label annotation and has an intuitive interface to visualize temporal annotations through timelines. It offers the following features: 43 | 44 | - Allows the annotation of temporal events in videos (not only from the DMD). 45 | - The annotations could be divided in up to 7 different annotation levels(group of labels). Within each group the labels are **mutually exclusive.** 46 | - Levels and labels of annotation can be defined in a **configuration file** and TaTo can work with many of these files (one at a time). 47 | - Annotations are represented by colors in two timelines. Each label has its own color. 48 | - The labels can be input either frame-by-frame or by frame-block interval. 49 | - There is a **validation property** per each frame that indicates how it was annotated: frame-by-frame or frame-block interval. 50 | - The output annotations are saved in VCD format in a **JSON file**. 51 | - For DMD default annotation modes, you can apply **automatic annotations**. That is, applying logically related annotations among levels. 52 | - To avoid loss of progress, the tool **autosaves** annotations. This autosaving is done in txt files to not affect the tool performance. 53 | 54 | ## Usage Instructions 55 | 56 | ### General functionality 57 | Its operation depends on a general configuration file **“config.json”**. The main options to configure are in the object **“tatoConfig”**, these are: 58 | 59 | #### Annotation mode 60 | DMD has predefined annotation modes. These annotation modes are defined by a group of labels related to an analysis dimension of the dataset. For example, there is the distraction annotation mode which includes labels like texting-left, drinking, etc. 61 | You can indicate the annotation mode in the “annotation_mode” field. To define a new annotation mode you must create a new configuration file and name it as: “config_$annModeName.json”. To create an annotation mode configuration file, we recommend duplicate and modify an existing one to avoid incompatibilities. Inside there must be the following JSON objects: 62 | 63 | - **level_names:** this object lists the names of the levels of annotation. A level is a category of labels. Each label must be identified by a number as a string (e.g. “0”: "driver_actions"). 64 | - **level_types:** This object specifies the nature of each level in the VCD (action, object, stream_properties). Must be the same number of items as in level_names, and must share the same id (e.g. “0”, “1”). 65 | - **$level_id:** For each level, there is an object to list its corresponding labels. The name of the object must match the id from level_names and level_types (e.g. “0”, “1”). In the same way, the labels within each level must be identified by a number as a string (e.g. “0”: “safe_drive”). There are two standard labels with their corresponding id: “99”: “--” that represents the absence of a label in one frame and “100”: “NAN” that means there is no frame information. 66 | - **level_defaults:** In this object, the default label for level is defined. When the VCD is initially created from cero, all frames will be annotated with a default label. This label should be the most recurrent one in the video so the annotation effort is less. There must be the same number of items as in level_names, and must share the same id (e.g. “0”, “1”). 67 | - **camera_dependencies:** The DMD annotation is done with a mosaic video that includes 3 perspective views. Some labels are camera-dependant, so when there is no frame showing from one perspective, there shouldn’t be an annotation from those perspective-dependant labels. To indicate these dependencies, there must be an array of the labels ids for each camera perspective (e.g. "face": [1,2]). 68 | 69 | #### Dataset 70 | This option is to identify if you are working with the DMD and load predefined configuration and validations of TaTo for this dataset. If it is different, then it loads a general configuration and allows annotating any video with VCD>=5.0. 71 | 72 | #### Pre_annotate 73 | We created automatic pre-annotations for the DMD, also the DMD has metadata and static annotations we had to include inside the VCD. To do this, we load them before creating a VCD. To activate this functionality, we added this configuration option. You must leave it with a value of 0 since pre-annotation only works internally. 74 | 75 | ### TaTo Window description 76 | The annotation tool TaTo opens with three windows: 77 | 78 | ![Annotation Tool](../docs/imgs/annotation_tool_info.png) 79 | 80 | The main window will display: 81 | - The **mosaic video** consisting of three camera streams (these streams were previously synchronized). 82 | - Frame information (current frame). The **current frame** is the mosaic frame. 83 | - The last time you saved your annotation (It is recommended to save your progress frequently). 84 | - The [**annotation info**](#select-the-annotation-level) panel shows a list of levels (annotation groups) and the current label for each level at the current frame. **The level you are currently annotating is indicated with the “->” sign and is written in bold.** 85 | 86 | A second window (with dark background) will show: 87 | - The **frame offsets** between video streams 88 | - The **video path** you are annotating. 89 | - A list of the available **labels** of the current selected annotation level. Each label has a colored box and a key to press for annotation in brackets [ ]. Check [annotation instructions](#annotating-with-tato) to know how to input annotation for each level. 90 | - An informative list of possible frame validation states. 91 | 92 | In the third window there are **two timelines** which show video annotations in colors, depending on the level: 93 | - The first is a full timeline from frame 0 to the total length of the video. 94 | - The second is a zoomed timeline around the current frame always keeping the actual frame in the center of the window. This timeline helps the visualization of individual frame label. 95 | 96 | The colors in the timeline representation are the same as the colors in the label description panel. The colors and labels depend on the selected annotation level. 97 | 98 | When there are 5 frames left, there will appear a “last frames” text and a “LAST FRAME!” text at the last frame. 99 | 100 | ### Annotating with TaTo 101 | The interaction with the tool is meant to be done using the keyboard. We have a thorough list of keys available for interaction and annotation. Once the tool is open, you can press P to open instructions. 102 | 103 | #### Select the annotation level 104 | An annotation level is a group of labels which are mutually exclusive (two or more labels of the same level cannot be assigned to the same frame). The definition of these levels and their corresponding labels is taken from a [config file](../annotation-tool/config_distraction.json). 105 | 106 | There are some annotation levels which require all frames to have a label. However, other levels can admit frames with no label. This depends on the nature of the annotation level. Annotation levels with empty labels will include this option in the config file. 107 | 108 | The first step to start annotating a video is to **select the annotation level**. You can see the current selected level in the main window's annotation info panel (see figure below). The current selected level will display an arrow "->" symbol and the current label will be highlighted in bold text. To change between levels use the Tab key. 109 | 110 | ![annotation info](../docs/imgs/level_panel.png "annotation_info") 111 | 112 | ##### Special labels 113 | All the annotations levels will have a group of text labels which are used to annotate the temporal events. However, you will see two special labels displayed in the [annotation info panel](../docs/imgs/level_panel.png): 114 | 115 | - **"NAN"**: This means there is not a frame from the camera stream used to annotate that level. This label is automatically set by the tool, it is not possible to change it manually. 116 | - **"--"**: This means absence of label. Some levels don't require to have continuos annotation labels so this "empty" label should be present. For those levels which requires continuos annotations, this "empty" label should not be present. Please consult the annotation criteria to know level characteristics. 117 | 118 | #### Annotation Modes 119 | The annotation tool has two modes of annotation: **frame-by-frame** and **block** annotation. 120 | 121 | ##### Frame-by-frame annotation 122 | This is the basic annotation mode. To annotate frame-by-frame you should first [select the annotation level](#select-the-annotation-level) you would want to annotate. Then, press the corresponding [label key](#annotation-keys) according to the list of available labels located in the window with dark background. The key to press will be in brackets [ ] on the left of each label. Once you press the key, the frame color in timeline will change. 123 | 124 | ##### Block annotation 125 | This is a handy way to annotate a frame interval. For this, you should first select the frame interval to be annotated. To do so, press the Z key to select the starting frame. Then, move forward or backward with the [navigation keys](#video-navigation). You will see in the timeline window your selection of the frame interval in green. After selecting the desired frame interval, press the corresponding [label key](#annotation-keys) to annotate all frames in the interval with that label. 126 | 127 | To unselect a frame interval press two times the Z key. 128 | 129 | ![block_annotation](../docs/imgs/block_annotation.png) 130 | 131 | ##### Automatic annotation 132 | In the case of DMD, there are logical relationships between different levels of annotation. For example, if the driver is performing the activity “Texting left”, then the annotation in hands_using_wheel level should be “Only Right”. There is a function that allows applying these logical annotations and change the related labels from different levels depending on the driver_actions level. 133 | 134 | It is important that you use this function after you have completed the annotations of driver_actions, then perform the automatic annotation changes to the rest of levels. The key to do this in the tool is x and is a just one-time operation. Mor info [here](https://github.com/Vicomtech/DMD-Driver-Monitoring-Dataset/wiki/DMD-distraction-related-action-annotation-criteria#apply-automatic-annotation-interpolation-warning). 135 | 136 | #### Keyboard Interaction 137 | ##### General keys 138 | | Keys | Function | 139 | | :--------------: | :----------------------------------------------- | 140 | | Esc | **Close** the tool, saving the current progress | 141 | | Enter | **Save** the current annotation progress | 142 | | P | Open a help window showing the available keys | 143 | 144 | ##### Video Navigation 145 | | Keys | Function | 146 | | :----------------: | :------------------------------------------------------- | 147 | | Any Key | Besides function specific keys, move forward **1 frame** | 148 | | E | Move **forward 50 frames** | 149 | | R | Move **forward 300 frames** | 150 | | W | Move **backwards 50 frames** | 151 | | Q | Move **backwards 300 frames** | 152 | | Space | Move **backwards 1 frame** | 153 | | S | **Jump forward** to nearest label change | 154 | | A | **Jump backwards** to nearest label change | 155 | 156 | ##### Playback keys 157 | This is a functionality where you can play and visualize the video along with the annotations from all levels. Can be used to check annotations or to navigate in timeline. 158 | 159 | | Keys | Function | 160 | | :------------------: | :------------------------------------------------------------------------------------------------------------ | 161 | | Backspace | Opens the **playback of the video** in a new window. | 162 | | Enter | In the playback window, closes the playback window returning to the main window at the **last frame played** | 163 | | Esc | In the playback window, closes the playback window returning to the main window at the **first frame played** | 164 | 165 | ##### Annotation keys 166 | | Keys | Function | 167 | | :--------------: | :------------------------------------------------------------------------ | 168 | | Tab | **Switch** between annotation levels | 169 | | Z | **Select/Unselect** the starting frame for block annotation (key-frame) | 170 | | 0...9, /, *, -, + | **Annotate** the frame or frame interval with the corresponding label. | 171 | | . | **Remove** the current label of the frame or frame interval | 172 | | X | Apply **automatic annotations** to other levels. :warning: **Caution: This is a destructive action, apply carefully.** | 173 | 174 | ## Saving annotations 175 | You can save the progress by pressing the Enter key. The tool also saves the progress when you exit the tool using the Esc key. 176 | 177 | **The tool saves the annotations in VCD 4 format in a JSON file.** 178 | 179 | The tool has an autosave functionality that creates two TXT files with the annotation information. If a VCD file is successfully saved, then autosave TXT files will be deleted automatically. The name of these files are: 180 | - [video_name]_autoSaveAnn-A.txt 181 | - [video_name]_autoSaveAnn-B.txt 182 | 183 | **In case something occurs and the tool exits without you manually saving the progress, you can recover your progress with these temporal TXT files.** 184 | 185 | If there was a failure in saving the VCD, you will have **both** JSON and TXT files in the video directory. If you try to run the tool again, you should receive an error telling you to keep the most recent file. If that is the case, **delete** the VCD (JSON) file and start TaTo again. 186 | 187 | You could know which file is taken by TaTo during start-up time, depending on the file the tool is reading the annotations from, it should print: 188 | 189 | - *"Loading VCD ..."* if the annotations are taken from a VCD json file. 190 | - *"Loading provisional txt annotation files ..."* if the autoSave txt files are loaded. 191 | 192 | ## Annotation criteria 193 | Depending on the annotation problem, different annotation criteria should be defined to guarantee all the annotators produce the same output annotations. 194 | 195 | We have defined the following criteria to be used with tool to produce consistent annotations: 196 | 197 | - [DMD Distraction-related actions](https://github.com/Vicomtech/DMD-Driver-Monitoring-Dataset/wiki/DMD-distraction-related-action-annotation-criteria) annotation 198 | 199 | ## Changelog 200 | For a complete list of changes check the [CHANGELOG.md](../CHANGELOG.md) file 201 | 202 | ## FAQs 203 | 204 | - **How can I change the labels name?** 205 | 206 | The number of labels per level and their names are specified at the [config file](../annotation-tool/config_distraction.json). You can change them there and restart the tool. You can also add more labels. If you delete labels, some problems of compatibility might appear. You can create your own config file and define your levels and labels of annotation. 207 | 208 | - **What if I forgot to save?** 209 | 210 | Relax, it happens. The tool saves progress when you press Esc and exit the tool. If you forgot to save and there was a sudden problem with the tool, there are autosave files that can help you recover your unsaved progress. Go to: [Saving annotations](#saving-annotations) 211 | 212 | - **The tool gives the error: Incompatible file name. Please check your file name or folder structure.** 213 | 214 | This error appears when the path or the folder structure of the video you had input is not valid or has not correct DMD nomenclature [DMD File Structure](https://github.com/Vicomtech/DMD-Driver-Monitoring-Dataset/wiki/DMD-file-struct). Also, make sure that the path you are inserting is the one form the **mosaic video** and not from the annotations. 215 | 216 | - **I have the error: There are two annotation files: VCD and txt. Please, keep only the most recent one. You can delete '..ann.json' file or '..autoSaveAnnA.txt and ..autoSaveAnnB.txt' files.** 217 | 218 | That error appears when there are two kinds of annotations files, the VCD file and autosave TXT files. This means that the VCD file has not been saved successfully for some reason and there is a backup of your unsaved progress in the txt files. You can check if there are annotations inside the TXT files and go ahead and delete the VCD. When you open the tool again, it will create a VCD from the TXT files and everything will be fine again :) 219 | 220 | - **I'm pressing the keys to navigate through timeline but it does not move** 221 | 222 | Check if any other support window apart from the 3 main windows is open, like instructions window. If so, you can close it with Esc or pressing the "x" directly on the window. 223 | 224 | - **If the main window goes dark while there is a "Saving..." sign, does it still saving?** 225 | 226 | Yes. Depending on the VCD file size, it can take a while to save the progress and the main windows turn dark. We understand that it seems like there has been a problem and the tool is not responding, but don't worry, is normal :) 227 | 228 | - **What are the static annotations shown in the console?** 229 | 230 | Besides of temporal annotations, each video has context annotations, driver info and properties that we call static annotations. These are within VCD file since this format allows to include all kinds of annotations. You can access the VCD file to use those annotations. 231 | 232 | 233 | ## Known Issues 234 | 235 | | Issue | Solution | 236 | | ------------------- | :------------------------------------------------------- | 237 | | When pressed Alt Gr the tool exists abruptly | This is caused due to a bug in the capture system dependency used in the tool. Please **Don't press** Alt Gr | 238 | 239 | 240 | :warning: If you find any bug with the tool or have ideas of new features please open a new issue using the [bug report template](../docs/issue_bug_template.md) or the [feature request template](../docs/issue_feature_template.md) :warning: 241 | -------------------------------------------------------------------------------- /annotation-tool/config.json: -------------------------------------------------------------------------------- 1 | { 2 | "tatoConfig": { 3 | "annotation_mode": "distraction", 4 | "dataset": "dmd", 5 | "pre_annotate": 0, 6 | "calculate_time":0 7 | }, 8 | "interfaceText": { 9 | "mainLevelDependency":{ 10 | "distraction":"depending on Driver Actions level!", 11 | "drowsiness":"depending on Eye State (Right) level!" 12 | }, 13 | "levelCompletedToAnnotate":{ 14 | "distraction": "Only do this when you have completed the Driver_Actions Annotations", 15 | "drowsiness": "Only do this when you have completed the Eye State (Right) Annotations" 16 | } 17 | }, 18 | "consoleText": { 19 | "video_path_dmd": { 20 | "True": "PATH of the video (/dmd/.../_.._mosaic.avi): ", 21 | "False": "PATH of the video: " 22 | } 23 | }, 24 | "dimensions":{ 25 | "total-width": 1280, 26 | "total-height":720 27 | 28 | }, 29 | "colors": { 30 | "textColorMain": [ 31 | 60, 32 | 60, 33 | 60 34 | ], 35 | "textColorLabels": [ 36 | 255, 37 | 255, 38 | 255 39 | ], 40 | "textColorInstructions": [ 41 | 60, 42 | 60, 43 | 60 44 | ], 45 | "backgroundColorMain": 255, 46 | "backgroundColorLabels": 40, 47 | "backgroundColorInstructions": 230, 48 | "keyFrameColor": [ 49 | 131, 50 | 255, 51 | 167 52 | ], 53 | "colorDict": { 54 | "0": [ 55 | 223, 56 | 215, 57 | 195 58 | ], 59 | "1": [ 60 | 105, 61 | 237, 62 | 249 63 | ], 64 | "2": [ 65 | 201, 66 | 193, 67 | 63 68 | ], 69 | "3": [ 70 | 6, 71 | 214, 72 | 160 73 | ], 74 | "4": [ 75 | 233, 76 | 187, 77 | 202 78 | ], 79 | "5": [ 80 | 133, 81 | 81, 82 | 252 83 | ], 84 | "6": [ 85 | 0, 86 | 154, 87 | 255 88 | ], 89 | "7": [ 90 | 181, 91 | 107, 92 | 69 93 | ], 94 | "8": [ 95 | 137, 96 | 171, 97 | 31 98 | ], 99 | "9": [ 100 | 224, 101 | 119, 102 | 125 103 | ], 104 | "10": [ 105 | 153, 106 | 153, 107 | 255 108 | ], 109 | "11": [ 110 | 83, 111 | 73, 112 | 193 113 | ], 114 | "12": [ 115 | 107, 116 | 79, 117 | 54 118 | ], 119 | "13": [ 120 | 106, 121 | 107, 122 | 131 123 | ], 124 | "99": [ 125 | 245, 126 | 245, 127 | 245 128 | ], 129 | "100": [ 130 | 80, 131 | 80, 132 | 80 133 | ], 134 | "val_0": [ 135 | 245, 136 | 245, 137 | 245 138 | ], 139 | "val_1": [ 140 | 223, 141 | 187, 142 | 185 143 | ], 144 | "val_2": [ 145 | 250, 146 | 210, 147 | 170 148 | ] 149 | } 150 | } 151 | } -------------------------------------------------------------------------------- /annotation-tool/config_distraction.json: -------------------------------------------------------------------------------- 1 | { 2 | "6": { 3 | "0": "safe_drive", 4 | "1": "texting_right", 5 | "2": "phonecall_right", 6 | "3": "texting_left", 7 | "4": "phonecall_left", 8 | "5": "radio", 9 | "6": "drinking", 10 | "7": "reach_side", 11 | "8": "hair_and_makeup", 12 | "9": "talking_to_passenger", 13 | "10": "reach_backseat", 14 | "11": "change_gear", 15 | "12": "standstill_or_waiting", 16 | "13": "unclassified", 17 | "100": "NAN" 18 | }, 19 | "5": { 20 | "0": "cellphone", 21 | "1": "hair_comb", 22 | "2": "bottle", 23 | "99": "--", 24 | "100": "NAN" 25 | }, 26 | "4": { 27 | "0": "hand_on_gear", 28 | "99": "--", 29 | "100": "NAN" 30 | }, 31 | "3": { 32 | "0": "both", 33 | "1": "only_right", 34 | "2": "only_left", 35 | "3": "none", 36 | "100": "NAN" 37 | }, 38 | "2": { 39 | "0": "talking", 40 | "99": "--", 41 | "100": "NAN" 42 | }, 43 | "1": { 44 | "0": "looking_road", 45 | "1": "not_looking_road", 46 | "100": "NAN" 47 | }, 48 | "0": { 49 | "0": "face_camera", 50 | "1": "body_camera", 51 | "2": "hands_camera", 52 | "99": "--", 53 | "100": "NAN" 54 | }, 55 | "level_names": { 56 | "0": "occlusion", 57 | "1": "gaze_on_road", 58 | "2": "talking", 59 | "3": "hands_using_wheel", 60 | "4": "hand_on_gear", 61 | "5": "objects_in_scene", 62 | "6": "driver_actions" 63 | }, 64 | "level_types": { 65 | "0": "stream_properties", 66 | "1": "action", 67 | "2": "action", 68 | "3": "action", 69 | "4": "action", 70 | "5": "object", 71 | "6": "action" 72 | }, 73 | "level_defaults": { 74 | "0": 99, 75 | "1": 0, 76 | "2": 99, 77 | "3": 0, 78 | "4": 99, 79 | "5": 99, 80 | "6": 0 81 | }, 82 | "camera_dependencies":{ 83 | "face": [1,2], 84 | "body": [5,6], 85 | "hands": [3,4] 86 | } 87 | } 88 | -------------------------------------------------------------------------------- /annotation-tool/config_drowsiness.json: -------------------------------------------------------------------------------- 1 | { 2 | "0": { 3 | "0": "face_camera", 4 | "1": "body_camera", 5 | "2": "hands_camera", 6 | "99": "--", 7 | "100": "NAN" 8 | }, 9 | "1":{ 10 | "0": "open", 11 | "1": "close", 12 | "2": "opening", 13 | "3": "closing", 14 | "4": "undefined", 15 | "100": "NAN" 16 | }, 17 | "2": { 18 | "0": "blinking", 19 | "99": "--", 20 | "100": "NAN" 21 | }, 22 | "3": { 23 | "0": "Yawning with hand", 24 | "1": "Yawning without hand", 25 | "99": "--", 26 | "100": "NAN" 27 | }, 28 | "level_names": { 29 | "0": "occlusion", 30 | "1": "eyes_state", 31 | "2": "blinks", 32 | "3": "yawning" 33 | }, 34 | "level_types": { 35 | "0": "stream_properties", 36 | "1": "action", 37 | "2": "action", 38 | "3": "action" 39 | }, 40 | "level_defaults": { 41 | "0": 99, 42 | "1": 0, 43 | "2": 99, 44 | "3": 99 45 | 46 | }, 47 | "camera_dependencies": { 48 | "face": [1,2,3], 49 | "body": [], 50 | "hands": [] 51 | } 52 | } 53 | -------------------------------------------------------------------------------- /annotation-tool/config_gaze.json: -------------------------------------------------------------------------------- 1 | { 2 | "2": { 3 | "0": "blinking", 4 | "99": "--", 5 | "100": "NAN" 6 | }, 7 | "1": { 8 | "0": "left_mirror", 9 | "1": "left", 10 | "2": "front", 11 | "3": "center_mirror", 12 | "4": "front_right", 13 | "5": "right_mirror", 14 | "6": "right", 15 | "7": "infotainment", 16 | "8": "steering_wheel", 17 | "9": "not_valid", 18 | "100": "NAN" 19 | }, 20 | "0": { 21 | "0": "face_camera", 22 | "1": "body_camera", 23 | "2": "hands_camera", 24 | "99": "--", 25 | "100": "NAN" 26 | }, 27 | "level_names": { 28 | "0": "occlusion", 29 | "1": "gaze_zone", 30 | "2": "blinks" 31 | }, 32 | "level_types": { 33 | "0": "stream_properties", 34 | "1": "action", 35 | "2": "action" 36 | }, 37 | "level_defaults": { 38 | "0": 99, 39 | "1": 2, 40 | "2": 99 41 | }, 42 | "camera_dependencies": { 43 | "face": [ 44 | 1, 45 | 2 46 | ], 47 | "body": [], 48 | "hands": [] 49 | } 50 | } -------------------------------------------------------------------------------- /annotation-tool/config_hands.json: -------------------------------------------------------------------------------- 1 | { 2 | "0": { 3 | "0": "face_camera", 4 | "1": "body_camera", 5 | "2": "hands_camera", 6 | "99": "--", 7 | "100": "NAN" 8 | }, 9 | "1": { 10 | "0": "both_hands", 11 | "1": "only_right", 12 | "2": "only_left", 13 | "3": "none", 14 | "100": "NAN" 15 | }, 16 | "2": { 17 | "0": "taking control", 18 | "1": "giving control", 19 | "99": "--", 20 | "100": "NAN" 21 | }, 22 | "3": { 23 | "0": "moving", 24 | "1": "not_moving", 25 | "100": "NAN" 26 | }, 27 | "level_names": { 28 | "0": "occlusion", 29 | "1": "hands_on_wheel", 30 | "2": "transition", 31 | "3": "moving_hands" 32 | }, 33 | "level_types": { 34 | "0": "stream_properties", 35 | "1": "action", 36 | "2": "action", 37 | "3": "action" 38 | }, 39 | "level_defaults": { 40 | "0": 99, 41 | "1": 0, 42 | "2": 99, 43 | "3": 1 44 | }, 45 | "camera_dependencies": { 46 | "face": [], 47 | "body": [ 48 | 1, 49 | 2, 50 | 3 51 | ], 52 | "hands": [] 53 | } 54 | } -------------------------------------------------------------------------------- /annotation-tool/config_statics.json: -------------------------------------------------------------------------------- 1 | { 2 | "static_dict": { 3 | "0": { 4 | "name": "age", 5 | "text": "Subject Age", 6 | "type": "num", 7 | "parent": { 8 | "element": "object", 9 | "type": "driver" 10 | } 11 | }, 12 | "1": { 13 | "name": "gender", 14 | "text": "Subject Gender", 15 | "type": "text", 16 | "options": { 17 | "0": "Male", 18 | "1": "Female" 19 | }, 20 | "parent": { 21 | "element": "object", 22 | "type": "driver" 23 | } 24 | }, 25 | "2": { 26 | "name": "glasses", 27 | "text": "Is wearing glasses?", 28 | "type": "boolean", 29 | "options": { 30 | "0": "No", 31 | "1": "Yes" 32 | }, 33 | "parent": { 34 | "element": "object", 35 | "type": "driver" 36 | } 37 | }, 38 | "3": { 39 | "name": "drive_freq", 40 | "text": "Driving Frecuency", 41 | "type": "text", 42 | "options": { 43 | "0": "Once a week or less", 44 | "1": "Between 2 or 5 times a week", 45 | "2": "Everyday" 46 | }, 47 | "parent": { 48 | "element": "object", 49 | "type": "driver" 50 | } 51 | }, 52 | "4": { 53 | "name": "experience", 54 | "text": "Driving Experience", 55 | "type": "text", 56 | "options": { 57 | "0": "Less than 1 year", 58 | "1": "Between 1 and 3 years", 59 | "2": "More than 3 years" 60 | }, 61 | "parent": { 62 | "element": "object", 63 | "type": "driver" 64 | } 65 | }, 66 | "5": { 67 | "name": "weather", 68 | "text": "Weather", 69 | "type": "text", 70 | "options": { 71 | "0": "Sunny", 72 | "1": "Rainy", 73 | "2": "Cloudy" 74 | }, 75 | "parent": { 76 | "element": "context", 77 | "type": "recording_context" 78 | } 79 | }, 80 | "6": { 81 | "name": "setup", 82 | "text": "Setup", 83 | "type": "text", 84 | "options": { 85 | "0": "Car Moving", 86 | "1": "Car Stopped", 87 | "2": "Simulator" 88 | }, 89 | "parent": { 90 | "element": "context", 91 | "type": "recording_context" 92 | } 93 | }, 94 | "7": { 95 | "name": "annotatorID", 96 | "text": "Annotator ID", 97 | "type": "num", 98 | "parent": { 99 | "element": "metadata", 100 | "type": "annotator" 101 | } 102 | } 103 | } 104 | } -------------------------------------------------------------------------------- /annotation-tool/setUp.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import json 3 | import re 4 | from pathlib import Path # To handle paths independent of OS 5 | import datetime 6 | 7 | 8 | def is_string_int(s): 9 | try: 10 | int(s) 11 | return True 12 | except ValueError: 13 | return False 14 | 15 | 16 | # Function to transform int keys to integer if possible 17 | def keys_to_int(x): 18 | r = {int(k) if is_string_int(k) else k: v for k, v in x} 19 | return r 20 | 21 | 22 | class ConfigTato: 23 | 24 | def __init__(self): 25 | """ Most of the paths specified are necessary for internal compatibility 26 | also for the construction of pre-annotations. 27 | If its running in external structure, the .json and .avi video paths 28 | should be enough 29 | Run the tool through the ./annotate.sh script to avoid confussions 30 | with paths 31 | """ 32 | 33 | # ----GLOBAL VARIABLES---- 34 | default_annotation_modes = ["distraction", "drowsiness", "gaze"] 35 | # @_external_struct: flag to know if is running in internal or external 36 | # extructure 37 | self._external_struct = True 38 | 39 | # ----LOAD CONFIG FROM JSON---- 40 | 41 | # Config dictionary path 42 | self._config_json = "config.json" 43 | # From json to python dictionaries 44 | with open(self._config_json) as config_file: 45 | config_dict = json.load(config_file, object_pairs_hook=keys_to_int) 46 | tatoConfig = config_dict["tatoConfig"] 47 | self._interfaceTexts = config_dict["interfaceText"] 48 | self._consoleTexts = config_dict["consoleText"] 49 | self._colorConfig = config_dict["colors"] 50 | self._dimensions = config_dict["dimensions"] 51 | # Config variables 52 | self._pre_annotate = bool(tatoConfig["pre_annotate"]) 53 | self._annotation_mode = tatoConfig["annotation_mode"] 54 | self._annotation_dataset = tatoConfig["dataset"] 55 | self._calculate_time = bool(tatoConfig["calculate_time"]) 56 | self._default_annotation_mode = self._annotation_mode in default_annotation_modes 57 | self._dataset_dmd = self._annotation_dataset == "dmd" 58 | 59 | 60 | #----GET CONSOLE INPUTS---- 61 | print("Welcome :)") 62 | #Capture video PATH 63 | self._video_file_path = Path(input(self._consoleTexts["video_path_dmd"][str(self._dataset_dmd)])) 64 | 65 | #Check if video exists 66 | if not self._video_file_path.exists(): 67 | raise RuntimeError("Video file doesn't exist: " + 68 | str(self._video_file_path.resolve())) 69 | else: 70 | print("Video from " + self._annotation_dataset + 71 | " loaded: " + self._video_file_path.name) 72 | 73 | #Check if config of annotation exists 74 | self._annConfig_file_path = Path("./config_"+self._annotation_mode+".json") 75 | if not self._annConfig_file_path.exists(): 76 | raise RuntimeError("Annotation config file doesn't exist: " + 77 | str(self._annConfig_file_path.resolve()) + " Please, define a config file for "+self._annotation_mode+" or change 'annotation_mode' option in "+self._config_json) 78 | else: 79 | print("TaTo is in "+self._annotation_mode+" annotation mode with " + 80 | self._annConfig_file_path.name+" annotation config file.") 81 | 82 | #----DEFINE FILES PATHS---- 83 | root_path = self._video_file_path.parent 84 | #If annotating dmd 85 | if self._dataset_dmd: 86 | # Build a regular expression for the mosaic name to be satisfied by the input mosaic file name 87 | regex_internal = '(?P[1-9]|[1-2][0-9]|[3][0-7])_(?P[a-z]{1,}|[a-z]{1,}[2])_'\ 88 | '(?Pmosaic|body|face|hands)_(?P(?P0[1-9]|1[012])-(?P0[1-9]|[12][0-9]|3[01]))' 89 | regex_external = '(?Pg[A-z]{1,})_(?P[1-9]|[1-2][0-9]|[3][0-7])_'\ 90 | '(?Ps[1-9]{1,})_(?P(?P(?P\d{4})-(?P0[1-9]|1[012])-'\ 91 | '(?P0[1-9]|[12][0-9]|3[01]))T(?P