├── .github └── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── android ├── apk │ └── app-debug.apk └── startup_linux │ ├── deepeye.sh │ └── rfcomm.service ├── custom_model └── OI_Dataset │ ├── README.md │ ├── requirements.txt │ ├── scripts │ ├── csv2tfrecord.py │ ├── txt2xml.py │ └── xml2csv.py │ └── tf_test.py ├── deepeye_app ├── app.py ├── calibration │ ├── config │ │ └── BW1098FFC.json │ └── depthai.calib ├── collision_avoidance.py ├── config.py ├── models │ ├── mobilenet-ssd.blob │ ├── mobilenet-ssd_depth.json │ ├── mobilenet-ssd_v1.blob │ └── mobilenet-ssd_v1.json ├── tracker.py └── txt2speech │ ├── README.md │ ├── s_images │ ├── txt2s.png │ └── txt2s_bt.png │ ├── sw_design │ ├── txt2s_BT │ └── txt2speech │ ├── txt-simulator.py │ └── txt2speech.py ├── images ├── BW1098FFC_R0M0E0_diag.png ├── BW1098FFC_R0M0E0_dims.png ├── BW1098FFC_R0M0E0_front.png ├── HLD.png ├── HLD_1.JPG ├── demo.PNG ├── hw.jpeg └── jetson-nano-dev-kit-top-r6-HR-B01.png ├── openvino_analysis ├── README.md ├── intel │ ├── object-detection │ │ ├── pedestrian-and-vehicle-detector-adas-0001 │ │ │ ├── __pycache__ │ │ │ │ └── inference.cpython-36.pyc │ │ │ ├── inference.py │ │ │ └── main.py │ │ ├── pedestrian-detection-adas-0002 │ │ │ ├── __pycache__ │ │ │ │ └── inference.cpython-36.pyc │ │ │ ├── inference.py │ │ │ └── main.py │ │ ├── pedestrian-detection-adas-binary-0001 │ │ │ ├── inference.py │ │ │ └── main.py │ │ └── vehicle-detection-adas │ │ │ ├── inference.py │ │ │ └── main.py │ └── semantic-segmentation │ │ ├── road-segmentation-adas-0001 │ │ ├── __pycache__ │ │ │ └── inference.cpython-36.pyc │ │ ├── inference.py │ │ └── main.py │ │ └── semantic-segmentation-adas-0001 │ │ ├── async │ │ ├── 009649.png │ │ ├── __pycache__ │ │ │ └── inference.cpython-36.pyc │ │ ├── inference.py │ │ └── main.py │ │ ├── core │ │ ├── __pycache__ │ │ │ └── semantic_detection.cpython-36.pyc │ │ └── semantic_detection.py │ │ ├── main.py │ │ └── utils │ │ ├── __pycache__ │ │ └── ie_module.cpython-36.pyc │ │ └── ie_module.py └── public │ ├── ssd_mobilenet_v2_coco │ ├── __pycache__ │ │ └── inference.cpython-36.pyc │ ├── async_multithread │ │ ├── __pycache__ │ │ │ └── inference.cpython-36.pyc │ │ ├── example.log │ │ ├── inference.py │ │ └── main.py │ ├── inference.py │ └── main.py │ └── yolo-v3 │ └── main.py ├── report └── OpenCV_Spatial_AI_Competition.pdf ├── requirements.txt └── scripts ├── inference_engine_native_myriad.sh ├── model_intel.sh ├── opencv.sh └── rpi_openvino_install-2020_1.sh /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Desktop (please complete the following information):** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | - Version [e.g. 22] 30 | 31 | **Smartphone (please complete the following information):** 32 | - Device: [e.g. iPhone6] 33 | - OS: [e.g. iOS8.1] 34 | - Browser [e.g. stock browser, safari] 35 | - Version [e.g. 22] 36 | 37 | **Additional context** 38 | Add any other context about the problem here. 39 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's: 2 | 3 | Reporting a bug 4 | Discussing the current state of the code 5 | Submitting a fix 6 | Proposing new features 7 | Becoming a maintainer 8 | We Develop with Github 9 | We use github to host code, to track issues and feature requests, as well as accept pull requests. 10 | 11 | We Use Github Flow, So All Code Changes Happen Through Pull Requests 12 | Pull requests are the best way to propose changes to the codebase (we use Github Flow). We actively welcome your pull requests: 13 | 14 | Fork the repo and create your branch from master. 15 | If you've added code that should be tested, add tests. 16 | If you've changed APIs, update the documentation. 17 | Ensure the test suite passes. 18 | Make sure your code lints. 19 | Issue that pull request! 20 | Any contributions you make will be under the MIT Software License 21 | In short, when you submit code changes, your submissions are understood to be under the same MIT License that covers the project. Feel free to contact the maintainers if that's a concern. 22 | 23 | Report bugs using Github's issues 24 | We use GitHub issues to track public bugs. Report a bug by opening a new issue; it's that easy! 25 | 26 | Write bug reports with detail, background, and sample code 27 | This is an example of a bug report I wrote, and I think it's not a bad model. Here's another example from Craig Hockenberry, an app developer whom I greatly respect. 28 | 29 | Great Bug Reports tend to have: 30 | 31 | A quick summary and/or background 32 | Steps to reproduce 33 | Be specific! 34 | Give sample code if you can. My stackoverflow question includes sample code that anyone with a base R setup can run to reproduce what I was seeing 35 | What you expected would happen 36 | What actually happens 37 | Notes (possibly including why you think this might be happening, or stuff you tried that didn't work) 38 | People love thorough bug reports. I'm not even kidding. 39 | 40 | Use a Consistent Coding Style 41 | I'm again borrowing these from Facebook's Guidelines 42 | 43 | 2 spaces for indentation rather than tabs 44 | You can try running npm run lint for style unification 45 | License 46 | By contributing, you agree that your contributions will be licensed under its MIT License. 47 | 48 | References 49 | This document was adapted from the open-source contribution guidelines for Facebook's Draft 50 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Jegathesan Shanmugam 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # deepEye - The third eye for Visually Impaired People 2 | 3 | [OpenCV](https://opencv.org/) announced its first Spatial AI Competition sponsored by Intel. As we know, OpenCV is a famous open-source computer vision library. They called for participants to solve real-world problems by using [OAK-D](https://www.kickstarter.com/projects/opencv/opencv-ai-kit) (OpenCV AI Kit with Depth) module. The OAK-D module has built-in Stereo cameras along with an RGB camera. It also has powerful visual processing unit (Myriad X from Intel) to enable deep neural network inferences on board. 4 | 5 | We decided to submit a project proposal for this competition back in July. Our group’s proposal was selected (among 32 out of 235). 6 | 7 | So, we propose to build an advanced assist system for the Visually Impaired People to perceive the environment in a better way and would provide seamless, reliable navigation for them at a low cost so that anyone can leverage the benefits of computer vision. 8 | 9 | ## Demo Videos 10 | 11 | | 👉 [deepEye Demo](https://www.youtube.com/watch?v=llE8l1IWpYA&feature=youtu.be) | 12 | | ------------------------------------------------------------ | 13 | | [![deepEye Demo](https://github.com/nullbyte91/deepEye/blob/master/images/demo.PNG)](https://www.youtube.com/watch?v=llE8l1IWpYA&feature=youtu.be) | 14 | 15 | ## Table of content 16 | 17 | - [deepEye - The third eye for Visually Impaired People](#deepEye-the-third-eye-for-visually-impaired-people) 18 | - [Demo Videos](#demo-videos) 19 | - [Table of content](#table-of-content) 20 | - [🎬 Software High Level Design](#software-high-level-design) 21 | - [🗃Project structure](#project-structure) 22 | - [💻 Hardware pre-requisite](#-hardware-pre-requisite) 23 | - [📦 Software pre-requisite](#1-software-pre-requisite-) 24 | - [For Jetson: Flash Jetson board to jetpack 4.4 ⚡️](#for-jetson-flash-jetson-board-to-jetpack-44-️) 25 | - [Depth AI Install](#depth-ai-install) 26 | - [Camera Calibration ](#camera-calibration) 27 | - [Robotic Operating System](#robotic-operating-system) 28 | - [Android RFCOMM Setup](#android-rfcomm-setup) 29 | - [Other Dependency](#other-dependency) 30 | - [🖖 Quick Start](#quick-start) 31 | - [🎛 Advanced uses](#-advanced-uses) 32 | - [Custom Object Detector](#custom-object-detector) 33 | - [🛠 Hardware Details](#hardware-details) 34 | - [💌 Acknowledgments](#-acknowledgments) 35 | 36 | ## 🎬 Software High Level Design 37 | 38 | ![HLD](images/HLD_1.JPG "Software Stack") 39 | 40 | 41 | ## 🗃 Project structure 42 | 43 | ```python 44 | . 45 | ├── android 46 | │ ├── apk # Android APK File 47 | │ │ └── app-debug.apk 48 | │ └── startup_linux 49 | │ ├── deepeye.sh # deepeye startup script to enable RFCOMM 50 | │ └── rfcomm.service # systemd service for RFCOMM 51 | | 52 | ├── custom_model 53 | │ └── OI_Dataset # Mobile Net SSD V2 Custom training on OpenImage Dataset V4 54 | │ ├── README.md 55 | │ ├── requirements.txt 56 | │ ├── scripts 57 | │ │ ├── csv2tfrecord.py # Tensorflow: CSV to TFrecord Converter 58 | │ │ ├── txt2xml.py # Tensorflow: TXT to XML Converter 59 | │ │ └── xml2csv.py # Tensorflow: XML to CSV Converter 60 | │ └── tf_test.py # Test script for Trained model inference 61 | | 62 | ├── deepeye_app # Deepeye core application 63 | │ ├── app.py # Object detection and post processing 64 | │ ├── calibration # Camera Callibration 65 | │ │ └── config 66 | │ │ └── BW1098FFC.json 67 | │ ├── collision_avoidance.py # Collision calculation 68 | │ ├── config.py 69 | │ ├── models # Mobilenet-ssd v2 trained model 70 | │ │ ├── mobilenet-ssd.blob 71 | │ │ └── mobilenet-ssd_depth.json 72 | │ ├── tracker.py # Object tracker 73 | │ └── txt2speech # txt2speech model 74 | │ ├── README.md 75 | │ ├── txt2speech.py 76 | │ └── txt-simulator.py 77 | ├── images 78 | ├── openvino_analysis # CNN model fom Intel and Opensouce ACC, FPS analysis 79 | │ ├── intel 80 | │ │ ├── object-detection 81 | │ │ └── semantic-segmentation 82 | │ ├── public 83 | │ │ ├── ssd_mobilenet_v2_coco 84 | │ │ └── yolo-v3 85 | │ └── README.md 86 | ├── README.md # Deepeye README 87 | ├── requirements.txt 88 | └── scripts # OpenVino Toolkit scripts 89 | ├── inference_engine_native_myriad.sh 90 | ├── model_intel.sh 91 | └── rpi_openvino_install-2020_1.sh 92 | ``` 93 | 94 | ## 💻 Hardware pre-requisite 95 | * [Jetson Nano](https://developer.nvidia.com/embedded/jetson-nano-developer-kit) 96 | * [BW1098FFC depthAI HW](https://docs.luxonis.com/products/bw1098ffc/) 97 | * Smartphone with Android OS 98 | 99 | ## 📦 Software pre-requisite 100 | 101 | ### For Jetson: Flash Jetson board to jetpack 4.4 ⚡️ 102 | 103 | microSD card Prepration: 104 | 1. Download Jetson Nano Developer Kit SD Card image [Jetpack4.4 Image](https://developer.nvidia.com/jetson-nano-sd-card-image). 105 | 2. Use [etcher](https://www.balena.io/etcher) to burn a image. 106 | 107 | CUDA Env PATH : 108 | ```bash 109 | if ! grep 'cuda/bin' ${HOME}/.bashrc > /dev/null ; then 110 | echo "** Add CUDA stuffs into ~/.bashrc" 111 | echo >> ${HOME}/.bashrc 112 | echo "export PATH=/usr/local/cuda/bin:\${PATH}" >> ${HOME}/.bashrc 113 | echo "export LD_LIBRARY_PATH=/usr/local/cuda/lib64:\${LD_LIBRARY_PATH}" >> ${HOME}/.bashrc 114 | fi 115 | source ${HOME}/.bashrc 116 | ``` 117 | 118 | System dependencies : 119 | ```bash 120 | sudo apt-get update 121 | sudo apt-get install -y build-essential make cmake cmake-curses-gui 122 | sudo apt-get install -y git g++ pkg-config curl libfreetype6-dev 123 | sudo apt-get install -y libcanberra-gtk-module libcanberra-gtk3-module 124 | sudo apt-get install -y python3-dev python3-testresources python3-pip 125 | sudo pip3 install -U pip 126 | ``` 127 | 128 | Performance Improvements: 129 | 130 | To set Jetson Nano to 10W performance mode (reference), execute the following from a terminal: 131 | ```bash 132 | sudo nvpmodel -m 0 133 | sudo jetson_clocks 134 | ``` 135 | Enable swap: 136 | ```bash 137 | sudo fallocate -l 8G /mnt/8GB.swap 138 | sudo mkswap /mnt/8GB.swap 139 | sudo swapon /mnt/8GB.swap 140 | if ! grep swap /etc/fstab > /dev/null; then \ 141 | echo "/mnt/8GB.swap none swap sw 0 0" | sudo tee -a /etc/fstab; \ 142 | fi 143 | ``` 144 | jetson performance analysis: 145 | ```bash 146 | pip3 install jetson-stats 147 | ``` 148 | 149 | Recompile a Jetson Linux kernel - Support RFCOMM TTY Support: 150 | 151 | We are using RFCOMM Serial protocol for Jetson-Android communication and the defauly kernel doesn't have a support for RFCOMM TTY. So, We have to recompile with new kernel config and update. 152 | 153 | ```bash 154 | # Basic Update 155 | sudo apt-get update 156 | sudo apt-get install -y libncurses5-dev 157 | 158 | # Downlaod Linux L4T(BSP) Source code from Nvidia Downlaod center 159 | wget https://developer.nvidia.com/embedded/L4T/r32_Release_v4.3/Sources/T210/public_sources.tbz2 160 | 161 | tar -xvf public_sources.tbz2 162 | 163 | cp Linux_for_Tegra/source/public/kernel_src.tbz2 ~/ 164 | 165 | pushd ~/ 166 | 167 | tar -xvf kernel_src.tbz2 168 | 169 | pushd ~/kernel/kernel-4.9 170 | 171 | zcat /proc/config.gz > .config 172 | 173 | # Enable RFCOMM TTY 174 | make menuconfig # Networking Support --> Bluetooth subsystem support ---> Select RFCOMM TTY Support ---> Save ---> Exit 175 | 176 | make prepare 177 | 178 | make modules_prepare 179 | 180 | # Compile kernel as an image file 181 | make -j5 Image 182 | 183 | # Compile all kernel modules 184 | make -j5 modules 185 | 186 | # Install modules and kernel image 187 | cd ~/kernel/kernel-4.9 188 | sudo make modules_install 189 | sudo cp arch/arm64/boot/Image /boot/Image 190 | 191 | # Reboot 192 | sudo reboot 193 | ``` 194 | 195 | ### Depth AI Python Interface Install 196 | ```bash 197 | 198 | # Install dep 199 | curl -fL http://docs.luxonis.com/install_dependencies.sh | bash 200 | sudo apt install libusb-1.0-0-dev 201 | 202 | # USB Udev 203 | echo 'SUBSYSTEM=="usb", ATTRS{idVendor}=="03e7", MODE="0666"' | sudo tee /etc/udev/rules.d/80-movidius.rules 204 | sudo udevadm control --reload-rules && sudo udevadm trigger 205 | 206 | git clone https://github.com/luxonis/depthai-python.git 207 | cd depthai-python 208 | git submodule update --init --recursive 209 | mkdir -p ~/depthai_v1 210 | python3 -m venv ~/depthai_v1 211 | python3 -m pip install -U pip 212 | python3 setup.py develop 213 | 214 | # Check the Installation 215 | python3 -c "import depthai" 216 | 217 | # Install opencv 218 | cd scripts 219 | bash opencv.sh 220 | cd .. 221 | ``` 222 | 223 | ### Camera Calibration 224 | ```bash 225 | mkdir -p ~/depthai/ && pushd ~/depthai/ 226 | git clone https://github.com/luxonis/depthai.git 227 | popd 228 | cp calibration/config/BW1098FFC.json depthAI/depthai/resources/boards/ 229 | pushd ~/depthai/ 230 | python3 calibrate.py -s 2 -brd BW1098FFC -ih 231 | ``` 232 | 233 | ### Robotic Operating System 234 | We use ROS framework multiprocess communication. 235 | ```bash 236 | sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list' 237 | sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654 238 | 239 | sudo apt update 240 | 241 | sudo apt install -y ros-melodic-ros-base 242 | 243 | # Env setup 244 | echo "source /opt/ros/melodic/setup.bash" >> ~/.bashrc 245 | source ~/.bashrc 246 | 247 | # Dep to build ROS Package 248 | sudo apt install python-rosdep python-rosinstall python-rosinstall-generator python-wstool build-essential 249 | 250 | # Install inside virtual env 251 | sudo apt install python-rosdep 252 | rosdep init 253 | 254 | rosdep update 255 | ``` 256 | 257 | ### Android RFCOMM Setup 258 | We need to configure rfcomm service in order to use the Android application for text to speech feature. 259 | 260 | ```bash 261 | sudo cp android/startup_linux/deepeye.sh /usr/bin/ 262 | sudo chmod a+x /usr/bin/deepeye.sh 263 | 264 | sudo cp android/startup_linux/rfcomm.service /etc/systemd/system/ 265 | sudo systemctl enable rfcomm 266 | ``` 267 | 268 | ### Other Dependency 269 | ```bash 270 | python3 -m pip install -r requirements.txt 271 | 272 | # SOX for txt 2 speech 273 | sudo apt-get install sox libsox-fmt-mp3 274 | 275 | ``` 276 | 277 | ## 🖖 Quick Start 278 | ```bash 279 | # Terminal one 280 | # ROS Master 281 | roscore & 282 | 283 | # Terminal Two 284 | # Deepeye core app 285 | pushd deepeye_app 286 | python3 app.py 287 | popd 288 | 289 | # Terminal three 290 | # Txt2speech middleware component 291 | pushd deepeye_app 292 | python3 txt2speech/txt2speech.py 293 | popd 294 | ``` 295 | 296 | ## 🎛 Advanced uses 297 | 298 | ### Custom Object Detector 299 | We have retrained an SSD MobileNet SSD-V2 with Open Image dataset. We picked up and trained all the object classes that help visually impaired people to navigate when they go to outdoor environments. 300 | 301 | We have added [README](https://github.com/nullbyte91/vision-system/tree/master/custom_model/OI_Dataset) for the end to end training and the OpenVino Conversion before loading to depth AI. 302 | 303 | ### 🛠 Hardware Details 304 | We plan use the DepthAI USB3 Modular Cameras[BW1098FFC] for POC. We are using RPI and Jeston. The AI/vision processing is done on the depthAI based on Myriad X Arch. 305 | 306 | ![depthAI](images/BW1098FFC_R0M0E0_diag.png "depthAI HW") 307 | 308 | 309 | Key Features of the device: 310 | 311 | * 2 BG0250TG mono camera module interfaces 312 | * 1 BG0249 RGB camera module interface 313 | * 5V power input via barrel jack 314 | * USB 3.1 Gen 1 Type-C 315 | * Pads for DepthAI SoM 1.8V SPI 316 | * Pads for DepthAI SoM 3.3V SDIO 317 | * Pads for DepthAI SoM 1.8V Aux Signals (I2C, UART, GPIO) 318 | * 5V Fan/Aux header 319 | * Pads for DepthAI SoM aux signals 320 | * Design files produced with Altium Designer 20 321 | 322 | ## 💌 Acknowledgments 323 | [DepthaAI Home Page](https://luxonis.com/depthai)
324 | [depthaAI core development](https://github.com/luxonis)
325 | [OpenVino toolkit development](https://github.com/openvinotoolkit/openvino)
326 | [BW1098FFC_DepthAI_USB3 HW](https://github.com/luxonis/depthai-hardware/tree/master/BW1098FFC_DepthAI_USB3)
327 | [OIDv4 ToolKit](https://github.com/EscVM/OIDv4_ToolKit)
328 | 329 | 330 | 331 | 332 | 333 | -------------------------------------------------------------------------------- /android/apk/app-debug.apk: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/android/apk/app-debug.apk -------------------------------------------------------------------------------- /android/startup_linux/deepeye.sh: -------------------------------------------------------------------------------- 1 | #! /bin/bash 2 | 3 | # Make BT Disoverable 4 | #hciconfig hci0 piscan 5 | 6 | # listen to RFCOMM Port 3 7 | rfcomm listen /dev/rfcomm0 3 & 8 | 9 | #chmod 777 /dev/rfcomm0 10 | -------------------------------------------------------------------------------- /android/startup_linux/rfcomm.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=rfcomm service 3 | 4 | [Service] 5 | Type=forking 6 | ExecStart=/bin/bash /usr/bin/deepeye.sh start 7 | KillMode=process 8 | 9 | [Install] 10 | WantedBy=multi-user.target 11 | -------------------------------------------------------------------------------- /custom_model/OI_Dataset/README.md: -------------------------------------------------------------------------------- 1 | # Custom Object Detector 2 | 3 | ## Dataset Download 4 | We are using OIDv4_ToolKit to download a Open Image Dataset. 5 | 6 | ```bash 7 | # Download a kit 8 | git clone https://github.com/EscVM/OIDv4_ToolKit.git 9 | 10 | cd OIDv4_ToolKit 11 | 12 | # Install dep 13 | pip3 install -r requirements.txt 14 | 15 | # Downlaod a dataset 16 | python3 main.py downloader --classes car --type_csv train 17 | python3 main.py downloader --classes car --type_csv test 18 | python3 main.py downloader --classes car --type_csv validation 19 | ``` 20 | 21 | ## Setup Object Detection Model 22 | ```bash 23 | python3 -m venv ODM 24 | source ODM/bin/activate 25 | pip3 install -r requirements.txt 26 | 27 | # Object Detection Models 28 | git clone --quiet https://github.com/tensorflow/models.git 29 | cd models 30 | git checkout 58d19c67e1d30d905dd5c6e5092348658fed80af 31 | cd research/ 32 | protoc object_detection/protos/*.proto --python_out=. 33 | export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim 34 | ``` 35 | 36 | ## TFRecord Preparation 37 | ```bash 38 | # Convert txt to xml 39 | python3 txt2xml.py 40 | 41 | # Convert xml to csv 42 | python3 xml2csv.py 43 | 44 | # Convert csv to tfrecord 45 | python3 csv2tfrecord.py 46 | ``` 47 | 48 | ## Training 49 | ```bash 50 | # Modify the ssd_mobilenet_v2_coco.config 51 | python3 model_main.py --pipeline_config_path=models/research/object_detection/samples/configs/ssd_mobilenet_v2_coco.config -num_train_steps=50 --num_eval_steps=10 --model_dir train_log 52 | ``` 53 | 54 | ## Export a Trained Inference Graph 55 | ```bash 56 | python3 export_inference_graph.py --input_type=image_tensor --pipeline_config_path=./models/research/object_detection/samples/configs/ssd_mobilenet_v2_coco.config --output_directory=./train_log --trained_checkpoint_prefix=model.ckpt-50.meta 57 | ``` 58 | 59 | ## Running Inference: Checking what the trained model can detect 60 | ```bash 61 | python3 tf_test.py 62 | ``` 63 | 64 | ## Tensorflow to Openvino model conversion 65 | ```bash 66 | sudo apt-get update && sudo apt-get install -y pciutils cpio 67 | 68 | # Download OpenVino and setup 69 | wget http://registrationcenter-download.intel.com/akdlm/irc_nas/16345/l_openvino_toolkit_p_2020.1.023.tgz 70 | 71 | chmod a+x install_GUI.sh 72 | sudo bash install_GUI.sh 73 | ``` 74 | 75 | OpenVino Model conversion: 76 | ```bash 77 | source /opt/intel/openvino/bin/setupvars.sh 78 | ``` 79 | 80 | SSD Changes: 81 | ```python 82 | with open('ssd_v2_support.json', 'r') as file : 83 | filedata = file.read() 84 | 85 | # Replace the target string 86 | filedata = filedata.replace('"Postprocessor/ToFloat"', '"Postprocessor/Cast_1"') 87 | 88 | # Write the file out again 89 | with open('ssd_v2_support.json', 'w') as file: 90 | file.write(filedata) 91 | ``` 92 | 93 | ```bash 94 | python /opt/intel/openvino/deployment_tools/model_optimizer/mo.py \ 95 | --input_model ./fine_tune_model/frozen_inference_graph.pb \ 96 | --tensorflow_use_custom_operations_config /opt/intel/openvino_2020.1.023/deployment_tools/model_optimizer/extensions/front/tf/ssd_v2_support.json \ 97 | --tensorflow_object_detection_api_pipeline_config ./fine_tune_model/pipeline.config \ 98 | --reverse_input_channels \ 99 | --data_type FP16 \ 100 | --output_dir ./openvino_out 101 | ``` 102 | 103 | OpenVino to depthAI blob conversion: 104 | ```bash 105 | python3 openvino_depthai-blob.py -x ~/Desktop/OI_dataset/l_openvino_toolkit_p_2020.1.023/output/frozen_inference_graph.xml -b ~/Desktop/OI_dataset/l_openvino_toolkit_p_2020.1.023/output/frozen_inference_graph.bin -o output 106 | ``` 107 | 108 | 109 | 110 | -------------------------------------------------------------------------------- /custom_model/OI_Dataset/requirements.txt: -------------------------------------------------------------------------------- 1 | tensorflow-gpu==1.15.4 2 | pandas 3 | tqdm 4 | opencv-python 5 | lxml 6 | tf_slim 7 | scipy 8 | pycocotools 9 | -------------------------------------------------------------------------------- /custom_model/OI_Dataset/scripts/csv2tfrecord.py: -------------------------------------------------------------------------------- 1 | """ 2 | Usage: 3 | # From tensorflow/models/ 4 | # Create train data: 5 | python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=train.record 6 | # Create test data: 7 | python generate_tfrecord.py --csv_input=data/test_labels.csv --output_path=test.record 8 | """ 9 | from __future__ import division 10 | from __future__ import print_function 11 | from __future__ import absolute_import 12 | 13 | import os 14 | import io 15 | import pandas as pd 16 | import tensorflow as tf 17 | 18 | from PIL import Image 19 | import sys 20 | sys.path.append('../') 21 | from object_detection.utils import dataset_util 22 | from collections import namedtuple, OrderedDict 23 | 24 | flags = tf.app.flags 25 | flags.DEFINE_string('csv_input', '', 'data/train_labels.csv') 26 | flags.DEFINE_string('output_path', '', 'data/train.record') 27 | flags.DEFINE_string('image_dir', '', 'images/train') 28 | FLAGS = flags.FLAGS 29 | 30 | 31 | # TO-DO replace this with label map 32 | def class_text_to_int(row_label): 33 | if row_label == 'Label1': 34 | return 1 35 | if row_label == 'Label2': 36 | return 2 37 | else: 38 | return 0 39 | 40 | 41 | def split(df, group): 42 | data = namedtuple('data', ['filename', 'object']) 43 | gb = df.groupby(group) 44 | return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)] 45 | 46 | 47 | def create_tf_example(group, path): 48 | with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid: 49 | encoded_jpg = fid.read() 50 | encoded_jpg_io = io.BytesIO(encoded_jpg) 51 | image = Image.open(encoded_jpg_io) 52 | width, height = image.size 53 | 54 | filename = group.filename.encode('utf8') 55 | image_format = b'jpg' 56 | xmins = [] 57 | xmaxs = [] 58 | ymins = [] 59 | ymaxs = [] 60 | classes_text = [] 61 | classes = [] 62 | 63 | for index, row in group.object.iterrows(): 64 | xmins.append(row['xmin'] / width) 65 | xmaxs.append(row['xmax'] / width) 66 | ymins.append(row['ymin'] / height) 67 | ymaxs.append(row['ymax'] / height) 68 | classes_text.append(row['class'].encode('utf8')) 69 | classes.append(class_text_to_int(row['class'])) 70 | 71 | tf_example = tf.train.Example(features=tf.train.Features(feature={ 72 | 'image/height': dataset_util.int64_feature(height), 73 | 'image/width': dataset_util.int64_feature(width), 74 | 'image/filename': dataset_util.bytes_feature(filename), 75 | 'image/source_id': dataset_util.bytes_feature(filename), 76 | 'image/encoded': dataset_util.bytes_feature(encoded_jpg), 77 | 'image/format': dataset_util.bytes_feature(image_format), 78 | 'image/object/bbox/xmin': dataset_util.float_list_feature(xmins), 79 | 'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs), 80 | 'image/object/bbox/ymin': dataset_util.float_list_feature(ymins), 81 | 'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs), 82 | 'image/object/class/text': dataset_util.bytes_list_feature(classes_text), 83 | 'image/object/class/label': dataset_util.int64_list_feature(classes), 84 | })) 85 | return tf_example 86 | 87 | 88 | def main(_): 89 | writer = tf.python_io.TFRecordWriter(FLAGS.output_path) 90 | path = os.path.join(FLAGS.image_dir) 91 | examples = pd.read_csv(FLAGS.csv_input) 92 | grouped = split(examples, 'filename') 93 | for group in grouped: 94 | tf_example = create_tf_example(group, path) 95 | writer.write(tf_example.SerializeToString()) 96 | 97 | writer.close() 98 | output_path = os.path.join(os.getcwd(), FLAGS.output_path) 99 | print('Successfully created the TFRecords: {}'.format(output_path)) 100 | 101 | 102 | if __name__ == '__main__': 103 | tf.app.run() -------------------------------------------------------------------------------- /custom_model/OI_Dataset/scripts/txt2xml.py: -------------------------------------------------------------------------------- 1 | import os 2 | from tqdm import tqdm 3 | from sys import exit 4 | import argparse 5 | import cv2 6 | from textwrap import dedent 7 | from lxml import etree 8 | from argparse import ArgumentParser 9 | 10 | XML_DIR = 'To_PASCAL_XML' 11 | 12 | def build_argparser(): 13 | """ 14 | Parse command line arguments. 15 | :return: command line arguments 16 | """ 17 | parser = ArgumentParser() 18 | parser.add_argument("-i", "--input", required=True, type=str, 19 | help="Dataset path") 20 | return parser 21 | 22 | args = build_argparser().parse_args() 23 | root = os.listdir(args.input) 24 | 25 | for dir in root: 26 | sub_dir = args.input + dir 27 | if os.path.isdir(sub_dir): 28 | os.chdir(sub_dir) 29 | class_dirs = os.listdir(os.getcwd()) 30 | for class_dir in class_dirs: 31 | if os.path.isdir(class_dir): 32 | os.chdir(class_dir) 33 | 34 | if not os.path.exists(XML_DIR): 35 | os.makedirs(XML_DIR) 36 | 37 | for filename in tqdm(os.listdir(os.getcwd())): 38 | if filename.endswith(".txt"): 39 | filename_str = str.split(filename, ".")[0] 40 | 41 | annotation = etree.Element("annotation") 42 | 43 | #os.chdir("..") 44 | folder = etree.Element("folder") 45 | folder.text = os.path.basename(os.getcwd()) 46 | annotation.append(folder) 47 | 48 | filename_xml = etree.Element("filename") 49 | filename_xml.text = filename_str + ".jpg" 50 | annotation.append(filename_xml) 51 | 52 | path = etree.Element("path") 53 | path.text = os.path.join(os.path.dirname(os.path.abspath(filename)), filename_str + ".jpg") 54 | annotation.append(path) 55 | 56 | source = etree.Element("source") 57 | annotation.append(source) 58 | 59 | database = etree.Element("database") 60 | database.text = "Unknown" 61 | source.append(database) 62 | 63 | size = etree.Element("size") 64 | annotation.append(size) 65 | 66 | width = etree.Element("width") 67 | height = etree.Element("height") 68 | depth = etree.Element("depth") 69 | 70 | img = cv2.imread(filename_xml.text) 71 | 72 | width.text = str(img.shape[1]) 73 | height.text = str(img.shape[0]) 74 | depth.text = str(img.shape[2]) 75 | 76 | size.append(width) 77 | size.append(height) 78 | size.append(depth) 79 | 80 | segmented = etree.Element("segmented") 81 | segmented.text = "0" 82 | annotation.append(segmented) 83 | 84 | label_original = open(filename, 'r') 85 | 86 | # Labels from OIDv4 Toolkit: name_of_class X_min Y_min X_max Y_max 87 | for line in label_original: 88 | line = line.strip() 89 | l = line.split(' ') 90 | len_str = len(l) 91 | if len_str == 6: 92 | class_name = l[0]+ "_" + l[1] 93 | xmin_l = str(int(float(l[2]))) 94 | ymin_l = str(int(float(l[3]))) 95 | xmax_l = str(int(float(l[4]))) 96 | ymax_l = str(int(float(l[5]))) 97 | else: 98 | class_name = l[0] 99 | xmin_l = str(int(float(l[1]))) 100 | ymin_l = str(int(float(l[2]))) 101 | xmax_l = str(int(float(l[3]))) 102 | ymax_l = str(int(float(l[4]))) 103 | 104 | obj = etree.Element("object") 105 | annotation.append(obj) 106 | 107 | name = etree.Element("name") 108 | name.text = class_name 109 | obj.append(name) 110 | 111 | pose = etree.Element("pose") 112 | pose.text = "Unspecified" 113 | obj.append(pose) 114 | 115 | truncated = etree.Element("truncated") 116 | truncated.text = "0" 117 | obj.append(truncated) 118 | 119 | difficult = etree.Element("difficult") 120 | difficult.text = "0" 121 | obj.append(difficult) 122 | 123 | bndbox = etree.Element("bndbox") 124 | obj.append(bndbox) 125 | 126 | xmin = etree.Element("xmin") 127 | xmin.text = xmin_l 128 | bndbox.append(xmin) 129 | 130 | ymin = etree.Element("ymin") 131 | ymin.text = ymin_l 132 | bndbox.append(ymin) 133 | 134 | xmax = etree.Element("xmax") 135 | xmax.text = xmax_l 136 | bndbox.append(xmax) 137 | 138 | ymax = etree.Element("ymax") 139 | ymax.text = ymax_l 140 | bndbox.append(ymax) 141 | 142 | os.chdir(XML_DIR) 143 | 144 | # write xml to file 145 | s = etree.tostring(annotation, pretty_print=True) 146 | with open(filename_str + ".xml", 'wb') as f: 147 | f.write(s) 148 | f.close() 149 | 150 | os.chdir("..") 151 | 152 | os.chdir("..") 153 | os.chdir("..") 154 | -------------------------------------------------------------------------------- /custom_model/OI_Dataset/scripts/xml2csv.py: -------------------------------------------------------------------------------- 1 | import os 2 | import glob 3 | import pandas as pd 4 | import xml.etree.ElementTree as ET 5 | from argparse import ArgumentParser 6 | 7 | dataset = " " 8 | 9 | def build_argparser(): 10 | """ 11 | Parse command line arguments. 12 | :return: command line arguments 13 | """ 14 | parser = ArgumentParser() 15 | parser.add_argument("-i", "--input", required=True, type=str, 16 | help="Dataset path") 17 | return parser 18 | 19 | 20 | def xml_to_csv(path): 21 | xml_list = [] 22 | classes_names = [] 23 | os.chdir(path) 24 | CLASS_DIRS = os.listdir(os.getcwd()) 25 | 26 | print(CLASS_DIRS) 27 | for CLASS_DIR in CLASS_DIRS: 28 | if os.path.isdir(CLASS_DIR): 29 | os.chdir(CLASS_DIR) 30 | print("Currently in Subdirectory:", os.path.join(os.getcwd())) 31 | path = os.path.join(os.getcwd()) + "/To_PASCAL_XML" 32 | print(path) 33 | for xml_file in glob.glob(path + '/*.xml'): 34 | #print(xml_file) 35 | tree = ET.parse(xml_file) 36 | #print(tree) 37 | root = tree.getroot() 38 | for member in root.findall('object'): 39 | classes_names.append(member[0].text) 40 | print(root.find('path').text) 41 | value = (root.find('path').text, 42 | int(root.find('size')[0].text), 43 | int(root.find('size')[1].text), 44 | member[0].text, 45 | int(member[4][0].text), 46 | int(member[4][1].text), 47 | int(member[4][2].text), 48 | int(member[4][3].text) 49 | ) 50 | xml_list.append(value) 51 | os.chdir("..") 52 | column_name = ['filename', 'width', 'height', 'class', 'xmin', 'ymin', 'xmax', 'ymax'] 53 | xml_df = pd.DataFrame(xml_list, columns=column_name) 54 | classes_names = list(set(classes_names)) 55 | classes_names.sort() 56 | return xml_df, classes_names 57 | 58 | def main(): 59 | args = build_argparser().parse_args() 60 | dataset = args.input 61 | os.chdir(dataset) 62 | for folder in ['train','test', 'validation']: 63 | image_path = os.path.join(os.getcwd(), (folder)) 64 | print(image_path) 65 | DIR = image_path 66 | if os.path.isdir(DIR): 67 | os.chdir(DIR) 68 | print("Currently in Subdirectory:", DIR) 69 | xml_df, classes_names = xml_to_csv(DIR) 70 | path = DIR + "/" + folder + '_labels.csv' 71 | print(path) 72 | xml_df.to_csv((path), index=None) 73 | print('Successfully converted xml to csv.') 74 | 75 | # Labelmap generation 76 | l_path = DIR + "/" + "label_map.pbtxt" 77 | pbtxt_content = "" 78 | for i, class_name in enumerate(classes_names): 79 | pbtxt_content = ( 80 | pbtxt_content 81 | + "item {{\n id: {0}\n name: '{1}'\n}}\n\n".format( 82 | i + 1, class_name 83 | ) 84 | ) 85 | pbtxt_content = pbtxt_content.strip() 86 | with open(l_path, "w") as f: 87 | f.write(pbtxt_content) 88 | os.chdir("..") 89 | os.chdir("..") 90 | main() -------------------------------------------------------------------------------- /custom_model/OI_Dataset/tf_test.py: -------------------------------------------------------------------------------- 1 | import os 2 | import glob 3 | 4 | import numpy as np 5 | import os 6 | import six.moves.urllib as urllib 7 | import sys 8 | import tarfile 9 | import tensorflow as tf 10 | import zipfile 11 | import cv2 12 | 13 | from collections import defaultdict 14 | from io import StringIO 15 | # This is needed to display the images. 16 | 17 | import matplotlib.pyplot as plt 18 | from PIL import Image 19 | from argparse import ArgumentParser 20 | 21 | # This is needed since the notebook is stored in the object_detection folder. 22 | sys.path.append("..") 23 | from object_detection.utils import ops as utils_ops 24 | 25 | from object_detection.utils import label_map_util 26 | 27 | from object_detection.utils import visualization_utils as vis_util 28 | 29 | def build_argparser(): 30 | """ 31 | Parse command line arguments. 32 | :return: command line arguments 33 | """ 34 | parser = ArgumentParser() 35 | parser.add_argument("-m", "--model", required=True, type=str, 36 | help="Model path") 37 | parser.add_argument("-l", "--label", required=True, type=str, 38 | help="Label path") 39 | parser.add_argument("-i", "--input", required=True, type=str, 40 | help="input images") 41 | 42 | return parser 43 | 44 | args = build_argparser().parse_args() 45 | 46 | output_directory = args.model 47 | PATH_TO_LABELS = args.label 48 | dataset = args.input 49 | 50 | pb_fname = os.path.join(os.path.abspath(output_directory), "frozen_inference_graph.pb") 51 | 52 | print(pb_fname) 53 | 54 | def get_num_classes(pbtxt_fname): 55 | from object_detection.utils import label_map_util 56 | label_map = label_map_util.load_labelmap(pbtxt_fname) 57 | categories = label_map_util.convert_label_map_to_categories( 58 | label_map, max_num_classes=90, use_display_name=True) 59 | category_index = label_map_util.create_category_index(categories) 60 | return len(category_index.keys()) 61 | 62 | num_classes = get_num_classes(PATH_TO_LABELS) 63 | 64 | PATH_TO_CKPT = pb_fname 65 | detection_graph = tf.Graph() 66 | with detection_graph.as_default(): 67 | od_graph_def = tf.GraphDef() 68 | with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid: 69 | serialized_graph = fid.read() 70 | od_graph_def.ParseFromString(serialized_graph) 71 | tf.import_graph_def(od_graph_def, name='') 72 | 73 | 74 | 75 | label_map = label_map_util.load_labelmap(PATH_TO_LABELS) 76 | categories = label_map_util.convert_label_map_to_categories( 77 | label_map, max_num_classes=num_classes, use_display_name=True) 78 | category_index = label_map_util.create_category_index(categories) 79 | 80 | 81 | def load_image_into_numpy_array(image): 82 | (im_width, im_height) = image.size 83 | return np.array(image.getdata()).reshape( 84 | (im_height, im_width, 3)).astype(np.uint8) 85 | 86 | # Size, in inches, of the output images. 87 | IMAGE_SIZE = (12, 8) 88 | 89 | 90 | def run_inference_for_single_image(image, graph): 91 | with graph.as_default(): 92 | with tf.Session() as sess: 93 | # Get handles to input and output tensors 94 | ops = tf.get_default_graph().get_operations() 95 | all_tensor_names = { 96 | output.name for op in ops for output in op.outputs} 97 | tensor_dict = {} 98 | for key in [ 99 | 'num_detections', 'detection_boxes', 'detection_scores', 100 | 'detection_classes', 'detection_masks' 101 | ]: 102 | tensor_name = key + ':0' 103 | if tensor_name in all_tensor_names: 104 | tensor_dict[key] = tf.get_default_graph().get_tensor_by_name( 105 | tensor_name) 106 | if 'detection_masks' in tensor_dict: 107 | # The following processing is only for single image 108 | detection_boxes = tf.squeeze( 109 | tensor_dict['detection_boxes'], [0]) 110 | detection_masks = tf.squeeze( 111 | tensor_dict['detection_masks'], [0]) 112 | # Reframe is required to translate mask from box coordinates to image coordinates and fit the image size. 113 | real_num_detection = tf.cast( 114 | tensor_dict['num_detections'][0], tf.int32) 115 | detection_boxes = tf.slice(detection_boxes, [0, 0], [ 116 | real_num_detection, -1]) 117 | detection_masks = tf.slice(detection_masks, [0, 0, 0], [ 118 | real_num_detection, -1, -1]) 119 | detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks( 120 | detection_masks, detection_boxes, image.shape[0], image.shape[1]) 121 | detection_masks_reframed = tf.cast( 122 | tf.greater(detection_masks_reframed, 0.5), tf.uint8) 123 | # Follow the convention by adding back the batch dimension 124 | tensor_dict['detection_masks'] = tf.expand_dims( 125 | detection_masks_reframed, 0) 126 | image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0') 127 | 128 | # Run inference 129 | output_dict = sess.run(tensor_dict, 130 | feed_dict={image_tensor: np.expand_dims(image, 0)}) 131 | 132 | # all outputs are float32 numpy arrays, so convert types as appropriate 133 | output_dict['num_detections'] = int( 134 | output_dict['num_detections'][0]) 135 | output_dict['detection_classes'] = output_dict[ 136 | 'detection_classes'][0].astype(np.uint8) 137 | output_dict['detection_boxes'] = output_dict['detection_boxes'][0] 138 | output_dict['detection_scores'] = output_dict['detection_scores'][0] 139 | if 'detection_masks' in output_dict: 140 | output_dict['detection_masks'] = output_dict['detection_masks'][0] 141 | return output_dict 142 | 143 | 144 | TEST_IMAGE_PATHS = glob.glob(os.path.join(dataset, "*.jpg*")) 145 | print(TEST_IMAGE_PATHS) 146 | 147 | for image_path in TEST_IMAGE_PATHS: 148 | image = Image.open(image_path) 149 | print(image_path) 150 | # the array based representation of the image will be used later in order to prepare the 151 | # result image with boxes and labels on it. 152 | image_np = load_image_into_numpy_array(image) 153 | # Expand dimensions since the model expects images to have shape: [1, None, None, 3] 154 | image_np_expanded = np.expand_dims(image_np, axis=0) 155 | # Actual detection. 156 | output_dict = run_inference_for_single_image(image_np, detection_graph) 157 | # Visualization of the results of a detection. 158 | vis_util.visualize_boxes_and_labels_on_image_array( 159 | image_np, 160 | output_dict['detection_boxes'], 161 | output_dict['detection_classes'], 162 | output_dict['detection_scores'], 163 | category_index, 164 | instance_masks=output_dict.get('detection_masks'), 165 | use_normalized_coordinates=True, 166 | line_thickness=8) 167 | #print(image_np) 168 | #exit(1) 169 | cv2.imshow("window", image_np) 170 | cv2.waitKey(0) 171 | 172 | # plt.figure(figsize=IMAGE_SIZE) 173 | # plt.imshow(image_np) 174 | # plt.show() -------------------------------------------------------------------------------- /deepeye_app/app.py: -------------------------------------------------------------------------------- 1 | import depthai #core packages 2 | import logging # Logger 3 | import cv2 # computer vision package 4 | import json #json lib 5 | from time import time, sleep, monotonic 6 | 7 | from pathlib import Path #Path lib 8 | from config import model, calib 9 | 10 | # Object Tracker 11 | from tracker import Tracker 12 | from collision_avoidance import CrashAvoidance 13 | 14 | log = logging.getLogger(__name__) 15 | 16 | DEBUG = 1 17 | class DepthAI: 18 | 19 | process_watchdog_timeout=10 #seconds 20 | 21 | def create_pipeline(self, config): 22 | self.device = depthai.Device('', False) 23 | log.info("Creating DepthAI pipeline...") 24 | 25 | self.pipeline = self.device.create_pipeline(config) 26 | if self.pipeline is None: 27 | raise RuntimeError("Pipeline was not created.") 28 | log.info("Pipeline created.") 29 | 30 | 31 | def reset_process_wd(self): 32 | global wd_cutoff 33 | wd_cutoff=monotonic()+self.process_watchdog_timeout 34 | return 35 | 36 | def __init__(self): 37 | self.threshold = 0.5 38 | 39 | self.config = config = { 40 | 'streams': ['metaout', 'previewout'], 41 | 'depth': 42 | {'calibration_file': str(Path(calib, 'depthai.calib').absolute()), 43 | 'left_mesh_file': '', 44 | 'right_mesh_file': '', 45 | 'padding_factor': 0.3, 46 | 'depth_limit_m': 10.0, 47 | 'median_kernel_size': 7, 48 | 'lr_check': False, 49 | 'warp_rectify': 50 | { 51 | 'use_mesh': False, 52 | 'mirror_frame': True, 53 | 'edge_fill_color': 0 54 | } 55 | }, 56 | 'ai': 57 | {'blob_file': str(Path(model, 'mobilenet-ssd_v1.blob').absolute()), 58 | 'blob_file_config': str(Path(model, 'mobilenet-ssd_v1.json').absolute()), 59 | 'blob_file2': '', 60 | 'blob_file_config2': '', 61 | 'calc_dist_to_bb': True, 62 | 'keep_aspect_ratio': True, 63 | 'camera_input': 'rgb', 64 | 'shaves': 14, 65 | 'cmx_slices': 14, 66 | 'NN_engines': 1 67 | }, 68 | 'ot': 69 | {'max_tracklets': 20, 70 | 'confidence_threshold': 0.5 71 | }, 72 | 'board_config': 73 | {'swap_left_and_right_cameras': True, 74 | 'left_fov_deg': 71.86, 75 | 'rgb_fov_deg': 68.7938, 76 | 'left_to_right_distance_cm': 9.0, 77 | 'left_to_rgb_distance_cm': 2.0, 78 | 'store_to_eeprom': False, 79 | 'clear_eeprom': False, 80 | 'override_eeprom': False 81 | }, 82 | 'camera': 83 | {'rgb': 84 | {'resolution_h': 1080, 85 | 'fps': 30.0}, 86 | 'mono': 87 | {'resolution_h': 720, 88 | 'fps': 30.0 89 | } 90 | }, 91 | 'app': 92 | {'sync_video_meta_streams': False, 93 | 'sync_sequence_numbers': False, 94 | 'usb_chunk_KiB': 64 95 | } 96 | } 97 | # Create a pipeline config 98 | self.create_pipeline(self.config) 99 | 100 | blob_file = str(Path(model, 'mobilenet-ssd_depth.json')) 101 | 102 | with open(blob_file) as f: 103 | self.data = json.load(f) 104 | 105 | self.detection = [] 106 | 107 | # Calculate collision_avoidance for bicycle 108 | # bus, car, dog, horse, motorbike, train 109 | self.collision_avoidance = [2.0, 6.0, 7.0, 10.0, 12.0, 13.0, 14.0, 17.0] 110 | # Label map 111 | self.label = { 112 | 0.0: "background", 113 | 1.0: "aeroplane", 114 | 2.0: "bicycle", 115 | 3.0: "bird", 116 | 4.0: "boat", 117 | 5.0: "bottle", 118 | 6.0: "bus", 119 | 7.0: "car", 120 | 8.0: "cat", 121 | 9.0: "chair", 122 | 10.0: "cow", 123 | 11.0: "diningtable", 124 | 12.0: "dog", 125 | 13.0: "horse", 126 | 14.0: "motorbike", 127 | 15.0: "person", 128 | 16.0: "pottedplant", 129 | 17.0: "sheep", 130 | 18.0: "sofa", 131 | 19.0: "train", 132 | 20.0: "tvmonitor", 133 | } 134 | 135 | self.reset_process_wd() 136 | 137 | def image_resize(self, image, width = None, height = None, inter = cv2.INTER_AREA): 138 | # initialize the dimensions of the image to be resized and 139 | # grab the image size 140 | dim = None 141 | (h, w) = image.shape[:2] 142 | 143 | # if both the width and height are None, then return the 144 | # original image 145 | if width is None and height is None: 146 | return image 147 | 148 | # check to see if the width is None 149 | if width is None: 150 | # calculate the ratio of the height and construct the 151 | # dimensions 152 | r = height / float(h) 153 | dim = (int(w * r), height) 154 | 155 | # otherwise, the height is None 156 | else: 157 | # calculate the ratio of the width and construct the 158 | # dimensions 159 | r = width / float(w) 160 | dim = (width, int(h * r)) 161 | 162 | # resize the image 163 | resized = cv2.resize(image, dim, interpolation = inter) 164 | 165 | # return the resized image 166 | return resized 167 | 168 | def capture(self): 169 | cv2.namedWindow("output", cv2.WINDOW_NORMAL) 170 | 171 | # Label mapping 172 | try: 173 | labels = self.data['mappings']['labels'] 174 | except: 175 | labels = None 176 | print("Labels not found in json!") 177 | 178 | while True: 179 | #print("debug") 180 | # retreive data from the device 181 | # data is stored in packets, there are nnet (Neural NETwork) packets which have additional functions for NNet result interpretation 182 | nnet_packets, data_packets = self.pipeline.get_available_nnet_and_data_packets() 183 | 184 | packets_len = len(nnet_packets) + len(data_packets) 185 | if packets_len != 0: 186 | self.reset_process_wd() 187 | else: 188 | cur_time=monotonic() 189 | if cur_time > wd_cutoff: 190 | print("process watchdog timeout") 191 | os._exit(10) 192 | 193 | # Get the detection 194 | for _, nnet_packet in enumerate(nnet_packets): 195 | self.detection = [] 196 | detections = nnet_packet.getDetectedObjects() 197 | for detection in detections: 198 | detection_dict = detection.get_dict() 199 | if detection_dict['confidence'] > 0.5: 200 | self.detection.append(detection_dict) 201 | 202 | # Post processing 203 | boxes = [] 204 | for packet in data_packets: 205 | if packet.stream_name == 'previewout': 206 | data = packet.getData() 207 | if data is None: 208 | continue 209 | # The format of previewout image is CHW (Chanel, Height, Width), but OpenCV needs HWC, so we 210 | # change shape (3, 300, 300) -> (300, 300, 3). 211 | data0 = data[0, :, :] 212 | data1 = data[1, :, :] 213 | data2 = data[2, :, :] 214 | frame = cv2.merge([data0, data1, data2]) 215 | 216 | img_h = frame.shape[0] 217 | img_w = frame.shape[1] 218 | 219 | frame_o = frame.copy() 220 | for e in self.detection: 221 | color = (0, 255, 0) # bgr 222 | label = e['label'] 223 | if label in self.collision_avoidance: 224 | # Create dic for tracking 225 | boxes.append({ 226 | 'detector': "MobileNet SSD", 227 | 'label': self.label[label], 228 | 'conf': e['confidence'], 229 | 'left': int(e['x_min'] * img_w), 230 | 'top': int(e['y_min'] * img_h), 231 | 'right': int(e['x_max'] * img_w), 232 | 'bottom': int(e['y_max'] * img_h), 233 | 'distance_x': e['depth_x'], 234 | 'distance_y': e['depth_y'], 235 | 'distance_z': e['depth_z'], 236 | }) 237 | color = (0, 0, 255) # bgr 238 | 239 | pt1 = int(e['x_min'] * img_w), int(e['y_min'] * img_h) 240 | pt2 = int(e['x_max'] * img_w), int(e['y_max'] * img_h) 241 | 242 | x1, y1 = pt1 243 | x2, y2 = pt2 244 | 245 | cv2.rectangle(frame, pt1, pt2, color) 246 | 247 | pt_t1 = x1, y1 + 20 248 | cv2.putText(frame, labels[int(e['label'])], pt_t1, cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2) 249 | 250 | pt_t2 = x1, y1 + 40 251 | cv2.putText(frame, '{:.2f}'.format(100*e['confidence']) + ' %', pt_t2, cv2.FONT_HERSHEY_SIMPLEX, 0.5, color) 252 | 253 | if DEBUG: 254 | if self.config['ai']['calc_dist_to_bb']: 255 | pt_t3 = x1, y1 + 60 256 | cv2.putText(frame, 'x1:' '{:7.3f}'.format(e['depth_x']) + ' m', pt_t3, cv2.FONT_HERSHEY_SIMPLEX, 0.5, color) 257 | 258 | pt_t4 = x1, y1 + 80 259 | cv2.putText(frame, 'y1:' '{:7.3f}'.format(e['depth_y']) + ' m', pt_t4, cv2.FONT_HERSHEY_SIMPLEX, 0.5, color) 260 | 261 | pt_t5 = x1, y1 + 100 262 | cv2.putText(frame, 'z1:' '{:7.3f}'.format(e['depth_z']) + ' m', pt_t5, cv2.FONT_HERSHEY_SIMPLEX, 0.5, color) 263 | 264 | # frame = self.image_resize(frame, 500) 265 | 266 | cv2.imshow("output", frame) 267 | 268 | if cv2.waitKey(1) == ord('q'): 269 | cv2.destroyAllWindows() 270 | exit(0) 271 | 272 | yield frame_o, boxes 273 | def main(): 274 | # depth AI 275 | di = DepthAI() 276 | 277 | # Tracker 278 | tracker = Tracker(log) 279 | # Cross avoidance 280 | crash_avoidance = CrashAvoidance() 281 | 282 | for frame, results in di.capture(): 283 | # Pass class along with Z and X coordinates 284 | pts_l = [(item['distance_x'], item['distance_z'], item['label']) for item in results] 285 | 286 | # Pass the points to tracker 287 | tracker_objs, obj_class = tracker.update(pts_l, log) 288 | 289 | #Pass the tracker objects to collision_avoidance 290 | crash_alert = crash_avoidance.parse(tracker_objs, obj_class) 291 | 292 | if __name__ == "__main__": 293 | main() -------------------------------------------------------------------------------- /deepeye_app/calibration/config/BW1098FFC.json: -------------------------------------------------------------------------------- 1 | { 2 | "board_config": 3 | { 4 | "name": "BW1098FFC", 5 | "revision": "R0M0E0", 6 | "swap_left_and_right_cameras": true, 7 | "left_fov_deg": 71.86, 8 | "rgb_fov_deg": 68.7938, 9 | "left_to_right_distance_cm": 5.0, 10 | "left_to_rgb_distance_cm": 2.5 11 | } 12 | } 13 | 14 | -------------------------------------------------------------------------------- /deepeye_app/calibration/depthai.calib: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/deepeye_app/calibration/depthai.calib -------------------------------------------------------------------------------- /deepeye_app/collision_avoidance.py: -------------------------------------------------------------------------------- 1 | import math 2 | import time 3 | from collections import OrderedDict 4 | 5 | import cv2 6 | import numpy as np 7 | 8 | 9 | # ROS 10 | import rospy 11 | from std_msgs.msg import String 12 | 13 | # To do: 14 | # Try to get the debug on and off from the config file 15 | 16 | DEBUG = 1 17 | 18 | class CrashAvoidance: 19 | def __init__(self, calculated_entries=5, collision_trajectory_threshold=0.05, collision_time_to_impact=4): 20 | self.calculated_entries = calculated_entries 21 | self.collision_trajectory_threshold = collision_trajectory_threshold 22 | self.collision_time_to_impact = collision_time_to_impact 23 | self.entries = OrderedDict() 24 | # ROS 25 | rospy.init_node('publisher', anonymous=True) 26 | self.pub = rospy.Publisher('depthai', String, queue_size=10) 27 | rate = rospy.Rate(10) # 10hz 28 | 29 | self.object_ID = None 30 | 31 | def best_fit_slope_and_intercept(self, objectID): 32 | points = [item['value'] for item in self.entries[objectID]] 33 | xs = np.array([item[0] for item in points]) 34 | zs = np.array([item[1] for item in points]) 35 | 36 | m, b = np.polyfit(xs, zs, 1) 37 | 38 | return m, b 39 | 40 | def is_dangerous_trajectory(self, objectID): 41 | last = self.entries[objectID][0] 42 | 43 | 44 | # Update the class name 45 | class_name = last['class'] 46 | class_name = class_name[0] 47 | try: 48 | m, b = self.best_fit_slope_and_intercept(objectID) 49 | except ValueError: 50 | return False 51 | distance = abs(b) / math.sqrt(math.pow(m, 2) + 1) 52 | 53 | if DEBUG: 54 | image = np.ones((200, 200)) * 255 55 | 56 | cv2.line(image, (0, int(-100 * m + b)), (200, int(100 * m + b)), (0, 0, 0)) 57 | cv2.putText(image, f"Distance: {round(distance, 2)}", (20, 20), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0)) 58 | string = "" 59 | if distance < self.collision_trajectory_threshold: 60 | # We need to cal and update the oppsite side object 61 | if (m < 0): 62 | string = "A " + class_name + " is on your" + "left side " 63 | else: 64 | string = "A " + class_name + " is on your" + " right side " 65 | 66 | cv2.putText(image, "DANGER", (20, 100), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255)) 67 | cv2.putText(image, class_name, (100, 180), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255)) 68 | 69 | # Track the existing object and dont update the string 70 | # until get the new object ID 71 | 72 | if self.object_ID != objectID: 73 | self.object_ID = objectID 74 | # Publish the trajectory object 75 | print(string) 76 | self.pub.publish(string) 77 | 78 | cv2.imshow(f"trajectory", image) 79 | cv2.waitKey(1) 80 | 81 | return distance < self.collision_trajectory_threshold 82 | 83 | def is_impact_close(self, objectID): 84 | last, first = self.entries[objectID][0], self.entries[objectID][-1] 85 | x1, z1 = last['value'] 86 | x2, z2 = first['value'] 87 | lf_distance = math.sqrt(math.pow(x1 - x2, 2) + math.pow(z1 - z2, 2)) 88 | if lf_distance == 0: 89 | return False 90 | timed = first['timestamp'] - last['timestamp'] 91 | speed = lf_distance / timed # m/s 92 | target_distance = math.sqrt(math.pow(x2, 2) + math.pow(z2, 2)) 93 | tti = target_distance / speed 94 | 95 | if DEBUG: 96 | image = np.ones((200, 200)) * 255 97 | cv2.putText(image, f"LFD: {round(lf_distance, 2)}", (20, 20), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0)) 98 | cv2.putText(image, f"SPD: {round(speed, 2)}", (20, 60), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0)) 99 | cv2.putText(image, f"TDT: {round(target_distance, 2)}", (20, 100), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0)) 100 | cv2.putText(image, f"TTI: {round(tti, 2)}", (20, 140), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0)) 101 | if tti < self.collision_trajectory_threshold: 102 | cv2.putText(image, "DANGER", (80, 180), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255)) 103 | cv2.imshow(f"impact", image) 104 | cv2.waitKey(1) 105 | return tti < self.collision_time_to_impact 106 | 107 | def parse(self, tracker_objects, obj_class): 108 | #print(type(tracker_objects)) 109 | for key in list(self.entries.keys()): 110 | #print(key) 111 | if key not in tracker_objects: 112 | del self.entries[key] 113 | 114 | for key in tracker_objects.keys(): 115 | # print(tracker_objects[key]) 116 | # print(obj_class[key]) 117 | item = { 118 | 'timestamp': time.time(), 119 | 'value': tracker_objects[key], 120 | 'class': obj_class[key] 121 | } 122 | 123 | if key not in self.entries: 124 | self.entries[key] = [item] 125 | else: 126 | self.entries[key] = (self.entries[key] + [item])[-self.calculated_entries:] 127 | 128 | if len(self.entries[key]) > 2 and self.is_dangerous_trajectory(key) and self.is_impact_close(key): 129 | return True 130 | else: 131 | return False 132 | 133 | 134 | 135 | -------------------------------------------------------------------------------- /deepeye_app/config.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import os 3 | import sys 4 | from pathlib import Path 5 | from concurrent_log_handler import ConcurrentRotatingFileHandler 6 | 7 | # Log configuration 8 | root = logging.getLogger() 9 | handler = logging.StreamHandler(sys.stdout) 10 | formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') 11 | handler.setFormatter(formatter) 12 | root.addHandler(handler) 13 | logfile = os.path.abspath("camera.log") 14 | 15 | # Rotate log after reaching 512K, keep 5 old copies. 16 | rotateHandler = ConcurrentRotatingFileHandler(logfile, "a", 512*1024, 5) 17 | rotateHandler.setFormatter(formatter) 18 | root.addHandler(rotateHandler) 19 | root.setLevel(logging.INFO) 20 | root.info("Logging system initialized, kept in file {}...".format(logfile)) 21 | 22 | # Model path 23 | model = str(Path('models/').resolve()) 24 | # calibration path 25 | calib = str(Path('calibration/').resolve()) 26 | -------------------------------------------------------------------------------- /deepeye_app/models/mobilenet-ssd.blob: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/deepeye_app/models/mobilenet-ssd.blob -------------------------------------------------------------------------------- /deepeye_app/models/mobilenet-ssd_depth.json: -------------------------------------------------------------------------------- 1 | { 2 | "tensors": 3 | [ 4 | { 5 | "output_tensor_name": "out", 6 | "output_dimensions": [1, 1, 100, 10], 7 | "output_entry_iteration_index": 2, 8 | "output_properties_dimensions": [3], 9 | "property_key_mapping": 10 | [ 11 | [], 12 | [], 13 | [], 14 | ["image_id", "label", "conf", "x_min", "y_min", "x_max", "y_max", "distance_x", "distance_y", "distance_z"] 15 | ], 16 | "output_properties_type": "f16" 17 | } 18 | ], 19 | "mappings": 20 | { 21 | "labels": 22 | [ 23 | "background", 24 | "aeroplane", 25 | "bicycle", 26 | "bird", 27 | "boat", 28 | "bottle", 29 | "bus", 30 | "car", 31 | "cat", 32 | "chair", 33 | "cow", 34 | "diningtable", 35 | "dog", 36 | "horse", 37 | "motorbike", 38 | "person", 39 | "pottedplant", 40 | "sheep", 41 | "sofa", 42 | "train", 43 | "tvmonitor" 44 | ] 45 | } 46 | } 47 | 48 | -------------------------------------------------------------------------------- /deepeye_app/models/mobilenet-ssd_v1.blob: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/deepeye_app/models/mobilenet-ssd_v1.blob -------------------------------------------------------------------------------- /deepeye_app/models/mobilenet-ssd_v1.json: -------------------------------------------------------------------------------- 1 | { 2 | "NN_config": 3 | { 4 | "output_format" : "detection", 5 | "NN_family" : "mobilenet", 6 | "confidence_threshold" : 0.5 7 | }, 8 | "mappings": 9 | { 10 | "labels": 11 | [ 12 | "background", 13 | "aeroplane", 14 | "bicycle", 15 | "bird", 16 | "boat", 17 | "bottle", 18 | "bus", 19 | "car", 20 | "cat", 21 | "chair", 22 | "cow", 23 | "diningtable", 24 | "dog", 25 | "horse", 26 | "motorbike", 27 | "person", 28 | "pottedplant", 29 | "sheep", 30 | "sofa", 31 | "train", 32 | "tvmonitor" 33 | ] 34 | } 35 | } 36 | 37 | -------------------------------------------------------------------------------- /deepeye_app/tracker.py: -------------------------------------------------------------------------------- 1 | from collections import OrderedDict 2 | 3 | import cv2 4 | from scipy.spatial import distance as dist 5 | 6 | import numpy as np 7 | 8 | 9 | class Tracker: 10 | def __init__(self, log, maxDisappeared=50, maxDistance=50, maxHistory=100, ): 11 | self.nextObjectID = 0 12 | self.objects = OrderedDict() 13 | self.disappeared = OrderedDict() 14 | self.maxDisappeared = maxDisappeared 15 | self.maxDistance = maxDistance 16 | self.maxHistory = maxHistory 17 | self.obj_class = OrderedDict() 18 | self.colors = OrderedDict() 19 | self.history = OrderedDict() 20 | 21 | self.log = log 22 | 23 | def register(self, centroid, obj_cls): 24 | self.history[self.nextObjectID] = [centroid] 25 | self.objects[self.nextObjectID] = centroid 26 | self.obj_class[self.nextObjectID] = obj_cls 27 | self.disappeared[self.nextObjectID] = 0 28 | self.colors[self.nextObjectID] = np.random.choice(range(256), size=3).astype('i').tolist() 29 | self.nextObjectID += 1 30 | 31 | def deregister(self, objectID): 32 | del self.objects[objectID] 33 | del self.disappeared[objectID] 34 | del self.obj_class[objectID] 35 | del self.colors[objectID] 36 | del self.history[objectID] 37 | 38 | def update(self, pts_l, log): 39 | pts = [] 40 | for item in pts_l: 41 | pts.append(item[0:2]) 42 | 43 | if len(pts) == 0: 44 | for objectID in list(self.disappeared.keys()): 45 | self.disappeared[objectID] += 1 46 | 47 | if self.disappeared[objectID] > self.maxDisappeared: 48 | self.deregister(objectID) 49 | 50 | return self.objects, self.obj_class 51 | 52 | if len(self.objects) == 0: 53 | for pt in pts_l: 54 | self.register(pt[0:2], pt[2:3]) 55 | 56 | else: 57 | # grab the set of object IDs and corresponding centroids 58 | objectIDs = list(self.objects.keys()) 59 | objectCentroids = list(self.objects.values()) 60 | 61 | D = dist.cdist(np.array(objectCentroids), pts) 62 | rows = D.min(axis=1).argsort() 63 | cols = D.argmin(axis=1)[rows] 64 | usedRows = set() 65 | usedCols = set() 66 | 67 | for (row, col) in zip(rows, cols): 68 | if row in usedRows or col in usedCols: 69 | continue 70 | 71 | if D[row, col] > self.maxDistance: 72 | continue 73 | 74 | objectID = objectIDs[row] 75 | self.objects[objectID] = pts[col] 76 | self.history[objectID] = (self.history[objectID] + [pts[col]])[-self.maxHistory:] 77 | self.disappeared[objectID] = 0 78 | 79 | usedRows.add(row) 80 | usedCols.add(col) 81 | 82 | unusedRows = set(range(0, D.shape[0])).difference(usedRows) 83 | unusedCols = set(range(0, D.shape[1])).difference(usedCols) 84 | 85 | if D.shape[0] >= D.shape[1]: 86 | for row in unusedRows: 87 | objectID = objectIDs[row] 88 | self.disappeared[objectID] += 1 89 | 90 | if self.disappeared[objectID] > self.maxDisappeared: 91 | self.deregister(objectID) 92 | else: 93 | for col in unusedCols: 94 | self.register(pts[col], "None") 95 | 96 | return self.objects, self.obj_class 97 | -------------------------------------------------------------------------------- /deepeye_app/txt2speech/README.md: -------------------------------------------------------------------------------- 1 | # Text to speech support for deepEye 2 | 3 | ## Introduction 4 | This software component is to detect the moving objects like bicycle, car, bus, dog, horse, motorbike towards the camera, and alert them through speaker/headphone if there is a possible collision. 5 | 6 | We plan to support this feature both edge device and mobile app conversion. For example, if the user connected deepEye device via BT then send the text to the mobile app directly and convert them if not the conversion happens on edge devices. 7 | 8 | ## Software block 9 | ### On Device conversion block: 10 | 11 | ![alt text](s_images/txt2s.png) 12 | 13 | ### Mobile Device conversion 14 | ![alt text](s_images/txt2s_bt.png) 15 | 16 | 17 | ### Dep 18 | ```bash 19 | # google speech 20 | sudo apt-get install sox libsox-fmt-mp3 21 | 22 | python3 -m pip install pip install google-speech sox 23 | 24 | 25 | # Mycroft Mimic V1 26 | git clone https://github.com/MycroftAI/mimic1.git 27 | cd mimic1 28 | ./dependencies.sh --prefix="/usr/local" 29 | ./autogen.sh 30 | ./configure --prefix="/usr/local" 31 | make 32 | make check 33 | ``` -------------------------------------------------------------------------------- /deepeye_app/txt2speech/s_images/txt2s.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/deepeye_app/txt2speech/s_images/txt2s.png -------------------------------------------------------------------------------- /deepeye_app/txt2speech/s_images/txt2s_bt.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/deepeye_app/txt2speech/s_images/txt2s_bt.png -------------------------------------------------------------------------------- /deepeye_app/txt2speech/sw_design/txt2s_BT: -------------------------------------------------------------------------------- 1 | 3Vhdb5swFP01PK4CDCR5bNKue1ikTenW9dHFt2DNYGScBvbrZwfznShrlZSpUh7w8Rf33HOuLrHQKinuBM7iNSfALNcmhYVuLNd1PNe19M8mZYXMEaqASFBiFrXAhv4BA9oG3VICeW+h5JxJmvXBkKcphLKHYSH4rr/smbP+rRmOYARsQszG6AMlMjZRuLMW/wI0iuubnWBRzSS4XmwiyWNM+K4DoVsLrQTnsnpKihUwTV7NS7Xv85HZ5sUEpPJfNiT3X32+vi9z/Fz8uFt+z+OH5Se/OuUFs60JOOSM0ZzyVBP4winBaQg6gSAVv3t8H40sa4oE36YE9C2OhZa7mErYZDjUszslCoXFMmFm2twHQkJxNBCnoUfpCngCUpRqidkQ2IZRIynPDHdtfpzAYHEnNzWGjSSi5uSWNfVgiHsFibMRiZtYRUMUtoaEmxv+J8oafibjbD7ibEQSpORaO1iNQobznIZ9XqCg8pd6tq98M3rszNzo0O16UJpBdQmQkekHTKoX4VsRwinrjBnvMOofILTGBDAs6Uv/NQ6xbG74xql6wTah/uLK76fUGaSqCsDs69aF4VHzI9qoD5JYRCBHB+3T3gT+diUsxu6RgqaR5QY40VZIn/KsSV1HH8oOsq+IXAr+G1accaGQlKdq5fKZMjaAMKNRqmWlkg4KX2pzUVX0r81EQgnR1xy0Zmte+zzunA3dicbu9A5oyb2UOWspdXIiC+nmGUAYK/w6yyYvae4x2U5W0urO5iw1rVPR2vp28Zo2m7Km+d4go/Omxr22pnmnj7pwVUPeSAwEILstofIPU+Wm6qYCpqvYk1BPkdyn80Tdm9ppzXg6p43b1nfpHhRhouxs0sPH7ly7bT+6gEPr0lzJ96QAp/IyWgx79Dd2J0MnI/99uxMUTCO1M0rmgygBBScOurQSxp95PznV38YftS9FA+95zsX6UjVs/wmpMtb+n4Ru/wI= -------------------------------------------------------------------------------- /deepeye_app/txt2speech/sw_design/txt2speech: -------------------------------------------------------------------------------- 1 | 3Vpbc9soGP01nuk+tCOBLMePcdptd2bTy7ptNo9UEIspAg3Csd1fX7DQFSe+xLZUz2QS8SGBOOe7KwN4kyzfS5TGtwITNgAeXg7g2wEAfgDAwPx4eJVLriDMBTNJsb2pEkzpL2KFnpXOKSZZ40YlBFM0bQojwTmJVEOGpBSL5m0PgjV3TdGMOIJphJgrvaNYxfYUYFTJPxA6i4ud/XCczySouNmeJIsRFouaCL4bwBsphMqvkuUNYQa8Apf8ub+fmC1fTBKudnkg+frvUNx+XWXoYfnt/eRLFt9NXlt2HhGb2wNP7/R4gVQUYzFbo23QjUn0U//9hysiOVEV4FRwezy1KjCTYs4xMdv6AzhZxFSRaYoiM7vQWqJlsUqYnbYvQKQiyydP5pd4aUUjIiFKrvQt9gFwZSG2OhbY4aIizC9YiGtkhVaGrI7MypUrGPWFRXIPVKGD6rcUI0W07D+SzZnGL9S/4OSH1Fczc/WqBm2mkJpnf/UOVz/sGtihA2wkGKPZWgs99CgoRjwyOGOieqKdodcz7Ry5Nh/r02AtuyWJsDv0CTIw6hqzKwczByTC8bUJOHoUMZRlNGriQpZU/a+vvTdDO7qvzbw1R/eKwcoO8k0IdmJUC0n9ImIuI7LddBSSM6K2aYfLTA354QbgC5kkDCn62HzdTWzYHT4LylWN+OH4zbDpc/wWpflB7XP1cLd9qbZ25Fg4S60VpDz64Tozdu1MScp1UA1RYoyG/8jSkuSaJmnDUU3dyZQUP8mNYEJqCRdc3zl5oIy1RIjRGTcKSEw00QJjhlRnM9d2IqEYm202GnFl5t6R7HjUCiBXrh0HG7QJnMqMC2WqcaKWCmQpIVGs5ddp2rnza0ddCHd0fr53MtjcNPEF7i8Yhzs7QDP4TCTV5zD6fGynCLp0djBoWcehrg7Crl2dH7xcQXaltSu6gnY2PD6QrvZC5Queiyw3i36OLBta/iCmgNcOO6V97MuVUwK5S52arfDC2QqOx5Zjoudn6wiFQt8Jc8zrUEfYXujsjtDN0D9xRrnpH0yJfKSmk9BxNugHT2C0LRsMTpUMFpvVULsnmQPUxRQxZSv6mS7YWYuY4gg1Aj6Ky8U/gH3D362GPvHXUUzTdcnxoBb6bJ07jgD0znG4bfHnYqObyeiTy1WthWaG9/W5qoZcj/bvom3tjuUhoy+RF7bLvJ0TpVZUgaPzRl6wX714flXoS5F5OMOjdk8AtNO0U3O8X5l5EanwwXQ5Tb8zN3DAflXmH0nW8WxrG+unJsv9gndxZIVhE+Og3dw+lCxnoVOT5RaZ/Qp1vf8meDRN6DjpKeqYSzZb91NEWb69/KtGcOYMBrqF9nexqSl0MbU2GDURHxa2Vi8a4Qb1OaDY1sPqn99yyqp/IYTvfgM= -------------------------------------------------------------------------------- /deepeye_app/txt2speech/txt-simulator.py: -------------------------------------------------------------------------------- 1 | import rospy 2 | from std_msgs.msg import String 3 | 4 | phrases_list = ["slow down a little", 5 | "take this right", 6 | "you missed your turn again!", 7 | "A car gonna take a left, slow down", 8 | "Red Signal on", 9 | "Green Signal on, You can walk now", 10 | "take this right", 11 | "you missed your turn again!", 12 | "A car gonna take a left, slow down", 13 | "Red Signal on", 14 | "Green Signal on, You can walk now", 15 | "take this right", 16 | "you missed your turn again!", 17 | "A car gonna take a left, slow down", 18 | "Red Signal on", 19 | "Green Signal on, You can walk now", 20 | "take this right", 21 | "you missed your turn again!", 22 | "A car gonna take a left, slow down", 23 | "Red Signal on", 24 | "Green Signal on, You can walk now", 25 | "take this right", 26 | "you missed your turn again!", 27 | "A car gonna take a left, slow down", 28 | "Red Signal on", 29 | "Green Signal on, You can walk now", 30 | "take this right", 31 | "you missed your turn again!", 32 | "A car gonna take a left, slow down", 33 | "Red Signal on", 34 | "Green Signal on, You can walk now", 35 | ] 36 | 37 | def txt_simulator(): 38 | rospy.init_node('publisher', anonymous=True) 39 | pub = rospy.Publisher('txt_simulator', String, queue_size=10) 40 | rate = rospy.Rate(10) # 10hz 41 | 42 | for i in range(len(phrases_list)): 43 | rospy.loginfo(phrases_list[i]) 44 | pub.publish(phrases_list[i]) 45 | rate.sleep() 46 | 47 | if __name__ == '__main__': 48 | try: 49 | txt_simulator() 50 | except rospy.ROSInterruptException: 51 | pass -------------------------------------------------------------------------------- /deepeye_app/txt2speech/txt2speech.py: -------------------------------------------------------------------------------- 1 | import socket 2 | import logging 3 | import sched 4 | import time 5 | from threading import Thread, Lock 6 | 7 | # ROS 8 | import rospy 9 | from std_msgs.msg import String 10 | 11 | # Text 2 speech 12 | from google_speech import Speech 13 | 14 | import subprocess 15 | 16 | lang = "en" 17 | 18 | # Init with False 19 | networkConnection = False 20 | 21 | # schedular instance is created 22 | s = sched.scheduler(time.time, time.sleep) 23 | 24 | # Mutex 25 | mutex = Lock() 26 | 27 | # BT Connected flag 28 | bt_flag = True 29 | 30 | def isInternetConnected(): 31 | global networkConnection 32 | 33 | # Mutex lock before CS 34 | mutex.acquire() 35 | 36 | try: 37 | # Try to ping google 38 | # To do: Is there way we can do local ping rather than google 39 | sock = socket.create_connection(("www.google.com", 80)) 40 | if sock is not None: 41 | sock.close 42 | # On Success update the status 43 | networkConnection = True 44 | print("Network ping ... success") 45 | except OSError: 46 | # On Failure update 47 | networkConnection = False 48 | 49 | # Mutex unlock after CS 50 | mutex.release() 51 | 52 | # Recursive call 53 | s.enter(1, 1, isInternetConnected, ()) 54 | 55 | def callback(data): 56 | global networkConnection 57 | print("Debug Data: ".format(data)) 58 | 59 | if bt_flag == False: 60 | if networkConnection == True: 61 | # Google speech block 62 | speech = Speech(data.data, lang) 63 | speech.play() 64 | time.sleep(0.5) 65 | else: 66 | # On device mycroft mimic v1 67 | p = subprocess.Popen(["/home/nullbyte/Desktop/myGit/mimic1/mimic", data.data], stdout = subprocess.PIPE) 68 | (output, err) = p.communicate() 69 | p_status = p.wait() 70 | else: 71 | cmd = "echo " + data.data + " > /dev/rfcomm0" 72 | print(cmd) 73 | p = subprocess.Popen(cmd, shell=True) 74 | (stderr,stdout) = p.communicate() 75 | print("stderr: ".format(stderr)) 76 | print("stdout: ".format(stdout)) 77 | 78 | def main(): 79 | format = "%(asctime)s: %(message)s" 80 | logging.basicConfig(format=format, level=logging.INFO, 81 | datefmt="%H:%M:%S") 82 | 83 | # ROS callback 84 | rospy.init_node('subscriper', anonymous=True) 85 | 86 | rospy.Subscriber("depthai", String, callback) 87 | 88 | s.enter(1, 1, isInternetConnected, ()) 89 | s.run() 90 | 91 | rospy.spin() 92 | 93 | if __name__ == "__main__": 94 | main() -------------------------------------------------------------------------------- /images/BW1098FFC_R0M0E0_diag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/BW1098FFC_R0M0E0_diag.png -------------------------------------------------------------------------------- /images/BW1098FFC_R0M0E0_dims.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/BW1098FFC_R0M0E0_dims.png -------------------------------------------------------------------------------- /images/BW1098FFC_R0M0E0_front.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/BW1098FFC_R0M0E0_front.png -------------------------------------------------------------------------------- /images/HLD.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/HLD.png -------------------------------------------------------------------------------- /images/HLD_1.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/HLD_1.JPG -------------------------------------------------------------------------------- /images/demo.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/demo.PNG -------------------------------------------------------------------------------- /images/hw.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/hw.jpeg -------------------------------------------------------------------------------- /images/jetson-nano-dev-kit-top-r6-HR-B01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/jetson-nano-dev-kit-top-r6-HR-B01.png -------------------------------------------------------------------------------- /openvino_analysis/README.md: -------------------------------------------------------------------------------- 1 | # Model Analysis 2 | 3 | > Hardware Configuration 4 | > CPU : i7-8665U 5 | > VPU : myriad x NCS2 6 | 7 | ## Intel Models 8 | |Model Name |Complexity - GFLOPs | Size - MP | CPU FPS - FP32 | Jetson + VPU FPS - FP16 | RPI + VPU FPS - FP16 9 | |--- |--- |--- |--- |--- |--- 10 | |pedestrian-detection-adas-0002 | 2.836 |1.165 |28 |6.86 |4.53 11 | |pedestrian-detection-adas-binary-0001 |0.945 |1.165|33 |Not Supported Layer Found |Not Supported Layer Found 12 | |pedestrian-and-vehicle-detector-adas-0001 | 3.974 |1.650 |24.09 |6.18 |4.37 13 | |vehicle-detection-adas-0002 |2.798 |1.079 |26 |7.51 |4.85 14 | |vehicle-detection-adas-binary-0001 |0.942 |1.079 |28 |Not Supported Layer Found |Not Supported Layer Found 15 | |road-segmentation-adas-0001 |4.770 |4.770 |8.48 |1.94 |1.02 16 | |semantic-segmentation-adas-0001 |58.572 |6.686 |2.06 |Memory Error |Memory Error 17 | 18 | ## Public Models 19 | |Model Name |Complexity - GFLOPs | Size - MP | CPU FPS - FP32 | Jetson + VPU FPS - FP16 | RPI + VPU FPS - FP16 20 | |--- |--- |--- |--- |--- |--- 21 | |YOLO v3 | 65.9843 |237 |1.86 |1.4 |Crash | 22 | |SSD with MobileNet |2.316~2.494 |65 |22 |10.4 |6.51 23 | 24 | ## Depth AI HW Model analysis 25 | | Model | SAHVES 4 - FPS | SAHVES 7 - FPS | GFlops | mParams | | 26 | |---|---|---|---|---|---| 27 | | ssd_mobilenet_v1_coco | 28-30 | | 2.316~2.494 | 5.783~6.807 | | 28 | | ssd_mobilenet_v2_coco | 12-13 | 14-15 | 3.775 | 16.818 | | 29 | | | | | | | | 30 | | | | | | | | -------------------------------------------------------------------------------- /openvino_analysis/intel/object-detection/pedestrian-and-vehicle-detector-adas-0001/__pycache__/inference.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/object-detection/pedestrian-and-vehicle-detector-adas-0001/__pycache__/inference.cpython-36.pyc -------------------------------------------------------------------------------- /openvino_analysis/intel/object-detection/pedestrian-and-vehicle-detector-adas-0001/inference.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import logging as log 4 | from openvino.inference_engine import IENetwork, IECore 5 | 6 | 7 | class Network: 8 | """ 9 | Load and configure inference plugins for the specified target devices 10 | and performs synchronous and asynchronous modes for the specified infer requests. 11 | """ 12 | 13 | def __init__(self): 14 | self.ie = None 15 | self.network = None 16 | self.input_blob = None 17 | self.output_blob = None 18 | self.exec_network = None 19 | self.infer_request = None 20 | 21 | 22 | def load_model(self, model, args, device="CPU", num_requests=0): 23 | ''' 24 | Load the model given IR files. 25 | Defaults to CPU as device for use in the workspace. 26 | Synchronous requests made within. 27 | ''' 28 | model_xml = model 29 | model_bin = os.path.splitext(model_xml)[0] + ".bin" 30 | 31 | 32 | # Initialize the plugin 33 | self.ie = IECore() 34 | 35 | # Read the IR as a IENetwork 36 | try: 37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin) 38 | except Exception as e: 39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?") 40 | 41 | if args.cpu_extension and "CPU" in args.device: 42 | ie.add_extension(args.cpu_extension, "CPU") 43 | log.info("CPU extension loaded: {}".format(args.cpu_extension)) 44 | 45 | if "CPU" in args.device: 46 | supported_layers = self.ie.query_network(self.network, "CPU") 47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers] 48 | if len(not_supported_layers) != 0: 49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}". 50 | format(args.device, ', '.join(not_supported_layers))) 51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l " 52 | "or --cpu_extension command line argument") 53 | sys.exit(1) 54 | 55 | # Load the IENetwork into the plugin 56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests) 57 | 58 | def get_input_shape(self): 59 | ''' 60 | Gets the input shape of the network 61 | ''' 62 | log.info("Preparing input blobs") 63 | input_shape = [] 64 | print("inputs number: " + str(len(self.network.input_info.keys()))) 65 | 66 | for input_key in self.network.input_info: 67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape)) 68 | print("input key: " + input_key) 69 | self.input_blob = input_key 70 | if len(self.network.input_info[input_key].input_data.layout) == 4: 71 | input_shape = self.network.input_info[input_key].input_data.shape 72 | 73 | return input_shape 74 | 75 | def get_output_name(self): 76 | ''' 77 | Gets the input shape of the network 78 | ''' 79 | log.info('Preparing output blobs') 80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))] 81 | for output_key in self.network.outputs: 82 | if self.network.layers[output_key].type == "DetectionOutput": 83 | self.output_blob, _ = output_key, self.network.outputs[output_key] 84 | 85 | if self.output_blob == "": 86 | log.error("Can't find a DetectionOutput layer in the topology") 87 | exit(-1) 88 | return self.output_blob 89 | 90 | def exec_net(self, image, request_id): 91 | ''' 92 | Makes an asynchronous inference request, given an input image. 93 | ''' 94 | self.exec_network.start_async(request_id=request_id, 95 | inputs={self.input_blob: image}) 96 | return 97 | 98 | 99 | def wait(self, request_id): 100 | ''' 101 | Checks the status of the inference request. 102 | ''' 103 | status = self.exec_network.requests[request_id].wait(-1) 104 | return status 105 | 106 | 107 | def get_output(self, request_id): 108 | ''' 109 | Returns a list of the results for the output layer of the network. 110 | ''' 111 | return self.exec_network.requests[request_id].outputs[self.output_blob] -------------------------------------------------------------------------------- /openvino_analysis/intel/object-detection/pedestrian-and-vehicle-detector-adas-0001/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import socket 5 | import json 6 | import cv2 7 | import numpy as np 8 | 9 | import logging as log 10 | 11 | from argparse import ArgumentParser 12 | from inference import Network 13 | 14 | from imutils.video import FPS 15 | 16 | def build_argparser(): 17 | """ 18 | Parse command line arguments. 19 | :return: command line arguments 20 | """ 21 | parser = ArgumentParser() 22 | parser.add_argument("-m", "--model", required=True, type=str, 23 | help="Path to an xml file with a trained model.") 24 | parser.add_argument("-i", "--input", required=True, type=str, 25 | help="Path to image or video file") 26 | parser.add_argument("-d", "--device", type=str, default="CPU", 27 | help="Specify the target device to infer on: " 28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample " 29 | "will look for a suitable plugin for device " 30 | "specified (CPU by default)") 31 | parser.add_argument("-l", "--cpu_extension", 32 | help="Optional. Required for CPU custom layers. " 33 | "Absolute path to a shared library with the kernels implementations.", 34 | type=str, default=None) 35 | return parser 36 | 37 | 38 | def preprocessing(frame, in_w, in_h): 39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA) 40 | image = np.moveaxis(image_resize, -1, 0) 41 | 42 | return image 43 | def infer_on_stream(args): 44 | """ 45 | Initialize the inference network, stream video to network, 46 | and output stats and video. 47 | :param args: Command line arguments parsed by `build_argparser()` 48 | :return: None 49 | """ 50 | # Initialize the Inference Engine 51 | infer_network = Network() 52 | 53 | # Load the model through `infer_network` 54 | infer_network.load_model(args.model, args, args.device, num_requests=0) 55 | 56 | # Get a Input blob shape 57 | _, _, in_h, in_w = infer_network.get_input_shape() 58 | 59 | 60 | # Get a output blob name 61 | output_name = infer_network.get_output_name() 62 | 63 | # Handle the input stream 64 | try: 65 | cap = cv2.VideoCapture(args.input) 66 | except FileNotFoundError: 67 | print("Cannot locate video file: "+ args.input) 68 | except Exception as e: 69 | print("Something else went wrong with the video file: ", e) 70 | 71 | cap.open(args.input) 72 | _, frame = cap.read() 73 | 74 | fh = frame.shape[0] 75 | fw = frame.shape[1] 76 | 77 | fps = FPS().start() 78 | 79 | # Process frames until the video ends, or process is exited 80 | while cap.isOpened(): 81 | # Read the next frame 82 | flag, frame = cap.read() 83 | if not flag: 84 | break 85 | 86 | key_pressed = cv2.waitKey(1) 87 | 88 | image = preprocessing(frame, in_w, in_h) 89 | 90 | # Perform inference on the frame 91 | infer_network.exec_net(image, request_id=0) 92 | 93 | # Get the output of inference 94 | if infer_network.wait(request_id=0) == 0: 95 | result = infer_network.get_output(request_id=0) 96 | for box in result[0][0]: # Output shape is 1x1x100x7 97 | conf = box[2] 98 | if conf >= 0.5: 99 | xmin = int(box[3] * fw) 100 | ymin = int(box[4] * fh) 101 | xmax = int(box[5] * fw) 102 | ymax = int(box[6] * fh) 103 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 0, 255), 3) 104 | 105 | cv2.imshow('frame', frame) 106 | if cv2.waitKey(1) & 0xFF == ord('q'): 107 | break 108 | 109 | #Break if escape key pressed 110 | if key_pressed == 27: 111 | break 112 | 113 | fps.update() 114 | 115 | # Release the out writer, capture, and destroy any OpenCV windows 116 | cap.release() 117 | 118 | cv2.destroyAllWindows() 119 | 120 | fps.stop() 121 | 122 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps())) 123 | 124 | def main(): 125 | """ 126 | Load the network and parse the output. 127 | :return: None 128 | """ 129 | # Set log to INFO 130 | log.basicConfig(level=log.CRITICAL) 131 | 132 | # Grab command line args 133 | args = build_argparser().parse_args() 134 | 135 | # Perform inference on the input stream 136 | infer_on_stream(args) 137 | 138 | if __name__ == '__main__': 139 | main() -------------------------------------------------------------------------------- /openvino_analysis/intel/object-detection/pedestrian-detection-adas-0002/__pycache__/inference.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/object-detection/pedestrian-detection-adas-0002/__pycache__/inference.cpython-36.pyc -------------------------------------------------------------------------------- /openvino_analysis/intel/object-detection/pedestrian-detection-adas-0002/inference.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import logging as log 4 | from openvino.inference_engine import IENetwork, IECore 5 | 6 | 7 | class Network: 8 | """ 9 | Load and configure inference plugins for the specified target devices 10 | and performs synchronous and asynchronous modes for the specified infer requests. 11 | """ 12 | 13 | def __init__(self): 14 | self.ie = None 15 | self.network = None 16 | self.input_blob = None 17 | self.output_blob = None 18 | self.exec_network = None 19 | self.infer_request = None 20 | 21 | 22 | def load_model(self, model, args, device="CPU", num_requests=0): 23 | ''' 24 | Load the model given IR files. 25 | Defaults to CPU as device for use in the workspace. 26 | Synchronous requests made within. 27 | ''' 28 | model_xml = model 29 | model_bin = os.path.splitext(model_xml)[0] + ".bin" 30 | 31 | 32 | # Initialize the plugin 33 | self.ie = IECore() 34 | 35 | # Read the IR as a IENetwork 36 | try: 37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin) 38 | except Exception as e: 39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?") 40 | 41 | if args.cpu_extension and "CPU" in args.device: 42 | ie.add_extension(args.cpu_extension, "CPU") 43 | log.info("CPU extension loaded: {}".format(args.cpu_extension)) 44 | 45 | if "CPU" in args.device: 46 | supported_layers = self.ie.query_network(self.network, "CPU") 47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers] 48 | if len(not_supported_layers) != 0: 49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}". 50 | format(args.device, ', '.join(not_supported_layers))) 51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l " 52 | "or --cpu_extension command line argument") 53 | sys.exit(1) 54 | 55 | # Load the IENetwork into the plugin 56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests) 57 | 58 | def get_input_shape(self): 59 | ''' 60 | Gets the input shape of the network 61 | ''' 62 | log.info("Preparing input blobs") 63 | input_shape = [] 64 | print("inputs number: " + str(len(self.network.input_info.keys()))) 65 | 66 | for input_key in self.network.input_info: 67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape)) 68 | print("input key: " + input_key) 69 | self.input_blob = input_key 70 | if len(self.network.input_info[input_key].input_data.layout) == 4: 71 | input_shape = self.network.input_info[input_key].input_data.shape 72 | 73 | return input_shape 74 | 75 | def get_output_name(self): 76 | ''' 77 | Gets the input shape of the network 78 | ''' 79 | log.info('Preparing output blobs') 80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))] 81 | for output_key in self.network.outputs: 82 | if self.network.layers[output_key].type == "DetectionOutput": 83 | self.output_blob, _ = output_key, self.network.outputs[output_key] 84 | 85 | if self.output_blob == "": 86 | log.error("Can't find a DetectionOutput layer in the topology") 87 | exit(-1) 88 | return self.output_blob 89 | 90 | def exec_net(self, image, request_id): 91 | ''' 92 | Makes an asynchronous inference request, given an input image. 93 | ''' 94 | self.exec_network.start_async(request_id=request_id, 95 | inputs={self.input_blob: image}) 96 | return 97 | 98 | 99 | def wait(self, request_id): 100 | ''' 101 | Checks the status of the inference request. 102 | ''' 103 | status = self.exec_network.requests[request_id].wait(-1) 104 | return status 105 | 106 | 107 | def get_output(self, request_id): 108 | ''' 109 | Returns a list of the results for the output layer of the network. 110 | ''' 111 | return self.exec_network.requests[request_id].outputs[self.output_blob] -------------------------------------------------------------------------------- /openvino_analysis/intel/object-detection/pedestrian-detection-adas-0002/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import socket 5 | import json 6 | import cv2 7 | import numpy as np 8 | 9 | import logging as log 10 | 11 | from argparse import ArgumentParser 12 | from inference import Network 13 | 14 | from imutils.video import FPS 15 | 16 | def build_argparser(): 17 | """ 18 | Parse command line arguments. 19 | :return: command line arguments 20 | """ 21 | parser = ArgumentParser() 22 | parser.add_argument("-m", "--model", required=True, type=str, 23 | help="Path to an xml file with a trained model.") 24 | parser.add_argument("-i", "--input", required=True, type=str, 25 | help="Path to image or video file") 26 | parser.add_argument("-d", "--device", type=str, default="CPU", 27 | help="Specify the target device to infer on: " 28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample " 29 | "will look for a suitable plugin for device " 30 | "specified (CPU by default)") 31 | parser.add_argument("-l", "--cpu_extension", 32 | help="Optional. Required for CPU custom layers. " 33 | "Absolute path to a shared library with the kernels implementations.", 34 | type=str, default=None) 35 | return parser 36 | 37 | 38 | def preprocessing(frame, in_w, in_h): 39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA) 40 | image = np.moveaxis(image_resize, -1, 0) 41 | 42 | return image 43 | def infer_on_stream(args): 44 | """ 45 | Initialize the inference network, stream video to network, 46 | and output stats and video. 47 | :param args: Command line arguments parsed by `build_argparser()` 48 | :return: None 49 | """ 50 | # Initialize the Inference Engine 51 | infer_network = Network() 52 | 53 | # Load the model through `infer_network` 54 | infer_network.load_model(args.model, args, args.device, num_requests=0) 55 | 56 | # Get a Input blob shape 57 | _, _, in_h, in_w = infer_network.get_input_shape() 58 | 59 | 60 | # Get a output blob name 61 | output_name = infer_network.get_output_name() 62 | 63 | # Handle the input stream 64 | try: 65 | cap = cv2.VideoCapture(args.input) 66 | except FileNotFoundError: 67 | print("Cannot locate video file: "+ args.input) 68 | except Exception as e: 69 | print("Something else went wrong with the video file: ", e) 70 | 71 | cap.open(args.input) 72 | _, frame = cap.read() 73 | 74 | fh = frame.shape[0] 75 | fw = frame.shape[1] 76 | 77 | fps = FPS().start() 78 | 79 | # Process frames until the video ends, or process is exited 80 | while cap.isOpened(): 81 | # Read the next frame 82 | flag, frame = cap.read() 83 | if not flag: 84 | break 85 | 86 | key_pressed = cv2.waitKey(1) 87 | 88 | image = preprocessing(frame, in_w, in_h) 89 | 90 | # Perform inference on the frame 91 | infer_network.exec_net(image, request_id=0) 92 | 93 | # Get the output of inference 94 | if infer_network.wait(request_id=0) == 0: 95 | result = infer_network.get_output(request_id=0) 96 | for box in result[0][0]: # Output shape is 1x1x100x7 97 | conf = box[2] 98 | if conf >= 0.5: 99 | xmin = int(box[3] * fw) 100 | ymin = int(box[4] * fh) 101 | xmax = int(box[5] * fw) 102 | ymax = int(box[6] * fh) 103 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 0, 255), 3) 104 | 105 | cv2.imshow('frame', frame) 106 | if cv2.waitKey(1) & 0xFF == ord('q'): 107 | break 108 | 109 | #Break if escape key pressed 110 | if key_pressed == 27: 111 | break 112 | 113 | fps.update() 114 | 115 | # Release the out writer, capture, and destroy any OpenCV windows 116 | cap.release() 117 | 118 | cv2.destroyAllWindows() 119 | 120 | fps.stop() 121 | 122 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps())) 123 | 124 | def main(): 125 | """ 126 | Load the network and parse the output. 127 | :return: None 128 | """ 129 | # Set log to INFO 130 | log.basicConfig(level=log.CRITICAL) 131 | 132 | # Grab command line args 133 | args = build_argparser().parse_args() 134 | 135 | # Perform inference on the input stream 136 | infer_on_stream(args) 137 | 138 | if __name__ == '__main__': 139 | main() -------------------------------------------------------------------------------- /openvino_analysis/intel/object-detection/pedestrian-detection-adas-binary-0001/inference.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import logging as log 4 | from openvino.inference_engine import IENetwork, IECore 5 | 6 | 7 | class Network: 8 | """ 9 | Load and configure inference plugins for the specified target devices 10 | and performs synchronous and asynchronous modes for the specified infer requests. 11 | """ 12 | 13 | def __init__(self): 14 | self.ie = None 15 | self.network = None 16 | self.input_blob = None 17 | self.output_blob = None 18 | self.exec_network = None 19 | self.infer_request = None 20 | 21 | 22 | def load_model(self, model, args, device="CPU", num_requests=0): 23 | ''' 24 | Load the model given IR files. 25 | Defaults to CPU as device for use in the workspace. 26 | Synchronous requests made within. 27 | ''' 28 | model_xml = model 29 | model_bin = os.path.splitext(model_xml)[0] + ".bin" 30 | 31 | 32 | # Initialize the plugin 33 | self.ie = IECore() 34 | 35 | # Read the IR as a IENetwork 36 | try: 37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin) 38 | except Exception as e: 39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?") 40 | 41 | if args.cpu_extension and "CPU" in args.device: 42 | ie.add_extension(args.cpu_extension, "CPU") 43 | log.info("CPU extension loaded: {}".format(args.cpu_extension)) 44 | 45 | if "CPU" in args.device: 46 | supported_layers = self.ie.query_network(self.network, "CPU") 47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers] 48 | if len(not_supported_layers) != 0: 49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}". 50 | format(args.device, ', '.join(not_supported_layers))) 51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l " 52 | "or --cpu_extension command line argument") 53 | sys.exit(1) 54 | 55 | # Load the IENetwork into the plugin 56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests) 57 | 58 | def get_input_shape(self): 59 | ''' 60 | Gets the input shape of the network 61 | ''' 62 | log.info("Preparing input blobs") 63 | input_shape = [] 64 | print("inputs number: " + str(len(self.network.input_info.keys()))) 65 | 66 | for input_key in self.network.input_info: 67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape)) 68 | print("input key: " + input_key) 69 | self.input_blob = input_key 70 | if len(self.network.input_info[input_key].input_data.layout) == 4: 71 | input_shape = self.network.input_info[input_key].input_data.shape 72 | 73 | return input_shape 74 | 75 | def get_output_name(self): 76 | ''' 77 | Gets the input shape of the network 78 | ''' 79 | log.info('Preparing output blobs') 80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))] 81 | for output_key in self.network.outputs: 82 | if self.network.layers[output_key].type == "DetectionOutput": 83 | self.output_blob, _ = output_key, self.network.outputs[output_key] 84 | 85 | if self.output_blob == "": 86 | log.error("Can't find a DetectionOutput layer in the topology") 87 | exit(-1) 88 | return self.output_blob 89 | 90 | def exec_net(self, image, request_id): 91 | ''' 92 | Makes an asynchronous inference request, given an input image. 93 | ''' 94 | self.exec_network.start_async(request_id=request_id, 95 | inputs={self.input_blob: image}) 96 | return 97 | 98 | 99 | def wait(self, request_id): 100 | ''' 101 | Checks the status of the inference request. 102 | ''' 103 | status = self.exec_network.requests[request_id].wait(-1) 104 | return status 105 | 106 | 107 | def get_output(self, request_id): 108 | ''' 109 | Returns a list of the results for the output layer of the network. 110 | ''' 111 | return self.exec_network.requests[request_id].outputs[self.output_blob] -------------------------------------------------------------------------------- /openvino_analysis/intel/object-detection/pedestrian-detection-adas-binary-0001/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import socket 5 | import json 6 | import cv2 7 | import numpy as np 8 | 9 | import logging as log 10 | 11 | from argparse import ArgumentParser 12 | from inference import Network 13 | 14 | from imutils.video import FPS 15 | 16 | def build_argparser(): 17 | """ 18 | Parse command line arguments. 19 | :return: command line arguments 20 | """ 21 | parser = ArgumentParser() 22 | parser.add_argument("-m", "--model", required=True, type=str, 23 | help="Path to an xml file with a trained model.") 24 | parser.add_argument("-i", "--input", required=True, type=str, 25 | help="Path to image or video file") 26 | parser.add_argument("-d", "--device", type=str, default="CPU", 27 | help="Specify the target device to infer on: " 28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample " 29 | "will look for a suitable plugin for device " 30 | "specified (CPU by default)") 31 | parser.add_argument("-l", "--cpu_extension", 32 | help="Optional. Required for CPU custom layers. " 33 | "Absolute path to a shared library with the kernels implementations.", 34 | type=str, default=None) 35 | return parser 36 | 37 | 38 | def preprocessing(frame, in_w, in_h): 39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA) 40 | image = np.moveaxis(image_resize, -1, 0) 41 | 42 | return image 43 | def infer_on_stream(args): 44 | """ 45 | Initialize the inference network, stream video to network, 46 | and output stats and video. 47 | :param args: Command line arguments parsed by `build_argparser()` 48 | :return: None 49 | """ 50 | # Initialize the Inference Engine 51 | infer_network = Network() 52 | 53 | # Load the model through `infer_network` 54 | infer_network.load_model(args.model, args, args.device, num_requests=0) 55 | 56 | # Get a Input blob shape 57 | _, _, in_h, in_w = infer_network.get_input_shape() 58 | 59 | 60 | # Get a output blob name 61 | output_name = infer_network.get_output_name() 62 | 63 | # Handle the input stream 64 | try: 65 | cap = cv2.VideoCapture(args.input) 66 | except FileNotFoundError: 67 | print("Cannot locate video file: "+ args.input) 68 | except Exception as e: 69 | print("Something else went wrong with the video file: ", e) 70 | 71 | cap.open(args.input) 72 | _, frame = cap.read() 73 | 74 | fh = frame.shape[0] 75 | fw = frame.shape[1] 76 | 77 | fps = FPS().start() 78 | 79 | # Process frames until the video ends, or process is exited 80 | while cap.isOpened(): 81 | # Read the next frame 82 | flag, frame = cap.read() 83 | if not flag: 84 | break 85 | 86 | key_pressed = cv2.waitKey(1) 87 | 88 | image = preprocessing(frame, in_w, in_h) 89 | 90 | # Perform inference on the frame 91 | infer_network.exec_net(image, request_id=0) 92 | 93 | # Get the output of inference 94 | if infer_network.wait(request_id=0) == 0: 95 | result = infer_network.get_output(request_id=0) 96 | for box in result[0][0]: # Output shape is 1x1x100x7 97 | conf = box[2] 98 | if conf >= 0.5: 99 | xmin = int(box[3] * fw) 100 | ymin = int(box[4] * fh) 101 | xmax = int(box[5] * fw) 102 | ymax = int(box[6] * fh) 103 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 0, 255), 3) 104 | 105 | cv2.imshow('frame', frame) 106 | if cv2.waitKey(1) & 0xFF == ord('q'): 107 | break 108 | 109 | #Break if escape key pressed 110 | if key_pressed == 27: 111 | break 112 | 113 | fps.update() 114 | 115 | # Release the out writer, capture, and destroy any OpenCV windows 116 | cap.release() 117 | 118 | cv2.destroyAllWindows() 119 | 120 | fps.stop() 121 | 122 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps())) 123 | 124 | def main(): 125 | """ 126 | Load the network and parse the output. 127 | :return: None 128 | """ 129 | # Set log to INFO 130 | log.basicConfig(level=log.CRITICAL) 131 | 132 | # Grab command line args 133 | args = build_argparser().parse_args() 134 | 135 | # Perform inference on the input stream 136 | infer_on_stream(args) 137 | 138 | if __name__ == '__main__': 139 | main() -------------------------------------------------------------------------------- /openvino_analysis/intel/object-detection/vehicle-detection-adas/inference.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import logging as log 4 | from openvino.inference_engine import IENetwork, IECore 5 | 6 | 7 | class Network: 8 | """ 9 | Load and configure inference plugins for the specified target devices 10 | and performs synchronous and asynchronous modes for the specified infer requests. 11 | """ 12 | 13 | def __init__(self): 14 | self.ie = None 15 | self.network = None 16 | self.input_blob = None 17 | self.output_blob = None 18 | self.exec_network = None 19 | self.infer_request = None 20 | 21 | 22 | def load_model(self, model, args, device="CPU", num_requests=0): 23 | ''' 24 | Load the model given IR files. 25 | Defaults to CPU as device for use in the workspace. 26 | Synchronous requests made within. 27 | ''' 28 | model_xml = model 29 | model_bin = os.path.splitext(model_xml)[0] + ".bin" 30 | 31 | 32 | # Initialize the plugin 33 | self.ie = IECore() 34 | 35 | # Read the IR as a IENetwork 36 | try: 37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin) 38 | except Exception as e: 39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?") 40 | 41 | if args.cpu_extension and "CPU" in args.device: 42 | ie.add_extension(args.cpu_extension, "CPU") 43 | log.info("CPU extension loaded: {}".format(args.cpu_extension)) 44 | 45 | if "CPU" in args.device: 46 | supported_layers = self.ie.query_network(self.network, "CPU") 47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers] 48 | if len(not_supported_layers) != 0: 49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}". 50 | format(args.device, ', '.join(not_supported_layers))) 51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l " 52 | "or --cpu_extension command line argument") 53 | sys.exit(1) 54 | 55 | # Load the IENetwork into the plugin 56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests) 57 | 58 | def get_input_shape(self): 59 | ''' 60 | Gets the input shape of the network 61 | ''' 62 | log.info("Preparing input blobs") 63 | input_shape = [] 64 | print("inputs number: " + str(len(self.network.input_info.keys()))) 65 | 66 | for input_key in self.network.input_info: 67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape)) 68 | print("input key: " + input_key) 69 | self.input_blob = input_key 70 | if len(self.network.input_info[input_key].input_data.layout) == 4: 71 | input_shape = self.network.input_info[input_key].input_data.shape 72 | 73 | return input_shape 74 | 75 | def get_output_name(self): 76 | ''' 77 | Gets the input shape of the network 78 | ''' 79 | log.info('Preparing output blobs') 80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))] 81 | for output_key in self.network.outputs: 82 | if self.network.layers[output_key].type == "DetectionOutput": 83 | self.output_blob, _ = output_key, self.network.outputs[output_key] 84 | 85 | if self.output_blob == "": 86 | log.error("Can't find a DetectionOutput layer in the topology") 87 | exit(-1) 88 | return self.output_blob 89 | 90 | def exec_net(self, image, request_id): 91 | ''' 92 | Makes an asynchronous inference request, given an input image. 93 | ''' 94 | self.exec_network.start_async(request_id=request_id, 95 | inputs={self.input_blob: image}) 96 | return 97 | 98 | 99 | def wait(self, request_id): 100 | ''' 101 | Checks the status of the inference request. 102 | ''' 103 | status = self.exec_network.requests[request_id].wait(-1) 104 | return status 105 | 106 | 107 | def get_output(self, request_id): 108 | ''' 109 | Returns a list of the results for the output layer of the network. 110 | ''' 111 | return self.exec_network.requests[request_id].outputs[self.output_blob] -------------------------------------------------------------------------------- /openvino_analysis/intel/object-detection/vehicle-detection-adas/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import socket 5 | import json 6 | import cv2 7 | import numpy as np 8 | 9 | import logging as log 10 | 11 | from argparse import ArgumentParser 12 | from inference import Network 13 | 14 | from imutils.video import FPS 15 | 16 | def build_argparser(): 17 | """ 18 | Parse command line arguments. 19 | :return: command line arguments 20 | """ 21 | parser = ArgumentParser() 22 | parser.add_argument("-m", "--model", required=True, type=str, 23 | help="Path to an xml file with a trained model.") 24 | parser.add_argument("-i", "--input", required=True, type=str, 25 | help="Path to image or video file") 26 | parser.add_argument("-d", "--device", type=str, default="CPU", 27 | help="Specify the target device to infer on: " 28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample " 29 | "will look for a suitable plugin for device " 30 | "specified (CPU by default)") 31 | parser.add_argument("-l", "--cpu_extension", 32 | help="Optional. Required for CPU custom layers. " 33 | "Absolute path to a shared library with the kernels implementations.", 34 | type=str, default=None) 35 | return parser 36 | 37 | 38 | def preprocessing(frame, in_w, in_h): 39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA) 40 | image = np.moveaxis(image_resize, -1, 0) 41 | 42 | return image 43 | def infer_on_stream(args): 44 | """ 45 | Initialize the inference network, stream video to network, 46 | and output stats and video. 47 | :param args: Command line arguments parsed by `build_argparser()` 48 | :return: None 49 | """ 50 | # Initialize the Inference Engine 51 | infer_network = Network() 52 | 53 | # Load the model through `infer_network` 54 | infer_network.load_model(args.model, args, args.device, num_requests=0) 55 | 56 | # Get a Input blob shape 57 | _, _, in_h, in_w = infer_network.get_input_shape() 58 | 59 | 60 | # Get a output blob name 61 | output_name = infer_network.get_output_name() 62 | 63 | # Handle the input stream 64 | try: 65 | cap = cv2.VideoCapture(args.input) 66 | except FileNotFoundError: 67 | print("Cannot locate video file: "+ args.input) 68 | except Exception as e: 69 | print("Something else went wrong with the video file: ", e) 70 | 71 | cap.open(args.input) 72 | _, frame = cap.read() 73 | 74 | fh = frame.shape[0] 75 | fw = frame.shape[1] 76 | 77 | fps = FPS().start() 78 | 79 | # Process frames until the video ends, or process is exited 80 | while cap.isOpened(): 81 | # Read the next frame 82 | flag, frame = cap.read() 83 | if not flag: 84 | break 85 | 86 | key_pressed = cv2.waitKey(1) 87 | 88 | image = preprocessing(frame, in_w, in_h) 89 | 90 | # Perform inference on the frame 91 | infer_network.exec_net(image, request_id=0) 92 | 93 | # Get the output of inference 94 | if infer_network.wait(request_id=0) == 0: 95 | result = infer_network.get_output(request_id=0) 96 | for box in result[0][0]: # Output shape is 1x1x100x7 97 | conf = box[2] 98 | if conf >= 0.5: 99 | xmin = int(box[3] * fw) 100 | ymin = int(box[4] * fh) 101 | xmax = int(box[5] * fw) 102 | ymax = int(box[6] * fh) 103 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 0, 255), 3) 104 | 105 | cv2.imshow('frame', frame) 106 | if cv2.waitKey(1) & 0xFF == ord('q'): 107 | break 108 | 109 | #Break if escape key pressed 110 | if key_pressed == 27: 111 | break 112 | 113 | fps.update() 114 | 115 | # Release the out writer, capture, and destroy any OpenCV windows 116 | cap.release() 117 | 118 | cv2.destroyAllWindows() 119 | 120 | fps.stop() 121 | 122 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps())) 123 | 124 | def main(): 125 | """ 126 | Load the network and parse the output. 127 | :return: None 128 | """ 129 | # Set log to INFO 130 | log.basicConfig(level=log.CRITICAL) 131 | 132 | # Grab command line args 133 | args = build_argparser().parse_args() 134 | 135 | # Perform inference on the input stream 136 | infer_on_stream(args) 137 | 138 | if __name__ == '__main__': 139 | main() -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/road-segmentation-adas-0001/__pycache__/inference.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/semantic-segmentation/road-segmentation-adas-0001/__pycache__/inference.cpython-36.pyc -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/road-segmentation-adas-0001/inference.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import logging as log 4 | from openvino.inference_engine import IENetwork, IECore 5 | 6 | 7 | class Network: 8 | """ 9 | Load and configure inference plugins for the specified target devices 10 | and performs synchronous and asynchronous modes for the specified infer requests. 11 | """ 12 | 13 | def __init__(self): 14 | self.ie = None 15 | self.network = None 16 | self.input_blob = None 17 | self.output_blob = None 18 | self.exec_network = None 19 | self.infer_request = None 20 | 21 | 22 | def load_model(self, model, args, device="CPU", num_requests=0): 23 | ''' 24 | Load the model given IR files. 25 | Defaults to CPU as device for use in the workspace. 26 | Synchronous requests made within. 27 | ''' 28 | model_xml = model 29 | model_bin = os.path.splitext(model_xml)[0] + ".bin" 30 | 31 | 32 | # Initialize the plugin 33 | self.ie = IECore() 34 | 35 | # Read the IR as a IENetwork 36 | try: 37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin) 38 | except Exception as e: 39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?") 40 | 41 | if args.cpu_extension and "CPU" in args.device: 42 | ie.add_extension(args.cpu_extension, "CPU") 43 | log.info("CPU extension loaded: {}".format(args.cpu_extension)) 44 | 45 | if "CPU" in args.device: 46 | supported_layers = self.ie.query_network(self.network, "CPU") 47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers] 48 | if len(not_supported_layers) != 0: 49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}". 50 | format(args.device, ', '.join(not_supported_layers))) 51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l " 52 | "or --cpu_extension command line argument") 53 | sys.exit(1) 54 | 55 | # Load the IENetwork into the plugin 56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests) 57 | 58 | def get_input_shape(self): 59 | ''' 60 | Gets the input shape of the network 61 | ''' 62 | log.info("Preparing input blobs") 63 | input_shape = [] 64 | print("inputs number: " + str(len(self.network.input_info.keys()))) 65 | 66 | for input_key in self.network.input_info: 67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape)) 68 | print("input key: " + input_key) 69 | self.input_blob = input_key 70 | if len(self.network.input_info[input_key].input_data.layout) == 4: 71 | input_shape = self.network.input_info[input_key].input_data.shape 72 | 73 | return input_shape 74 | 75 | def get_output_name(self): 76 | ''' 77 | Gets the input shape of the network 78 | ''' 79 | log.info('Preparing output blobs') 80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))] 81 | for output_key in self.network.outputs: 82 | if self.network.layers[output_key].type == "SoftMax": 83 | self.output_blob, _ = output_key, self.network.outputs[output_key] 84 | 85 | if self.output_blob == "": 86 | log.error("Can't find a L0317_ReWeight_SoftMax layer in the topology") 87 | exit(-1) 88 | return self.output_blob 89 | 90 | def exec_net(self, image, request_id): 91 | ''' 92 | Makes an asynchronous inference request, given an input image. 93 | ''' 94 | self.exec_network.start_async(request_id=request_id, 95 | inputs={self.input_blob: image}) 96 | return 97 | 98 | 99 | def wait(self, request_id): 100 | ''' 101 | Checks the status of the inference request. 102 | ''' 103 | status = self.exec_network.requests[request_id].wait(-1) 104 | return status 105 | 106 | 107 | def get_output(self, request_id): 108 | ''' 109 | Returns a list of the results for the output layer of the network. 110 | ''' 111 | return self.exec_network.requests[request_id].outputs[self.output_blob] -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/road-segmentation-adas-0001/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import socket 5 | import json 6 | import cv2 7 | import numpy as np 8 | 9 | import logging as log 10 | 11 | from argparse import ArgumentParser 12 | from inference import Network 13 | 14 | from imutils.video import FPS 15 | 16 | def build_argparser(): 17 | """ 18 | Parse command line arguments. 19 | :return: command line arguments 20 | """ 21 | parser = ArgumentParser() 22 | parser.add_argument("-m", "--model", required=True, type=str, 23 | help="Path to an xml file with a trained model.") 24 | parser.add_argument("-i", "--input", required=True, type=str, 25 | help="Path to image or video file") 26 | parser.add_argument("-d", "--device", type=str, default="CPU", 27 | help="Specify the target device to infer on: " 28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample " 29 | "will look for a suitable plugin for device " 30 | "specified (CPU by default)") 31 | parser.add_argument("-l", "--cpu_extension", 32 | help="Optional. Required for CPU custom layers. " 33 | "Absolute path to a shared library with the kernels implementations.", 34 | type=str, default=None) 35 | return parser 36 | 37 | 38 | def preprocessing(frame, in_w, in_h): 39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA) 40 | image = np.moveaxis(image_resize, -1, 0) 41 | 42 | return image 43 | def infer_on_stream(args): 44 | """ 45 | Initialize the inference network, stream video to network, 46 | and output stats and video. 47 | :param args: Command line arguments parsed by `build_argparser()` 48 | :return: None 49 | """ 50 | # Initialize the Inference Engine 51 | infer_network = Network() 52 | 53 | # Load the model through `infer_network` 54 | infer_network.load_model(args.model, args, args.device, num_requests=0) 55 | 56 | # Get a Input blob shape 57 | _, _, in_h, in_w = infer_network.get_input_shape() 58 | 59 | 60 | # Get a output blob name 61 | output_name = infer_network.get_output_name() 62 | 63 | # Handle the input stream 64 | try: 65 | cap = cv2.VideoCapture(args.input) 66 | except FileNotFoundError: 67 | print("Cannot locate video file: "+ args.input) 68 | except Exception as e: 69 | print("Something else went wrong with the video file: ", e) 70 | 71 | cap.open(args.input) 72 | _, frame = cap.read() 73 | 74 | fh = frame.shape[0] 75 | fw = frame.shape[1] 76 | 77 | fps = FPS().start() 78 | 79 | # Process frames until the video ends, or process is exited 80 | while cap.isOpened(): 81 | # Read the next frame 82 | flag, frame = cap.read() 83 | if not flag: 84 | break 85 | 86 | key_pressed = cv2.waitKey(1) 87 | 88 | image = preprocessing(frame, in_w, in_h) 89 | 90 | # Perform inference on the frame 91 | infer_network.exec_net(image, request_id=0) 92 | 93 | # Get the output of inference 94 | if infer_network.wait(request_id=0) == 0: 95 | result = infer_network.get_output(request_id=0) 96 | print(result.shape) 97 | 98 | # Analyze road segmentation results 99 | result = np.squeeze(result, 0) 100 | result = result . transpose ( 1 , 2 , 0 ) # HWC 101 | result = np.argmax(result, 2) 102 | 103 | hh, ww = result.shape 104 | mask = np.zeros((hh, ww, 3), dtype=np.uint8) 105 | mask[np.where(result > 0)] = (0, 255, 255) # yellow 106 | mask[np.where(result > 1)] = (255, 0, 255) # 107 | 108 | #cv2.imshow("mask", mask) 109 | 110 | # Superimpose output results 111 | mask = cv2.resize(mask, dsize=(frame.shape[1], frame.shape[0])) 112 | frame = cv2.addWeighted(mask, 0.2, frame, 0.8, 0) 113 | 114 | 115 | cv2.imshow('frame', frame) 116 | if cv2.waitKey(1) & 0xFF == ord('q'): 117 | break 118 | 119 | #Break if escape key pressed 120 | if key_pressed == 27: 121 | break 122 | 123 | fps.update() 124 | 125 | # Release the out writer, capture, and destroy any OpenCV windows 126 | cap.release() 127 | 128 | cv2.destroyAllWindows() 129 | 130 | fps.stop() 131 | 132 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps())) 133 | 134 | def main(): 135 | """ 136 | Load the network and parse the output. 137 | :return: None 138 | """ 139 | # Set log to INFO 140 | log.basicConfig(level=log.CRITICAL) 141 | 142 | # Grab command line args 143 | args = build_argparser().parse_args() 144 | 145 | # Perform inference on the input stream 146 | infer_on_stream(args) 147 | 148 | if __name__ == '__main__': 149 | main() -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/009649.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/009649.png -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/__pycache__/inference.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/__pycache__/inference.cpython-36.pyc -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/inference.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import logging as log 4 | from openvino.inference_engine import IENetwork, IECore 5 | 6 | 7 | class Network: 8 | """ 9 | Load and configure inference plugins for the specified target devices 10 | and performs synchronous and asynchronous modes for the specified infer requests. 11 | """ 12 | 13 | def __init__(self): 14 | self.ie = None 15 | self.network = None 16 | self.input_blob = None 17 | self.output_blob = None 18 | self.exec_network = None 19 | self.infer_request = None 20 | 21 | 22 | def load_model(self, model, args, device="CPU", num_requests=0): 23 | ''' 24 | Load the model given IR files. 25 | Defaults to CPU as device for use in the workspace. 26 | Synchronous requests made within. 27 | ''' 28 | model_xml = model 29 | model_bin = os.path.splitext(model_xml)[0] + ".bin" 30 | 31 | 32 | # Initialize the plugin 33 | self.ie = IECore() 34 | 35 | # Read the IR as a IENetwork 36 | try: 37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin) 38 | except Exception as e: 39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?") 40 | 41 | if args.cpu_extension and "CPU" in args.device: 42 | ie.add_extension(args.cpu_extension, "CPU") 43 | log.info("CPU extension loaded: {}".format(args.cpu_extension)) 44 | 45 | if "CPU" in args.device: 46 | supported_layers = self.ie.query_network(self.network, "CPU") 47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers] 48 | if len(not_supported_layers) != 0: 49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}". 50 | format(args.device, ', '.join(not_supported_layers))) 51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l " 52 | "or --cpu_extension command line argument") 53 | sys.exit(1) 54 | 55 | # Load the IENetwork into the plugin 56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests) 57 | 58 | def get_input_shape(self): 59 | ''' 60 | Gets the input shape of the network 61 | ''' 62 | log.info("Preparing input blobs") 63 | input_shape = [] 64 | print("inputs number: " + str(len(self.network.input_info.keys()))) 65 | 66 | for input_key in self.network.input_info: 67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape)) 68 | print("input key: " + input_key) 69 | self.input_blob = input_key 70 | if len(self.network.input_info[input_key].input_data.layout) == 4: 71 | input_shape = self.network.input_info[input_key].input_data.shape 72 | 73 | return input_shape 74 | 75 | def get_output_name(self): 76 | ''' 77 | Gets the input shape of the network 78 | ''' 79 | log.info('Preparing output blobs') 80 | 81 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))] 82 | for output_key in self.network.outputs: 83 | if self.network.layers[output_key].type == "Convert": 84 | self.output_blob, _ = output_key, self.network.outputs[output_key] 85 | 86 | if self.output_blob == "": 87 | log.error("Can't find a L0317_ReWeight_SoftMax layer in the topology") 88 | exit(-1) 89 | return self.output_blob 90 | 91 | def exec_net(self, image, request_id): 92 | ''' 93 | Makes an asynchronous inference request, given an input image. 94 | ''' 95 | self.exec_network.start_async(request_id=request_id, 96 | inputs={self.input_blob: image}) 97 | return 98 | 99 | 100 | def wait(self, request_id): 101 | ''' 102 | Checks the status of the inference request. 103 | ''' 104 | status = self.exec_network.requests[request_id].wait(-1) 105 | return status 106 | 107 | 108 | def get_output(self, request_id): 109 | ''' 110 | Returns a list of the results for the output layer of the network. 111 | ''' 112 | return self.exec_network.requests[request_id].outputs[self.output_blob] -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import socket 5 | import json 6 | import cv2 7 | import numpy as np 8 | 9 | import logging as log 10 | 11 | from argparse import ArgumentParser 12 | from inference import Network 13 | from PIL import Image 14 | 15 | from imutils.video import FPS 16 | 17 | def build_argparser(): 18 | """ 19 | Parse command line arguments. 20 | :return: command line arguments 21 | """ 22 | parser = ArgumentParser() 23 | parser.add_argument("-m", "--model", required=True, type=str, 24 | help="Path to an xml file with a trained model.") 25 | parser.add_argument("-i", "--input", required=True, type=str, 26 | help="Path to image or video file") 27 | parser.add_argument("-d", "--device", type=str, default="CPU", 28 | help="Specify the target device to infer on: " 29 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample " 30 | "will look for a suitable plugin for device " 31 | "specified (CPU by default)") 32 | parser.add_argument("-l", "--cpu_extension", 33 | help="Optional. Required for CPU custom layers. " 34 | "Absolute path to a shared library with the kernels implementations.", 35 | type=str, default=None) 36 | return parser 37 | 38 | 39 | def preprocessing(frame, in_w, in_h): 40 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA) 41 | image = np.moveaxis(image_resize, -1, 0) 42 | 43 | return image 44 | def infer_on_stream(args): 45 | """ 46 | Initialize the inference network, stream video to network, 47 | and output stats and video. 48 | :param args: Command line arguments parsed by `build_argparser()` 49 | :return: None 50 | """ 51 | # Initialize the Inference Engine 52 | infer_network = Network() 53 | 54 | # Load the model through `infer_network` 55 | infer_network.load_model(args.model, args, args.device, num_requests=0) 56 | 57 | # Get a Input blob shape 58 | _, _, in_h, in_w = infer_network.get_input_shape() 59 | 60 | 61 | # Get a output blob name 62 | output_name = infer_network.get_output_name() 63 | 64 | # Handle the input stream 65 | try: 66 | cap = cv2.VideoCapture(args.input) 67 | except FileNotFoundError: 68 | print("Cannot locate video file: "+ args.input) 69 | except Exception as e: 70 | print("Something else went wrong with the video file: ", e) 71 | 72 | cap.open(args.input) 73 | _, frame = cap.read() 74 | 75 | fh = frame.shape[0] 76 | fw = frame.shape[1] 77 | 78 | fps = FPS().start() 79 | 80 | # Process frames until the video ends, or process is exited 81 | while cap.isOpened(): 82 | # Read the next frame 83 | flag, frame = cap.read() 84 | if not flag: 85 | break 86 | 87 | key_pressed = cv2.waitKey(1) 88 | 89 | image = preprocessing(frame, in_w, in_h) 90 | 91 | # Perform inference on the frame 92 | infer_network.exec_net(image, request_id=0) 93 | 94 | seg_image = Image.open("./009649.png") 95 | palette = seg_image.getpalette() # Get a color palette 96 | # Get the output of inference 97 | if infer_network.wait(request_id=0) == 0: 98 | result = infer_network.get_output(request_id=0) 99 | output = result[0][0] 100 | output = np.array(output, dtype='uint8') 101 | output = cv2.resize(output, (fw, fh)) 102 | 103 | image = Image.fromarray(np.uint8(output), mode="P") 104 | image.putpalette(palette) 105 | image = image.convert("RGB") 106 | 107 | image = np.asarray(image) 108 | image = cv2.addWeighted(frame, 1, image, 0.9, 0) 109 | 110 | cv2.imshow('frame', image) 111 | if cv2.waitKey(1) & 0xFF == ord('q'): 112 | break 113 | 114 | #Break if escape key pressed 115 | if key_pressed == 27: 116 | break 117 | 118 | fps.update() 119 | 120 | # Release the out writer, capture, and destroy any OpenCV windows 121 | cap.release() 122 | 123 | cv2.destroyAllWindows() 124 | 125 | fps.stop() 126 | 127 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps())) 128 | 129 | def main(): 130 | """ 131 | Load the network and parse the output. 132 | :return: None 133 | """ 134 | # Set log to INFO 135 | log.basicConfig(level=log.CRITICAL) 136 | 137 | # Grab command line args 138 | args = build_argparser().parse_args() 139 | 140 | # Perform inference on the input stream 141 | infer_on_stream(args) 142 | 143 | if __name__ == '__main__': 144 | main() -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/core/__pycache__/semantic_detection.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/core/__pycache__/semantic_detection.cpython-36.pyc -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/core/semantic_detection.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import cv2 3 | from PIL import Image 4 | import matplotlib.pyplot as plt 5 | 6 | from numpy import clip 7 | from utils.ie_module import Module 8 | 9 | class SemanticSegmentation(Module): 10 | def __init__(self, model): 11 | super(SemanticSegmentation, self).__init__(model) 12 | 13 | self.input_blob = next(iter(model.inputs)) 14 | self.input_name = next(iter(model.input_info)) 15 | 16 | self.output_blob = next(iter(model.outputs)) 17 | self.input_shape = model.inputs[self.input_blob].shape 18 | self.output_shape = model.outputs[self.output_blob].shape 19 | 20 | print("Input Shape: {}".format(self.input_shape)) 21 | print("Output Shape: {}".format(self.output_shape)) 22 | 23 | def resize_input(self, frame, target_shape): 24 | assert len(frame.shape) == len(target_shape), \ 25 | "Expected a frame with %s dimensions, but got %s" % \ 26 | (len(target_shape), len(frame.shape)) 27 | 28 | assert frame.shape[0] == 1, "Only batch size 1 is supported" 29 | n, c, h, w = target_shape 30 | 31 | input = frame[0] 32 | if not np.array_equal(target_shape[-2:], frame.shape[-2:]): 33 | input = input.transpose((1, 2, 0)) # to HWC 34 | input = cv2.resize(input, (w, h)) 35 | input = input.transpose((2, 0, 1)) # to CHW 36 | 37 | return input.reshape((n, c, h, w)) 38 | 39 | def preprocess(self, frame): 40 | assert len(frame.shape) == 4, "Frame shape should be [1, c, h, w]" 41 | assert frame.shape[0] == 1 42 | assert frame.shape[1] == 3 43 | input = self.resize_input(frame, self.input_shape) 44 | return input 45 | 46 | def draw_masks(self, result, width, height): 47 | ''' 48 | Draw semantic mask classes onto the frame. 49 | ''' 50 | # Create a mask with color by class 51 | classes = cv2.resize(result[0].transpose((1, 2, 0)), (width, height), 52 | interpolation=cv2.INTER_NEAREST) 53 | unique_classes = np.unique(classes) 54 | out_mask = classes * (255 / 20) 55 | 56 | # Stack the mask so FFmpeg understands it 57 | out_mask = np.dstack((out_mask, out_mask, out_mask)) 58 | out_mask = np.uint8(out_mask) 59 | 60 | return out_mask, unique_classes 61 | 62 | def post_processing(self, frame): 63 | print(frame.shape) 64 | H, W, C = frame.shape 65 | seg_image = Image.open("../../../demo/009649.png") 66 | palette = seg_image.getpalette() # Get a color palette 67 | outputs = self.get_outputs()[0][self.output_blob] 68 | print(outputs.shape) 69 | output = outputs[0][0] 70 | print(output.shape) 71 | print(type(output)) 72 | output = np.array(output, dtype='uint8') 73 | print(type(output)) 74 | print("Height: {}".format(H)) 75 | print("Weight: {}".format(W)) 76 | output = cv2.resize(output, (W, H)) 77 | 78 | image = Image.fromarray(np.uint8(output), mode="P") 79 | image.putpalette(palette) 80 | image = image.convert("RGB") 81 | 82 | image = np.asarray(image) 83 | print("Frame size: {}".format(frame.shape)) 84 | print("image size: {}".format(image.shape)) 85 | image = cv2.addWeighted(frame, 1, image, 0.9, 0) 86 | # plt.imshow(image) 87 | # plt.pause(0.001) 88 | #plt.show() 89 | # plt.show() 90 | cv2.imshow("Result", image) 91 | cv2.waitKey(1) 92 | 93 | def start_async(self, frame): 94 | input = self.preprocess(frame) 95 | input_data = {self.input_name: input} 96 | self.enqueue(input_data) 97 | 98 | -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/main.py: -------------------------------------------------------------------------------- 1 | import logging as log 2 | import sys 3 | import time 4 | import os.path as osp 5 | import cv2 6 | import numpy as np 7 | 8 | from openvino.inference_engine import IENetwork 9 | from argparse import ArgumentParser 10 | 11 | from utils.ie_module import InferenceContext 12 | from core.semantic_detection import SemanticSegmentation 13 | 14 | #Global 15 | DEVICE_KINDS = ['CPU', 'GPU', 'FPGA', 'MYRIAD', 'HETERO', 'HDDL'] 16 | 17 | def build_argparser(): 18 | """ 19 | Parse command line arguments. 20 | :return: command line arguments 21 | """ 22 | 23 | parser = ArgumentParser() 24 | parser.add_argument("-input", "--input", required=True, type=str, 25 | help="Path to image or video file or enter cam for webcam") 26 | parser.add_argument("-model", "--detection_model", required=True, type=str, 27 | help="Path to an .xml file with a trained Face Detection model") 28 | parser.add_argument('-device', default='CPU', choices=DEVICE_KINDS, 29 | help="(optional) Target device for the " \ 30 | "detection model (default: %(default)s)") 31 | parser.add_argument('-v', '--verbose', action='store_true', 32 | help="(optional) Be more verbose") 33 | parser.add_argument('-l', '--cpu_lib', metavar="PATH", default="", 34 | help="(optional) For MKLDNN (CPU)-targeted custom layers, if any. " \ 35 | "Path to a shared library with custom layers implementations") 36 | parser.add_argument('-c', '--gpu_lib', metavar="PATH", default="", 37 | help="(optional) For clDNN (GPU)-targeted custom layers, if any. " \ 38 | "Path to the XML file with descriptions of the kernels") 39 | parser.add_argument('-pc', '--perf_stats', action='store_true', 40 | help="(optional) Output detailed per-layer performance stats") 41 | return parser 42 | 43 | class ProcessOnFrame: 44 | # Queue size will be used to put frames in a queue for 45 | # Inference Engine 46 | QUEUE_SIZE = 10 47 | 48 | def __init__(self, args): 49 | used_devices = set([args.device]) 50 | 51 | # Create a Inference Engine Context 52 | self.context = InferenceContext() 53 | context = self.context 54 | 55 | # Load OpenVino Plugin based on device selection 56 | context.load_plugins(used_devices, args.cpu_lib, args.gpu_lib) 57 | for d in used_devices: 58 | context.get_plugin(d).set_config({ 59 | "PERF_COUNT": "YES" if args.perf_stats else "NO"}) 60 | 61 | log.info("Loading models") 62 | start_time = time.perf_counter() 63 | 64 | # Load face detection model on Inference Engine 65 | segmentation = self.load_model(args.detection_model) 66 | 67 | stop_time = time.perf_counter() 68 | print("[INFO] Model Load Time: {}".format(stop_time - start_time)) 69 | 70 | self.segmentation = SemanticSegmentation(segmentation) 71 | 72 | self.segmentation.deploy(args.device, context) 73 | 74 | log.info("Models are loaded") 75 | 76 | def load_model(self, model_path): 77 | """ 78 | Initializing IENetwork(Inference Enginer) object from IR files: 79 | 80 | Args: 81 | Model path - This should contain both .xml and .bin file 82 | :return Instance of IENetwork class 83 | """ 84 | 85 | model_path = osp.abspath(model_path) 86 | model_description_path = model_path 87 | model_weights_path = osp.splitext(model_path)[0] + ".bin" 88 | 89 | print(model_weights_path) 90 | log.info("Loading the model from '%s'" % (model_description_path)) 91 | assert osp.isfile(model_description_path), \ 92 | "Model description is not found at '%s'" % (model_description_path) 93 | assert osp.isfile(model_weights_path), \ 94 | "Model weights are not found at '%s'" % (model_weights_path) 95 | 96 | # Load model on IE 97 | model = IENetwork(model_description_path, model_weights_path) 98 | log.info("Model is loaded") 99 | 100 | return model 101 | 102 | def frame_pre_process(self, frame): 103 | """ 104 | Pre-Process the input frame given to model 105 | Args: 106 | frame: Input frame from video stream 107 | Return: 108 | frame: Pre-Processed frame 109 | """ 110 | assert len(frame.shape) == 3, \ 111 | "Expected input frame in (H, W, C) format" 112 | assert frame.shape[2] in [3, 4], \ 113 | "Expected BGR or BGRA input" 114 | 115 | orig_image = frame.copy() 116 | frame = frame.transpose((2, 0, 1)) # HWC to CHW 117 | frame = np.expand_dims(frame, axis=0) 118 | return frame 119 | 120 | def detection(self, frame): 121 | image = frame.copy() 122 | frame = self.frame_pre_process(frame) 123 | 124 | self.segmentation.clear() 125 | 126 | self.segmentation.start_async(frame) 127 | self.segmentation.post_processing(image) 128 | 129 | 130 | class VisionSystem: 131 | BREAK_KEY_LABELS = "q(Q) or Escape" 132 | BREAK_KEYS = {ord('q'), ord('Q'), 27} 133 | 134 | def __init__(self, args): 135 | self.frame_processor = ProcessOnFrame(args) 136 | self.print_perf_stats = args.perf_stats 137 | 138 | def process(self): 139 | input_stream = cv2.VideoCapture("../../../demo/Paris.mp4") 140 | 141 | frame_count = 0 142 | 143 | while input_stream.isOpened(): 144 | has_frame, frame = input_stream.read() 145 | if not has_frame: 146 | break 147 | 148 | frame_count+=1 149 | print("Frame cout: {}".format(frame_count)) 150 | 151 | self.frame_processor.detection(frame) 152 | def run(self, args): 153 | if args.input == "cam": 154 | path = "0" 155 | else: 156 | path = args.input 157 | 158 | # input_stream = VisionSystem.open_input_stream(path) 159 | 160 | # if input_stream is None or not input_stream.isOpened(): 161 | # log.error("Cannot open input stream: %s" % args.input) 162 | 163 | # # FPS init 164 | # fps = input_stream.get(cv2.CAP_PROP_FPS) 165 | 166 | # # Get the Frame org size 167 | # frame_size = (int(input_stream.get(cv2.CAP_PROP_FRAME_WIDTH)), 168 | # int(input_stream.get(cv2.CAP_PROP_FRAME_HEIGHT))) 169 | 170 | # # Get the frame count if its a video 171 | # self.frame_count = int(input_stream.get(cv2.CAP_PROP_FRAME_COUNT)) 172 | 173 | self.process() 174 | 175 | @staticmethod 176 | def open_input_stream(path): 177 | """ 178 | Open the input stream 179 | """ 180 | log.info("Reading input data from '%s'" % (path)) 181 | stream = path 182 | try: 183 | stream = int(path) 184 | except ValueError: 185 | pass 186 | return cv2.VideoCapture(stream) 187 | 188 | def main(): 189 | args = build_argparser().parse_args() 190 | 191 | log.basicConfig(format="[ %(levelname)s ] %(asctime)-15s %(message)s", 192 | level=log.INFO if not args.verbose else log.DEBUG, stream=sys.stdout) 193 | detection = VisionSystem(args) 194 | detection.run(args) 195 | 196 | if __name__ == "__main__": 197 | main() -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/utils/__pycache__/ie_module.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/utils/__pycache__/ie_module.cpython-36.pyc -------------------------------------------------------------------------------- /openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/utils/ie_module.py: -------------------------------------------------------------------------------- 1 | import logging as log 2 | import os.path as osp 3 | 4 | from openvino.inference_engine import IEPlugin 5 | 6 | class InferenceContext: 7 | def __init__(self): 8 | self.plugins = {} 9 | 10 | def load_plugins(self, devices, cpu_ext="", gpu_ext=""): 11 | log.info("Loading plugins for devices: %s" % (devices)) 12 | 13 | plugins = { d: IEPlugin(d) for d in devices } 14 | if 'CPU' in plugins and not len(cpu_ext) == 0: 15 | log.info("Using CPU extensions library '%s'" % (cpu_ext)) 16 | assert osp.isfile(cpu_ext), "Failed to open CPU extensions library" 17 | plugins['CPU'].add_cpu_extension(cpu_ext) 18 | 19 | if 'GPU' in plugins and not len(gpu_ext) == 0: 20 | assert osp.isfile(gpu_ext), "Failed to open GPU definitions file" 21 | plugins['GPU'].set_config({"CONFIG_FILE": gpu_ext}) 22 | 23 | self.plugins = plugins 24 | 25 | log.info("Plugins are loaded") 26 | 27 | def get_plugin(self, device): 28 | return self.plugins.get(device, None) 29 | 30 | def check_model_support(self, net, device): 31 | plugin = self.plugins[device] 32 | 33 | if plugin.device == "CPU": 34 | supported_layers = plugin.get_supported_layers(net) 35 | not_supported_layers = [l for l in net.layers.keys() \ 36 | if l not in supported_layers] 37 | if len(not_supported_layers) != 0: 38 | log.error("The following layers are not supported " \ 39 | "by the plugin for the specified device {}:\n {}" \ 40 | .format(plugin.device, ', '.join(not_supported_layers))) 41 | log.error("Please try to specify cpu extensions " \ 42 | "library path in the command line parameters using " \ 43 | "the '-l' parameter") 44 | raise NotImplementedError( 45 | "Some layers are not supported on the device") 46 | 47 | def load_model(self, model, device, max_requests=1): 48 | self.check_model_support(model, device) 49 | plugin = self.plugins[device] 50 | deployed_model = plugin.load(network=model, num_requests=max_requests) 51 | return deployed_model 52 | 53 | class Module(object): 54 | def __init__(self, model): 55 | self.model = model 56 | self.device_model = None 57 | 58 | self.max_requests = 0 59 | self.active_requests = 0 60 | 61 | self.clear() 62 | 63 | def deploy(self, device, context, queue_size=1): 64 | self.context = context 65 | self.max_requests = queue_size 66 | self.device_model = context.load_model( 67 | self.model, device, self.max_requests) 68 | self.model = None 69 | 70 | def enqueue(self, input): 71 | self.clear() 72 | 73 | if self.max_requests <= self.active_requests: 74 | log.warning("Processing request rejected - too many requests") 75 | return False 76 | 77 | self.device_model.start_async(self.active_requests, input) 78 | self.active_requests += 1 79 | return True 80 | 81 | def wait(self): 82 | if self.active_requests <= 0: 83 | return 84 | 85 | self.perf_stats = [None, ] * self.active_requests 86 | self.outputs = [None, ] * self.active_requests 87 | for i in range(self.active_requests): 88 | self.device_model.requests[i].wait() 89 | self.outputs[i] = self.device_model.requests[i].outputs 90 | self.perf_stats[i] = self.device_model.requests[i].get_perf_counts() 91 | 92 | self.active_requests = 0 93 | 94 | def get_outputs(self): 95 | self.wait() 96 | return self.outputs 97 | 98 | def get_performance_stats(self): 99 | return self.perf_stats 100 | 101 | 102 | def clear(self): 103 | self.perf_stats = [] 104 | self.outputs = [] -------------------------------------------------------------------------------- /openvino_analysis/public/ssd_mobilenet_v2_coco/__pycache__/inference.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/public/ssd_mobilenet_v2_coco/__pycache__/inference.cpython-36.pyc -------------------------------------------------------------------------------- /openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/__pycache__/inference.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/__pycache__/inference.cpython-36.pyc -------------------------------------------------------------------------------- /openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/example.log: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/example.log -------------------------------------------------------------------------------- /openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/inference.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import logging as log 4 | from openvino.inference_engine import IENetwork, IECore 5 | 6 | 7 | class Network: 8 | """ 9 | Load and configure inference plugins for the specified target devices 10 | and performs synchronous and asynchronous modes for the specified infer requests. 11 | """ 12 | 13 | def __init__(self): 14 | self.ie = None 15 | self.network = None 16 | self.input_blob = None 17 | self.output_blob = None 18 | self.exec_network = None 19 | self.infer_request = None 20 | 21 | def load_model(self, model, device="CPU", cpu_extension=None, num_requests=0): 22 | ''' 23 | Load the model given IR files. 24 | Defaults to CPU as device for use in the workspace. 25 | Synchronous requests made within. 26 | ''' 27 | model_xml = model 28 | model_bin = os.path.splitext(model_xml)[0] + ".bin" 29 | 30 | 31 | # Initialize the plugin 32 | self.ie = IECore() 33 | 34 | # Read the IR as a IENetwork 35 | self.network = self.ie.read_network(model=model_xml, weights=model_bin) 36 | 37 | # Load the IENetwork into the plugin 38 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests) 39 | 40 | return 41 | 42 | def get_input_shape(self): 43 | ''' 44 | Gets the input shape of the network 45 | ''' 46 | log.info("Preparing input blobs") 47 | input_shape = [] 48 | print("inputs number: " + str(len(self.network.input_info.keys()))) 49 | 50 | for input_key in self.network.input_info: 51 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape)) 52 | print("input key: " + input_key) 53 | self.input_blob = input_key 54 | if len(self.network.input_info[input_key].input_data.layout) == 4: 55 | input_shape = self.network.input_info[input_key].input_data.shape 56 | 57 | return input_shape 58 | 59 | def get_output_name(self): 60 | ''' 61 | Gets the input shape of the network 62 | ''' 63 | log.info('Preparing output blobs') 64 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))] 65 | for output_key in self.network.outputs: 66 | print(output_key) 67 | if self.network.layers[output_key].type == "DetectionOutput": 68 | self.output_blob, _ = output_key, self.network.outputs[output_key] 69 | 70 | if self.output_blob == "": 71 | log.error("Can't find a DetectionOutput layer in the topology") 72 | exit(-1) 73 | 74 | return self.output_blob 75 | 76 | def exec_net_async(self, image, request_id=0): 77 | ''' 78 | Makes an asynchronous inference request, given an input image. 79 | ''' 80 | self.exec_network.start_async(request_id=request_id, 81 | inputs={self.input_blob: image}) 82 | return 83 | 84 | def exec_net_sync(self, image): 85 | ''' 86 | Makes an asynchronous inference request, given an input image. 87 | ''' 88 | return self.exec_network.infer(inputs={self.input_blob: image}) 89 | 90 | def wait(self, request_id=0): 91 | ''' 92 | Checks the status of the inference request. 93 | ''' 94 | status = self.exec_network.requests[request_id].wait(-1) 95 | return status 96 | 97 | 98 | def get_output(self, request_id=0): 99 | ''' 100 | Returns a list of the results for the output layer of the network. 101 | ''' 102 | return self.exec_network.requests[request_id].outputs[self.output_blob] -------------------------------------------------------------------------------- /openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import socket 5 | import json 6 | import cv2 7 | import numpy as np 8 | 9 | 10 | import logging as log 11 | from imutils.video import FPS 12 | 13 | from argparse import ArgumentParser 14 | from inference import Network 15 | 16 | from multiprocessing import Process, Queue 17 | import multiprocessing 18 | import threading 19 | import queue 20 | 21 | # Browser and OpenCV Window toggle 22 | CPU_EXTENSION = "/opt/intel/openvino/deployment_tools/inference_engine/lib/intel64/libcpu_extension_sse4.so" 23 | 24 | def build_argparser(): 25 | """ 26 | Parse command line arguments. 27 | :return: command line arguments 28 | """ 29 | parser = ArgumentParser() 30 | parser.add_argument("-m", "--model", required=True, type=str, 31 | help="Path to an xml file with a trained model.") 32 | parser.add_argument("-i", "--input", required=True, type=str, 33 | help="Path to image or video file") 34 | parser.add_argument("-l", "--cpu_extension", required=False, type=str, 35 | default=None, 36 | help="MKLDNN (CPU)-targeted custom layers." 37 | "Absolute path to a shared library with the" 38 | "kernels impl.") 39 | parser.add_argument("-d", "--device", type=str, default="CPU", 40 | help="Specify the target device to infer on: " 41 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample " 42 | "will look for a suitable plugin for device " 43 | "specified (CPU by default)") 44 | parser.add_argument("-pt", "--prob_threshold", type=float, default=0.2, 45 | help="Probability threshold for detections filtering" 46 | "(0.5 by default)") 47 | return parser 48 | 49 | 50 | def image_process_worker(cap, frame_queue, image_queue, in_n, in_c, in_h, in_w): 51 | # Process frames until the video ends, or process is exited 52 | while cap.isOpened(): 53 | # Read the next frame 54 | flag, frame = cap.read() 55 | if not flag: 56 | frame_queue.put(None) 57 | image_queue.put(None) 58 | break 59 | 60 | # Pre-process the frame 61 | image_resize = cv2.resize(frame, (in_w, in_h)) 62 | image = image_resize.transpose((2,0,1)) 63 | image = image.reshape(in_n, in_c, in_h, in_w) 64 | 65 | frame_queue.put(frame) 66 | image_queue.put(image) 67 | 68 | 69 | def network_inference(infer_network, frame_queue, image_queue, 70 | fw, fh, prob_threshold, fps): 71 | current_inference, next_inference = 0, 1 72 | count = 0 73 | while True: 74 | image = image_queue.get() 75 | if image is None: 76 | break 77 | count += 1 78 | frame = frame_queue.get() 79 | # Perform inference on the frame 80 | infer_network.exec_net_async(image, request_id=current_inference) 81 | 82 | # Get the output of inference 83 | if infer_network.wait(next_inference) == 0: 84 | detection = infer_network.get_output(next_inference) 85 | for box in detection[0][0]: # Output shape is 1x1x100x7 86 | conf = box[2] 87 | if conf >= prob_threshold: 88 | xmin = int(box[3] * fw) 89 | ymin = int(box[4] * fh) 90 | xmax = int(box[5] * fw) 91 | ymax = int(box[6] * fh) 92 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 125, 255), 3) 93 | current_inference, next_inference = next_inference, current_inference 94 | 95 | cv2.imshow('frame', frame) 96 | if cv2.waitKey(1) & 0xFF == ord('q'): 97 | break 98 | fps.update() 99 | if count == 100: 100 | fps.stop() 101 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps())) 102 | 103 | 104 | def infer_on_stream(args): 105 | """ 106 | Initialize the inference network, stream video to network, 107 | and output stats and video. 108 | :param args: Command line arguments parsed by `build_argparser()` 109 | :return: None 110 | """ 111 | 112 | frame_queue = queue.Queue(maxsize= 4) 113 | image_queue = queue.Queue(maxsize= 4) 114 | 115 | # Initialize the Inference Engine 116 | infer_network = Network() 117 | 118 | # Set Probability threshold for detections 119 | prob_threshold = args.prob_threshold 120 | 121 | # Load the model through `infer_network` 122 | infer_network.load_model(args.model, args.device, CPU_EXTENSION, num_requests=2) 123 | 124 | # Get a Input blob shape 125 | in_n, in_c, in_h, in_w = infer_network.get_input_shape() 126 | 127 | # Get a output blob name 128 | _ = infer_network.get_output_name() 129 | 130 | # Handle the input stream 131 | cap = cv2.VideoCapture(args.input) 132 | cap.open(args.input) 133 | _, frame = cap.read() 134 | 135 | 136 | fps = FPS().start() 137 | 138 | _s, frame = cap.read() 139 | fh = frame.shape[0] 140 | fw = frame.shape[1] 141 | 142 | preprocess_thread = None 143 | 144 | preprocess_thread = threading.Thread(target=image_process_worker, 145 | args=(cap, frame_queue, image_queue, in_n, in_c, in_h, in_w)) 146 | 147 | preprocess_thread.start() 148 | 149 | network_inference(infer_network, frame_queue, image_queue, fw, fh, prob_threshold, fps) 150 | 151 | preprocess_thread.join() 152 | 153 | # Release the out writer, capture, and destroy any OpenCV windows 154 | cap.release() 155 | cv2.destroyAllWindows() 156 | fps.stop() 157 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps())) 158 | 159 | 160 | def main(): 161 | """ 162 | Load the network and parse the output. 163 | :return: None 164 | """ 165 | # set log level 166 | log.basicConfig(filename='example.log',level=log.CRITICAL) 167 | 168 | # Grab command line args 169 | args = build_argparser().parse_args() 170 | 171 | # Inference 172 | infer_on_stream(args) 173 | 174 | if __name__ == '__main__': 175 | main() -------------------------------------------------------------------------------- /openvino_analysis/public/ssd_mobilenet_v2_coco/inference.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import logging as log 4 | from openvino.inference_engine import IENetwork, IECore 5 | 6 | 7 | class Network: 8 | """ 9 | Load and configure inference plugins for the specified target devices 10 | and performs synchronous and asynchronous modes for the specified infer requests. 11 | """ 12 | 13 | def __init__(self): 14 | self.ie = None 15 | self.network = None 16 | self.input_blob = None 17 | self.output_blob = None 18 | self.exec_network = None 19 | self.infer_request = None 20 | 21 | 22 | def load_model(self, model, args, device="CPU", num_requests=0): 23 | ''' 24 | Load the model given IR files. 25 | Defaults to CPU as device for use in the workspace. 26 | Synchronous requests made within. 27 | ''' 28 | model_xml = model 29 | model_bin = os.path.splitext(model_xml)[0] + ".bin" 30 | 31 | 32 | # Initialize the plugin 33 | self.ie = IECore() 34 | 35 | # Read the IR as a IENetwork 36 | try: 37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin) 38 | except Exception as e: 39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?") 40 | 41 | if args.cpu_extension and "CPU" in args.device: 42 | ie.add_extension(args.cpu_extension, "CPU") 43 | log.info("CPU extension loaded: {}".format(args.cpu_extension)) 44 | 45 | if "CPU" in args.device: 46 | supported_layers = self.ie.query_network(self.network, "CPU") 47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers] 48 | if len(not_supported_layers) != 0: 49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}". 50 | format(args.device, ', '.join(not_supported_layers))) 51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l " 52 | "or --cpu_extension command line argument") 53 | sys.exit(1) 54 | 55 | # Load the IENetwork into the plugin 56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests) 57 | 58 | def get_input_shape(self): 59 | ''' 60 | Gets the input shape of the network 61 | ''' 62 | log.info("Preparing input blobs") 63 | input_shape = [] 64 | print("inputs number: " + str(len(self.network.input_info.keys()))) 65 | 66 | for input_key in self.network.input_info: 67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape)) 68 | print("input key: " + input_key) 69 | self.input_blob = input_key 70 | if len(self.network.input_info[input_key].input_data.layout) == 4: 71 | input_shape = self.network.input_info[input_key].input_data.shape 72 | 73 | return input_shape 74 | 75 | def get_output_name(self): 76 | ''' 77 | Gets the input shape of the network 78 | ''' 79 | log.info('Preparing output blobs') 80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))] 81 | for output_key in self.network.outputs: 82 | print(output_key) 83 | if self.network.layers[output_key].type == "DetectionOutput": 84 | self.output_blob, _ = output_key, self.network.outputs[output_key] 85 | 86 | if self.output_blob == "": 87 | log.error("Can't find a DetectionOutput layer in the topology") 88 | exit(-1) 89 | 90 | return self.output_blob 91 | 92 | def exec_net(self, image, request_id): 93 | ''' 94 | Makes an asynchronous inference request, given an input image. 95 | ''' 96 | self.exec_network.start_async(request_id=request_id, 97 | inputs={self.input_blob: image}) 98 | return 99 | 100 | 101 | def wait(self, request_id): 102 | ''' 103 | Checks the status of the inference request. 104 | ''' 105 | status = self.exec_network.requests[request_id].wait(-1) 106 | return status 107 | 108 | 109 | def get_output(self, request_id): 110 | ''' 111 | Returns a list of the results for the output layer of the network. 112 | ''' 113 | return self.exec_network.requests[request_id].outputs[self.output_blob] -------------------------------------------------------------------------------- /openvino_analysis/public/ssd_mobilenet_v2_coco/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import socket 5 | import json 6 | import cv2 7 | import numpy as np 8 | 9 | import logging as log 10 | 11 | from argparse import ArgumentParser 12 | from inference import Network 13 | 14 | from imutils.video import FPS 15 | 16 | def build_argparser(): 17 | """ 18 | Parse command line arguments. 19 | :return: command line arguments 20 | """ 21 | parser = ArgumentParser() 22 | parser.add_argument("-m", "--model", required=True, type=str, 23 | help="Path to an xml file with a trained model.") 24 | parser.add_argument("-i", "--input", required=True, type=str, 25 | help="Path to image or video file") 26 | parser.add_argument("-d", "--device", type=str, default="CPU", 27 | help="Specify the target device to infer on: " 28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample " 29 | "will look for a suitable plugin for device " 30 | "specified (CPU by default)") 31 | parser.add_argument("-l", "--cpu_extension", 32 | help="Optional. Required for CPU custom layers. " 33 | "Absolute path to a shared library with the kernels implementations.", 34 | type=str, default=None) 35 | return parser 36 | 37 | 38 | def preprocessing(frame, in_w, in_h): 39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA) 40 | image = np.moveaxis(image_resize, -1, 0) 41 | 42 | return image 43 | def infer_on_stream(args): 44 | """ 45 | Initialize the inference network, stream video to network, 46 | and output stats and video. 47 | :param args: Command line arguments parsed by `build_argparser()` 48 | :return: None 49 | """ 50 | # Initialize the Inference Engine 51 | infer_network = Network() 52 | 53 | # Load the model through `infer_network` 54 | infer_network.load_model(args.model, args, args.device, num_requests=0) 55 | 56 | # Get a Input blob shape 57 | _, _, in_h, in_w = infer_network.get_input_shape() 58 | 59 | 60 | # Get a output blob name 61 | output_name = infer_network.get_output_name() 62 | 63 | # Handle the input stream 64 | try: 65 | cap = cv2.VideoCapture(args.input) 66 | except FileNotFoundError: 67 | print("Cannot locate video file: "+ args.input) 68 | except Exception as e: 69 | print("Something else went wrong with the video file: ", e) 70 | 71 | cap.open(args.input) 72 | _, frame = cap.read() 73 | 74 | fh = frame.shape[0] 75 | fw = frame.shape[1] 76 | 77 | fps = FPS().start() 78 | 79 | # Process frames until the video ends, or process is exited 80 | while cap.isOpened(): 81 | # Read the next frame 82 | flag, frame = cap.read() 83 | if not flag: 84 | break 85 | 86 | key_pressed = cv2.waitKey(1) 87 | 88 | image = preprocessing(frame, in_w, in_h) 89 | 90 | # Perform inference on the frame 91 | infer_network.exec_net(image, request_id=0) 92 | 93 | # Get the output of inference 94 | if infer_network.wait(request_id=0) == 0: 95 | result = infer_network.get_output(request_id=0) 96 | for box in result[0][0]: # Output shape is 1x1x100x7 97 | conf = box[2] 98 | if conf >= 0.5: 99 | xmin = int(box[3] * fw) 100 | ymin = int(box[4] * fh) 101 | xmax = int(box[5] * fw) 102 | ymax = int(box[6] * fh) 103 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 0, 255), 3) 104 | 105 | cv2.imshow('frame', frame) 106 | if cv2.waitKey(1) & 0xFF == ord('q'): 107 | break 108 | 109 | #Break if escape key pressed 110 | if key_pressed == 27: 111 | break 112 | 113 | fps.update() 114 | 115 | # Release the out writer, capture, and destroy any OpenCV windows 116 | cap.release() 117 | 118 | cv2.destroyAllWindows() 119 | 120 | fps.stop() 121 | 122 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps())) 123 | 124 | def main(): 125 | """ 126 | Load the network and parse the output. 127 | :return: None 128 | """ 129 | # Set log to INFO 130 | log.basicConfig(level=log.CRITICAL) 131 | 132 | # Grab command line args 133 | args = build_argparser().parse_args() 134 | 135 | # Perform inference on the input stream 136 | infer_on_stream(args) 137 | 138 | if __name__ == '__main__': 139 | main() -------------------------------------------------------------------------------- /openvino_analysis/public/yolo-v3/main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | Copyright (C) 2018-2020 Intel Corporation 4 | 5 | Licensed under the Apache License, Version 2.0 (the "License"); 6 | you may not use this file except in compliance with the License. 7 | You may obtain a copy of the License at 8 | 9 | http://www.apache.org/licenses/LICENSE-2.0 10 | 11 | Unless required by applicable law or agreed to in writing, software 12 | distributed under the License is distributed on an "AS IS" BASIS, 13 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | See the License for the specific language governing permissions and 15 | limitations under the License. 16 | """ 17 | 18 | import logging 19 | import threading 20 | import os 21 | import sys 22 | from collections import deque 23 | from argparse import ArgumentParser, SUPPRESS 24 | from math import exp as exp 25 | from time import perf_counter 26 | from enum import Enum 27 | 28 | import cv2 29 | import numpy as np 30 | from openvino.inference_engine import IECore 31 | 32 | sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'common')) 33 | # import monitors 34 | 35 | 36 | logging.basicConfig(format="[ %(levelname)s ] %(message)s", level=logging.INFO, stream=sys.stdout) 37 | log = logging.getLogger() 38 | 39 | def build_argparser(): 40 | parser = ArgumentParser(add_help=False) 41 | args = parser.add_argument_group('Options') 42 | args.add_argument('-h', '--help', action='help', default=SUPPRESS, help='Show this help message and exit.') 43 | args.add_argument("-m", "--model", help="Required. Path to an .xml file with a trained model.", 44 | required=True, type=str) 45 | args.add_argument("-i", "--input", help="Required. Path to an image/video file. (Specify 'cam' to work with " 46 | "camera)", required=True, type=str) 47 | args.add_argument("-l", "--cpu_extension", 48 | help="Optional. Required for CPU custom layers. Absolute path to a shared library with " 49 | "the kernels implementations.", type=str, default=None) 50 | args.add_argument("-d", "--device", 51 | help="Optional. Specify the target device to infer on; CPU, GPU, FPGA, HDDL or MYRIAD is" 52 | " acceptable. The sample will look for a suitable plugin for device specified. " 53 | "Default value is CPU", default="CPU", type=str) 54 | args.add_argument("--labels", help="Optional. Labels mapping file", default=None, type=str) 55 | args.add_argument("-t", "--prob_threshold", help="Optional. Probability threshold for detections filtering", 56 | default=0.5, type=float) 57 | args.add_argument("-iout", "--iou_threshold", help="Optional. Intersection over union threshold for overlapping " 58 | "detections filtering", default=0.4, type=float) 59 | args.add_argument("-r", "--raw_output_message", help="Optional. Output inference results raw values showing", 60 | default=False, action="store_true") 61 | args.add_argument("-nireq", "--num_infer_requests", help="Optional. Number of infer requests", 62 | default=1, type=int) 63 | args.add_argument("-nstreams", "--num_streams", 64 | help="Optional. Number of streams to use for inference on the CPU or/and GPU in throughput mode " 65 | "(for HETERO and MULTI device cases use format :,: " 66 | "or just )", 67 | default="", type=str) 68 | args.add_argument("-nthreads", "--number_threads", 69 | help="Optional. Number of threads to use for inference on CPU (including HETERO cases)", 70 | default=None, type=int) 71 | args.add_argument("-loop_input", "--loop_input", help="Optional. Iterate over input infinitely", 72 | action='store_true') 73 | args.add_argument("-no_show", "--no_show", help="Optional. Don't show output", action='store_true') 74 | args.add_argument('-u', '--utilization_monitors', default='', type=str, 75 | help='Optional. List of monitors to show initially.') 76 | args.add_argument("--keep_aspect_ratio", action="store_true", default=False, 77 | help='Optional. Keeps aspect ratio on resize.') 78 | return parser 79 | 80 | 81 | class YoloParams: 82 | # ------------------------------------------- Extracting layer parameters ------------------------------------------ 83 | # Magic numbers are copied from yolo samples 84 | def __init__(self, param, side): 85 | self.num = 3 if 'num' not in param else int(param['num']) 86 | self.coords = 4 if 'coords' not in param else int(param['coords']) 87 | self.classes = 80 if 'classes' not in param else int(param['classes']) 88 | self.side = side 89 | self.anchors = [10.0, 13.0, 16.0, 30.0, 33.0, 23.0, 30.0, 61.0, 62.0, 45.0, 59.0, 119.0, 116.0, 90.0, 156.0, 90 | 198.0, 91 | 373.0, 326.0] if 'anchors' not in param else [float(a) for a in param['anchors'].split(',')] 92 | 93 | self.isYoloV3 = False 94 | 95 | if param.get('mask'): 96 | mask = [int(idx) for idx in param['mask'].split(',')] 97 | self.num = len(mask) 98 | 99 | maskedAnchors = [] 100 | for idx in mask: 101 | maskedAnchors += [self.anchors[idx * 2], self.anchors[idx * 2 + 1]] 102 | self.anchors = maskedAnchors 103 | 104 | self.isYoloV3 = True # Weak way to determine but the only one. 105 | 106 | 107 | class Modes(Enum): 108 | USER_SPECIFIED = 0 109 | MIN_LATENCY = 1 110 | 111 | 112 | class Mode(): 113 | def __init__(self, value): 114 | self.current = value 115 | 116 | def next(self): 117 | if self.current.value + 1 < len(Modes): 118 | self.current = Modes(self.current.value + 1) 119 | else: 120 | self.current = Modes(0) 121 | 122 | 123 | class ModeInfo(): 124 | def __init__(self): 125 | self.last_start_time = perf_counter() 126 | self.last_end_time = None 127 | self.frames_count = 0 128 | self.latency_sum = 0 129 | 130 | 131 | def scale_bbox(x, y, height, width, class_id, confidence, im_h, im_w, is_proportional): 132 | if is_proportional: 133 | scale = np.array([min(im_w/im_h, 1), min(im_h/im_w, 1)]) 134 | offset = 0.5*(np.ones(2) - scale) 135 | x, y = (np.array([x, y]) - offset) / scale 136 | width, height = np.array([width, height]) / scale 137 | xmin = int((x - width / 2) * im_w) 138 | ymin = int((y - height / 2) * im_h) 139 | xmax = int(xmin + width * im_w) 140 | ymax = int(ymin + height * im_h) 141 | # Method item() used here to convert NumPy types to native types for compatibility with functions, which don't 142 | # support Numpy types (e.g., cv2.rectangle doesn't support int64 in color parameter) 143 | return dict(xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax, class_id=class_id.item(), confidence=confidence.item()) 144 | 145 | 146 | def parse_yolo_region(predictions, resized_image_shape, original_im_shape, params, threshold, is_proportional): 147 | # ------------------------------------------ Validating output parameters ------------------------------------------ 148 | _, _, out_blob_h, out_blob_w = predictions.shape 149 | assert out_blob_w == out_blob_h, "Invalid size of output blob. It sould be in NCHW layout and height should " \ 150 | "be equal to width. Current height = {}, current width = {}" \ 151 | "".format(out_blob_h, out_blob_w) 152 | 153 | # ------------------------------------------ Extracting layer parameters ------------------------------------------- 154 | orig_im_h, orig_im_w = original_im_shape 155 | resized_image_h, resized_image_w = resized_image_shape 156 | objects = list() 157 | size_normalizer = (resized_image_w, resized_image_h) if params.isYoloV3 else (params.side, params.side) 158 | bbox_size = params.coords + 1 + params.classes 159 | # ------------------------------------------- Parsing YOLO Region output ------------------------------------------- 160 | for row, col, n in np.ndindex(params.side, params.side, params.num): 161 | # Getting raw values for each detection bounding box 162 | bbox = predictions[0, n*bbox_size:(n+1)*bbox_size, row, col] 163 | x, y, width, height, object_probability = bbox[:5] 164 | class_probabilities = bbox[5:] 165 | if object_probability < threshold: 166 | continue 167 | # Process raw value 168 | x = (col + x) / params.side 169 | y = (row + y) / params.side 170 | # Value for exp is very big number in some cases so following construction is using here 171 | try: 172 | width = exp(width) 173 | height = exp(height) 174 | except OverflowError: 175 | continue 176 | # Depends on topology we need to normalize sizes by feature maps (up to YOLOv3) or by input shape (YOLOv3) 177 | width = width * params.anchors[2 * n] / size_normalizer[0] 178 | height = height * params.anchors[2 * n + 1] / size_normalizer[1] 179 | 180 | class_id = np.argmax(class_probabilities) 181 | confidence = class_probabilities[class_id]*object_probability 182 | if confidence < threshold: 183 | continue 184 | objects.append(scale_bbox(x=x, y=y, height=height, width=width, class_id=class_id, confidence=confidence, 185 | im_h=orig_im_h, im_w=orig_im_w, is_proportional=is_proportional)) 186 | return objects 187 | 188 | 189 | def intersection_over_union(box_1, box_2): 190 | width_of_overlap_area = min(box_1['xmax'], box_2['xmax']) - max(box_1['xmin'], box_2['xmin']) 191 | height_of_overlap_area = min(box_1['ymax'], box_2['ymax']) - max(box_1['ymin'], box_2['ymin']) 192 | if width_of_overlap_area < 0 or height_of_overlap_area < 0: 193 | area_of_overlap = 0 194 | else: 195 | area_of_overlap = width_of_overlap_area * height_of_overlap_area 196 | box_1_area = (box_1['ymax'] - box_1['ymin']) * (box_1['xmax'] - box_1['xmin']) 197 | box_2_area = (box_2['ymax'] - box_2['ymin']) * (box_2['xmax'] - box_2['xmin']) 198 | area_of_union = box_1_area + box_2_area - area_of_overlap 199 | if area_of_union == 0: 200 | return 0 201 | return area_of_overlap / area_of_union 202 | 203 | 204 | def resize(image, size, keep_aspect_ratio, interpolation=cv2.INTER_LINEAR): 205 | if not keep_aspect_ratio: 206 | return cv2.resize(image, size, interpolation=interpolation) 207 | 208 | iw, ih = image.shape[0:2][::-1] 209 | w, h = size 210 | scale = min(w/iw, h/ih) 211 | nw = int(iw*scale) 212 | nh = int(ih*scale) 213 | image = cv2.resize(image, (nw, nh), interpolation=interpolation) 214 | new_image = np.full((size[1], size[0], 3), 128, dtype=np.uint8) 215 | dx = (w-nw)//2 216 | dy = (h-nh)//2 217 | new_image[dy:dy+nh, dx:dx+nw, :] = image 218 | return new_image 219 | 220 | 221 | def preprocess_frame(frame, input_height, input_width, nchw_shape, keep_aspect_ratio): 222 | in_frame = resize(frame, (input_width, input_height), keep_aspect_ratio) 223 | if nchw_shape: 224 | in_frame = in_frame.transpose((2, 0, 1)) # Change data layout from HWC to CHW 225 | in_frame = np.expand_dims(in_frame, axis=0) 226 | return in_frame 227 | 228 | 229 | def get_objects(output, net, new_frame_height_width, source_height_width, prob_threshold, is_proportional): 230 | objects = list() 231 | 232 | for layer_name, out_blob in output.items(): 233 | out_blob = out_blob.buffer.reshape(net.layers[net.layers[layer_name].parents[0]].out_data[0].shape) 234 | layer_params = YoloParams(net.layers[layer_name].params, out_blob.shape[2]) 235 | objects += parse_yolo_region(out_blob, new_frame_height_width, source_height_width, layer_params, 236 | prob_threshold, is_proportional) 237 | 238 | return objects 239 | 240 | 241 | def filter_objects(objects, iou_threshold, prob_threshold): 242 | # Filtering overlapping boxes with respect to the --iou_threshold CLI parameter 243 | objects = sorted(objects, key=lambda obj : obj['confidence'], reverse=True) 244 | for i in range(len(objects)): 245 | if objects[i]['confidence'] == 0: 246 | continue 247 | for j in range(i + 1, len(objects)): 248 | if intersection_over_union(objects[i], objects[j]) > iou_threshold: 249 | objects[j]['confidence'] = 0 250 | 251 | return tuple(obj for obj in objects if obj['confidence'] >= prob_threshold) 252 | 253 | 254 | def async_callback(status, callback_args): 255 | request, frame_id, frame_mode, frame, start_time, completed_request_results, empty_requests, \ 256 | mode, event, callback_exceptions = callback_args 257 | 258 | try: 259 | if status != 0: 260 | raise RuntimeError('Infer Request has returned status code {}'.format(status)) 261 | 262 | completed_request_results[frame_id] = (frame, request.output_blobs, start_time, frame_mode == mode.current) 263 | 264 | if mode.current == frame_mode: 265 | empty_requests.append(request) 266 | except Exception as e: 267 | callback_exceptions.append(e) 268 | 269 | event.set() 270 | 271 | 272 | def put_highlighted_text(frame, message, position, font_face, font_scale, color, thickness): 273 | cv2.putText(frame, message, position, font_face, font_scale, (255, 255, 255), thickness + 1) # white border 274 | cv2.putText(frame, message, position, font_face, font_scale, color, thickness) 275 | 276 | 277 | def await_requests_completion(requests): 278 | for request in requests: 279 | request.wait() 280 | 281 | 282 | def main(): 283 | args = build_argparser().parse_args() 284 | 285 | # ------------- 1. Plugin initialization for specified device and load extensions library if specified ------------- 286 | log.info("Creating Inference Engine...") 287 | ie = IECore() 288 | 289 | config_user_specified = {} 290 | config_min_latency = {} 291 | 292 | devices_nstreams = {} 293 | if args.num_streams: 294 | devices_nstreams = {device: args.num_streams for device in ['CPU', 'GPU'] if device in args.device} \ 295 | if args.num_streams.isdigit() \ 296 | else dict([device.split(':') for device in args.num_streams.split(',')]) 297 | 298 | if 'CPU' in args.device: 299 | if args.cpu_extension: 300 | ie.add_extension(args.cpu_extension, 'CPU') 301 | if args.number_threads is not None: 302 | config_user_specified['CPU_THREADS_NUM'] = str(args.number_threads) 303 | if 'CPU' in devices_nstreams: 304 | config_user_specified['CPU_THROUGHPUT_STREAMS'] = devices_nstreams['CPU'] \ 305 | if int(devices_nstreams['CPU']) > 0 \ 306 | else 'CPU_THROUGHPUT_AUTO' 307 | 308 | config_min_latency['CPU_THROUGHPUT_STREAMS'] = '1' 309 | 310 | if 'GPU' in args.device: 311 | if 'GPU' in devices_nstreams: 312 | config_user_specified['GPU_THROUGHPUT_STREAMS'] = devices_nstreams['GPU'] \ 313 | if int(devices_nstreams['GPU']) > 0 \ 314 | else 'GPU_THROUGHPUT_AUTO' 315 | 316 | config_min_latency['GPU_THROUGHPUT_STREAMS'] = '1' 317 | 318 | # -------------------- 2. Reading the IR generated by the Model Optimizer (.xml and .bin files) -------------------- 319 | log.info("Loading network") 320 | net = ie.read_network(args.model, os.path.splitext(args.model)[0] + ".bin") 321 | 322 | # ---------------------------------- 3. Load CPU extension for support specific layer ------------------------------ 323 | if "CPU" in args.device: 324 | supported_layers = ie.query_network(net, "CPU") 325 | not_supported_layers = [l for l in net.layers.keys() if l not in supported_layers] 326 | if len(not_supported_layers) != 0: 327 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}". 328 | format(args.device, ', '.join(not_supported_layers))) 329 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l " 330 | "or --cpu_extension command line argument") 331 | sys.exit(1) 332 | 333 | assert len(net.input_info) == 1, "Sample supports only YOLO V3 based single input topologies" 334 | 335 | # ---------------------------------------------- 4. Preparing inputs ----------------------------------------------- 336 | log.info("Preparing inputs") 337 | input_blob = next(iter(net.input_info)) 338 | 339 | # Read and pre-process input images 340 | if net.input_info[input_blob].input_data.shape[1] == 3: 341 | input_height, input_width = net.input_info[input_blob].input_data.shape[2:] 342 | nchw_shape = True 343 | else: 344 | input_height, input_width = net.input_info[input_blob].input_data.shape[1:3] 345 | nchw_shape = False 346 | 347 | if args.labels: 348 | with open(args.labels, 'r') as f: 349 | labels_map = [x.strip() for x in f] 350 | else: 351 | labels_map = None 352 | 353 | input_stream = 0 if args.input == "cam" else args.input 354 | 355 | mode = Mode(Modes.USER_SPECIFIED) 356 | cap = cv2.VideoCapture(input_stream) 357 | wait_key_time = 1 358 | 359 | # ----------------------------------------- 5. Loading model to the plugin ----------------------------------------- 360 | log.info("Loading model to the plugin") 361 | exec_nets = {} 362 | 363 | exec_nets[Modes.USER_SPECIFIED] = ie.load_network(network=net, device_name=args.device, 364 | config=config_user_specified, 365 | num_requests=args.num_infer_requests) 366 | exec_nets[Modes.MIN_LATENCY] = ie.load_network(network=net, device_name=args.device.split(":")[-1].split(",")[0], 367 | config=config_min_latency, 368 | num_requests=1) 369 | 370 | empty_requests = deque(exec_nets[mode.current].requests) 371 | completed_request_results = {} 372 | next_frame_id = 0 373 | next_frame_id_to_show = 0 374 | mode_info = { mode.current: ModeInfo() } 375 | event = threading.Event() 376 | callback_exceptions = [] 377 | 378 | # ----------------------------------------------- 6. Doing inference ----------------------------------------------- 379 | log.info("Starting inference...") 380 | print("To close the application, press 'CTRL+C' here or switch to the output window and press ESC key") 381 | print("To switch between min_latency/user_specified modes, press TAB key in the output window") 382 | 383 | # presenter = monitors.Presenter(args.utilization_monitors, 55, 384 | # (round(cap.get(cv2.CAP_PROP_FRAME_WIDTH) / 4), round(cap.get(cv2.CAP_PROP_FRAME_HEIGHT) / 8))) 385 | 386 | while (cap.isOpened() \ 387 | or completed_request_results \ 388 | or len(empty_requests) < len(exec_nets[mode.current].requests)) \ 389 | and not callback_exceptions: 390 | if next_frame_id_to_show in completed_request_results: 391 | frame, output, start_time, is_same_mode = completed_request_results.pop(next_frame_id_to_show) 392 | 393 | next_frame_id_to_show += 1 394 | if is_same_mode: 395 | mode_info[mode.current].frames_count += 1 396 | 397 | objects = get_objects(output, net, (input_height, input_width), frame.shape[:-1], args.prob_threshold, 398 | args.keep_aspect_ratio) 399 | objects = filter_objects(objects, args.iou_threshold, args.prob_threshold) 400 | 401 | if len(objects) and args.raw_output_message: 402 | log.info(" Class ID | Confidence | XMIN | YMIN | XMAX | YMAX | COLOR ") 403 | 404 | origin_im_size = frame.shape[:-1] 405 | # presenter.drawGraphs(frame) 406 | for obj in objects: 407 | # Validation bbox of detected object 408 | obj['xmax'] = min(obj['xmax'], origin_im_size[1]) 409 | obj['ymax'] = min(obj['ymax'], origin_im_size[0]) 410 | obj['xmin'] = max(obj['xmin'], 0) 411 | obj['ymin'] = max(obj['ymin'], 0) 412 | color = (min(obj['class_id'] * 12.5, 255), 413 | min(obj['class_id'] * 7, 255), 414 | min(obj['class_id'] * 5, 255)) 415 | det_label = labels_map[obj['class_id']] if labels_map and len(labels_map) >= obj['class_id'] else \ 416 | str(obj['class_id']) 417 | 418 | if args.raw_output_message: 419 | log.info( 420 | "{:^9} | {:10f} | {:4} | {:4} | {:4} | {:4} | {} ".format(det_label, obj['confidence'], 421 | obj['xmin'], obj['ymin'], obj['xmax'], 422 | obj['ymax'], 423 | color)) 424 | 425 | cv2.rectangle(frame, (obj['xmin'], obj['ymin']), (obj['xmax'], obj['ymax']), color, 2) 426 | cv2.putText(frame, 427 | "#" + det_label + ' ' + str(round(obj['confidence'] * 100, 1)) + ' %', 428 | (obj['xmin'], obj['ymin'] - 7), cv2.FONT_HERSHEY_COMPLEX, 0.6, color, 1) 429 | 430 | # Draw performance stats over frame 431 | if mode_info[mode.current].frames_count != 0: 432 | fps_message = "FPS: {:.1f}".format(mode_info[mode.current].frames_count / \ 433 | (perf_counter() - mode_info[mode.current].last_start_time)) 434 | mode_info[mode.current].latency_sum += perf_counter() - start_time 435 | latency_message = "Latency: {:.1f} ms".format((mode_info[mode.current].latency_sum / \ 436 | mode_info[mode.current].frames_count) * 1e3) 437 | 438 | put_highlighted_text(frame, fps_message, (15, 20), cv2.FONT_HERSHEY_COMPLEX, 0.75, (200, 10, 10), 2) 439 | put_highlighted_text(frame, latency_message, (15, 50), cv2.FONT_HERSHEY_COMPLEX, 0.75, (200, 10, 10), 2) 440 | 441 | mode_message = "{} mode".format(mode.current.name) 442 | put_highlighted_text(frame, mode_message, (10, int(origin_im_size[0] - 20)), 443 | cv2.FONT_HERSHEY_COMPLEX, 0.75, (10, 10, 200), 2) 444 | 445 | if not args.no_show: 446 | cv2.imshow("Detection Results", frame) 447 | key = cv2.waitKey(wait_key_time) 448 | 449 | if key in {ord("q"), ord("Q"), 27}: # ESC key 450 | break 451 | if key == 9: # Tab key 452 | prev_mode = mode.current 453 | mode.next() 454 | 455 | await_requests_completion(exec_nets[prev_mode].requests) 456 | empty_requests.clear() 457 | empty_requests.extend(exec_nets[mode.current].requests) 458 | 459 | mode_info[prev_mode].last_end_time = perf_counter() 460 | mode_info[mode.current] = ModeInfo() 461 | # else: 462 | # presenter.handleKey(key) 463 | 464 | elif empty_requests and cap.isOpened(): 465 | start_time = perf_counter() 466 | ret, frame = cap.read() 467 | if not ret: 468 | if args.loop_input: 469 | cap.open(input_stream) 470 | else: 471 | cap.release() 472 | continue 473 | 474 | request = empty_requests.popleft() 475 | 476 | # resize input_frame to network size 477 | in_frame = preprocess_frame(frame, input_height, input_width, nchw_shape, args.keep_aspect_ratio) 478 | 479 | # Start inference 480 | request.set_completion_callback(py_callback=async_callback, 481 | py_data=(request, 482 | next_frame_id, 483 | mode.current, 484 | frame, 485 | start_time, 486 | completed_request_results, 487 | empty_requests, 488 | mode, 489 | event, 490 | callback_exceptions)) 491 | request.async_infer(inputs={input_blob: in_frame}) 492 | next_frame_id += 1 493 | 494 | else: 495 | event.wait() 496 | 497 | if callback_exceptions: 498 | raise callback_exceptions[0] 499 | 500 | for mode_value in mode_info.keys(): 501 | log.info("") 502 | log.info("Mode: {}".format(mode_value.name)) 503 | 504 | end_time = mode_info[mode_value].last_end_time if mode_value in mode_info \ 505 | and mode_info[mode_value].last_end_time is not None \ 506 | else perf_counter() 507 | log.info("FPS: {:.1f}".format(mode_info[mode_value].frames_count / \ 508 | (end_time - mode_info[mode_value].last_start_time))) 509 | log.info("Latency: {:.1f} ms".format((mode_info[mode_value].latency_sum / \ 510 | mode_info[mode_value].frames_count) * 1e3)) 511 | # print(presenter.reportMeans()) 512 | 513 | for exec_net in exec_nets.values(): 514 | await_requests_completion(exec_net.requests) 515 | 516 | 517 | if __name__ == '__main__': 518 | sys.exit(main() or 0) 519 | -------------------------------------------------------------------------------- /report/OpenCV_Spatial_AI_Competition.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/report/OpenCV_Spatial_AI_Competition.pdf -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # depthai dep 2 | opencv-python==4.2.0.34; platform_machine != "armv7l" 3 | opencv-python==4.1.0.25; platform_machine == "armv7l" 4 | concurrent-log-handler 5 | scipy 6 | 7 | # txt2speech 8 | google-speech -------------------------------------------------------------------------------- /scripts/inference_engine_native_myriad.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # A script to build inference engine for ARM arch - Tested on Jetson Nano - 4.3 Jetpack 3 | 4 | # System update and Dep Install 5 | sudo apt update && sudo apt upgrade -y 6 | sudo apt install build-essential 7 | 8 | # Update cmake from source 9 | cd ~/ 10 | wget https://github.com/Kitware/CMake/releases/download/v3.14.4/cmake-3.14.4.tar.gz 11 | tar xvzf cmake-3.14.4.tar.gz 12 | cd ~/cmake-3.14.4 13 | 14 | ## Configure and install 15 | ./bootstrap 16 | make -j4 17 | sudo make install 18 | 19 | # Install OpenCV 20 | cd ~/ 21 | git clone https://github.com/opencv/opencv.git 22 | cd opencv && mkdir build && cd build 23 | cmake –DCMAKE_BUILD_TYPE=Release –DCMAKE_INSTALL_PREFIX=/usr/local .. 24 | make -j4 25 | sudo make install 26 | 27 | # Install OpenVino Inference Engine 28 | cd ~/ 29 | git clone https://github.com/openvinotoolkit/openvino.git 30 | 31 | cd ~/openvino/inference-engine 32 | git submodule update --init --recursive 33 | 34 | ## Dep install 35 | cd ~/openvino 36 | sh ./install_dependencies.sh 37 | 38 | ## Build 39 | export OpenCV_DIR=/usr/local/lib 40 | 41 | cd ~/openvino 42 | mkdir build && cd build 43 | cmake -DCMAKE_BUILD_TYPE=Release \ 44 | -DENABLE_MKL_DNN=OFF \ 45 | -DENABLE_CLDNN=ON \ 46 | -DENABLE_GNA=OFF \ 47 | -DENABLE_SSE42=OFF \ 48 | -DTHREADING=SEQ \ 49 | -DENABLE_SAMPLES=ON \ 50 | -DENABLE_PYTHON=ON \ 51 | -DPYTHON_EXECUTABLE=`which python3.6` \ 52 | -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.6m.so \ 53 | -DPYTHON_INCLUDE_DIR=/usr/include/python3.6 54 | .. 55 | make -j4 56 | 57 | # Update env 58 | echo "export PYTHONPATH=$PYTHONPATH:~/openvino/bin/aarch64/Release/lib/python_api/python3.6/" >> ~/.bashrc 59 | echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/openvino/bin/aarch64/Release/lib/" >> ~/.bashrc -------------------------------------------------------------------------------- /scripts/model_intel.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Use this only when you have issue with Windows download env 3 | # Otherwise use download.py from openvino toolkit 4 | 5 | dModel="./model/" 6 | 7 | # Main starts here 8 | mkdir -p ${dModel} 9 | # pedestrian-detection-adas-0002 10 | pushd ${dModel} 11 | mkdir -p pedestrian-detection-adas-0002 12 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-detection-adas-0002/FP32/pedestrian-detection-adas-0002.xml -P \ 13 | pedestrian-detection-adas-0002 14 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-detection-adas-0002/FP32/pedestrian-detection-adas-0002.bin -P \ 15 | pedestrian-detection-adas-0002 16 | 17 | # pedestrian-detection-adas-binary-0001 18 | mkdir -p pedestrian-detection-adas-binary-0001 19 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-detection-adas-binary-0001/FP32-INT1/pedestrian-detection-adas-binary-0001.xml -P \ 20 | pedestrian-detection-adas-binary-0001 21 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-detection-adas-binary-0001/FP32-INT1/pedestrian-detection-adas-binary-0001.bin -P \ 22 | pedestrian-detection-adas-binary-0001 23 | 24 | # pedestrian-and-vehicle-detector-adas-0001 25 | mkdir -p pedestrian-and-vehicle-detector-adas-0001 26 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-and-vehicle-detector-adas-0001/FP32/pedestrian-and-vehicle-detector-adas-0001.xml -P \ 27 | pedestrian-and-vehicle-detector-adas-0001 28 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-and-vehicle-detector-adas-0001/FP32/pedestrian-and-vehicle-detector-adas-0001.bin -P \ 29 | pedestrian-and-vehicle-detector-adas-0001 30 | 31 | # vehicle-detection-adas-0002 32 | mkdir -p vehicle-detection-adas-0002 33 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/vehicle-detection-adas-0002/FP32/vehicle-detection-adas-0002.xml -P \ 34 | vehicle-detection-adas-0002 35 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/vehicle-detection-adas-0002/FP32/vehicle-detection-adas-0002.bin -P \ 36 | vehicle-detection-adas-0002 37 | 38 | # vehicle-detection-adas-binary-0001 39 | mkdir -p vehicle-detection-adas-binary-0001 40 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/vehicle-detection-adas-binary-0001/FP32-INT1/vehicle-detection-adas-binary-0001.xml -P \ 41 | vehicle-detection-adas-binary-0001 42 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/vehicle-detection-adas-binary-0001/FP32-INT1/vehicle-detection-adas-binary-0001.bin -P \ 43 | vehicle-detection-adas-binary-0001 44 | 45 | # road-segmentation-adas-0001 46 | mkdir -p road-segmentation-adas-0001 47 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/road-segmentation-adas-0001/FP32/road-segmentation-adas-0001.xml -P \ 48 | road-segmentation-adas-0001 49 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/road-segmentation-adas-0001/FP32/road-segmentation-adas-0001.bin -P \ 50 | road-segmentation-adas-0001 51 | 52 | # semantic-segmentation-adas-0001 53 | mkdir -p semantic-segmentation-adas-0001 54 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/semantic-segmentation-adas-0001/FP32/semantic-segmentation-adas-0001.xml -P \ 55 | semantic-segmentation-adas-0001 56 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/semantic-segmentation-adas-0001/FP32/semantic-segmentation-adas-0001.bin -P \ 57 | semantic-segmentation-adas-0001 58 | 59 | pushd -------------------------------------------------------------------------------- /scripts/opencv.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | 5 | # Main start from here 6 | chip_id=$(cat /sys/module/tegra_fuse/parameters/tegra_chip_id) 7 | case ${chip_id} in 8 | "33" ) # Nano and TX1 9 | cuda_compute=5.3 10 | ;; 11 | "24" ) # TX2 12 | cuda_compute=6.2 13 | ;; 14 | "25" ) # AGX Xavier 15 | cuda_compute=7.2 16 | ;; 17 | * ) # default 18 | cuda_compute=5.3,6.2,7.2 19 | ;; 20 | esac 21 | 22 | py3_ver=$(python3 -c "import sys; print(sys.version_info[1])") 23 | 24 | folder=${HOME}/src 25 | mkdir -p $folder 26 | 27 | echo "** Purge old opencv installation" 28 | sudo apt-get purge -y libopencv* 29 | 30 | echo "** Install requirements" 31 | sudo apt-get update 32 | sudo apt-get install -y build-essential make cmake cmake-curses-gui git g++ pkg-config curl 33 | sudo apt-get install -y libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libeigen3-dev libglew-dev libgtk2.0-dev 34 | sudo apt-get install -y libtbb2 libtbb-dev libv4l-dev v4l-utils qv4l2 v4l2ucp 35 | sudo apt-get install -y libdc1394-22-dev libxine2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev 36 | # sudo apt-get install -y libjasper-dev 37 | sudo apt-get install -y libjpeg8-dev libjpeg-turbo8-dev libtiff-dev libpng-dev 38 | sudo apt-get install -y libxvidcore-dev libx264-dev libgtk-3-dev 39 | sudo apt-get install -y libatlas-base-dev libopenblas-dev liblapack-dev liblapacke-dev gfortran 40 | sudo apt-get install -y qt5-default 41 | 42 | sudo apt-get install -y python3-dev python3-testresources 43 | rm -f $folder/get-pip.py 44 | wget https://bootstrap.pypa.io/get-pip.py -O $folder/get-pip.py 45 | sudo python3 $folder/get-pip.py 46 | sudo pip3 install protobuf 47 | sudo pip3 install -U numpy matplotlib 48 | 49 | if [ ! -f /usr/local/cuda/include/cuda_gl_interop.h.bak ]; then 50 | sudo cp /usr/local/cuda/include/cuda_gl_interop.h /usr/local/cuda/include/cuda_gl_interop.h.bak 51 | fi 52 | sudo patch -N -r - /usr/local/cuda/include/cuda_gl_interop.h < opencv/cuda_gl_interop.h.patch && echo "** '/usr/local/cuda/include/cuda_gl_interop.h' appears to be patched already. Continue..." 53 | 54 | echo "** Download opencv-4.1.2" 55 | cd $folder 56 | wget -O opencv.zip https://github.com/opencv/opencv/archive/4.1.2.zip 57 | wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.1.2.zip 58 | unzip opencv.zip 59 | unzip opencv_contrib.zip 60 | mv opencv-4.1.2 opencv 61 | mv opencv_contrib-4.1.2 opencv_contrib 62 | cd opencv 63 | mkdir build 64 | cd build 65 | cmake -D CMAKE_BUILD_TYPE=RELEASE \ 66 | -D CMAKE_INSTALL_PREFIX=/usr/local \ 67 | -D WITH_CUDA=ON \ 68 | -D CUDA_ARCH_PTX="" \ 69 | -D CUDA_ARCH_BIN=${cuda_compute} \ 70 | -D WITH_CUBLAS=ON \ 71 | -D WITH_LIBV4L=ON \ 72 | -D ENABLE_FAST_MATH=ON \ 73 | -D CUDA_FAST_MATH=ON \ 74 | -D EIGEN_INCLUDE_PATH=/usr/include/eigen3 \ 75 | -D BUILD_opencv_python3=ON \ 76 | -D BUILD_opencv_python2=OFF \ 77 | -D BUILD_opencv_java=OFF \ 78 | -D WITH_GSTREAMER=ON \ 79 | -D WITH_GTK=ON \ 80 | -D BUILD_TESTS=OFF \ 81 | -D BUILD_PERF_TESTS=OFF \ 82 | -D BUILD_EXAMPLES=OFF \ 83 | -D OPENCV_ENABLE_NONFREE=ON \ 84 | -D OPENCV_EXTRA_MODULES_PATH=$folder/opencv_contrib/modules .. 85 | make -j3 86 | sudo make install 87 | sudo ldconfig 88 | 89 | pushd ~/depthai_v1/lib/python3.6/site-packages/ 90 | ln -s /usr/local/lib/python3.6/site-packages/cv2/python-3.6/cv2.cpython-36m-aarch64-linux-gnu.so 91 | popd 92 | 93 | python3 -c 'import cv2; print("python3 cv2 version: %s" % cv2.__version__)' -------------------------------------------------------------------------------- /scripts/rpi_openvino_install-2020_1.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # A script to install OpenVino toolchain 2020 1 version 4 | 5 | #Main starts from here 6 | # Downlaod openvino tool chain 7 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.1/l_openvino_toolkit_runtime_raspbian_p_2020.1.023.tgz 8 | 9 | # Create openvino install path 10 | sudo mkdir -p /opt/intel/openvino 11 | 12 | # Unzip the toolchain 13 | sudo tar -xvf l_openvino_toolkit_runtime_raspbian_p_2020.1.023.tgz --strip 1 -C /opt/intel/openvino 14 | 15 | # Install cmake 16 | sudo apt install cmake 17 | 18 | # Export a path 19 | echo "source /opt/intel/openvino/bin/setupvars.sh" >> ~/.bashrc 20 | 21 | # Add use for USB 22 | sudo usermod -a -G users "$(whoami)" 23 | --------------------------------------------------------------------------------