├── .github
└── ISSUE_TEMPLATE
│ ├── bug_report.md
│ └── feature_request.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── android
├── apk
│ └── app-debug.apk
└── startup_linux
│ ├── deepeye.sh
│ └── rfcomm.service
├── custom_model
└── OI_Dataset
│ ├── README.md
│ ├── requirements.txt
│ ├── scripts
│ ├── csv2tfrecord.py
│ ├── txt2xml.py
│ └── xml2csv.py
│ └── tf_test.py
├── deepeye_app
├── app.py
├── calibration
│ ├── config
│ │ └── BW1098FFC.json
│ └── depthai.calib
├── collision_avoidance.py
├── config.py
├── models
│ ├── mobilenet-ssd.blob
│ ├── mobilenet-ssd_depth.json
│ ├── mobilenet-ssd_v1.blob
│ └── mobilenet-ssd_v1.json
├── tracker.py
└── txt2speech
│ ├── README.md
│ ├── s_images
│ ├── txt2s.png
│ └── txt2s_bt.png
│ ├── sw_design
│ ├── txt2s_BT
│ └── txt2speech
│ ├── txt-simulator.py
│ └── txt2speech.py
├── images
├── BW1098FFC_R0M0E0_diag.png
├── BW1098FFC_R0M0E0_dims.png
├── BW1098FFC_R0M0E0_front.png
├── HLD.png
├── HLD_1.JPG
├── demo.PNG
├── hw.jpeg
└── jetson-nano-dev-kit-top-r6-HR-B01.png
├── openvino_analysis
├── README.md
├── intel
│ ├── object-detection
│ │ ├── pedestrian-and-vehicle-detector-adas-0001
│ │ │ ├── __pycache__
│ │ │ │ └── inference.cpython-36.pyc
│ │ │ ├── inference.py
│ │ │ └── main.py
│ │ ├── pedestrian-detection-adas-0002
│ │ │ ├── __pycache__
│ │ │ │ └── inference.cpython-36.pyc
│ │ │ ├── inference.py
│ │ │ └── main.py
│ │ ├── pedestrian-detection-adas-binary-0001
│ │ │ ├── inference.py
│ │ │ └── main.py
│ │ └── vehicle-detection-adas
│ │ │ ├── inference.py
│ │ │ └── main.py
│ └── semantic-segmentation
│ │ ├── road-segmentation-adas-0001
│ │ ├── __pycache__
│ │ │ └── inference.cpython-36.pyc
│ │ ├── inference.py
│ │ └── main.py
│ │ └── semantic-segmentation-adas-0001
│ │ ├── async
│ │ ├── 009649.png
│ │ ├── __pycache__
│ │ │ └── inference.cpython-36.pyc
│ │ ├── inference.py
│ │ └── main.py
│ │ ├── core
│ │ ├── __pycache__
│ │ │ └── semantic_detection.cpython-36.pyc
│ │ └── semantic_detection.py
│ │ ├── main.py
│ │ └── utils
│ │ ├── __pycache__
│ │ └── ie_module.cpython-36.pyc
│ │ └── ie_module.py
└── public
│ ├── ssd_mobilenet_v2_coco
│ ├── __pycache__
│ │ └── inference.cpython-36.pyc
│ ├── async_multithread
│ │ ├── __pycache__
│ │ │ └── inference.cpython-36.pyc
│ │ ├── example.log
│ │ ├── inference.py
│ │ └── main.py
│ ├── inference.py
│ └── main.py
│ └── yolo-v3
│ └── main.py
├── report
└── OpenCV_Spatial_AI_Competition.pdf
├── requirements.txt
└── scripts
├── inference_engine_native_myriad.sh
├── model_intel.sh
├── opencv.sh
└── rpi_openvino_install-2020_1.sh
/.github/ISSUE_TEMPLATE/bug_report.md:
--------------------------------------------------------------------------------
1 | ---
2 | name: Bug report
3 | about: Create a report to help us improve
4 | title: ''
5 | labels: ''
6 | assignees: ''
7 |
8 | ---
9 |
10 | **Describe the bug**
11 | A clear and concise description of what the bug is.
12 |
13 | **To Reproduce**
14 | Steps to reproduce the behavior:
15 | 1. Go to '...'
16 | 2. Click on '....'
17 | 3. Scroll down to '....'
18 | 4. See error
19 |
20 | **Expected behavior**
21 | A clear and concise description of what you expected to happen.
22 |
23 | **Screenshots**
24 | If applicable, add screenshots to help explain your problem.
25 |
26 | **Desktop (please complete the following information):**
27 | - OS: [e.g. iOS]
28 | - Browser [e.g. chrome, safari]
29 | - Version [e.g. 22]
30 |
31 | **Smartphone (please complete the following information):**
32 | - Device: [e.g. iPhone6]
33 | - OS: [e.g. iOS8.1]
34 | - Browser [e.g. stock browser, safari]
35 | - Version [e.g. 22]
36 |
37 | **Additional context**
38 | Add any other context about the problem here.
39 |
--------------------------------------------------------------------------------
/.github/ISSUE_TEMPLATE/feature_request.md:
--------------------------------------------------------------------------------
1 | ---
2 | name: Feature request
3 | about: Suggest an idea for this project
4 | title: ''
5 | labels: ''
6 | assignees: ''
7 |
8 | ---
9 |
10 | **Is your feature request related to a problem? Please describe.**
11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
12 |
13 | **Describe the solution you'd like**
14 | A clear and concise description of what you want to happen.
15 |
16 | **Describe alternatives you've considered**
17 | A clear and concise description of any alternative solutions or features you've considered.
18 |
19 | **Additional context**
20 | Add any other context or screenshots about the feature request here.
21 |
--------------------------------------------------------------------------------
/CONTRIBUTING.md:
--------------------------------------------------------------------------------
1 | We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:
2 |
3 | Reporting a bug
4 | Discussing the current state of the code
5 | Submitting a fix
6 | Proposing new features
7 | Becoming a maintainer
8 | We Develop with Github
9 | We use github to host code, to track issues and feature requests, as well as accept pull requests.
10 |
11 | We Use Github Flow, So All Code Changes Happen Through Pull Requests
12 | Pull requests are the best way to propose changes to the codebase (we use Github Flow). We actively welcome your pull requests:
13 |
14 | Fork the repo and create your branch from master.
15 | If you've added code that should be tested, add tests.
16 | If you've changed APIs, update the documentation.
17 | Ensure the test suite passes.
18 | Make sure your code lints.
19 | Issue that pull request!
20 | Any contributions you make will be under the MIT Software License
21 | In short, when you submit code changes, your submissions are understood to be under the same MIT License that covers the project. Feel free to contact the maintainers if that's a concern.
22 |
23 | Report bugs using Github's issues
24 | We use GitHub issues to track public bugs. Report a bug by opening a new issue; it's that easy!
25 |
26 | Write bug reports with detail, background, and sample code
27 | This is an example of a bug report I wrote, and I think it's not a bad model. Here's another example from Craig Hockenberry, an app developer whom I greatly respect.
28 |
29 | Great Bug Reports tend to have:
30 |
31 | A quick summary and/or background
32 | Steps to reproduce
33 | Be specific!
34 | Give sample code if you can. My stackoverflow question includes sample code that anyone with a base R setup can run to reproduce what I was seeing
35 | What you expected would happen
36 | What actually happens
37 | Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
38 | People love thorough bug reports. I'm not even kidding.
39 |
40 | Use a Consistent Coding Style
41 | I'm again borrowing these from Facebook's Guidelines
42 |
43 | 2 spaces for indentation rather than tabs
44 | You can try running npm run lint for style unification
45 | License
46 | By contributing, you agree that your contributions will be licensed under its MIT License.
47 |
48 | References
49 | This document was adapted from the open-source contribution guidelines for Facebook's Draft
50 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2020 Jegathesan Shanmugam
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # deepEye - The third eye for Visually Impaired People
2 |
3 | [OpenCV](https://opencv.org/) announced its first Spatial AI Competition sponsored by Intel. As we know, OpenCV is a famous open-source computer vision library. They called for participants to solve real-world problems by using [OAK-D](https://www.kickstarter.com/projects/opencv/opencv-ai-kit) (OpenCV AI Kit with Depth) module. The OAK-D module has built-in Stereo cameras along with an RGB camera. It also has powerful visual processing unit (Myriad X from Intel) to enable deep neural network inferences on board.
4 |
5 | We decided to submit a project proposal for this competition back in July. Our group’s proposal was selected (among 32 out of 235).
6 |
7 | So, we propose to build an advanced assist system for the Visually Impaired People to perceive the environment in a better way and would provide seamless, reliable navigation for them at a low cost so that anyone can leverage the benefits of computer vision.
8 |
9 | ## Demo Videos
10 |
11 | | 👉 [deepEye Demo](https://www.youtube.com/watch?v=llE8l1IWpYA&feature=youtu.be) |
12 | | ------------------------------------------------------------ |
13 | | [](https://www.youtube.com/watch?v=llE8l1IWpYA&feature=youtu.be) |
14 |
15 | ## Table of content
16 |
17 | - [deepEye - The third eye for Visually Impaired People](#deepEye-the-third-eye-for-visually-impaired-people)
18 | - [Demo Videos](#demo-videos)
19 | - [Table of content](#table-of-content)
20 | - [🎬 Software High Level Design](#software-high-level-design)
21 | - [🗃Project structure](#project-structure)
22 | - [💻 Hardware pre-requisite](#-hardware-pre-requisite)
23 | - [📦 Software pre-requisite](#1-software-pre-requisite-)
24 | - [For Jetson: Flash Jetson board to jetpack 4.4 ⚡️](#for-jetson-flash-jetson-board-to-jetpack-44-️)
25 | - [Depth AI Install](#depth-ai-install)
26 | - [Camera Calibration ](#camera-calibration)
27 | - [Robotic Operating System](#robotic-operating-system)
28 | - [Android RFCOMM Setup](#android-rfcomm-setup)
29 | - [Other Dependency](#other-dependency)
30 | - [🖖 Quick Start](#quick-start)
31 | - [🎛 Advanced uses](#-advanced-uses)
32 | - [Custom Object Detector](#custom-object-detector)
33 | - [🛠 Hardware Details](#hardware-details)
34 | - [💌 Acknowledgments](#-acknowledgments)
35 |
36 | ## 🎬 Software High Level Design
37 |
38 | 
39 |
40 |
41 | ## 🗃 Project structure
42 |
43 | ```python
44 | .
45 | ├── android
46 | │ ├── apk # Android APK File
47 | │ │ └── app-debug.apk
48 | │ └── startup_linux
49 | │ ├── deepeye.sh # deepeye startup script to enable RFCOMM
50 | │ └── rfcomm.service # systemd service for RFCOMM
51 | |
52 | ├── custom_model
53 | │ └── OI_Dataset # Mobile Net SSD V2 Custom training on OpenImage Dataset V4
54 | │ ├── README.md
55 | │ ├── requirements.txt
56 | │ ├── scripts
57 | │ │ ├── csv2tfrecord.py # Tensorflow: CSV to TFrecord Converter
58 | │ │ ├── txt2xml.py # Tensorflow: TXT to XML Converter
59 | │ │ └── xml2csv.py # Tensorflow: XML to CSV Converter
60 | │ └── tf_test.py # Test script for Trained model inference
61 | |
62 | ├── deepeye_app # Deepeye core application
63 | │ ├── app.py # Object detection and post processing
64 | │ ├── calibration # Camera Callibration
65 | │ │ └── config
66 | │ │ └── BW1098FFC.json
67 | │ ├── collision_avoidance.py # Collision calculation
68 | │ ├── config.py
69 | │ ├── models # Mobilenet-ssd v2 trained model
70 | │ │ ├── mobilenet-ssd.blob
71 | │ │ └── mobilenet-ssd_depth.json
72 | │ ├── tracker.py # Object tracker
73 | │ └── txt2speech # txt2speech model
74 | │ ├── README.md
75 | │ ├── txt2speech.py
76 | │ └── txt-simulator.py
77 | ├── images
78 | ├── openvino_analysis # CNN model fom Intel and Opensouce ACC, FPS analysis
79 | │ ├── intel
80 | │ │ ├── object-detection
81 | │ │ └── semantic-segmentation
82 | │ ├── public
83 | │ │ ├── ssd_mobilenet_v2_coco
84 | │ │ └── yolo-v3
85 | │ └── README.md
86 | ├── README.md # Deepeye README
87 | ├── requirements.txt
88 | └── scripts # OpenVino Toolkit scripts
89 | ├── inference_engine_native_myriad.sh
90 | ├── model_intel.sh
91 | └── rpi_openvino_install-2020_1.sh
92 | ```
93 |
94 | ## 💻 Hardware pre-requisite
95 | * [Jetson Nano](https://developer.nvidia.com/embedded/jetson-nano-developer-kit)
96 | * [BW1098FFC depthAI HW](https://docs.luxonis.com/products/bw1098ffc/)
97 | * Smartphone with Android OS
98 |
99 | ## 📦 Software pre-requisite
100 |
101 | ### For Jetson: Flash Jetson board to jetpack 4.4 ⚡️
102 |
103 | microSD card Prepration:
104 | 1. Download Jetson Nano Developer Kit SD Card image [Jetpack4.4 Image](https://developer.nvidia.com/jetson-nano-sd-card-image).
105 | 2. Use [etcher](https://www.balena.io/etcher) to burn a image.
106 |
107 | CUDA Env PATH :
108 | ```bash
109 | if ! grep 'cuda/bin' ${HOME}/.bashrc > /dev/null ; then
110 | echo "** Add CUDA stuffs into ~/.bashrc"
111 | echo >> ${HOME}/.bashrc
112 | echo "export PATH=/usr/local/cuda/bin:\${PATH}" >> ${HOME}/.bashrc
113 | echo "export LD_LIBRARY_PATH=/usr/local/cuda/lib64:\${LD_LIBRARY_PATH}" >> ${HOME}/.bashrc
114 | fi
115 | source ${HOME}/.bashrc
116 | ```
117 |
118 | System dependencies :
119 | ```bash
120 | sudo apt-get update
121 | sudo apt-get install -y build-essential make cmake cmake-curses-gui
122 | sudo apt-get install -y git g++ pkg-config curl libfreetype6-dev
123 | sudo apt-get install -y libcanberra-gtk-module libcanberra-gtk3-module
124 | sudo apt-get install -y python3-dev python3-testresources python3-pip
125 | sudo pip3 install -U pip
126 | ```
127 |
128 | Performance Improvements:
129 |
130 | To set Jetson Nano to 10W performance mode (reference), execute the following from a terminal:
131 | ```bash
132 | sudo nvpmodel -m 0
133 | sudo jetson_clocks
134 | ```
135 | Enable swap:
136 | ```bash
137 | sudo fallocate -l 8G /mnt/8GB.swap
138 | sudo mkswap /mnt/8GB.swap
139 | sudo swapon /mnt/8GB.swap
140 | if ! grep swap /etc/fstab > /dev/null; then \
141 | echo "/mnt/8GB.swap none swap sw 0 0" | sudo tee -a /etc/fstab; \
142 | fi
143 | ```
144 | jetson performance analysis:
145 | ```bash
146 | pip3 install jetson-stats
147 | ```
148 |
149 | Recompile a Jetson Linux kernel - Support RFCOMM TTY Support:
150 |
151 | We are using RFCOMM Serial protocol for Jetson-Android communication and the defauly kernel doesn't have a support for RFCOMM TTY. So, We have to recompile with new kernel config and update.
152 |
153 | ```bash
154 | # Basic Update
155 | sudo apt-get update
156 | sudo apt-get install -y libncurses5-dev
157 |
158 | # Downlaod Linux L4T(BSP) Source code from Nvidia Downlaod center
159 | wget https://developer.nvidia.com/embedded/L4T/r32_Release_v4.3/Sources/T210/public_sources.tbz2
160 |
161 | tar -xvf public_sources.tbz2
162 |
163 | cp Linux_for_Tegra/source/public/kernel_src.tbz2 ~/
164 |
165 | pushd ~/
166 |
167 | tar -xvf kernel_src.tbz2
168 |
169 | pushd ~/kernel/kernel-4.9
170 |
171 | zcat /proc/config.gz > .config
172 |
173 | # Enable RFCOMM TTY
174 | make menuconfig # Networking Support --> Bluetooth subsystem support ---> Select RFCOMM TTY Support ---> Save ---> Exit
175 |
176 | make prepare
177 |
178 | make modules_prepare
179 |
180 | # Compile kernel as an image file
181 | make -j5 Image
182 |
183 | # Compile all kernel modules
184 | make -j5 modules
185 |
186 | # Install modules and kernel image
187 | cd ~/kernel/kernel-4.9
188 | sudo make modules_install
189 | sudo cp arch/arm64/boot/Image /boot/Image
190 |
191 | # Reboot
192 | sudo reboot
193 | ```
194 |
195 | ### Depth AI Python Interface Install
196 | ```bash
197 |
198 | # Install dep
199 | curl -fL http://docs.luxonis.com/install_dependencies.sh | bash
200 | sudo apt install libusb-1.0-0-dev
201 |
202 | # USB Udev
203 | echo 'SUBSYSTEM=="usb", ATTRS{idVendor}=="03e7", MODE="0666"' | sudo tee /etc/udev/rules.d/80-movidius.rules
204 | sudo udevadm control --reload-rules && sudo udevadm trigger
205 |
206 | git clone https://github.com/luxonis/depthai-python.git
207 | cd depthai-python
208 | git submodule update --init --recursive
209 | mkdir -p ~/depthai_v1
210 | python3 -m venv ~/depthai_v1
211 | python3 -m pip install -U pip
212 | python3 setup.py develop
213 |
214 | # Check the Installation
215 | python3 -c "import depthai"
216 |
217 | # Install opencv
218 | cd scripts
219 | bash opencv.sh
220 | cd ..
221 | ```
222 |
223 | ### Camera Calibration
224 | ```bash
225 | mkdir -p ~/depthai/ && pushd ~/depthai/
226 | git clone https://github.com/luxonis/depthai.git
227 | popd
228 | cp calibration/config/BW1098FFC.json depthAI/depthai/resources/boards/
229 | pushd ~/depthai/
230 | python3 calibrate.py -s 2 -brd BW1098FFC -ih
231 | ```
232 |
233 | ### Robotic Operating System
234 | We use ROS framework multiprocess communication.
235 | ```bash
236 | sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
237 | sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
238 |
239 | sudo apt update
240 |
241 | sudo apt install -y ros-melodic-ros-base
242 |
243 | # Env setup
244 | echo "source /opt/ros/melodic/setup.bash" >> ~/.bashrc
245 | source ~/.bashrc
246 |
247 | # Dep to build ROS Package
248 | sudo apt install python-rosdep python-rosinstall python-rosinstall-generator python-wstool build-essential
249 |
250 | # Install inside virtual env
251 | sudo apt install python-rosdep
252 | rosdep init
253 |
254 | rosdep update
255 | ```
256 |
257 | ### Android RFCOMM Setup
258 | We need to configure rfcomm service in order to use the Android application for text to speech feature.
259 |
260 | ```bash
261 | sudo cp android/startup_linux/deepeye.sh /usr/bin/
262 | sudo chmod a+x /usr/bin/deepeye.sh
263 |
264 | sudo cp android/startup_linux/rfcomm.service /etc/systemd/system/
265 | sudo systemctl enable rfcomm
266 | ```
267 |
268 | ### Other Dependency
269 | ```bash
270 | python3 -m pip install -r requirements.txt
271 |
272 | # SOX for txt 2 speech
273 | sudo apt-get install sox libsox-fmt-mp3
274 |
275 | ```
276 |
277 | ## 🖖 Quick Start
278 | ```bash
279 | # Terminal one
280 | # ROS Master
281 | roscore &
282 |
283 | # Terminal Two
284 | # Deepeye core app
285 | pushd deepeye_app
286 | python3 app.py
287 | popd
288 |
289 | # Terminal three
290 | # Txt2speech middleware component
291 | pushd deepeye_app
292 | python3 txt2speech/txt2speech.py
293 | popd
294 | ```
295 |
296 | ## 🎛 Advanced uses
297 |
298 | ### Custom Object Detector
299 | We have retrained an SSD MobileNet SSD-V2 with Open Image dataset. We picked up and trained all the object classes that help visually impaired people to navigate when they go to outdoor environments.
300 |
301 | We have added [README](https://github.com/nullbyte91/vision-system/tree/master/custom_model/OI_Dataset) for the end to end training and the OpenVino Conversion before loading to depth AI.
302 |
303 | ### 🛠 Hardware Details
304 | We plan use the DepthAI USB3 Modular Cameras[BW1098FFC] for POC. We are using RPI and Jeston. The AI/vision processing is done on the depthAI based on Myriad X Arch.
305 |
306 | 
307 |
308 |
309 | Key Features of the device:
310 |
311 | * 2 BG0250TG mono camera module interfaces
312 | * 1 BG0249 RGB camera module interface
313 | * 5V power input via barrel jack
314 | * USB 3.1 Gen 1 Type-C
315 | * Pads for DepthAI SoM 1.8V SPI
316 | * Pads for DepthAI SoM 3.3V SDIO
317 | * Pads for DepthAI SoM 1.8V Aux Signals (I2C, UART, GPIO)
318 | * 5V Fan/Aux header
319 | * Pads for DepthAI SoM aux signals
320 | * Design files produced with Altium Designer 20
321 |
322 | ## 💌 Acknowledgments
323 | [DepthaAI Home Page](https://luxonis.com/depthai)
324 | [depthaAI core development](https://github.com/luxonis)
325 | [OpenVino toolkit development](https://github.com/openvinotoolkit/openvino)
326 | [BW1098FFC_DepthAI_USB3 HW](https://github.com/luxonis/depthai-hardware/tree/master/BW1098FFC_DepthAI_USB3)
327 | [OIDv4 ToolKit](https://github.com/EscVM/OIDv4_ToolKit)
328 |
329 |
330 |
331 |
332 |
333 |
--------------------------------------------------------------------------------
/android/apk/app-debug.apk:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/android/apk/app-debug.apk
--------------------------------------------------------------------------------
/android/startup_linux/deepeye.sh:
--------------------------------------------------------------------------------
1 | #! /bin/bash
2 |
3 | # Make BT Disoverable
4 | #hciconfig hci0 piscan
5 |
6 | # listen to RFCOMM Port 3
7 | rfcomm listen /dev/rfcomm0 3 &
8 |
9 | #chmod 777 /dev/rfcomm0
10 |
--------------------------------------------------------------------------------
/android/startup_linux/rfcomm.service:
--------------------------------------------------------------------------------
1 | [Unit]
2 | Description=rfcomm service
3 |
4 | [Service]
5 | Type=forking
6 | ExecStart=/bin/bash /usr/bin/deepeye.sh start
7 | KillMode=process
8 |
9 | [Install]
10 | WantedBy=multi-user.target
11 |
--------------------------------------------------------------------------------
/custom_model/OI_Dataset/README.md:
--------------------------------------------------------------------------------
1 | # Custom Object Detector
2 |
3 | ## Dataset Download
4 | We are using OIDv4_ToolKit to download a Open Image Dataset.
5 |
6 | ```bash
7 | # Download a kit
8 | git clone https://github.com/EscVM/OIDv4_ToolKit.git
9 |
10 | cd OIDv4_ToolKit
11 |
12 | # Install dep
13 | pip3 install -r requirements.txt
14 |
15 | # Downlaod a dataset
16 | python3 main.py downloader --classes car --type_csv train
17 | python3 main.py downloader --classes car --type_csv test
18 | python3 main.py downloader --classes car --type_csv validation
19 | ```
20 |
21 | ## Setup Object Detection Model
22 | ```bash
23 | python3 -m venv ODM
24 | source ODM/bin/activate
25 | pip3 install -r requirements.txt
26 |
27 | # Object Detection Models
28 | git clone --quiet https://github.com/tensorflow/models.git
29 | cd models
30 | git checkout 58d19c67e1d30d905dd5c6e5092348658fed80af
31 | cd research/
32 | protoc object_detection/protos/*.proto --python_out=.
33 | export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
34 | ```
35 |
36 | ## TFRecord Preparation
37 | ```bash
38 | # Convert txt to xml
39 | python3 txt2xml.py
40 |
41 | # Convert xml to csv
42 | python3 xml2csv.py
43 |
44 | # Convert csv to tfrecord
45 | python3 csv2tfrecord.py
46 | ```
47 |
48 | ## Training
49 | ```bash
50 | # Modify the ssd_mobilenet_v2_coco.config
51 | python3 model_main.py --pipeline_config_path=models/research/object_detection/samples/configs/ssd_mobilenet_v2_coco.config -num_train_steps=50 --num_eval_steps=10 --model_dir train_log
52 | ```
53 |
54 | ## Export a Trained Inference Graph
55 | ```bash
56 | python3 export_inference_graph.py --input_type=image_tensor --pipeline_config_path=./models/research/object_detection/samples/configs/ssd_mobilenet_v2_coco.config --output_directory=./train_log --trained_checkpoint_prefix=model.ckpt-50.meta
57 | ```
58 |
59 | ## Running Inference: Checking what the trained model can detect
60 | ```bash
61 | python3 tf_test.py
62 | ```
63 |
64 | ## Tensorflow to Openvino model conversion
65 | ```bash
66 | sudo apt-get update && sudo apt-get install -y pciutils cpio
67 |
68 | # Download OpenVino and setup
69 | wget http://registrationcenter-download.intel.com/akdlm/irc_nas/16345/l_openvino_toolkit_p_2020.1.023.tgz
70 |
71 | chmod a+x install_GUI.sh
72 | sudo bash install_GUI.sh
73 | ```
74 |
75 | OpenVino Model conversion:
76 | ```bash
77 | source /opt/intel/openvino/bin/setupvars.sh
78 | ```
79 |
80 | SSD Changes:
81 | ```python
82 | with open('ssd_v2_support.json', 'r') as file :
83 | filedata = file.read()
84 |
85 | # Replace the target string
86 | filedata = filedata.replace('"Postprocessor/ToFloat"', '"Postprocessor/Cast_1"')
87 |
88 | # Write the file out again
89 | with open('ssd_v2_support.json', 'w') as file:
90 | file.write(filedata)
91 | ```
92 |
93 | ```bash
94 | python /opt/intel/openvino/deployment_tools/model_optimizer/mo.py \
95 | --input_model ./fine_tune_model/frozen_inference_graph.pb \
96 | --tensorflow_use_custom_operations_config /opt/intel/openvino_2020.1.023/deployment_tools/model_optimizer/extensions/front/tf/ssd_v2_support.json \
97 | --tensorflow_object_detection_api_pipeline_config ./fine_tune_model/pipeline.config \
98 | --reverse_input_channels \
99 | --data_type FP16 \
100 | --output_dir ./openvino_out
101 | ```
102 |
103 | OpenVino to depthAI blob conversion:
104 | ```bash
105 | python3 openvino_depthai-blob.py -x ~/Desktop/OI_dataset/l_openvino_toolkit_p_2020.1.023/output/frozen_inference_graph.xml -b ~/Desktop/OI_dataset/l_openvino_toolkit_p_2020.1.023/output/frozen_inference_graph.bin -o output
106 | ```
107 |
108 |
109 |
110 |
--------------------------------------------------------------------------------
/custom_model/OI_Dataset/requirements.txt:
--------------------------------------------------------------------------------
1 | tensorflow-gpu==1.15.4
2 | pandas
3 | tqdm
4 | opencv-python
5 | lxml
6 | tf_slim
7 | scipy
8 | pycocotools
9 |
--------------------------------------------------------------------------------
/custom_model/OI_Dataset/scripts/csv2tfrecord.py:
--------------------------------------------------------------------------------
1 | """
2 | Usage:
3 | # From tensorflow/models/
4 | # Create train data:
5 | python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=train.record
6 | # Create test data:
7 | python generate_tfrecord.py --csv_input=data/test_labels.csv --output_path=test.record
8 | """
9 | from __future__ import division
10 | from __future__ import print_function
11 | from __future__ import absolute_import
12 |
13 | import os
14 | import io
15 | import pandas as pd
16 | import tensorflow as tf
17 |
18 | from PIL import Image
19 | import sys
20 | sys.path.append('../')
21 | from object_detection.utils import dataset_util
22 | from collections import namedtuple, OrderedDict
23 |
24 | flags = tf.app.flags
25 | flags.DEFINE_string('csv_input', '', 'data/train_labels.csv')
26 | flags.DEFINE_string('output_path', '', 'data/train.record')
27 | flags.DEFINE_string('image_dir', '', 'images/train')
28 | FLAGS = flags.FLAGS
29 |
30 |
31 | # TO-DO replace this with label map
32 | def class_text_to_int(row_label):
33 | if row_label == 'Label1':
34 | return 1
35 | if row_label == 'Label2':
36 | return 2
37 | else:
38 | return 0
39 |
40 |
41 | def split(df, group):
42 | data = namedtuple('data', ['filename', 'object'])
43 | gb = df.groupby(group)
44 | return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]
45 |
46 |
47 | def create_tf_example(group, path):
48 | with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid:
49 | encoded_jpg = fid.read()
50 | encoded_jpg_io = io.BytesIO(encoded_jpg)
51 | image = Image.open(encoded_jpg_io)
52 | width, height = image.size
53 |
54 | filename = group.filename.encode('utf8')
55 | image_format = b'jpg'
56 | xmins = []
57 | xmaxs = []
58 | ymins = []
59 | ymaxs = []
60 | classes_text = []
61 | classes = []
62 |
63 | for index, row in group.object.iterrows():
64 | xmins.append(row['xmin'] / width)
65 | xmaxs.append(row['xmax'] / width)
66 | ymins.append(row['ymin'] / height)
67 | ymaxs.append(row['ymax'] / height)
68 | classes_text.append(row['class'].encode('utf8'))
69 | classes.append(class_text_to_int(row['class']))
70 |
71 | tf_example = tf.train.Example(features=tf.train.Features(feature={
72 | 'image/height': dataset_util.int64_feature(height),
73 | 'image/width': dataset_util.int64_feature(width),
74 | 'image/filename': dataset_util.bytes_feature(filename),
75 | 'image/source_id': dataset_util.bytes_feature(filename),
76 | 'image/encoded': dataset_util.bytes_feature(encoded_jpg),
77 | 'image/format': dataset_util.bytes_feature(image_format),
78 | 'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
79 | 'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
80 | 'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
81 | 'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
82 | 'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
83 | 'image/object/class/label': dataset_util.int64_list_feature(classes),
84 | }))
85 | return tf_example
86 |
87 |
88 | def main(_):
89 | writer = tf.python_io.TFRecordWriter(FLAGS.output_path)
90 | path = os.path.join(FLAGS.image_dir)
91 | examples = pd.read_csv(FLAGS.csv_input)
92 | grouped = split(examples, 'filename')
93 | for group in grouped:
94 | tf_example = create_tf_example(group, path)
95 | writer.write(tf_example.SerializeToString())
96 |
97 | writer.close()
98 | output_path = os.path.join(os.getcwd(), FLAGS.output_path)
99 | print('Successfully created the TFRecords: {}'.format(output_path))
100 |
101 |
102 | if __name__ == '__main__':
103 | tf.app.run()
--------------------------------------------------------------------------------
/custom_model/OI_Dataset/scripts/txt2xml.py:
--------------------------------------------------------------------------------
1 | import os
2 | from tqdm import tqdm
3 | from sys import exit
4 | import argparse
5 | import cv2
6 | from textwrap import dedent
7 | from lxml import etree
8 | from argparse import ArgumentParser
9 |
10 | XML_DIR = 'To_PASCAL_XML'
11 |
12 | def build_argparser():
13 | """
14 | Parse command line arguments.
15 | :return: command line arguments
16 | """
17 | parser = ArgumentParser()
18 | parser.add_argument("-i", "--input", required=True, type=str,
19 | help="Dataset path")
20 | return parser
21 |
22 | args = build_argparser().parse_args()
23 | root = os.listdir(args.input)
24 |
25 | for dir in root:
26 | sub_dir = args.input + dir
27 | if os.path.isdir(sub_dir):
28 | os.chdir(sub_dir)
29 | class_dirs = os.listdir(os.getcwd())
30 | for class_dir in class_dirs:
31 | if os.path.isdir(class_dir):
32 | os.chdir(class_dir)
33 |
34 | if not os.path.exists(XML_DIR):
35 | os.makedirs(XML_DIR)
36 |
37 | for filename in tqdm(os.listdir(os.getcwd())):
38 | if filename.endswith(".txt"):
39 | filename_str = str.split(filename, ".")[0]
40 |
41 | annotation = etree.Element("annotation")
42 |
43 | #os.chdir("..")
44 | folder = etree.Element("folder")
45 | folder.text = os.path.basename(os.getcwd())
46 | annotation.append(folder)
47 |
48 | filename_xml = etree.Element("filename")
49 | filename_xml.text = filename_str + ".jpg"
50 | annotation.append(filename_xml)
51 |
52 | path = etree.Element("path")
53 | path.text = os.path.join(os.path.dirname(os.path.abspath(filename)), filename_str + ".jpg")
54 | annotation.append(path)
55 |
56 | source = etree.Element("source")
57 | annotation.append(source)
58 |
59 | database = etree.Element("database")
60 | database.text = "Unknown"
61 | source.append(database)
62 |
63 | size = etree.Element("size")
64 | annotation.append(size)
65 |
66 | width = etree.Element("width")
67 | height = etree.Element("height")
68 | depth = etree.Element("depth")
69 |
70 | img = cv2.imread(filename_xml.text)
71 |
72 | width.text = str(img.shape[1])
73 | height.text = str(img.shape[0])
74 | depth.text = str(img.shape[2])
75 |
76 | size.append(width)
77 | size.append(height)
78 | size.append(depth)
79 |
80 | segmented = etree.Element("segmented")
81 | segmented.text = "0"
82 | annotation.append(segmented)
83 |
84 | label_original = open(filename, 'r')
85 |
86 | # Labels from OIDv4 Toolkit: name_of_class X_min Y_min X_max Y_max
87 | for line in label_original:
88 | line = line.strip()
89 | l = line.split(' ')
90 | len_str = len(l)
91 | if len_str == 6:
92 | class_name = l[0]+ "_" + l[1]
93 | xmin_l = str(int(float(l[2])))
94 | ymin_l = str(int(float(l[3])))
95 | xmax_l = str(int(float(l[4])))
96 | ymax_l = str(int(float(l[5])))
97 | else:
98 | class_name = l[0]
99 | xmin_l = str(int(float(l[1])))
100 | ymin_l = str(int(float(l[2])))
101 | xmax_l = str(int(float(l[3])))
102 | ymax_l = str(int(float(l[4])))
103 |
104 | obj = etree.Element("object")
105 | annotation.append(obj)
106 |
107 | name = etree.Element("name")
108 | name.text = class_name
109 | obj.append(name)
110 |
111 | pose = etree.Element("pose")
112 | pose.text = "Unspecified"
113 | obj.append(pose)
114 |
115 | truncated = etree.Element("truncated")
116 | truncated.text = "0"
117 | obj.append(truncated)
118 |
119 | difficult = etree.Element("difficult")
120 | difficult.text = "0"
121 | obj.append(difficult)
122 |
123 | bndbox = etree.Element("bndbox")
124 | obj.append(bndbox)
125 |
126 | xmin = etree.Element("xmin")
127 | xmin.text = xmin_l
128 | bndbox.append(xmin)
129 |
130 | ymin = etree.Element("ymin")
131 | ymin.text = ymin_l
132 | bndbox.append(ymin)
133 |
134 | xmax = etree.Element("xmax")
135 | xmax.text = xmax_l
136 | bndbox.append(xmax)
137 |
138 | ymax = etree.Element("ymax")
139 | ymax.text = ymax_l
140 | bndbox.append(ymax)
141 |
142 | os.chdir(XML_DIR)
143 |
144 | # write xml to file
145 | s = etree.tostring(annotation, pretty_print=True)
146 | with open(filename_str + ".xml", 'wb') as f:
147 | f.write(s)
148 | f.close()
149 |
150 | os.chdir("..")
151 |
152 | os.chdir("..")
153 | os.chdir("..")
154 |
--------------------------------------------------------------------------------
/custom_model/OI_Dataset/scripts/xml2csv.py:
--------------------------------------------------------------------------------
1 | import os
2 | import glob
3 | import pandas as pd
4 | import xml.etree.ElementTree as ET
5 | from argparse import ArgumentParser
6 |
7 | dataset = " "
8 |
9 | def build_argparser():
10 | """
11 | Parse command line arguments.
12 | :return: command line arguments
13 | """
14 | parser = ArgumentParser()
15 | parser.add_argument("-i", "--input", required=True, type=str,
16 | help="Dataset path")
17 | return parser
18 |
19 |
20 | def xml_to_csv(path):
21 | xml_list = []
22 | classes_names = []
23 | os.chdir(path)
24 | CLASS_DIRS = os.listdir(os.getcwd())
25 |
26 | print(CLASS_DIRS)
27 | for CLASS_DIR in CLASS_DIRS:
28 | if os.path.isdir(CLASS_DIR):
29 | os.chdir(CLASS_DIR)
30 | print("Currently in Subdirectory:", os.path.join(os.getcwd()))
31 | path = os.path.join(os.getcwd()) + "/To_PASCAL_XML"
32 | print(path)
33 | for xml_file in glob.glob(path + '/*.xml'):
34 | #print(xml_file)
35 | tree = ET.parse(xml_file)
36 | #print(tree)
37 | root = tree.getroot()
38 | for member in root.findall('object'):
39 | classes_names.append(member[0].text)
40 | print(root.find('path').text)
41 | value = (root.find('path').text,
42 | int(root.find('size')[0].text),
43 | int(root.find('size')[1].text),
44 | member[0].text,
45 | int(member[4][0].text),
46 | int(member[4][1].text),
47 | int(member[4][2].text),
48 | int(member[4][3].text)
49 | )
50 | xml_list.append(value)
51 | os.chdir("..")
52 | column_name = ['filename', 'width', 'height', 'class', 'xmin', 'ymin', 'xmax', 'ymax']
53 | xml_df = pd.DataFrame(xml_list, columns=column_name)
54 | classes_names = list(set(classes_names))
55 | classes_names.sort()
56 | return xml_df, classes_names
57 |
58 | def main():
59 | args = build_argparser().parse_args()
60 | dataset = args.input
61 | os.chdir(dataset)
62 | for folder in ['train','test', 'validation']:
63 | image_path = os.path.join(os.getcwd(), (folder))
64 | print(image_path)
65 | DIR = image_path
66 | if os.path.isdir(DIR):
67 | os.chdir(DIR)
68 | print("Currently in Subdirectory:", DIR)
69 | xml_df, classes_names = xml_to_csv(DIR)
70 | path = DIR + "/" + folder + '_labels.csv'
71 | print(path)
72 | xml_df.to_csv((path), index=None)
73 | print('Successfully converted xml to csv.')
74 |
75 | # Labelmap generation
76 | l_path = DIR + "/" + "label_map.pbtxt"
77 | pbtxt_content = ""
78 | for i, class_name in enumerate(classes_names):
79 | pbtxt_content = (
80 | pbtxt_content
81 | + "item {{\n id: {0}\n name: '{1}'\n}}\n\n".format(
82 | i + 1, class_name
83 | )
84 | )
85 | pbtxt_content = pbtxt_content.strip()
86 | with open(l_path, "w") as f:
87 | f.write(pbtxt_content)
88 | os.chdir("..")
89 | os.chdir("..")
90 | main()
--------------------------------------------------------------------------------
/custom_model/OI_Dataset/tf_test.py:
--------------------------------------------------------------------------------
1 | import os
2 | import glob
3 |
4 | import numpy as np
5 | import os
6 | import six.moves.urllib as urllib
7 | import sys
8 | import tarfile
9 | import tensorflow as tf
10 | import zipfile
11 | import cv2
12 |
13 | from collections import defaultdict
14 | from io import StringIO
15 | # This is needed to display the images.
16 |
17 | import matplotlib.pyplot as plt
18 | from PIL import Image
19 | from argparse import ArgumentParser
20 |
21 | # This is needed since the notebook is stored in the object_detection folder.
22 | sys.path.append("..")
23 | from object_detection.utils import ops as utils_ops
24 |
25 | from object_detection.utils import label_map_util
26 |
27 | from object_detection.utils import visualization_utils as vis_util
28 |
29 | def build_argparser():
30 | """
31 | Parse command line arguments.
32 | :return: command line arguments
33 | """
34 | parser = ArgumentParser()
35 | parser.add_argument("-m", "--model", required=True, type=str,
36 | help="Model path")
37 | parser.add_argument("-l", "--label", required=True, type=str,
38 | help="Label path")
39 | parser.add_argument("-i", "--input", required=True, type=str,
40 | help="input images")
41 |
42 | return parser
43 |
44 | args = build_argparser().parse_args()
45 |
46 | output_directory = args.model
47 | PATH_TO_LABELS = args.label
48 | dataset = args.input
49 |
50 | pb_fname = os.path.join(os.path.abspath(output_directory), "frozen_inference_graph.pb")
51 |
52 | print(pb_fname)
53 |
54 | def get_num_classes(pbtxt_fname):
55 | from object_detection.utils import label_map_util
56 | label_map = label_map_util.load_labelmap(pbtxt_fname)
57 | categories = label_map_util.convert_label_map_to_categories(
58 | label_map, max_num_classes=90, use_display_name=True)
59 | category_index = label_map_util.create_category_index(categories)
60 | return len(category_index.keys())
61 |
62 | num_classes = get_num_classes(PATH_TO_LABELS)
63 |
64 | PATH_TO_CKPT = pb_fname
65 | detection_graph = tf.Graph()
66 | with detection_graph.as_default():
67 | od_graph_def = tf.GraphDef()
68 | with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
69 | serialized_graph = fid.read()
70 | od_graph_def.ParseFromString(serialized_graph)
71 | tf.import_graph_def(od_graph_def, name='')
72 |
73 |
74 |
75 | label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
76 | categories = label_map_util.convert_label_map_to_categories(
77 | label_map, max_num_classes=num_classes, use_display_name=True)
78 | category_index = label_map_util.create_category_index(categories)
79 |
80 |
81 | def load_image_into_numpy_array(image):
82 | (im_width, im_height) = image.size
83 | return np.array(image.getdata()).reshape(
84 | (im_height, im_width, 3)).astype(np.uint8)
85 |
86 | # Size, in inches, of the output images.
87 | IMAGE_SIZE = (12, 8)
88 |
89 |
90 | def run_inference_for_single_image(image, graph):
91 | with graph.as_default():
92 | with tf.Session() as sess:
93 | # Get handles to input and output tensors
94 | ops = tf.get_default_graph().get_operations()
95 | all_tensor_names = {
96 | output.name for op in ops for output in op.outputs}
97 | tensor_dict = {}
98 | for key in [
99 | 'num_detections', 'detection_boxes', 'detection_scores',
100 | 'detection_classes', 'detection_masks'
101 | ]:
102 | tensor_name = key + ':0'
103 | if tensor_name in all_tensor_names:
104 | tensor_dict[key] = tf.get_default_graph().get_tensor_by_name(
105 | tensor_name)
106 | if 'detection_masks' in tensor_dict:
107 | # The following processing is only for single image
108 | detection_boxes = tf.squeeze(
109 | tensor_dict['detection_boxes'], [0])
110 | detection_masks = tf.squeeze(
111 | tensor_dict['detection_masks'], [0])
112 | # Reframe is required to translate mask from box coordinates to image coordinates and fit the image size.
113 | real_num_detection = tf.cast(
114 | tensor_dict['num_detections'][0], tf.int32)
115 | detection_boxes = tf.slice(detection_boxes, [0, 0], [
116 | real_num_detection, -1])
117 | detection_masks = tf.slice(detection_masks, [0, 0, 0], [
118 | real_num_detection, -1, -1])
119 | detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(
120 | detection_masks, detection_boxes, image.shape[0], image.shape[1])
121 | detection_masks_reframed = tf.cast(
122 | tf.greater(detection_masks_reframed, 0.5), tf.uint8)
123 | # Follow the convention by adding back the batch dimension
124 | tensor_dict['detection_masks'] = tf.expand_dims(
125 | detection_masks_reframed, 0)
126 | image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')
127 |
128 | # Run inference
129 | output_dict = sess.run(tensor_dict,
130 | feed_dict={image_tensor: np.expand_dims(image, 0)})
131 |
132 | # all outputs are float32 numpy arrays, so convert types as appropriate
133 | output_dict['num_detections'] = int(
134 | output_dict['num_detections'][0])
135 | output_dict['detection_classes'] = output_dict[
136 | 'detection_classes'][0].astype(np.uint8)
137 | output_dict['detection_boxes'] = output_dict['detection_boxes'][0]
138 | output_dict['detection_scores'] = output_dict['detection_scores'][0]
139 | if 'detection_masks' in output_dict:
140 | output_dict['detection_masks'] = output_dict['detection_masks'][0]
141 | return output_dict
142 |
143 |
144 | TEST_IMAGE_PATHS = glob.glob(os.path.join(dataset, "*.jpg*"))
145 | print(TEST_IMAGE_PATHS)
146 |
147 | for image_path in TEST_IMAGE_PATHS:
148 | image = Image.open(image_path)
149 | print(image_path)
150 | # the array based representation of the image will be used later in order to prepare the
151 | # result image with boxes and labels on it.
152 | image_np = load_image_into_numpy_array(image)
153 | # Expand dimensions since the model expects images to have shape: [1, None, None, 3]
154 | image_np_expanded = np.expand_dims(image_np, axis=0)
155 | # Actual detection.
156 | output_dict = run_inference_for_single_image(image_np, detection_graph)
157 | # Visualization of the results of a detection.
158 | vis_util.visualize_boxes_and_labels_on_image_array(
159 | image_np,
160 | output_dict['detection_boxes'],
161 | output_dict['detection_classes'],
162 | output_dict['detection_scores'],
163 | category_index,
164 | instance_masks=output_dict.get('detection_masks'),
165 | use_normalized_coordinates=True,
166 | line_thickness=8)
167 | #print(image_np)
168 | #exit(1)
169 | cv2.imshow("window", image_np)
170 | cv2.waitKey(0)
171 |
172 | # plt.figure(figsize=IMAGE_SIZE)
173 | # plt.imshow(image_np)
174 | # plt.show()
--------------------------------------------------------------------------------
/deepeye_app/app.py:
--------------------------------------------------------------------------------
1 | import depthai #core packages
2 | import logging # Logger
3 | import cv2 # computer vision package
4 | import json #json lib
5 | from time import time, sleep, monotonic
6 |
7 | from pathlib import Path #Path lib
8 | from config import model, calib
9 |
10 | # Object Tracker
11 | from tracker import Tracker
12 | from collision_avoidance import CrashAvoidance
13 |
14 | log = logging.getLogger(__name__)
15 |
16 | DEBUG = 1
17 | class DepthAI:
18 |
19 | process_watchdog_timeout=10 #seconds
20 |
21 | def create_pipeline(self, config):
22 | self.device = depthai.Device('', False)
23 | log.info("Creating DepthAI pipeline...")
24 |
25 | self.pipeline = self.device.create_pipeline(config)
26 | if self.pipeline is None:
27 | raise RuntimeError("Pipeline was not created.")
28 | log.info("Pipeline created.")
29 |
30 |
31 | def reset_process_wd(self):
32 | global wd_cutoff
33 | wd_cutoff=monotonic()+self.process_watchdog_timeout
34 | return
35 |
36 | def __init__(self):
37 | self.threshold = 0.5
38 |
39 | self.config = config = {
40 | 'streams': ['metaout', 'previewout'],
41 | 'depth':
42 | {'calibration_file': str(Path(calib, 'depthai.calib').absolute()),
43 | 'left_mesh_file': '',
44 | 'right_mesh_file': '',
45 | 'padding_factor': 0.3,
46 | 'depth_limit_m': 10.0,
47 | 'median_kernel_size': 7,
48 | 'lr_check': False,
49 | 'warp_rectify':
50 | {
51 | 'use_mesh': False,
52 | 'mirror_frame': True,
53 | 'edge_fill_color': 0
54 | }
55 | },
56 | 'ai':
57 | {'blob_file': str(Path(model, 'mobilenet-ssd_v1.blob').absolute()),
58 | 'blob_file_config': str(Path(model, 'mobilenet-ssd_v1.json').absolute()),
59 | 'blob_file2': '',
60 | 'blob_file_config2': '',
61 | 'calc_dist_to_bb': True,
62 | 'keep_aspect_ratio': True,
63 | 'camera_input': 'rgb',
64 | 'shaves': 14,
65 | 'cmx_slices': 14,
66 | 'NN_engines': 1
67 | },
68 | 'ot':
69 | {'max_tracklets': 20,
70 | 'confidence_threshold': 0.5
71 | },
72 | 'board_config':
73 | {'swap_left_and_right_cameras': True,
74 | 'left_fov_deg': 71.86,
75 | 'rgb_fov_deg': 68.7938,
76 | 'left_to_right_distance_cm': 9.0,
77 | 'left_to_rgb_distance_cm': 2.0,
78 | 'store_to_eeprom': False,
79 | 'clear_eeprom': False,
80 | 'override_eeprom': False
81 | },
82 | 'camera':
83 | {'rgb':
84 | {'resolution_h': 1080,
85 | 'fps': 30.0},
86 | 'mono':
87 | {'resolution_h': 720,
88 | 'fps': 30.0
89 | }
90 | },
91 | 'app':
92 | {'sync_video_meta_streams': False,
93 | 'sync_sequence_numbers': False,
94 | 'usb_chunk_KiB': 64
95 | }
96 | }
97 | # Create a pipeline config
98 | self.create_pipeline(self.config)
99 |
100 | blob_file = str(Path(model, 'mobilenet-ssd_depth.json'))
101 |
102 | with open(blob_file) as f:
103 | self.data = json.load(f)
104 |
105 | self.detection = []
106 |
107 | # Calculate collision_avoidance for bicycle
108 | # bus, car, dog, horse, motorbike, train
109 | self.collision_avoidance = [2.0, 6.0, 7.0, 10.0, 12.0, 13.0, 14.0, 17.0]
110 | # Label map
111 | self.label = {
112 | 0.0: "background",
113 | 1.0: "aeroplane",
114 | 2.0: "bicycle",
115 | 3.0: "bird",
116 | 4.0: "boat",
117 | 5.0: "bottle",
118 | 6.0: "bus",
119 | 7.0: "car",
120 | 8.0: "cat",
121 | 9.0: "chair",
122 | 10.0: "cow",
123 | 11.0: "diningtable",
124 | 12.0: "dog",
125 | 13.0: "horse",
126 | 14.0: "motorbike",
127 | 15.0: "person",
128 | 16.0: "pottedplant",
129 | 17.0: "sheep",
130 | 18.0: "sofa",
131 | 19.0: "train",
132 | 20.0: "tvmonitor",
133 | }
134 |
135 | self.reset_process_wd()
136 |
137 | def image_resize(self, image, width = None, height = None, inter = cv2.INTER_AREA):
138 | # initialize the dimensions of the image to be resized and
139 | # grab the image size
140 | dim = None
141 | (h, w) = image.shape[:2]
142 |
143 | # if both the width and height are None, then return the
144 | # original image
145 | if width is None and height is None:
146 | return image
147 |
148 | # check to see if the width is None
149 | if width is None:
150 | # calculate the ratio of the height and construct the
151 | # dimensions
152 | r = height / float(h)
153 | dim = (int(w * r), height)
154 |
155 | # otherwise, the height is None
156 | else:
157 | # calculate the ratio of the width and construct the
158 | # dimensions
159 | r = width / float(w)
160 | dim = (width, int(h * r))
161 |
162 | # resize the image
163 | resized = cv2.resize(image, dim, interpolation = inter)
164 |
165 | # return the resized image
166 | return resized
167 |
168 | def capture(self):
169 | cv2.namedWindow("output", cv2.WINDOW_NORMAL)
170 |
171 | # Label mapping
172 | try:
173 | labels = self.data['mappings']['labels']
174 | except:
175 | labels = None
176 | print("Labels not found in json!")
177 |
178 | while True:
179 | #print("debug")
180 | # retreive data from the device
181 | # data is stored in packets, there are nnet (Neural NETwork) packets which have additional functions for NNet result interpretation
182 | nnet_packets, data_packets = self.pipeline.get_available_nnet_and_data_packets()
183 |
184 | packets_len = len(nnet_packets) + len(data_packets)
185 | if packets_len != 0:
186 | self.reset_process_wd()
187 | else:
188 | cur_time=monotonic()
189 | if cur_time > wd_cutoff:
190 | print("process watchdog timeout")
191 | os._exit(10)
192 |
193 | # Get the detection
194 | for _, nnet_packet in enumerate(nnet_packets):
195 | self.detection = []
196 | detections = nnet_packet.getDetectedObjects()
197 | for detection in detections:
198 | detection_dict = detection.get_dict()
199 | if detection_dict['confidence'] > 0.5:
200 | self.detection.append(detection_dict)
201 |
202 | # Post processing
203 | boxes = []
204 | for packet in data_packets:
205 | if packet.stream_name == 'previewout':
206 | data = packet.getData()
207 | if data is None:
208 | continue
209 | # The format of previewout image is CHW (Chanel, Height, Width), but OpenCV needs HWC, so we
210 | # change shape (3, 300, 300) -> (300, 300, 3).
211 | data0 = data[0, :, :]
212 | data1 = data[1, :, :]
213 | data2 = data[2, :, :]
214 | frame = cv2.merge([data0, data1, data2])
215 |
216 | img_h = frame.shape[0]
217 | img_w = frame.shape[1]
218 |
219 | frame_o = frame.copy()
220 | for e in self.detection:
221 | color = (0, 255, 0) # bgr
222 | label = e['label']
223 | if label in self.collision_avoidance:
224 | # Create dic for tracking
225 | boxes.append({
226 | 'detector': "MobileNet SSD",
227 | 'label': self.label[label],
228 | 'conf': e['confidence'],
229 | 'left': int(e['x_min'] * img_w),
230 | 'top': int(e['y_min'] * img_h),
231 | 'right': int(e['x_max'] * img_w),
232 | 'bottom': int(e['y_max'] * img_h),
233 | 'distance_x': e['depth_x'],
234 | 'distance_y': e['depth_y'],
235 | 'distance_z': e['depth_z'],
236 | })
237 | color = (0, 0, 255) # bgr
238 |
239 | pt1 = int(e['x_min'] * img_w), int(e['y_min'] * img_h)
240 | pt2 = int(e['x_max'] * img_w), int(e['y_max'] * img_h)
241 |
242 | x1, y1 = pt1
243 | x2, y2 = pt2
244 |
245 | cv2.rectangle(frame, pt1, pt2, color)
246 |
247 | pt_t1 = x1, y1 + 20
248 | cv2.putText(frame, labels[int(e['label'])], pt_t1, cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2)
249 |
250 | pt_t2 = x1, y1 + 40
251 | cv2.putText(frame, '{:.2f}'.format(100*e['confidence']) + ' %', pt_t2, cv2.FONT_HERSHEY_SIMPLEX, 0.5, color)
252 |
253 | if DEBUG:
254 | if self.config['ai']['calc_dist_to_bb']:
255 | pt_t3 = x1, y1 + 60
256 | cv2.putText(frame, 'x1:' '{:7.3f}'.format(e['depth_x']) + ' m', pt_t3, cv2.FONT_HERSHEY_SIMPLEX, 0.5, color)
257 |
258 | pt_t4 = x1, y1 + 80
259 | cv2.putText(frame, 'y1:' '{:7.3f}'.format(e['depth_y']) + ' m', pt_t4, cv2.FONT_HERSHEY_SIMPLEX, 0.5, color)
260 |
261 | pt_t5 = x1, y1 + 100
262 | cv2.putText(frame, 'z1:' '{:7.3f}'.format(e['depth_z']) + ' m', pt_t5, cv2.FONT_HERSHEY_SIMPLEX, 0.5, color)
263 |
264 | # frame = self.image_resize(frame, 500)
265 |
266 | cv2.imshow("output", frame)
267 |
268 | if cv2.waitKey(1) == ord('q'):
269 | cv2.destroyAllWindows()
270 | exit(0)
271 |
272 | yield frame_o, boxes
273 | def main():
274 | # depth AI
275 | di = DepthAI()
276 |
277 | # Tracker
278 | tracker = Tracker(log)
279 | # Cross avoidance
280 | crash_avoidance = CrashAvoidance()
281 |
282 | for frame, results in di.capture():
283 | # Pass class along with Z and X coordinates
284 | pts_l = [(item['distance_x'], item['distance_z'], item['label']) for item in results]
285 |
286 | # Pass the points to tracker
287 | tracker_objs, obj_class = tracker.update(pts_l, log)
288 |
289 | #Pass the tracker objects to collision_avoidance
290 | crash_alert = crash_avoidance.parse(tracker_objs, obj_class)
291 |
292 | if __name__ == "__main__":
293 | main()
--------------------------------------------------------------------------------
/deepeye_app/calibration/config/BW1098FFC.json:
--------------------------------------------------------------------------------
1 | {
2 | "board_config":
3 | {
4 | "name": "BW1098FFC",
5 | "revision": "R0M0E0",
6 | "swap_left_and_right_cameras": true,
7 | "left_fov_deg": 71.86,
8 | "rgb_fov_deg": 68.7938,
9 | "left_to_right_distance_cm": 5.0,
10 | "left_to_rgb_distance_cm": 2.5
11 | }
12 | }
13 |
14 |
--------------------------------------------------------------------------------
/deepeye_app/calibration/depthai.calib:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/deepeye_app/calibration/depthai.calib
--------------------------------------------------------------------------------
/deepeye_app/collision_avoidance.py:
--------------------------------------------------------------------------------
1 | import math
2 | import time
3 | from collections import OrderedDict
4 |
5 | import cv2
6 | import numpy as np
7 |
8 |
9 | # ROS
10 | import rospy
11 | from std_msgs.msg import String
12 |
13 | # To do:
14 | # Try to get the debug on and off from the config file
15 |
16 | DEBUG = 1
17 |
18 | class CrashAvoidance:
19 | def __init__(self, calculated_entries=5, collision_trajectory_threshold=0.05, collision_time_to_impact=4):
20 | self.calculated_entries = calculated_entries
21 | self.collision_trajectory_threshold = collision_trajectory_threshold
22 | self.collision_time_to_impact = collision_time_to_impact
23 | self.entries = OrderedDict()
24 | # ROS
25 | rospy.init_node('publisher', anonymous=True)
26 | self.pub = rospy.Publisher('depthai', String, queue_size=10)
27 | rate = rospy.Rate(10) # 10hz
28 |
29 | self.object_ID = None
30 |
31 | def best_fit_slope_and_intercept(self, objectID):
32 | points = [item['value'] for item in self.entries[objectID]]
33 | xs = np.array([item[0] for item in points])
34 | zs = np.array([item[1] for item in points])
35 |
36 | m, b = np.polyfit(xs, zs, 1)
37 |
38 | return m, b
39 |
40 | def is_dangerous_trajectory(self, objectID):
41 | last = self.entries[objectID][0]
42 |
43 |
44 | # Update the class name
45 | class_name = last['class']
46 | class_name = class_name[0]
47 | try:
48 | m, b = self.best_fit_slope_and_intercept(objectID)
49 | except ValueError:
50 | return False
51 | distance = abs(b) / math.sqrt(math.pow(m, 2) + 1)
52 |
53 | if DEBUG:
54 | image = np.ones((200, 200)) * 255
55 |
56 | cv2.line(image, (0, int(-100 * m + b)), (200, int(100 * m + b)), (0, 0, 0))
57 | cv2.putText(image, f"Distance: {round(distance, 2)}", (20, 20), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0))
58 | string = ""
59 | if distance < self.collision_trajectory_threshold:
60 | # We need to cal and update the oppsite side object
61 | if (m < 0):
62 | string = "A " + class_name + " is on your" + "left side "
63 | else:
64 | string = "A " + class_name + " is on your" + " right side "
65 |
66 | cv2.putText(image, "DANGER", (20, 100), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255))
67 | cv2.putText(image, class_name, (100, 180), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255))
68 |
69 | # Track the existing object and dont update the string
70 | # until get the new object ID
71 |
72 | if self.object_ID != objectID:
73 | self.object_ID = objectID
74 | # Publish the trajectory object
75 | print(string)
76 | self.pub.publish(string)
77 |
78 | cv2.imshow(f"trajectory", image)
79 | cv2.waitKey(1)
80 |
81 | return distance < self.collision_trajectory_threshold
82 |
83 | def is_impact_close(self, objectID):
84 | last, first = self.entries[objectID][0], self.entries[objectID][-1]
85 | x1, z1 = last['value']
86 | x2, z2 = first['value']
87 | lf_distance = math.sqrt(math.pow(x1 - x2, 2) + math.pow(z1 - z2, 2))
88 | if lf_distance == 0:
89 | return False
90 | timed = first['timestamp'] - last['timestamp']
91 | speed = lf_distance / timed # m/s
92 | target_distance = math.sqrt(math.pow(x2, 2) + math.pow(z2, 2))
93 | tti = target_distance / speed
94 |
95 | if DEBUG:
96 | image = np.ones((200, 200)) * 255
97 | cv2.putText(image, f"LFD: {round(lf_distance, 2)}", (20, 20), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0))
98 | cv2.putText(image, f"SPD: {round(speed, 2)}", (20, 60), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0))
99 | cv2.putText(image, f"TDT: {round(target_distance, 2)}", (20, 100), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0))
100 | cv2.putText(image, f"TTI: {round(tti, 2)}", (20, 140), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0))
101 | if tti < self.collision_trajectory_threshold:
102 | cv2.putText(image, "DANGER", (80, 180), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255))
103 | cv2.imshow(f"impact", image)
104 | cv2.waitKey(1)
105 | return tti < self.collision_time_to_impact
106 |
107 | def parse(self, tracker_objects, obj_class):
108 | #print(type(tracker_objects))
109 | for key in list(self.entries.keys()):
110 | #print(key)
111 | if key not in tracker_objects:
112 | del self.entries[key]
113 |
114 | for key in tracker_objects.keys():
115 | # print(tracker_objects[key])
116 | # print(obj_class[key])
117 | item = {
118 | 'timestamp': time.time(),
119 | 'value': tracker_objects[key],
120 | 'class': obj_class[key]
121 | }
122 |
123 | if key not in self.entries:
124 | self.entries[key] = [item]
125 | else:
126 | self.entries[key] = (self.entries[key] + [item])[-self.calculated_entries:]
127 |
128 | if len(self.entries[key]) > 2 and self.is_dangerous_trajectory(key) and self.is_impact_close(key):
129 | return True
130 | else:
131 | return False
132 |
133 |
134 |
135 |
--------------------------------------------------------------------------------
/deepeye_app/config.py:
--------------------------------------------------------------------------------
1 | import logging
2 | import os
3 | import sys
4 | from pathlib import Path
5 | from concurrent_log_handler import ConcurrentRotatingFileHandler
6 |
7 | # Log configuration
8 | root = logging.getLogger()
9 | handler = logging.StreamHandler(sys.stdout)
10 | formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
11 | handler.setFormatter(formatter)
12 | root.addHandler(handler)
13 | logfile = os.path.abspath("camera.log")
14 |
15 | # Rotate log after reaching 512K, keep 5 old copies.
16 | rotateHandler = ConcurrentRotatingFileHandler(logfile, "a", 512*1024, 5)
17 | rotateHandler.setFormatter(formatter)
18 | root.addHandler(rotateHandler)
19 | root.setLevel(logging.INFO)
20 | root.info("Logging system initialized, kept in file {}...".format(logfile))
21 |
22 | # Model path
23 | model = str(Path('models/').resolve())
24 | # calibration path
25 | calib = str(Path('calibration/').resolve())
26 |
--------------------------------------------------------------------------------
/deepeye_app/models/mobilenet-ssd.blob:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/deepeye_app/models/mobilenet-ssd.blob
--------------------------------------------------------------------------------
/deepeye_app/models/mobilenet-ssd_depth.json:
--------------------------------------------------------------------------------
1 | {
2 | "tensors":
3 | [
4 | {
5 | "output_tensor_name": "out",
6 | "output_dimensions": [1, 1, 100, 10],
7 | "output_entry_iteration_index": 2,
8 | "output_properties_dimensions": [3],
9 | "property_key_mapping":
10 | [
11 | [],
12 | [],
13 | [],
14 | ["image_id", "label", "conf", "x_min", "y_min", "x_max", "y_max", "distance_x", "distance_y", "distance_z"]
15 | ],
16 | "output_properties_type": "f16"
17 | }
18 | ],
19 | "mappings":
20 | {
21 | "labels":
22 | [
23 | "background",
24 | "aeroplane",
25 | "bicycle",
26 | "bird",
27 | "boat",
28 | "bottle",
29 | "bus",
30 | "car",
31 | "cat",
32 | "chair",
33 | "cow",
34 | "diningtable",
35 | "dog",
36 | "horse",
37 | "motorbike",
38 | "person",
39 | "pottedplant",
40 | "sheep",
41 | "sofa",
42 | "train",
43 | "tvmonitor"
44 | ]
45 | }
46 | }
47 |
48 |
--------------------------------------------------------------------------------
/deepeye_app/models/mobilenet-ssd_v1.blob:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/deepeye_app/models/mobilenet-ssd_v1.blob
--------------------------------------------------------------------------------
/deepeye_app/models/mobilenet-ssd_v1.json:
--------------------------------------------------------------------------------
1 | {
2 | "NN_config":
3 | {
4 | "output_format" : "detection",
5 | "NN_family" : "mobilenet",
6 | "confidence_threshold" : 0.5
7 | },
8 | "mappings":
9 | {
10 | "labels":
11 | [
12 | "background",
13 | "aeroplane",
14 | "bicycle",
15 | "bird",
16 | "boat",
17 | "bottle",
18 | "bus",
19 | "car",
20 | "cat",
21 | "chair",
22 | "cow",
23 | "diningtable",
24 | "dog",
25 | "horse",
26 | "motorbike",
27 | "person",
28 | "pottedplant",
29 | "sheep",
30 | "sofa",
31 | "train",
32 | "tvmonitor"
33 | ]
34 | }
35 | }
36 |
37 |
--------------------------------------------------------------------------------
/deepeye_app/tracker.py:
--------------------------------------------------------------------------------
1 | from collections import OrderedDict
2 |
3 | import cv2
4 | from scipy.spatial import distance as dist
5 |
6 | import numpy as np
7 |
8 |
9 | class Tracker:
10 | def __init__(self, log, maxDisappeared=50, maxDistance=50, maxHistory=100, ):
11 | self.nextObjectID = 0
12 | self.objects = OrderedDict()
13 | self.disappeared = OrderedDict()
14 | self.maxDisappeared = maxDisappeared
15 | self.maxDistance = maxDistance
16 | self.maxHistory = maxHistory
17 | self.obj_class = OrderedDict()
18 | self.colors = OrderedDict()
19 | self.history = OrderedDict()
20 |
21 | self.log = log
22 |
23 | def register(self, centroid, obj_cls):
24 | self.history[self.nextObjectID] = [centroid]
25 | self.objects[self.nextObjectID] = centroid
26 | self.obj_class[self.nextObjectID] = obj_cls
27 | self.disappeared[self.nextObjectID] = 0
28 | self.colors[self.nextObjectID] = np.random.choice(range(256), size=3).astype('i').tolist()
29 | self.nextObjectID += 1
30 |
31 | def deregister(self, objectID):
32 | del self.objects[objectID]
33 | del self.disappeared[objectID]
34 | del self.obj_class[objectID]
35 | del self.colors[objectID]
36 | del self.history[objectID]
37 |
38 | def update(self, pts_l, log):
39 | pts = []
40 | for item in pts_l:
41 | pts.append(item[0:2])
42 |
43 | if len(pts) == 0:
44 | for objectID in list(self.disappeared.keys()):
45 | self.disappeared[objectID] += 1
46 |
47 | if self.disappeared[objectID] > self.maxDisappeared:
48 | self.deregister(objectID)
49 |
50 | return self.objects, self.obj_class
51 |
52 | if len(self.objects) == 0:
53 | for pt in pts_l:
54 | self.register(pt[0:2], pt[2:3])
55 |
56 | else:
57 | # grab the set of object IDs and corresponding centroids
58 | objectIDs = list(self.objects.keys())
59 | objectCentroids = list(self.objects.values())
60 |
61 | D = dist.cdist(np.array(objectCentroids), pts)
62 | rows = D.min(axis=1).argsort()
63 | cols = D.argmin(axis=1)[rows]
64 | usedRows = set()
65 | usedCols = set()
66 |
67 | for (row, col) in zip(rows, cols):
68 | if row in usedRows or col in usedCols:
69 | continue
70 |
71 | if D[row, col] > self.maxDistance:
72 | continue
73 |
74 | objectID = objectIDs[row]
75 | self.objects[objectID] = pts[col]
76 | self.history[objectID] = (self.history[objectID] + [pts[col]])[-self.maxHistory:]
77 | self.disappeared[objectID] = 0
78 |
79 | usedRows.add(row)
80 | usedCols.add(col)
81 |
82 | unusedRows = set(range(0, D.shape[0])).difference(usedRows)
83 | unusedCols = set(range(0, D.shape[1])).difference(usedCols)
84 |
85 | if D.shape[0] >= D.shape[1]:
86 | for row in unusedRows:
87 | objectID = objectIDs[row]
88 | self.disappeared[objectID] += 1
89 |
90 | if self.disappeared[objectID] > self.maxDisappeared:
91 | self.deregister(objectID)
92 | else:
93 | for col in unusedCols:
94 | self.register(pts[col], "None")
95 |
96 | return self.objects, self.obj_class
97 |
--------------------------------------------------------------------------------
/deepeye_app/txt2speech/README.md:
--------------------------------------------------------------------------------
1 | # Text to speech support for deepEye
2 |
3 | ## Introduction
4 | This software component is to detect the moving objects like bicycle, car, bus, dog, horse, motorbike towards the camera, and alert them through speaker/headphone if there is a possible collision.
5 |
6 | We plan to support this feature both edge device and mobile app conversion. For example, if the user connected deepEye device via BT then send the text to the mobile app directly and convert them if not the conversion happens on edge devices.
7 |
8 | ## Software block
9 | ### On Device conversion block:
10 |
11 | 
12 |
13 | ### Mobile Device conversion
14 | 
15 |
16 |
17 | ### Dep
18 | ```bash
19 | # google speech
20 | sudo apt-get install sox libsox-fmt-mp3
21 |
22 | python3 -m pip install pip install google-speech sox
23 |
24 |
25 | # Mycroft Mimic V1
26 | git clone https://github.com/MycroftAI/mimic1.git
27 | cd mimic1
28 | ./dependencies.sh --prefix="/usr/local"
29 | ./autogen.sh
30 | ./configure --prefix="/usr/local"
31 | make
32 | make check
33 | ```
--------------------------------------------------------------------------------
/deepeye_app/txt2speech/s_images/txt2s.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/deepeye_app/txt2speech/s_images/txt2s.png
--------------------------------------------------------------------------------
/deepeye_app/txt2speech/s_images/txt2s_bt.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/deepeye_app/txt2speech/s_images/txt2s_bt.png
--------------------------------------------------------------------------------
/deepeye_app/txt2speech/sw_design/txt2s_BT:
--------------------------------------------------------------------------------
1 | 3Vhdb5swFP01PK4CDCR5bNKue1ikTenW9dHFt2DNYGScBvbrZwfznShrlZSpUh7w8Rf33HOuLrHQKinuBM7iNSfALNcmhYVuLNd1PNe19M8mZYXMEaqASFBiFrXAhv4BA9oG3VICeW+h5JxJmvXBkKcphLKHYSH4rr/smbP+rRmOYARsQszG6AMlMjZRuLMW/wI0iuubnWBRzSS4XmwiyWNM+K4DoVsLrQTnsnpKihUwTV7NS7Xv85HZ5sUEpPJfNiT3X32+vi9z/Fz8uFt+z+OH5Se/OuUFs60JOOSM0ZzyVBP4winBaQg6gSAVv3t8H40sa4oE36YE9C2OhZa7mErYZDjUszslCoXFMmFm2twHQkJxNBCnoUfpCngCUpRqidkQ2IZRIynPDHdtfpzAYHEnNzWGjSSi5uSWNfVgiHsFibMRiZtYRUMUtoaEmxv+J8oafibjbD7ibEQSpORaO1iNQobznIZ9XqCg8pd6tq98M3rszNzo0O16UJpBdQmQkekHTKoX4VsRwinrjBnvMOofILTGBDAs6Uv/NQ6xbG74xql6wTah/uLK76fUGaSqCsDs69aF4VHzI9qoD5JYRCBHB+3T3gT+diUsxu6RgqaR5QY40VZIn/KsSV1HH8oOsq+IXAr+G1accaGQlKdq5fKZMjaAMKNRqmWlkg4KX2pzUVX0r81EQgnR1xy0Zmte+zzunA3dicbu9A5oyb2UOWspdXIiC+nmGUAYK/w6yyYvae4x2U5W0urO5iw1rVPR2vp28Zo2m7Km+d4go/Omxr22pnmnj7pwVUPeSAwEILstofIPU+Wm6qYCpqvYk1BPkdyn80Tdm9ppzXg6p43b1nfpHhRhouxs0sPH7ly7bT+6gEPr0lzJ96QAp/IyWgx79Dd2J0MnI/99uxMUTCO1M0rmgygBBScOurQSxp95PznV38YftS9FA+95zsX6UjVs/wmpMtb+n4Ru/wI=
--------------------------------------------------------------------------------
/deepeye_app/txt2speech/sw_design/txt2speech:
--------------------------------------------------------------------------------
1 | 3Vpbc9soGP01nuk+tCOBLMePcdptd2bTy7ptNo9UEIspAg3Csd1fX7DQFSe+xLZUz2QS8SGBOOe7KwN4kyzfS5TGtwITNgAeXg7g2wEAfgDAwPx4eJVLriDMBTNJsb2pEkzpL2KFnpXOKSZZ40YlBFM0bQojwTmJVEOGpBSL5m0PgjV3TdGMOIJphJgrvaNYxfYUYFTJPxA6i4ud/XCczySouNmeJIsRFouaCL4bwBsphMqvkuUNYQa8Apf8ub+fmC1fTBKudnkg+frvUNx+XWXoYfnt/eRLFt9NXlt2HhGb2wNP7/R4gVQUYzFbo23QjUn0U//9hysiOVEV4FRwezy1KjCTYs4xMdv6AzhZxFSRaYoiM7vQWqJlsUqYnbYvQKQiyydP5pd4aUUjIiFKrvQt9gFwZSG2OhbY4aIizC9YiGtkhVaGrI7MypUrGPWFRXIPVKGD6rcUI0W07D+SzZnGL9S/4OSH1Fczc/WqBm2mkJpnf/UOVz/sGtihA2wkGKPZWgs99CgoRjwyOGOieqKdodcz7Ry5Nh/r02AtuyWJsDv0CTIw6hqzKwczByTC8bUJOHoUMZRlNGriQpZU/a+vvTdDO7qvzbw1R/eKwcoO8k0IdmJUC0n9ImIuI7LddBSSM6K2aYfLTA354QbgC5kkDCn62HzdTWzYHT4LylWN+OH4zbDpc/wWpflB7XP1cLd9qbZ25Fg4S60VpDz64Tozdu1MScp1UA1RYoyG/8jSkuSaJmnDUU3dyZQUP8mNYEJqCRdc3zl5oIy1RIjRGTcKSEw00QJjhlRnM9d2IqEYm202GnFl5t6R7HjUCiBXrh0HG7QJnMqMC2WqcaKWCmQpIVGs5ddp2rnza0ddCHd0fr53MtjcNPEF7i8Yhzs7QDP4TCTV5zD6fGynCLp0djBoWcehrg7Crl2dH7xcQXaltSu6gnY2PD6QrvZC5Queiyw3i36OLBta/iCmgNcOO6V97MuVUwK5S52arfDC2QqOx5Zjoudn6wiFQt8Jc8zrUEfYXujsjtDN0D9xRrnpH0yJfKSmk9BxNugHT2C0LRsMTpUMFpvVULsnmQPUxRQxZSv6mS7YWYuY4gg1Aj6Ky8U/gH3D362GPvHXUUzTdcnxoBb6bJ07jgD0znG4bfHnYqObyeiTy1WthWaG9/W5qoZcj/bvom3tjuUhoy+RF7bLvJ0TpVZUgaPzRl6wX714flXoS5F5OMOjdk8AtNO0U3O8X5l5EanwwXQ5Tb8zN3DAflXmH0nW8WxrG+unJsv9gndxZIVhE+Og3dw+lCxnoVOT5RaZ/Qp1vf8meDRN6DjpKeqYSzZb91NEWb69/KtGcOYMBrqF9nexqSl0MbU2GDURHxa2Vi8a4Qb1OaDY1sPqn99yyqp/IYTvfgM=
--------------------------------------------------------------------------------
/deepeye_app/txt2speech/txt-simulator.py:
--------------------------------------------------------------------------------
1 | import rospy
2 | from std_msgs.msg import String
3 |
4 | phrases_list = ["slow down a little",
5 | "take this right",
6 | "you missed your turn again!",
7 | "A car gonna take a left, slow down",
8 | "Red Signal on",
9 | "Green Signal on, You can walk now",
10 | "take this right",
11 | "you missed your turn again!",
12 | "A car gonna take a left, slow down",
13 | "Red Signal on",
14 | "Green Signal on, You can walk now",
15 | "take this right",
16 | "you missed your turn again!",
17 | "A car gonna take a left, slow down",
18 | "Red Signal on",
19 | "Green Signal on, You can walk now",
20 | "take this right",
21 | "you missed your turn again!",
22 | "A car gonna take a left, slow down",
23 | "Red Signal on",
24 | "Green Signal on, You can walk now",
25 | "take this right",
26 | "you missed your turn again!",
27 | "A car gonna take a left, slow down",
28 | "Red Signal on",
29 | "Green Signal on, You can walk now",
30 | "take this right",
31 | "you missed your turn again!",
32 | "A car gonna take a left, slow down",
33 | "Red Signal on",
34 | "Green Signal on, You can walk now",
35 | ]
36 |
37 | def txt_simulator():
38 | rospy.init_node('publisher', anonymous=True)
39 | pub = rospy.Publisher('txt_simulator', String, queue_size=10)
40 | rate = rospy.Rate(10) # 10hz
41 |
42 | for i in range(len(phrases_list)):
43 | rospy.loginfo(phrases_list[i])
44 | pub.publish(phrases_list[i])
45 | rate.sleep()
46 |
47 | if __name__ == '__main__':
48 | try:
49 | txt_simulator()
50 | except rospy.ROSInterruptException:
51 | pass
--------------------------------------------------------------------------------
/deepeye_app/txt2speech/txt2speech.py:
--------------------------------------------------------------------------------
1 | import socket
2 | import logging
3 | import sched
4 | import time
5 | from threading import Thread, Lock
6 |
7 | # ROS
8 | import rospy
9 | from std_msgs.msg import String
10 |
11 | # Text 2 speech
12 | from google_speech import Speech
13 |
14 | import subprocess
15 |
16 | lang = "en"
17 |
18 | # Init with False
19 | networkConnection = False
20 |
21 | # schedular instance is created
22 | s = sched.scheduler(time.time, time.sleep)
23 |
24 | # Mutex
25 | mutex = Lock()
26 |
27 | # BT Connected flag
28 | bt_flag = True
29 |
30 | def isInternetConnected():
31 | global networkConnection
32 |
33 | # Mutex lock before CS
34 | mutex.acquire()
35 |
36 | try:
37 | # Try to ping google
38 | # To do: Is there way we can do local ping rather than google
39 | sock = socket.create_connection(("www.google.com", 80))
40 | if sock is not None:
41 | sock.close
42 | # On Success update the status
43 | networkConnection = True
44 | print("Network ping ... success")
45 | except OSError:
46 | # On Failure update
47 | networkConnection = False
48 |
49 | # Mutex unlock after CS
50 | mutex.release()
51 |
52 | # Recursive call
53 | s.enter(1, 1, isInternetConnected, ())
54 |
55 | def callback(data):
56 | global networkConnection
57 | print("Debug Data: ".format(data))
58 |
59 | if bt_flag == False:
60 | if networkConnection == True:
61 | # Google speech block
62 | speech = Speech(data.data, lang)
63 | speech.play()
64 | time.sleep(0.5)
65 | else:
66 | # On device mycroft mimic v1
67 | p = subprocess.Popen(["/home/nullbyte/Desktop/myGit/mimic1/mimic", data.data], stdout = subprocess.PIPE)
68 | (output, err) = p.communicate()
69 | p_status = p.wait()
70 | else:
71 | cmd = "echo " + data.data + " > /dev/rfcomm0"
72 | print(cmd)
73 | p = subprocess.Popen(cmd, shell=True)
74 | (stderr,stdout) = p.communicate()
75 | print("stderr: ".format(stderr))
76 | print("stdout: ".format(stdout))
77 |
78 | def main():
79 | format = "%(asctime)s: %(message)s"
80 | logging.basicConfig(format=format, level=logging.INFO,
81 | datefmt="%H:%M:%S")
82 |
83 | # ROS callback
84 | rospy.init_node('subscriper', anonymous=True)
85 |
86 | rospy.Subscriber("depthai", String, callback)
87 |
88 | s.enter(1, 1, isInternetConnected, ())
89 | s.run()
90 |
91 | rospy.spin()
92 |
93 | if __name__ == "__main__":
94 | main()
--------------------------------------------------------------------------------
/images/BW1098FFC_R0M0E0_diag.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/BW1098FFC_R0M0E0_diag.png
--------------------------------------------------------------------------------
/images/BW1098FFC_R0M0E0_dims.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/BW1098FFC_R0M0E0_dims.png
--------------------------------------------------------------------------------
/images/BW1098FFC_R0M0E0_front.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/BW1098FFC_R0M0E0_front.png
--------------------------------------------------------------------------------
/images/HLD.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/HLD.png
--------------------------------------------------------------------------------
/images/HLD_1.JPG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/HLD_1.JPG
--------------------------------------------------------------------------------
/images/demo.PNG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/demo.PNG
--------------------------------------------------------------------------------
/images/hw.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/hw.jpeg
--------------------------------------------------------------------------------
/images/jetson-nano-dev-kit-top-r6-HR-B01.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/images/jetson-nano-dev-kit-top-r6-HR-B01.png
--------------------------------------------------------------------------------
/openvino_analysis/README.md:
--------------------------------------------------------------------------------
1 | # Model Analysis
2 |
3 | > Hardware Configuration
4 | > CPU : i7-8665U
5 | > VPU : myriad x NCS2
6 |
7 | ## Intel Models
8 | |Model Name |Complexity - GFLOPs | Size - MP | CPU FPS - FP32 | Jetson + VPU FPS - FP16 | RPI + VPU FPS - FP16
9 | |--- |--- |--- |--- |--- |---
10 | |pedestrian-detection-adas-0002 | 2.836 |1.165 |28 |6.86 |4.53
11 | |pedestrian-detection-adas-binary-0001 |0.945 |1.165|33 |Not Supported Layer Found |Not Supported Layer Found
12 | |pedestrian-and-vehicle-detector-adas-0001 | 3.974 |1.650 |24.09 |6.18 |4.37
13 | |vehicle-detection-adas-0002 |2.798 |1.079 |26 |7.51 |4.85
14 | |vehicle-detection-adas-binary-0001 |0.942 |1.079 |28 |Not Supported Layer Found |Not Supported Layer Found
15 | |road-segmentation-adas-0001 |4.770 |4.770 |8.48 |1.94 |1.02
16 | |semantic-segmentation-adas-0001 |58.572 |6.686 |2.06 |Memory Error |Memory Error
17 |
18 | ## Public Models
19 | |Model Name |Complexity - GFLOPs | Size - MP | CPU FPS - FP32 | Jetson + VPU FPS - FP16 | RPI + VPU FPS - FP16
20 | |--- |--- |--- |--- |--- |---
21 | |YOLO v3 | 65.9843 |237 |1.86 |1.4 |Crash |
22 | |SSD with MobileNet |2.316~2.494 |65 |22 |10.4 |6.51
23 |
24 | ## Depth AI HW Model analysis
25 | | Model | SAHVES 4 - FPS | SAHVES 7 - FPS | GFlops | mParams | |
26 | |---|---|---|---|---|---|
27 | | ssd_mobilenet_v1_coco | 28-30 | | 2.316~2.494 | 5.783~6.807 | |
28 | | ssd_mobilenet_v2_coco | 12-13 | 14-15 | 3.775 | 16.818 | |
29 | | | | | | | |
30 | | | | | | | |
--------------------------------------------------------------------------------
/openvino_analysis/intel/object-detection/pedestrian-and-vehicle-detector-adas-0001/__pycache__/inference.cpython-36.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/object-detection/pedestrian-and-vehicle-detector-adas-0001/__pycache__/inference.cpython-36.pyc
--------------------------------------------------------------------------------
/openvino_analysis/intel/object-detection/pedestrian-and-vehicle-detector-adas-0001/inference.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import logging as log
4 | from openvino.inference_engine import IENetwork, IECore
5 |
6 |
7 | class Network:
8 | """
9 | Load and configure inference plugins for the specified target devices
10 | and performs synchronous and asynchronous modes for the specified infer requests.
11 | """
12 |
13 | def __init__(self):
14 | self.ie = None
15 | self.network = None
16 | self.input_blob = None
17 | self.output_blob = None
18 | self.exec_network = None
19 | self.infer_request = None
20 |
21 |
22 | def load_model(self, model, args, device="CPU", num_requests=0):
23 | '''
24 | Load the model given IR files.
25 | Defaults to CPU as device for use in the workspace.
26 | Synchronous requests made within.
27 | '''
28 | model_xml = model
29 | model_bin = os.path.splitext(model_xml)[0] + ".bin"
30 |
31 |
32 | # Initialize the plugin
33 | self.ie = IECore()
34 |
35 | # Read the IR as a IENetwork
36 | try:
37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin)
38 | except Exception as e:
39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?")
40 |
41 | if args.cpu_extension and "CPU" in args.device:
42 | ie.add_extension(args.cpu_extension, "CPU")
43 | log.info("CPU extension loaded: {}".format(args.cpu_extension))
44 |
45 | if "CPU" in args.device:
46 | supported_layers = self.ie.query_network(self.network, "CPU")
47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers]
48 | if len(not_supported_layers) != 0:
49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}".
50 | format(args.device, ', '.join(not_supported_layers)))
51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l "
52 | "or --cpu_extension command line argument")
53 | sys.exit(1)
54 |
55 | # Load the IENetwork into the plugin
56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests)
57 |
58 | def get_input_shape(self):
59 | '''
60 | Gets the input shape of the network
61 | '''
62 | log.info("Preparing input blobs")
63 | input_shape = []
64 | print("inputs number: " + str(len(self.network.input_info.keys())))
65 |
66 | for input_key in self.network.input_info:
67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape))
68 | print("input key: " + input_key)
69 | self.input_blob = input_key
70 | if len(self.network.input_info[input_key].input_data.layout) == 4:
71 | input_shape = self.network.input_info[input_key].input_data.shape
72 |
73 | return input_shape
74 |
75 | def get_output_name(self):
76 | '''
77 | Gets the input shape of the network
78 | '''
79 | log.info('Preparing output blobs')
80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))]
81 | for output_key in self.network.outputs:
82 | if self.network.layers[output_key].type == "DetectionOutput":
83 | self.output_blob, _ = output_key, self.network.outputs[output_key]
84 |
85 | if self.output_blob == "":
86 | log.error("Can't find a DetectionOutput layer in the topology")
87 | exit(-1)
88 | return self.output_blob
89 |
90 | def exec_net(self, image, request_id):
91 | '''
92 | Makes an asynchronous inference request, given an input image.
93 | '''
94 | self.exec_network.start_async(request_id=request_id,
95 | inputs={self.input_blob: image})
96 | return
97 |
98 |
99 | def wait(self, request_id):
100 | '''
101 | Checks the status of the inference request.
102 | '''
103 | status = self.exec_network.requests[request_id].wait(-1)
104 | return status
105 |
106 |
107 | def get_output(self, request_id):
108 | '''
109 | Returns a list of the results for the output layer of the network.
110 | '''
111 | return self.exec_network.requests[request_id].outputs[self.output_blob]
--------------------------------------------------------------------------------
/openvino_analysis/intel/object-detection/pedestrian-and-vehicle-detector-adas-0001/main.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import socket
5 | import json
6 | import cv2
7 | import numpy as np
8 |
9 | import logging as log
10 |
11 | from argparse import ArgumentParser
12 | from inference import Network
13 |
14 | from imutils.video import FPS
15 |
16 | def build_argparser():
17 | """
18 | Parse command line arguments.
19 | :return: command line arguments
20 | """
21 | parser = ArgumentParser()
22 | parser.add_argument("-m", "--model", required=True, type=str,
23 | help="Path to an xml file with a trained model.")
24 | parser.add_argument("-i", "--input", required=True, type=str,
25 | help="Path to image or video file")
26 | parser.add_argument("-d", "--device", type=str, default="CPU",
27 | help="Specify the target device to infer on: "
28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample "
29 | "will look for a suitable plugin for device "
30 | "specified (CPU by default)")
31 | parser.add_argument("-l", "--cpu_extension",
32 | help="Optional. Required for CPU custom layers. "
33 | "Absolute path to a shared library with the kernels implementations.",
34 | type=str, default=None)
35 | return parser
36 |
37 |
38 | def preprocessing(frame, in_w, in_h):
39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA)
40 | image = np.moveaxis(image_resize, -1, 0)
41 |
42 | return image
43 | def infer_on_stream(args):
44 | """
45 | Initialize the inference network, stream video to network,
46 | and output stats and video.
47 | :param args: Command line arguments parsed by `build_argparser()`
48 | :return: None
49 | """
50 | # Initialize the Inference Engine
51 | infer_network = Network()
52 |
53 | # Load the model through `infer_network`
54 | infer_network.load_model(args.model, args, args.device, num_requests=0)
55 |
56 | # Get a Input blob shape
57 | _, _, in_h, in_w = infer_network.get_input_shape()
58 |
59 |
60 | # Get a output blob name
61 | output_name = infer_network.get_output_name()
62 |
63 | # Handle the input stream
64 | try:
65 | cap = cv2.VideoCapture(args.input)
66 | except FileNotFoundError:
67 | print("Cannot locate video file: "+ args.input)
68 | except Exception as e:
69 | print("Something else went wrong with the video file: ", e)
70 |
71 | cap.open(args.input)
72 | _, frame = cap.read()
73 |
74 | fh = frame.shape[0]
75 | fw = frame.shape[1]
76 |
77 | fps = FPS().start()
78 |
79 | # Process frames until the video ends, or process is exited
80 | while cap.isOpened():
81 | # Read the next frame
82 | flag, frame = cap.read()
83 | if not flag:
84 | break
85 |
86 | key_pressed = cv2.waitKey(1)
87 |
88 | image = preprocessing(frame, in_w, in_h)
89 |
90 | # Perform inference on the frame
91 | infer_network.exec_net(image, request_id=0)
92 |
93 | # Get the output of inference
94 | if infer_network.wait(request_id=0) == 0:
95 | result = infer_network.get_output(request_id=0)
96 | for box in result[0][0]: # Output shape is 1x1x100x7
97 | conf = box[2]
98 | if conf >= 0.5:
99 | xmin = int(box[3] * fw)
100 | ymin = int(box[4] * fh)
101 | xmax = int(box[5] * fw)
102 | ymax = int(box[6] * fh)
103 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 0, 255), 3)
104 |
105 | cv2.imshow('frame', frame)
106 | if cv2.waitKey(1) & 0xFF == ord('q'):
107 | break
108 |
109 | #Break if escape key pressed
110 | if key_pressed == 27:
111 | break
112 |
113 | fps.update()
114 |
115 | # Release the out writer, capture, and destroy any OpenCV windows
116 | cap.release()
117 |
118 | cv2.destroyAllWindows()
119 |
120 | fps.stop()
121 |
122 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
123 |
124 | def main():
125 | """
126 | Load the network and parse the output.
127 | :return: None
128 | """
129 | # Set log to INFO
130 | log.basicConfig(level=log.CRITICAL)
131 |
132 | # Grab command line args
133 | args = build_argparser().parse_args()
134 |
135 | # Perform inference on the input stream
136 | infer_on_stream(args)
137 |
138 | if __name__ == '__main__':
139 | main()
--------------------------------------------------------------------------------
/openvino_analysis/intel/object-detection/pedestrian-detection-adas-0002/__pycache__/inference.cpython-36.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/object-detection/pedestrian-detection-adas-0002/__pycache__/inference.cpython-36.pyc
--------------------------------------------------------------------------------
/openvino_analysis/intel/object-detection/pedestrian-detection-adas-0002/inference.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import logging as log
4 | from openvino.inference_engine import IENetwork, IECore
5 |
6 |
7 | class Network:
8 | """
9 | Load and configure inference plugins for the specified target devices
10 | and performs synchronous and asynchronous modes for the specified infer requests.
11 | """
12 |
13 | def __init__(self):
14 | self.ie = None
15 | self.network = None
16 | self.input_blob = None
17 | self.output_blob = None
18 | self.exec_network = None
19 | self.infer_request = None
20 |
21 |
22 | def load_model(self, model, args, device="CPU", num_requests=0):
23 | '''
24 | Load the model given IR files.
25 | Defaults to CPU as device for use in the workspace.
26 | Synchronous requests made within.
27 | '''
28 | model_xml = model
29 | model_bin = os.path.splitext(model_xml)[0] + ".bin"
30 |
31 |
32 | # Initialize the plugin
33 | self.ie = IECore()
34 |
35 | # Read the IR as a IENetwork
36 | try:
37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin)
38 | except Exception as e:
39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?")
40 |
41 | if args.cpu_extension and "CPU" in args.device:
42 | ie.add_extension(args.cpu_extension, "CPU")
43 | log.info("CPU extension loaded: {}".format(args.cpu_extension))
44 |
45 | if "CPU" in args.device:
46 | supported_layers = self.ie.query_network(self.network, "CPU")
47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers]
48 | if len(not_supported_layers) != 0:
49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}".
50 | format(args.device, ', '.join(not_supported_layers)))
51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l "
52 | "or --cpu_extension command line argument")
53 | sys.exit(1)
54 |
55 | # Load the IENetwork into the plugin
56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests)
57 |
58 | def get_input_shape(self):
59 | '''
60 | Gets the input shape of the network
61 | '''
62 | log.info("Preparing input blobs")
63 | input_shape = []
64 | print("inputs number: " + str(len(self.network.input_info.keys())))
65 |
66 | for input_key in self.network.input_info:
67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape))
68 | print("input key: " + input_key)
69 | self.input_blob = input_key
70 | if len(self.network.input_info[input_key].input_data.layout) == 4:
71 | input_shape = self.network.input_info[input_key].input_data.shape
72 |
73 | return input_shape
74 |
75 | def get_output_name(self):
76 | '''
77 | Gets the input shape of the network
78 | '''
79 | log.info('Preparing output blobs')
80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))]
81 | for output_key in self.network.outputs:
82 | if self.network.layers[output_key].type == "DetectionOutput":
83 | self.output_blob, _ = output_key, self.network.outputs[output_key]
84 |
85 | if self.output_blob == "":
86 | log.error("Can't find a DetectionOutput layer in the topology")
87 | exit(-1)
88 | return self.output_blob
89 |
90 | def exec_net(self, image, request_id):
91 | '''
92 | Makes an asynchronous inference request, given an input image.
93 | '''
94 | self.exec_network.start_async(request_id=request_id,
95 | inputs={self.input_blob: image})
96 | return
97 |
98 |
99 | def wait(self, request_id):
100 | '''
101 | Checks the status of the inference request.
102 | '''
103 | status = self.exec_network.requests[request_id].wait(-1)
104 | return status
105 |
106 |
107 | def get_output(self, request_id):
108 | '''
109 | Returns a list of the results for the output layer of the network.
110 | '''
111 | return self.exec_network.requests[request_id].outputs[self.output_blob]
--------------------------------------------------------------------------------
/openvino_analysis/intel/object-detection/pedestrian-detection-adas-0002/main.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import socket
5 | import json
6 | import cv2
7 | import numpy as np
8 |
9 | import logging as log
10 |
11 | from argparse import ArgumentParser
12 | from inference import Network
13 |
14 | from imutils.video import FPS
15 |
16 | def build_argparser():
17 | """
18 | Parse command line arguments.
19 | :return: command line arguments
20 | """
21 | parser = ArgumentParser()
22 | parser.add_argument("-m", "--model", required=True, type=str,
23 | help="Path to an xml file with a trained model.")
24 | parser.add_argument("-i", "--input", required=True, type=str,
25 | help="Path to image or video file")
26 | parser.add_argument("-d", "--device", type=str, default="CPU",
27 | help="Specify the target device to infer on: "
28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample "
29 | "will look for a suitable plugin for device "
30 | "specified (CPU by default)")
31 | parser.add_argument("-l", "--cpu_extension",
32 | help="Optional. Required for CPU custom layers. "
33 | "Absolute path to a shared library with the kernels implementations.",
34 | type=str, default=None)
35 | return parser
36 |
37 |
38 | def preprocessing(frame, in_w, in_h):
39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA)
40 | image = np.moveaxis(image_resize, -1, 0)
41 |
42 | return image
43 | def infer_on_stream(args):
44 | """
45 | Initialize the inference network, stream video to network,
46 | and output stats and video.
47 | :param args: Command line arguments parsed by `build_argparser()`
48 | :return: None
49 | """
50 | # Initialize the Inference Engine
51 | infer_network = Network()
52 |
53 | # Load the model through `infer_network`
54 | infer_network.load_model(args.model, args, args.device, num_requests=0)
55 |
56 | # Get a Input blob shape
57 | _, _, in_h, in_w = infer_network.get_input_shape()
58 |
59 |
60 | # Get a output blob name
61 | output_name = infer_network.get_output_name()
62 |
63 | # Handle the input stream
64 | try:
65 | cap = cv2.VideoCapture(args.input)
66 | except FileNotFoundError:
67 | print("Cannot locate video file: "+ args.input)
68 | except Exception as e:
69 | print("Something else went wrong with the video file: ", e)
70 |
71 | cap.open(args.input)
72 | _, frame = cap.read()
73 |
74 | fh = frame.shape[0]
75 | fw = frame.shape[1]
76 |
77 | fps = FPS().start()
78 |
79 | # Process frames until the video ends, or process is exited
80 | while cap.isOpened():
81 | # Read the next frame
82 | flag, frame = cap.read()
83 | if not flag:
84 | break
85 |
86 | key_pressed = cv2.waitKey(1)
87 |
88 | image = preprocessing(frame, in_w, in_h)
89 |
90 | # Perform inference on the frame
91 | infer_network.exec_net(image, request_id=0)
92 |
93 | # Get the output of inference
94 | if infer_network.wait(request_id=0) == 0:
95 | result = infer_network.get_output(request_id=0)
96 | for box in result[0][0]: # Output shape is 1x1x100x7
97 | conf = box[2]
98 | if conf >= 0.5:
99 | xmin = int(box[3] * fw)
100 | ymin = int(box[4] * fh)
101 | xmax = int(box[5] * fw)
102 | ymax = int(box[6] * fh)
103 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 0, 255), 3)
104 |
105 | cv2.imshow('frame', frame)
106 | if cv2.waitKey(1) & 0xFF == ord('q'):
107 | break
108 |
109 | #Break if escape key pressed
110 | if key_pressed == 27:
111 | break
112 |
113 | fps.update()
114 |
115 | # Release the out writer, capture, and destroy any OpenCV windows
116 | cap.release()
117 |
118 | cv2.destroyAllWindows()
119 |
120 | fps.stop()
121 |
122 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
123 |
124 | def main():
125 | """
126 | Load the network and parse the output.
127 | :return: None
128 | """
129 | # Set log to INFO
130 | log.basicConfig(level=log.CRITICAL)
131 |
132 | # Grab command line args
133 | args = build_argparser().parse_args()
134 |
135 | # Perform inference on the input stream
136 | infer_on_stream(args)
137 |
138 | if __name__ == '__main__':
139 | main()
--------------------------------------------------------------------------------
/openvino_analysis/intel/object-detection/pedestrian-detection-adas-binary-0001/inference.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import logging as log
4 | from openvino.inference_engine import IENetwork, IECore
5 |
6 |
7 | class Network:
8 | """
9 | Load and configure inference plugins for the specified target devices
10 | and performs synchronous and asynchronous modes for the specified infer requests.
11 | """
12 |
13 | def __init__(self):
14 | self.ie = None
15 | self.network = None
16 | self.input_blob = None
17 | self.output_blob = None
18 | self.exec_network = None
19 | self.infer_request = None
20 |
21 |
22 | def load_model(self, model, args, device="CPU", num_requests=0):
23 | '''
24 | Load the model given IR files.
25 | Defaults to CPU as device for use in the workspace.
26 | Synchronous requests made within.
27 | '''
28 | model_xml = model
29 | model_bin = os.path.splitext(model_xml)[0] + ".bin"
30 |
31 |
32 | # Initialize the plugin
33 | self.ie = IECore()
34 |
35 | # Read the IR as a IENetwork
36 | try:
37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin)
38 | except Exception as e:
39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?")
40 |
41 | if args.cpu_extension and "CPU" in args.device:
42 | ie.add_extension(args.cpu_extension, "CPU")
43 | log.info("CPU extension loaded: {}".format(args.cpu_extension))
44 |
45 | if "CPU" in args.device:
46 | supported_layers = self.ie.query_network(self.network, "CPU")
47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers]
48 | if len(not_supported_layers) != 0:
49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}".
50 | format(args.device, ', '.join(not_supported_layers)))
51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l "
52 | "or --cpu_extension command line argument")
53 | sys.exit(1)
54 |
55 | # Load the IENetwork into the plugin
56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests)
57 |
58 | def get_input_shape(self):
59 | '''
60 | Gets the input shape of the network
61 | '''
62 | log.info("Preparing input blobs")
63 | input_shape = []
64 | print("inputs number: " + str(len(self.network.input_info.keys())))
65 |
66 | for input_key in self.network.input_info:
67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape))
68 | print("input key: " + input_key)
69 | self.input_blob = input_key
70 | if len(self.network.input_info[input_key].input_data.layout) == 4:
71 | input_shape = self.network.input_info[input_key].input_data.shape
72 |
73 | return input_shape
74 |
75 | def get_output_name(self):
76 | '''
77 | Gets the input shape of the network
78 | '''
79 | log.info('Preparing output blobs')
80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))]
81 | for output_key in self.network.outputs:
82 | if self.network.layers[output_key].type == "DetectionOutput":
83 | self.output_blob, _ = output_key, self.network.outputs[output_key]
84 |
85 | if self.output_blob == "":
86 | log.error("Can't find a DetectionOutput layer in the topology")
87 | exit(-1)
88 | return self.output_blob
89 |
90 | def exec_net(self, image, request_id):
91 | '''
92 | Makes an asynchronous inference request, given an input image.
93 | '''
94 | self.exec_network.start_async(request_id=request_id,
95 | inputs={self.input_blob: image})
96 | return
97 |
98 |
99 | def wait(self, request_id):
100 | '''
101 | Checks the status of the inference request.
102 | '''
103 | status = self.exec_network.requests[request_id].wait(-1)
104 | return status
105 |
106 |
107 | def get_output(self, request_id):
108 | '''
109 | Returns a list of the results for the output layer of the network.
110 | '''
111 | return self.exec_network.requests[request_id].outputs[self.output_blob]
--------------------------------------------------------------------------------
/openvino_analysis/intel/object-detection/pedestrian-detection-adas-binary-0001/main.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import socket
5 | import json
6 | import cv2
7 | import numpy as np
8 |
9 | import logging as log
10 |
11 | from argparse import ArgumentParser
12 | from inference import Network
13 |
14 | from imutils.video import FPS
15 |
16 | def build_argparser():
17 | """
18 | Parse command line arguments.
19 | :return: command line arguments
20 | """
21 | parser = ArgumentParser()
22 | parser.add_argument("-m", "--model", required=True, type=str,
23 | help="Path to an xml file with a trained model.")
24 | parser.add_argument("-i", "--input", required=True, type=str,
25 | help="Path to image or video file")
26 | parser.add_argument("-d", "--device", type=str, default="CPU",
27 | help="Specify the target device to infer on: "
28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample "
29 | "will look for a suitable plugin for device "
30 | "specified (CPU by default)")
31 | parser.add_argument("-l", "--cpu_extension",
32 | help="Optional. Required for CPU custom layers. "
33 | "Absolute path to a shared library with the kernels implementations.",
34 | type=str, default=None)
35 | return parser
36 |
37 |
38 | def preprocessing(frame, in_w, in_h):
39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA)
40 | image = np.moveaxis(image_resize, -1, 0)
41 |
42 | return image
43 | def infer_on_stream(args):
44 | """
45 | Initialize the inference network, stream video to network,
46 | and output stats and video.
47 | :param args: Command line arguments parsed by `build_argparser()`
48 | :return: None
49 | """
50 | # Initialize the Inference Engine
51 | infer_network = Network()
52 |
53 | # Load the model through `infer_network`
54 | infer_network.load_model(args.model, args, args.device, num_requests=0)
55 |
56 | # Get a Input blob shape
57 | _, _, in_h, in_w = infer_network.get_input_shape()
58 |
59 |
60 | # Get a output blob name
61 | output_name = infer_network.get_output_name()
62 |
63 | # Handle the input stream
64 | try:
65 | cap = cv2.VideoCapture(args.input)
66 | except FileNotFoundError:
67 | print("Cannot locate video file: "+ args.input)
68 | except Exception as e:
69 | print("Something else went wrong with the video file: ", e)
70 |
71 | cap.open(args.input)
72 | _, frame = cap.read()
73 |
74 | fh = frame.shape[0]
75 | fw = frame.shape[1]
76 |
77 | fps = FPS().start()
78 |
79 | # Process frames until the video ends, or process is exited
80 | while cap.isOpened():
81 | # Read the next frame
82 | flag, frame = cap.read()
83 | if not flag:
84 | break
85 |
86 | key_pressed = cv2.waitKey(1)
87 |
88 | image = preprocessing(frame, in_w, in_h)
89 |
90 | # Perform inference on the frame
91 | infer_network.exec_net(image, request_id=0)
92 |
93 | # Get the output of inference
94 | if infer_network.wait(request_id=0) == 0:
95 | result = infer_network.get_output(request_id=0)
96 | for box in result[0][0]: # Output shape is 1x1x100x7
97 | conf = box[2]
98 | if conf >= 0.5:
99 | xmin = int(box[3] * fw)
100 | ymin = int(box[4] * fh)
101 | xmax = int(box[5] * fw)
102 | ymax = int(box[6] * fh)
103 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 0, 255), 3)
104 |
105 | cv2.imshow('frame', frame)
106 | if cv2.waitKey(1) & 0xFF == ord('q'):
107 | break
108 |
109 | #Break if escape key pressed
110 | if key_pressed == 27:
111 | break
112 |
113 | fps.update()
114 |
115 | # Release the out writer, capture, and destroy any OpenCV windows
116 | cap.release()
117 |
118 | cv2.destroyAllWindows()
119 |
120 | fps.stop()
121 |
122 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
123 |
124 | def main():
125 | """
126 | Load the network and parse the output.
127 | :return: None
128 | """
129 | # Set log to INFO
130 | log.basicConfig(level=log.CRITICAL)
131 |
132 | # Grab command line args
133 | args = build_argparser().parse_args()
134 |
135 | # Perform inference on the input stream
136 | infer_on_stream(args)
137 |
138 | if __name__ == '__main__':
139 | main()
--------------------------------------------------------------------------------
/openvino_analysis/intel/object-detection/vehicle-detection-adas/inference.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import logging as log
4 | from openvino.inference_engine import IENetwork, IECore
5 |
6 |
7 | class Network:
8 | """
9 | Load and configure inference plugins for the specified target devices
10 | and performs synchronous and asynchronous modes for the specified infer requests.
11 | """
12 |
13 | def __init__(self):
14 | self.ie = None
15 | self.network = None
16 | self.input_blob = None
17 | self.output_blob = None
18 | self.exec_network = None
19 | self.infer_request = None
20 |
21 |
22 | def load_model(self, model, args, device="CPU", num_requests=0):
23 | '''
24 | Load the model given IR files.
25 | Defaults to CPU as device for use in the workspace.
26 | Synchronous requests made within.
27 | '''
28 | model_xml = model
29 | model_bin = os.path.splitext(model_xml)[0] + ".bin"
30 |
31 |
32 | # Initialize the plugin
33 | self.ie = IECore()
34 |
35 | # Read the IR as a IENetwork
36 | try:
37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin)
38 | except Exception as e:
39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?")
40 |
41 | if args.cpu_extension and "CPU" in args.device:
42 | ie.add_extension(args.cpu_extension, "CPU")
43 | log.info("CPU extension loaded: {}".format(args.cpu_extension))
44 |
45 | if "CPU" in args.device:
46 | supported_layers = self.ie.query_network(self.network, "CPU")
47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers]
48 | if len(not_supported_layers) != 0:
49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}".
50 | format(args.device, ', '.join(not_supported_layers)))
51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l "
52 | "or --cpu_extension command line argument")
53 | sys.exit(1)
54 |
55 | # Load the IENetwork into the plugin
56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests)
57 |
58 | def get_input_shape(self):
59 | '''
60 | Gets the input shape of the network
61 | '''
62 | log.info("Preparing input blobs")
63 | input_shape = []
64 | print("inputs number: " + str(len(self.network.input_info.keys())))
65 |
66 | for input_key in self.network.input_info:
67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape))
68 | print("input key: " + input_key)
69 | self.input_blob = input_key
70 | if len(self.network.input_info[input_key].input_data.layout) == 4:
71 | input_shape = self.network.input_info[input_key].input_data.shape
72 |
73 | return input_shape
74 |
75 | def get_output_name(self):
76 | '''
77 | Gets the input shape of the network
78 | '''
79 | log.info('Preparing output blobs')
80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))]
81 | for output_key in self.network.outputs:
82 | if self.network.layers[output_key].type == "DetectionOutput":
83 | self.output_blob, _ = output_key, self.network.outputs[output_key]
84 |
85 | if self.output_blob == "":
86 | log.error("Can't find a DetectionOutput layer in the topology")
87 | exit(-1)
88 | return self.output_blob
89 |
90 | def exec_net(self, image, request_id):
91 | '''
92 | Makes an asynchronous inference request, given an input image.
93 | '''
94 | self.exec_network.start_async(request_id=request_id,
95 | inputs={self.input_blob: image})
96 | return
97 |
98 |
99 | def wait(self, request_id):
100 | '''
101 | Checks the status of the inference request.
102 | '''
103 | status = self.exec_network.requests[request_id].wait(-1)
104 | return status
105 |
106 |
107 | def get_output(self, request_id):
108 | '''
109 | Returns a list of the results for the output layer of the network.
110 | '''
111 | return self.exec_network.requests[request_id].outputs[self.output_blob]
--------------------------------------------------------------------------------
/openvino_analysis/intel/object-detection/vehicle-detection-adas/main.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import socket
5 | import json
6 | import cv2
7 | import numpy as np
8 |
9 | import logging as log
10 |
11 | from argparse import ArgumentParser
12 | from inference import Network
13 |
14 | from imutils.video import FPS
15 |
16 | def build_argparser():
17 | """
18 | Parse command line arguments.
19 | :return: command line arguments
20 | """
21 | parser = ArgumentParser()
22 | parser.add_argument("-m", "--model", required=True, type=str,
23 | help="Path to an xml file with a trained model.")
24 | parser.add_argument("-i", "--input", required=True, type=str,
25 | help="Path to image or video file")
26 | parser.add_argument("-d", "--device", type=str, default="CPU",
27 | help="Specify the target device to infer on: "
28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample "
29 | "will look for a suitable plugin for device "
30 | "specified (CPU by default)")
31 | parser.add_argument("-l", "--cpu_extension",
32 | help="Optional. Required for CPU custom layers. "
33 | "Absolute path to a shared library with the kernels implementations.",
34 | type=str, default=None)
35 | return parser
36 |
37 |
38 | def preprocessing(frame, in_w, in_h):
39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA)
40 | image = np.moveaxis(image_resize, -1, 0)
41 |
42 | return image
43 | def infer_on_stream(args):
44 | """
45 | Initialize the inference network, stream video to network,
46 | and output stats and video.
47 | :param args: Command line arguments parsed by `build_argparser()`
48 | :return: None
49 | """
50 | # Initialize the Inference Engine
51 | infer_network = Network()
52 |
53 | # Load the model through `infer_network`
54 | infer_network.load_model(args.model, args, args.device, num_requests=0)
55 |
56 | # Get a Input blob shape
57 | _, _, in_h, in_w = infer_network.get_input_shape()
58 |
59 |
60 | # Get a output blob name
61 | output_name = infer_network.get_output_name()
62 |
63 | # Handle the input stream
64 | try:
65 | cap = cv2.VideoCapture(args.input)
66 | except FileNotFoundError:
67 | print("Cannot locate video file: "+ args.input)
68 | except Exception as e:
69 | print("Something else went wrong with the video file: ", e)
70 |
71 | cap.open(args.input)
72 | _, frame = cap.read()
73 |
74 | fh = frame.shape[0]
75 | fw = frame.shape[1]
76 |
77 | fps = FPS().start()
78 |
79 | # Process frames until the video ends, or process is exited
80 | while cap.isOpened():
81 | # Read the next frame
82 | flag, frame = cap.read()
83 | if not flag:
84 | break
85 |
86 | key_pressed = cv2.waitKey(1)
87 |
88 | image = preprocessing(frame, in_w, in_h)
89 |
90 | # Perform inference on the frame
91 | infer_network.exec_net(image, request_id=0)
92 |
93 | # Get the output of inference
94 | if infer_network.wait(request_id=0) == 0:
95 | result = infer_network.get_output(request_id=0)
96 | for box in result[0][0]: # Output shape is 1x1x100x7
97 | conf = box[2]
98 | if conf >= 0.5:
99 | xmin = int(box[3] * fw)
100 | ymin = int(box[4] * fh)
101 | xmax = int(box[5] * fw)
102 | ymax = int(box[6] * fh)
103 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 0, 255), 3)
104 |
105 | cv2.imshow('frame', frame)
106 | if cv2.waitKey(1) & 0xFF == ord('q'):
107 | break
108 |
109 | #Break if escape key pressed
110 | if key_pressed == 27:
111 | break
112 |
113 | fps.update()
114 |
115 | # Release the out writer, capture, and destroy any OpenCV windows
116 | cap.release()
117 |
118 | cv2.destroyAllWindows()
119 |
120 | fps.stop()
121 |
122 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
123 |
124 | def main():
125 | """
126 | Load the network and parse the output.
127 | :return: None
128 | """
129 | # Set log to INFO
130 | log.basicConfig(level=log.CRITICAL)
131 |
132 | # Grab command line args
133 | args = build_argparser().parse_args()
134 |
135 | # Perform inference on the input stream
136 | infer_on_stream(args)
137 |
138 | if __name__ == '__main__':
139 | main()
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/road-segmentation-adas-0001/__pycache__/inference.cpython-36.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/semantic-segmentation/road-segmentation-adas-0001/__pycache__/inference.cpython-36.pyc
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/road-segmentation-adas-0001/inference.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import logging as log
4 | from openvino.inference_engine import IENetwork, IECore
5 |
6 |
7 | class Network:
8 | """
9 | Load and configure inference plugins for the specified target devices
10 | and performs synchronous and asynchronous modes for the specified infer requests.
11 | """
12 |
13 | def __init__(self):
14 | self.ie = None
15 | self.network = None
16 | self.input_blob = None
17 | self.output_blob = None
18 | self.exec_network = None
19 | self.infer_request = None
20 |
21 |
22 | def load_model(self, model, args, device="CPU", num_requests=0):
23 | '''
24 | Load the model given IR files.
25 | Defaults to CPU as device for use in the workspace.
26 | Synchronous requests made within.
27 | '''
28 | model_xml = model
29 | model_bin = os.path.splitext(model_xml)[0] + ".bin"
30 |
31 |
32 | # Initialize the plugin
33 | self.ie = IECore()
34 |
35 | # Read the IR as a IENetwork
36 | try:
37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin)
38 | except Exception as e:
39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?")
40 |
41 | if args.cpu_extension and "CPU" in args.device:
42 | ie.add_extension(args.cpu_extension, "CPU")
43 | log.info("CPU extension loaded: {}".format(args.cpu_extension))
44 |
45 | if "CPU" in args.device:
46 | supported_layers = self.ie.query_network(self.network, "CPU")
47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers]
48 | if len(not_supported_layers) != 0:
49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}".
50 | format(args.device, ', '.join(not_supported_layers)))
51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l "
52 | "or --cpu_extension command line argument")
53 | sys.exit(1)
54 |
55 | # Load the IENetwork into the plugin
56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests)
57 |
58 | def get_input_shape(self):
59 | '''
60 | Gets the input shape of the network
61 | '''
62 | log.info("Preparing input blobs")
63 | input_shape = []
64 | print("inputs number: " + str(len(self.network.input_info.keys())))
65 |
66 | for input_key in self.network.input_info:
67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape))
68 | print("input key: " + input_key)
69 | self.input_blob = input_key
70 | if len(self.network.input_info[input_key].input_data.layout) == 4:
71 | input_shape = self.network.input_info[input_key].input_data.shape
72 |
73 | return input_shape
74 |
75 | def get_output_name(self):
76 | '''
77 | Gets the input shape of the network
78 | '''
79 | log.info('Preparing output blobs')
80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))]
81 | for output_key in self.network.outputs:
82 | if self.network.layers[output_key].type == "SoftMax":
83 | self.output_blob, _ = output_key, self.network.outputs[output_key]
84 |
85 | if self.output_blob == "":
86 | log.error("Can't find a L0317_ReWeight_SoftMax layer in the topology")
87 | exit(-1)
88 | return self.output_blob
89 |
90 | def exec_net(self, image, request_id):
91 | '''
92 | Makes an asynchronous inference request, given an input image.
93 | '''
94 | self.exec_network.start_async(request_id=request_id,
95 | inputs={self.input_blob: image})
96 | return
97 |
98 |
99 | def wait(self, request_id):
100 | '''
101 | Checks the status of the inference request.
102 | '''
103 | status = self.exec_network.requests[request_id].wait(-1)
104 | return status
105 |
106 |
107 | def get_output(self, request_id):
108 | '''
109 | Returns a list of the results for the output layer of the network.
110 | '''
111 | return self.exec_network.requests[request_id].outputs[self.output_blob]
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/road-segmentation-adas-0001/main.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import socket
5 | import json
6 | import cv2
7 | import numpy as np
8 |
9 | import logging as log
10 |
11 | from argparse import ArgumentParser
12 | from inference import Network
13 |
14 | from imutils.video import FPS
15 |
16 | def build_argparser():
17 | """
18 | Parse command line arguments.
19 | :return: command line arguments
20 | """
21 | parser = ArgumentParser()
22 | parser.add_argument("-m", "--model", required=True, type=str,
23 | help="Path to an xml file with a trained model.")
24 | parser.add_argument("-i", "--input", required=True, type=str,
25 | help="Path to image or video file")
26 | parser.add_argument("-d", "--device", type=str, default="CPU",
27 | help="Specify the target device to infer on: "
28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample "
29 | "will look for a suitable plugin for device "
30 | "specified (CPU by default)")
31 | parser.add_argument("-l", "--cpu_extension",
32 | help="Optional. Required for CPU custom layers. "
33 | "Absolute path to a shared library with the kernels implementations.",
34 | type=str, default=None)
35 | return parser
36 |
37 |
38 | def preprocessing(frame, in_w, in_h):
39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA)
40 | image = np.moveaxis(image_resize, -1, 0)
41 |
42 | return image
43 | def infer_on_stream(args):
44 | """
45 | Initialize the inference network, stream video to network,
46 | and output stats and video.
47 | :param args: Command line arguments parsed by `build_argparser()`
48 | :return: None
49 | """
50 | # Initialize the Inference Engine
51 | infer_network = Network()
52 |
53 | # Load the model through `infer_network`
54 | infer_network.load_model(args.model, args, args.device, num_requests=0)
55 |
56 | # Get a Input blob shape
57 | _, _, in_h, in_w = infer_network.get_input_shape()
58 |
59 |
60 | # Get a output blob name
61 | output_name = infer_network.get_output_name()
62 |
63 | # Handle the input stream
64 | try:
65 | cap = cv2.VideoCapture(args.input)
66 | except FileNotFoundError:
67 | print("Cannot locate video file: "+ args.input)
68 | except Exception as e:
69 | print("Something else went wrong with the video file: ", e)
70 |
71 | cap.open(args.input)
72 | _, frame = cap.read()
73 |
74 | fh = frame.shape[0]
75 | fw = frame.shape[1]
76 |
77 | fps = FPS().start()
78 |
79 | # Process frames until the video ends, or process is exited
80 | while cap.isOpened():
81 | # Read the next frame
82 | flag, frame = cap.read()
83 | if not flag:
84 | break
85 |
86 | key_pressed = cv2.waitKey(1)
87 |
88 | image = preprocessing(frame, in_w, in_h)
89 |
90 | # Perform inference on the frame
91 | infer_network.exec_net(image, request_id=0)
92 |
93 | # Get the output of inference
94 | if infer_network.wait(request_id=0) == 0:
95 | result = infer_network.get_output(request_id=0)
96 | print(result.shape)
97 |
98 | # Analyze road segmentation results
99 | result = np.squeeze(result, 0)
100 | result = result . transpose ( 1 , 2 , 0 ) # HWC
101 | result = np.argmax(result, 2)
102 |
103 | hh, ww = result.shape
104 | mask = np.zeros((hh, ww, 3), dtype=np.uint8)
105 | mask[np.where(result > 0)] = (0, 255, 255) # yellow
106 | mask[np.where(result > 1)] = (255, 0, 255) #
107 |
108 | #cv2.imshow("mask", mask)
109 |
110 | # Superimpose output results
111 | mask = cv2.resize(mask, dsize=(frame.shape[1], frame.shape[0]))
112 | frame = cv2.addWeighted(mask, 0.2, frame, 0.8, 0)
113 |
114 |
115 | cv2.imshow('frame', frame)
116 | if cv2.waitKey(1) & 0xFF == ord('q'):
117 | break
118 |
119 | #Break if escape key pressed
120 | if key_pressed == 27:
121 | break
122 |
123 | fps.update()
124 |
125 | # Release the out writer, capture, and destroy any OpenCV windows
126 | cap.release()
127 |
128 | cv2.destroyAllWindows()
129 |
130 | fps.stop()
131 |
132 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
133 |
134 | def main():
135 | """
136 | Load the network and parse the output.
137 | :return: None
138 | """
139 | # Set log to INFO
140 | log.basicConfig(level=log.CRITICAL)
141 |
142 | # Grab command line args
143 | args = build_argparser().parse_args()
144 |
145 | # Perform inference on the input stream
146 | infer_on_stream(args)
147 |
148 | if __name__ == '__main__':
149 | main()
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/009649.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/009649.png
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/__pycache__/inference.cpython-36.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/__pycache__/inference.cpython-36.pyc
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/inference.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import logging as log
4 | from openvino.inference_engine import IENetwork, IECore
5 |
6 |
7 | class Network:
8 | """
9 | Load and configure inference plugins for the specified target devices
10 | and performs synchronous and asynchronous modes for the specified infer requests.
11 | """
12 |
13 | def __init__(self):
14 | self.ie = None
15 | self.network = None
16 | self.input_blob = None
17 | self.output_blob = None
18 | self.exec_network = None
19 | self.infer_request = None
20 |
21 |
22 | def load_model(self, model, args, device="CPU", num_requests=0):
23 | '''
24 | Load the model given IR files.
25 | Defaults to CPU as device for use in the workspace.
26 | Synchronous requests made within.
27 | '''
28 | model_xml = model
29 | model_bin = os.path.splitext(model_xml)[0] + ".bin"
30 |
31 |
32 | # Initialize the plugin
33 | self.ie = IECore()
34 |
35 | # Read the IR as a IENetwork
36 | try:
37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin)
38 | except Exception as e:
39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?")
40 |
41 | if args.cpu_extension and "CPU" in args.device:
42 | ie.add_extension(args.cpu_extension, "CPU")
43 | log.info("CPU extension loaded: {}".format(args.cpu_extension))
44 |
45 | if "CPU" in args.device:
46 | supported_layers = self.ie.query_network(self.network, "CPU")
47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers]
48 | if len(not_supported_layers) != 0:
49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}".
50 | format(args.device, ', '.join(not_supported_layers)))
51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l "
52 | "or --cpu_extension command line argument")
53 | sys.exit(1)
54 |
55 | # Load the IENetwork into the plugin
56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests)
57 |
58 | def get_input_shape(self):
59 | '''
60 | Gets the input shape of the network
61 | '''
62 | log.info("Preparing input blobs")
63 | input_shape = []
64 | print("inputs number: " + str(len(self.network.input_info.keys())))
65 |
66 | for input_key in self.network.input_info:
67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape))
68 | print("input key: " + input_key)
69 | self.input_blob = input_key
70 | if len(self.network.input_info[input_key].input_data.layout) == 4:
71 | input_shape = self.network.input_info[input_key].input_data.shape
72 |
73 | return input_shape
74 |
75 | def get_output_name(self):
76 | '''
77 | Gets the input shape of the network
78 | '''
79 | log.info('Preparing output blobs')
80 |
81 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))]
82 | for output_key in self.network.outputs:
83 | if self.network.layers[output_key].type == "Convert":
84 | self.output_blob, _ = output_key, self.network.outputs[output_key]
85 |
86 | if self.output_blob == "":
87 | log.error("Can't find a L0317_ReWeight_SoftMax layer in the topology")
88 | exit(-1)
89 | return self.output_blob
90 |
91 | def exec_net(self, image, request_id):
92 | '''
93 | Makes an asynchronous inference request, given an input image.
94 | '''
95 | self.exec_network.start_async(request_id=request_id,
96 | inputs={self.input_blob: image})
97 | return
98 |
99 |
100 | def wait(self, request_id):
101 | '''
102 | Checks the status of the inference request.
103 | '''
104 | status = self.exec_network.requests[request_id].wait(-1)
105 | return status
106 |
107 |
108 | def get_output(self, request_id):
109 | '''
110 | Returns a list of the results for the output layer of the network.
111 | '''
112 | return self.exec_network.requests[request_id].outputs[self.output_blob]
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/async/main.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import socket
5 | import json
6 | import cv2
7 | import numpy as np
8 |
9 | import logging as log
10 |
11 | from argparse import ArgumentParser
12 | from inference import Network
13 | from PIL import Image
14 |
15 | from imutils.video import FPS
16 |
17 | def build_argparser():
18 | """
19 | Parse command line arguments.
20 | :return: command line arguments
21 | """
22 | parser = ArgumentParser()
23 | parser.add_argument("-m", "--model", required=True, type=str,
24 | help="Path to an xml file with a trained model.")
25 | parser.add_argument("-i", "--input", required=True, type=str,
26 | help="Path to image or video file")
27 | parser.add_argument("-d", "--device", type=str, default="CPU",
28 | help="Specify the target device to infer on: "
29 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample "
30 | "will look for a suitable plugin for device "
31 | "specified (CPU by default)")
32 | parser.add_argument("-l", "--cpu_extension",
33 | help="Optional. Required for CPU custom layers. "
34 | "Absolute path to a shared library with the kernels implementations.",
35 | type=str, default=None)
36 | return parser
37 |
38 |
39 | def preprocessing(frame, in_w, in_h):
40 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA)
41 | image = np.moveaxis(image_resize, -1, 0)
42 |
43 | return image
44 | def infer_on_stream(args):
45 | """
46 | Initialize the inference network, stream video to network,
47 | and output stats and video.
48 | :param args: Command line arguments parsed by `build_argparser()`
49 | :return: None
50 | """
51 | # Initialize the Inference Engine
52 | infer_network = Network()
53 |
54 | # Load the model through `infer_network`
55 | infer_network.load_model(args.model, args, args.device, num_requests=0)
56 |
57 | # Get a Input blob shape
58 | _, _, in_h, in_w = infer_network.get_input_shape()
59 |
60 |
61 | # Get a output blob name
62 | output_name = infer_network.get_output_name()
63 |
64 | # Handle the input stream
65 | try:
66 | cap = cv2.VideoCapture(args.input)
67 | except FileNotFoundError:
68 | print("Cannot locate video file: "+ args.input)
69 | except Exception as e:
70 | print("Something else went wrong with the video file: ", e)
71 |
72 | cap.open(args.input)
73 | _, frame = cap.read()
74 |
75 | fh = frame.shape[0]
76 | fw = frame.shape[1]
77 |
78 | fps = FPS().start()
79 |
80 | # Process frames until the video ends, or process is exited
81 | while cap.isOpened():
82 | # Read the next frame
83 | flag, frame = cap.read()
84 | if not flag:
85 | break
86 |
87 | key_pressed = cv2.waitKey(1)
88 |
89 | image = preprocessing(frame, in_w, in_h)
90 |
91 | # Perform inference on the frame
92 | infer_network.exec_net(image, request_id=0)
93 |
94 | seg_image = Image.open("./009649.png")
95 | palette = seg_image.getpalette() # Get a color palette
96 | # Get the output of inference
97 | if infer_network.wait(request_id=0) == 0:
98 | result = infer_network.get_output(request_id=0)
99 | output = result[0][0]
100 | output = np.array(output, dtype='uint8')
101 | output = cv2.resize(output, (fw, fh))
102 |
103 | image = Image.fromarray(np.uint8(output), mode="P")
104 | image.putpalette(palette)
105 | image = image.convert("RGB")
106 |
107 | image = np.asarray(image)
108 | image = cv2.addWeighted(frame, 1, image, 0.9, 0)
109 |
110 | cv2.imshow('frame', image)
111 | if cv2.waitKey(1) & 0xFF == ord('q'):
112 | break
113 |
114 | #Break if escape key pressed
115 | if key_pressed == 27:
116 | break
117 |
118 | fps.update()
119 |
120 | # Release the out writer, capture, and destroy any OpenCV windows
121 | cap.release()
122 |
123 | cv2.destroyAllWindows()
124 |
125 | fps.stop()
126 |
127 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
128 |
129 | def main():
130 | """
131 | Load the network and parse the output.
132 | :return: None
133 | """
134 | # Set log to INFO
135 | log.basicConfig(level=log.CRITICAL)
136 |
137 | # Grab command line args
138 | args = build_argparser().parse_args()
139 |
140 | # Perform inference on the input stream
141 | infer_on_stream(args)
142 |
143 | if __name__ == '__main__':
144 | main()
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/core/__pycache__/semantic_detection.cpython-36.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/core/__pycache__/semantic_detection.cpython-36.pyc
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/core/semantic_detection.py:
--------------------------------------------------------------------------------
1 | import numpy as np
2 | import cv2
3 | from PIL import Image
4 | import matplotlib.pyplot as plt
5 |
6 | from numpy import clip
7 | from utils.ie_module import Module
8 |
9 | class SemanticSegmentation(Module):
10 | def __init__(self, model):
11 | super(SemanticSegmentation, self).__init__(model)
12 |
13 | self.input_blob = next(iter(model.inputs))
14 | self.input_name = next(iter(model.input_info))
15 |
16 | self.output_blob = next(iter(model.outputs))
17 | self.input_shape = model.inputs[self.input_blob].shape
18 | self.output_shape = model.outputs[self.output_blob].shape
19 |
20 | print("Input Shape: {}".format(self.input_shape))
21 | print("Output Shape: {}".format(self.output_shape))
22 |
23 | def resize_input(self, frame, target_shape):
24 | assert len(frame.shape) == len(target_shape), \
25 | "Expected a frame with %s dimensions, but got %s" % \
26 | (len(target_shape), len(frame.shape))
27 |
28 | assert frame.shape[0] == 1, "Only batch size 1 is supported"
29 | n, c, h, w = target_shape
30 |
31 | input = frame[0]
32 | if not np.array_equal(target_shape[-2:], frame.shape[-2:]):
33 | input = input.transpose((1, 2, 0)) # to HWC
34 | input = cv2.resize(input, (w, h))
35 | input = input.transpose((2, 0, 1)) # to CHW
36 |
37 | return input.reshape((n, c, h, w))
38 |
39 | def preprocess(self, frame):
40 | assert len(frame.shape) == 4, "Frame shape should be [1, c, h, w]"
41 | assert frame.shape[0] == 1
42 | assert frame.shape[1] == 3
43 | input = self.resize_input(frame, self.input_shape)
44 | return input
45 |
46 | def draw_masks(self, result, width, height):
47 | '''
48 | Draw semantic mask classes onto the frame.
49 | '''
50 | # Create a mask with color by class
51 | classes = cv2.resize(result[0].transpose((1, 2, 0)), (width, height),
52 | interpolation=cv2.INTER_NEAREST)
53 | unique_classes = np.unique(classes)
54 | out_mask = classes * (255 / 20)
55 |
56 | # Stack the mask so FFmpeg understands it
57 | out_mask = np.dstack((out_mask, out_mask, out_mask))
58 | out_mask = np.uint8(out_mask)
59 |
60 | return out_mask, unique_classes
61 |
62 | def post_processing(self, frame):
63 | print(frame.shape)
64 | H, W, C = frame.shape
65 | seg_image = Image.open("../../../demo/009649.png")
66 | palette = seg_image.getpalette() # Get a color palette
67 | outputs = self.get_outputs()[0][self.output_blob]
68 | print(outputs.shape)
69 | output = outputs[0][0]
70 | print(output.shape)
71 | print(type(output))
72 | output = np.array(output, dtype='uint8')
73 | print(type(output))
74 | print("Height: {}".format(H))
75 | print("Weight: {}".format(W))
76 | output = cv2.resize(output, (W, H))
77 |
78 | image = Image.fromarray(np.uint8(output), mode="P")
79 | image.putpalette(palette)
80 | image = image.convert("RGB")
81 |
82 | image = np.asarray(image)
83 | print("Frame size: {}".format(frame.shape))
84 | print("image size: {}".format(image.shape))
85 | image = cv2.addWeighted(frame, 1, image, 0.9, 0)
86 | # plt.imshow(image)
87 | # plt.pause(0.001)
88 | #plt.show()
89 | # plt.show()
90 | cv2.imshow("Result", image)
91 | cv2.waitKey(1)
92 |
93 | def start_async(self, frame):
94 | input = self.preprocess(frame)
95 | input_data = {self.input_name: input}
96 | self.enqueue(input_data)
97 |
98 |
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/main.py:
--------------------------------------------------------------------------------
1 | import logging as log
2 | import sys
3 | import time
4 | import os.path as osp
5 | import cv2
6 | import numpy as np
7 |
8 | from openvino.inference_engine import IENetwork
9 | from argparse import ArgumentParser
10 |
11 | from utils.ie_module import InferenceContext
12 | from core.semantic_detection import SemanticSegmentation
13 |
14 | #Global
15 | DEVICE_KINDS = ['CPU', 'GPU', 'FPGA', 'MYRIAD', 'HETERO', 'HDDL']
16 |
17 | def build_argparser():
18 | """
19 | Parse command line arguments.
20 | :return: command line arguments
21 | """
22 |
23 | parser = ArgumentParser()
24 | parser.add_argument("-input", "--input", required=True, type=str,
25 | help="Path to image or video file or enter cam for webcam")
26 | parser.add_argument("-model", "--detection_model", required=True, type=str,
27 | help="Path to an .xml file with a trained Face Detection model")
28 | parser.add_argument('-device', default='CPU', choices=DEVICE_KINDS,
29 | help="(optional) Target device for the " \
30 | "detection model (default: %(default)s)")
31 | parser.add_argument('-v', '--verbose', action='store_true',
32 | help="(optional) Be more verbose")
33 | parser.add_argument('-l', '--cpu_lib', metavar="PATH", default="",
34 | help="(optional) For MKLDNN (CPU)-targeted custom layers, if any. " \
35 | "Path to a shared library with custom layers implementations")
36 | parser.add_argument('-c', '--gpu_lib', metavar="PATH", default="",
37 | help="(optional) For clDNN (GPU)-targeted custom layers, if any. " \
38 | "Path to the XML file with descriptions of the kernels")
39 | parser.add_argument('-pc', '--perf_stats', action='store_true',
40 | help="(optional) Output detailed per-layer performance stats")
41 | return parser
42 |
43 | class ProcessOnFrame:
44 | # Queue size will be used to put frames in a queue for
45 | # Inference Engine
46 | QUEUE_SIZE = 10
47 |
48 | def __init__(self, args):
49 | used_devices = set([args.device])
50 |
51 | # Create a Inference Engine Context
52 | self.context = InferenceContext()
53 | context = self.context
54 |
55 | # Load OpenVino Plugin based on device selection
56 | context.load_plugins(used_devices, args.cpu_lib, args.gpu_lib)
57 | for d in used_devices:
58 | context.get_plugin(d).set_config({
59 | "PERF_COUNT": "YES" if args.perf_stats else "NO"})
60 |
61 | log.info("Loading models")
62 | start_time = time.perf_counter()
63 |
64 | # Load face detection model on Inference Engine
65 | segmentation = self.load_model(args.detection_model)
66 |
67 | stop_time = time.perf_counter()
68 | print("[INFO] Model Load Time: {}".format(stop_time - start_time))
69 |
70 | self.segmentation = SemanticSegmentation(segmentation)
71 |
72 | self.segmentation.deploy(args.device, context)
73 |
74 | log.info("Models are loaded")
75 |
76 | def load_model(self, model_path):
77 | """
78 | Initializing IENetwork(Inference Enginer) object from IR files:
79 |
80 | Args:
81 | Model path - This should contain both .xml and .bin file
82 | :return Instance of IENetwork class
83 | """
84 |
85 | model_path = osp.abspath(model_path)
86 | model_description_path = model_path
87 | model_weights_path = osp.splitext(model_path)[0] + ".bin"
88 |
89 | print(model_weights_path)
90 | log.info("Loading the model from '%s'" % (model_description_path))
91 | assert osp.isfile(model_description_path), \
92 | "Model description is not found at '%s'" % (model_description_path)
93 | assert osp.isfile(model_weights_path), \
94 | "Model weights are not found at '%s'" % (model_weights_path)
95 |
96 | # Load model on IE
97 | model = IENetwork(model_description_path, model_weights_path)
98 | log.info("Model is loaded")
99 |
100 | return model
101 |
102 | def frame_pre_process(self, frame):
103 | """
104 | Pre-Process the input frame given to model
105 | Args:
106 | frame: Input frame from video stream
107 | Return:
108 | frame: Pre-Processed frame
109 | """
110 | assert len(frame.shape) == 3, \
111 | "Expected input frame in (H, W, C) format"
112 | assert frame.shape[2] in [3, 4], \
113 | "Expected BGR or BGRA input"
114 |
115 | orig_image = frame.copy()
116 | frame = frame.transpose((2, 0, 1)) # HWC to CHW
117 | frame = np.expand_dims(frame, axis=0)
118 | return frame
119 |
120 | def detection(self, frame):
121 | image = frame.copy()
122 | frame = self.frame_pre_process(frame)
123 |
124 | self.segmentation.clear()
125 |
126 | self.segmentation.start_async(frame)
127 | self.segmentation.post_processing(image)
128 |
129 |
130 | class VisionSystem:
131 | BREAK_KEY_LABELS = "q(Q) or Escape"
132 | BREAK_KEYS = {ord('q'), ord('Q'), 27}
133 |
134 | def __init__(self, args):
135 | self.frame_processor = ProcessOnFrame(args)
136 | self.print_perf_stats = args.perf_stats
137 |
138 | def process(self):
139 | input_stream = cv2.VideoCapture("../../../demo/Paris.mp4")
140 |
141 | frame_count = 0
142 |
143 | while input_stream.isOpened():
144 | has_frame, frame = input_stream.read()
145 | if not has_frame:
146 | break
147 |
148 | frame_count+=1
149 | print("Frame cout: {}".format(frame_count))
150 |
151 | self.frame_processor.detection(frame)
152 | def run(self, args):
153 | if args.input == "cam":
154 | path = "0"
155 | else:
156 | path = args.input
157 |
158 | # input_stream = VisionSystem.open_input_stream(path)
159 |
160 | # if input_stream is None or not input_stream.isOpened():
161 | # log.error("Cannot open input stream: %s" % args.input)
162 |
163 | # # FPS init
164 | # fps = input_stream.get(cv2.CAP_PROP_FPS)
165 |
166 | # # Get the Frame org size
167 | # frame_size = (int(input_stream.get(cv2.CAP_PROP_FRAME_WIDTH)),
168 | # int(input_stream.get(cv2.CAP_PROP_FRAME_HEIGHT)))
169 |
170 | # # Get the frame count if its a video
171 | # self.frame_count = int(input_stream.get(cv2.CAP_PROP_FRAME_COUNT))
172 |
173 | self.process()
174 |
175 | @staticmethod
176 | def open_input_stream(path):
177 | """
178 | Open the input stream
179 | """
180 | log.info("Reading input data from '%s'" % (path))
181 | stream = path
182 | try:
183 | stream = int(path)
184 | except ValueError:
185 | pass
186 | return cv2.VideoCapture(stream)
187 |
188 | def main():
189 | args = build_argparser().parse_args()
190 |
191 | log.basicConfig(format="[ %(levelname)s ] %(asctime)-15s %(message)s",
192 | level=log.INFO if not args.verbose else log.DEBUG, stream=sys.stdout)
193 | detection = VisionSystem(args)
194 | detection.run(args)
195 |
196 | if __name__ == "__main__":
197 | main()
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/utils/__pycache__/ie_module.cpython-36.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/utils/__pycache__/ie_module.cpython-36.pyc
--------------------------------------------------------------------------------
/openvino_analysis/intel/semantic-segmentation/semantic-segmentation-adas-0001/utils/ie_module.py:
--------------------------------------------------------------------------------
1 | import logging as log
2 | import os.path as osp
3 |
4 | from openvino.inference_engine import IEPlugin
5 |
6 | class InferenceContext:
7 | def __init__(self):
8 | self.plugins = {}
9 |
10 | def load_plugins(self, devices, cpu_ext="", gpu_ext=""):
11 | log.info("Loading plugins for devices: %s" % (devices))
12 |
13 | plugins = { d: IEPlugin(d) for d in devices }
14 | if 'CPU' in plugins and not len(cpu_ext) == 0:
15 | log.info("Using CPU extensions library '%s'" % (cpu_ext))
16 | assert osp.isfile(cpu_ext), "Failed to open CPU extensions library"
17 | plugins['CPU'].add_cpu_extension(cpu_ext)
18 |
19 | if 'GPU' in plugins and not len(gpu_ext) == 0:
20 | assert osp.isfile(gpu_ext), "Failed to open GPU definitions file"
21 | plugins['GPU'].set_config({"CONFIG_FILE": gpu_ext})
22 |
23 | self.plugins = plugins
24 |
25 | log.info("Plugins are loaded")
26 |
27 | def get_plugin(self, device):
28 | return self.plugins.get(device, None)
29 |
30 | def check_model_support(self, net, device):
31 | plugin = self.plugins[device]
32 |
33 | if plugin.device == "CPU":
34 | supported_layers = plugin.get_supported_layers(net)
35 | not_supported_layers = [l for l in net.layers.keys() \
36 | if l not in supported_layers]
37 | if len(not_supported_layers) != 0:
38 | log.error("The following layers are not supported " \
39 | "by the plugin for the specified device {}:\n {}" \
40 | .format(plugin.device, ', '.join(not_supported_layers)))
41 | log.error("Please try to specify cpu extensions " \
42 | "library path in the command line parameters using " \
43 | "the '-l' parameter")
44 | raise NotImplementedError(
45 | "Some layers are not supported on the device")
46 |
47 | def load_model(self, model, device, max_requests=1):
48 | self.check_model_support(model, device)
49 | plugin = self.plugins[device]
50 | deployed_model = plugin.load(network=model, num_requests=max_requests)
51 | return deployed_model
52 |
53 | class Module(object):
54 | def __init__(self, model):
55 | self.model = model
56 | self.device_model = None
57 |
58 | self.max_requests = 0
59 | self.active_requests = 0
60 |
61 | self.clear()
62 |
63 | def deploy(self, device, context, queue_size=1):
64 | self.context = context
65 | self.max_requests = queue_size
66 | self.device_model = context.load_model(
67 | self.model, device, self.max_requests)
68 | self.model = None
69 |
70 | def enqueue(self, input):
71 | self.clear()
72 |
73 | if self.max_requests <= self.active_requests:
74 | log.warning("Processing request rejected - too many requests")
75 | return False
76 |
77 | self.device_model.start_async(self.active_requests, input)
78 | self.active_requests += 1
79 | return True
80 |
81 | def wait(self):
82 | if self.active_requests <= 0:
83 | return
84 |
85 | self.perf_stats = [None, ] * self.active_requests
86 | self.outputs = [None, ] * self.active_requests
87 | for i in range(self.active_requests):
88 | self.device_model.requests[i].wait()
89 | self.outputs[i] = self.device_model.requests[i].outputs
90 | self.perf_stats[i] = self.device_model.requests[i].get_perf_counts()
91 |
92 | self.active_requests = 0
93 |
94 | def get_outputs(self):
95 | self.wait()
96 | return self.outputs
97 |
98 | def get_performance_stats(self):
99 | return self.perf_stats
100 |
101 |
102 | def clear(self):
103 | self.perf_stats = []
104 | self.outputs = []
--------------------------------------------------------------------------------
/openvino_analysis/public/ssd_mobilenet_v2_coco/__pycache__/inference.cpython-36.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/public/ssd_mobilenet_v2_coco/__pycache__/inference.cpython-36.pyc
--------------------------------------------------------------------------------
/openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/__pycache__/inference.cpython-36.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/__pycache__/inference.cpython-36.pyc
--------------------------------------------------------------------------------
/openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/example.log:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/example.log
--------------------------------------------------------------------------------
/openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/inference.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import logging as log
4 | from openvino.inference_engine import IENetwork, IECore
5 |
6 |
7 | class Network:
8 | """
9 | Load and configure inference plugins for the specified target devices
10 | and performs synchronous and asynchronous modes for the specified infer requests.
11 | """
12 |
13 | def __init__(self):
14 | self.ie = None
15 | self.network = None
16 | self.input_blob = None
17 | self.output_blob = None
18 | self.exec_network = None
19 | self.infer_request = None
20 |
21 | def load_model(self, model, device="CPU", cpu_extension=None, num_requests=0):
22 | '''
23 | Load the model given IR files.
24 | Defaults to CPU as device for use in the workspace.
25 | Synchronous requests made within.
26 | '''
27 | model_xml = model
28 | model_bin = os.path.splitext(model_xml)[0] + ".bin"
29 |
30 |
31 | # Initialize the plugin
32 | self.ie = IECore()
33 |
34 | # Read the IR as a IENetwork
35 | self.network = self.ie.read_network(model=model_xml, weights=model_bin)
36 |
37 | # Load the IENetwork into the plugin
38 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests)
39 |
40 | return
41 |
42 | def get_input_shape(self):
43 | '''
44 | Gets the input shape of the network
45 | '''
46 | log.info("Preparing input blobs")
47 | input_shape = []
48 | print("inputs number: " + str(len(self.network.input_info.keys())))
49 |
50 | for input_key in self.network.input_info:
51 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape))
52 | print("input key: " + input_key)
53 | self.input_blob = input_key
54 | if len(self.network.input_info[input_key].input_data.layout) == 4:
55 | input_shape = self.network.input_info[input_key].input_data.shape
56 |
57 | return input_shape
58 |
59 | def get_output_name(self):
60 | '''
61 | Gets the input shape of the network
62 | '''
63 | log.info('Preparing output blobs')
64 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))]
65 | for output_key in self.network.outputs:
66 | print(output_key)
67 | if self.network.layers[output_key].type == "DetectionOutput":
68 | self.output_blob, _ = output_key, self.network.outputs[output_key]
69 |
70 | if self.output_blob == "":
71 | log.error("Can't find a DetectionOutput layer in the topology")
72 | exit(-1)
73 |
74 | return self.output_blob
75 |
76 | def exec_net_async(self, image, request_id=0):
77 | '''
78 | Makes an asynchronous inference request, given an input image.
79 | '''
80 | self.exec_network.start_async(request_id=request_id,
81 | inputs={self.input_blob: image})
82 | return
83 |
84 | def exec_net_sync(self, image):
85 | '''
86 | Makes an asynchronous inference request, given an input image.
87 | '''
88 | return self.exec_network.infer(inputs={self.input_blob: image})
89 |
90 | def wait(self, request_id=0):
91 | '''
92 | Checks the status of the inference request.
93 | '''
94 | status = self.exec_network.requests[request_id].wait(-1)
95 | return status
96 |
97 |
98 | def get_output(self, request_id=0):
99 | '''
100 | Returns a list of the results for the output layer of the network.
101 | '''
102 | return self.exec_network.requests[request_id].outputs[self.output_blob]
--------------------------------------------------------------------------------
/openvino_analysis/public/ssd_mobilenet_v2_coco/async_multithread/main.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import socket
5 | import json
6 | import cv2
7 | import numpy as np
8 |
9 |
10 | import logging as log
11 | from imutils.video import FPS
12 |
13 | from argparse import ArgumentParser
14 | from inference import Network
15 |
16 | from multiprocessing import Process, Queue
17 | import multiprocessing
18 | import threading
19 | import queue
20 |
21 | # Browser and OpenCV Window toggle
22 | CPU_EXTENSION = "/opt/intel/openvino/deployment_tools/inference_engine/lib/intel64/libcpu_extension_sse4.so"
23 |
24 | def build_argparser():
25 | """
26 | Parse command line arguments.
27 | :return: command line arguments
28 | """
29 | parser = ArgumentParser()
30 | parser.add_argument("-m", "--model", required=True, type=str,
31 | help="Path to an xml file with a trained model.")
32 | parser.add_argument("-i", "--input", required=True, type=str,
33 | help="Path to image or video file")
34 | parser.add_argument("-l", "--cpu_extension", required=False, type=str,
35 | default=None,
36 | help="MKLDNN (CPU)-targeted custom layers."
37 | "Absolute path to a shared library with the"
38 | "kernels impl.")
39 | parser.add_argument("-d", "--device", type=str, default="CPU",
40 | help="Specify the target device to infer on: "
41 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample "
42 | "will look for a suitable plugin for device "
43 | "specified (CPU by default)")
44 | parser.add_argument("-pt", "--prob_threshold", type=float, default=0.2,
45 | help="Probability threshold for detections filtering"
46 | "(0.5 by default)")
47 | return parser
48 |
49 |
50 | def image_process_worker(cap, frame_queue, image_queue, in_n, in_c, in_h, in_w):
51 | # Process frames until the video ends, or process is exited
52 | while cap.isOpened():
53 | # Read the next frame
54 | flag, frame = cap.read()
55 | if not flag:
56 | frame_queue.put(None)
57 | image_queue.put(None)
58 | break
59 |
60 | # Pre-process the frame
61 | image_resize = cv2.resize(frame, (in_w, in_h))
62 | image = image_resize.transpose((2,0,1))
63 | image = image.reshape(in_n, in_c, in_h, in_w)
64 |
65 | frame_queue.put(frame)
66 | image_queue.put(image)
67 |
68 |
69 | def network_inference(infer_network, frame_queue, image_queue,
70 | fw, fh, prob_threshold, fps):
71 | current_inference, next_inference = 0, 1
72 | count = 0
73 | while True:
74 | image = image_queue.get()
75 | if image is None:
76 | break
77 | count += 1
78 | frame = frame_queue.get()
79 | # Perform inference on the frame
80 | infer_network.exec_net_async(image, request_id=current_inference)
81 |
82 | # Get the output of inference
83 | if infer_network.wait(next_inference) == 0:
84 | detection = infer_network.get_output(next_inference)
85 | for box in detection[0][0]: # Output shape is 1x1x100x7
86 | conf = box[2]
87 | if conf >= prob_threshold:
88 | xmin = int(box[3] * fw)
89 | ymin = int(box[4] * fh)
90 | xmax = int(box[5] * fw)
91 | ymax = int(box[6] * fh)
92 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 125, 255), 3)
93 | current_inference, next_inference = next_inference, current_inference
94 |
95 | cv2.imshow('frame', frame)
96 | if cv2.waitKey(1) & 0xFF == ord('q'):
97 | break
98 | fps.update()
99 | if count == 100:
100 | fps.stop()
101 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
102 |
103 |
104 | def infer_on_stream(args):
105 | """
106 | Initialize the inference network, stream video to network,
107 | and output stats and video.
108 | :param args: Command line arguments parsed by `build_argparser()`
109 | :return: None
110 | """
111 |
112 | frame_queue = queue.Queue(maxsize= 4)
113 | image_queue = queue.Queue(maxsize= 4)
114 |
115 | # Initialize the Inference Engine
116 | infer_network = Network()
117 |
118 | # Set Probability threshold for detections
119 | prob_threshold = args.prob_threshold
120 |
121 | # Load the model through `infer_network`
122 | infer_network.load_model(args.model, args.device, CPU_EXTENSION, num_requests=2)
123 |
124 | # Get a Input blob shape
125 | in_n, in_c, in_h, in_w = infer_network.get_input_shape()
126 |
127 | # Get a output blob name
128 | _ = infer_network.get_output_name()
129 |
130 | # Handle the input stream
131 | cap = cv2.VideoCapture(args.input)
132 | cap.open(args.input)
133 | _, frame = cap.read()
134 |
135 |
136 | fps = FPS().start()
137 |
138 | _s, frame = cap.read()
139 | fh = frame.shape[0]
140 | fw = frame.shape[1]
141 |
142 | preprocess_thread = None
143 |
144 | preprocess_thread = threading.Thread(target=image_process_worker,
145 | args=(cap, frame_queue, image_queue, in_n, in_c, in_h, in_w))
146 |
147 | preprocess_thread.start()
148 |
149 | network_inference(infer_network, frame_queue, image_queue, fw, fh, prob_threshold, fps)
150 |
151 | preprocess_thread.join()
152 |
153 | # Release the out writer, capture, and destroy any OpenCV windows
154 | cap.release()
155 | cv2.destroyAllWindows()
156 | fps.stop()
157 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
158 |
159 |
160 | def main():
161 | """
162 | Load the network and parse the output.
163 | :return: None
164 | """
165 | # set log level
166 | log.basicConfig(filename='example.log',level=log.CRITICAL)
167 |
168 | # Grab command line args
169 | args = build_argparser().parse_args()
170 |
171 | # Inference
172 | infer_on_stream(args)
173 |
174 | if __name__ == '__main__':
175 | main()
--------------------------------------------------------------------------------
/openvino_analysis/public/ssd_mobilenet_v2_coco/inference.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import logging as log
4 | from openvino.inference_engine import IENetwork, IECore
5 |
6 |
7 | class Network:
8 | """
9 | Load and configure inference plugins for the specified target devices
10 | and performs synchronous and asynchronous modes for the specified infer requests.
11 | """
12 |
13 | def __init__(self):
14 | self.ie = None
15 | self.network = None
16 | self.input_blob = None
17 | self.output_blob = None
18 | self.exec_network = None
19 | self.infer_request = None
20 |
21 |
22 | def load_model(self, model, args, device="CPU", num_requests=0):
23 | '''
24 | Load the model given IR files.
25 | Defaults to CPU as device for use in the workspace.
26 | Synchronous requests made within.
27 | '''
28 | model_xml = model
29 | model_bin = os.path.splitext(model_xml)[0] + ".bin"
30 |
31 |
32 | # Initialize the plugin
33 | self.ie = IECore()
34 |
35 | # Read the IR as a IENetwork
36 | try:
37 | self.network = self.ie.read_network(model=model_xml, weights=model_bin)
38 | except Exception as e:
39 | raise ValueError("Could not Initialise the network. Have you enterred the correct model path?")
40 |
41 | if args.cpu_extension and "CPU" in args.device:
42 | ie.add_extension(args.cpu_extension, "CPU")
43 | log.info("CPU extension loaded: {}".format(args.cpu_extension))
44 |
45 | if "CPU" in args.device:
46 | supported_layers = self.ie.query_network(self.network, "CPU")
47 | not_supported_layers = [l for l in self.network.layers.keys() if l not in supported_layers]
48 | if len(not_supported_layers) != 0:
49 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}".
50 | format(args.device, ', '.join(not_supported_layers)))
51 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l "
52 | "or --cpu_extension command line argument")
53 | sys.exit(1)
54 |
55 | # Load the IENetwork into the plugin
56 | self.exec_network = self.ie.load_network(self.network, device, num_requests=num_requests)
57 |
58 | def get_input_shape(self):
59 | '''
60 | Gets the input shape of the network
61 | '''
62 | log.info("Preparing input blobs")
63 | input_shape = []
64 | print("inputs number: " + str(len(self.network.input_info.keys())))
65 |
66 | for input_key in self.network.input_info:
67 | print("input shape: " + str(self.network.input_info[input_key].input_data.shape))
68 | print("input key: " + input_key)
69 | self.input_blob = input_key
70 | if len(self.network.input_info[input_key].input_data.layout) == 4:
71 | input_shape = self.network.input_info[input_key].input_data.shape
72 |
73 | return input_shape
74 |
75 | def get_output_name(self):
76 | '''
77 | Gets the input shape of the network
78 | '''
79 | log.info('Preparing output blobs')
80 | output_name, _ = "", self.network.outputs[next(iter(self.network.outputs.keys()))]
81 | for output_key in self.network.outputs:
82 | print(output_key)
83 | if self.network.layers[output_key].type == "DetectionOutput":
84 | self.output_blob, _ = output_key, self.network.outputs[output_key]
85 |
86 | if self.output_blob == "":
87 | log.error("Can't find a DetectionOutput layer in the topology")
88 | exit(-1)
89 |
90 | return self.output_blob
91 |
92 | def exec_net(self, image, request_id):
93 | '''
94 | Makes an asynchronous inference request, given an input image.
95 | '''
96 | self.exec_network.start_async(request_id=request_id,
97 | inputs={self.input_blob: image})
98 | return
99 |
100 |
101 | def wait(self, request_id):
102 | '''
103 | Checks the status of the inference request.
104 | '''
105 | status = self.exec_network.requests[request_id].wait(-1)
106 | return status
107 |
108 |
109 | def get_output(self, request_id):
110 | '''
111 | Returns a list of the results for the output layer of the network.
112 | '''
113 | return self.exec_network.requests[request_id].outputs[self.output_blob]
--------------------------------------------------------------------------------
/openvino_analysis/public/ssd_mobilenet_v2_coco/main.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import socket
5 | import json
6 | import cv2
7 | import numpy as np
8 |
9 | import logging as log
10 |
11 | from argparse import ArgumentParser
12 | from inference import Network
13 |
14 | from imutils.video import FPS
15 |
16 | def build_argparser():
17 | """
18 | Parse command line arguments.
19 | :return: command line arguments
20 | """
21 | parser = ArgumentParser()
22 | parser.add_argument("-m", "--model", required=True, type=str,
23 | help="Path to an xml file with a trained model.")
24 | parser.add_argument("-i", "--input", required=True, type=str,
25 | help="Path to image or video file")
26 | parser.add_argument("-d", "--device", type=str, default="CPU",
27 | help="Specify the target device to infer on: "
28 | "CPU, GPU, FPGA or MYRIAD is acceptable. Sample "
29 | "will look for a suitable plugin for device "
30 | "specified (CPU by default)")
31 | parser.add_argument("-l", "--cpu_extension",
32 | help="Optional. Required for CPU custom layers. "
33 | "Absolute path to a shared library with the kernels implementations.",
34 | type=str, default=None)
35 | return parser
36 |
37 |
38 | def preprocessing(frame, in_w, in_h):
39 | image_resize = cv2.resize(frame, (in_w, in_h), interpolation = cv2.INTER_AREA)
40 | image = np.moveaxis(image_resize, -1, 0)
41 |
42 | return image
43 | def infer_on_stream(args):
44 | """
45 | Initialize the inference network, stream video to network,
46 | and output stats and video.
47 | :param args: Command line arguments parsed by `build_argparser()`
48 | :return: None
49 | """
50 | # Initialize the Inference Engine
51 | infer_network = Network()
52 |
53 | # Load the model through `infer_network`
54 | infer_network.load_model(args.model, args, args.device, num_requests=0)
55 |
56 | # Get a Input blob shape
57 | _, _, in_h, in_w = infer_network.get_input_shape()
58 |
59 |
60 | # Get a output blob name
61 | output_name = infer_network.get_output_name()
62 |
63 | # Handle the input stream
64 | try:
65 | cap = cv2.VideoCapture(args.input)
66 | except FileNotFoundError:
67 | print("Cannot locate video file: "+ args.input)
68 | except Exception as e:
69 | print("Something else went wrong with the video file: ", e)
70 |
71 | cap.open(args.input)
72 | _, frame = cap.read()
73 |
74 | fh = frame.shape[0]
75 | fw = frame.shape[1]
76 |
77 | fps = FPS().start()
78 |
79 | # Process frames until the video ends, or process is exited
80 | while cap.isOpened():
81 | # Read the next frame
82 | flag, frame = cap.read()
83 | if not flag:
84 | break
85 |
86 | key_pressed = cv2.waitKey(1)
87 |
88 | image = preprocessing(frame, in_w, in_h)
89 |
90 | # Perform inference on the frame
91 | infer_network.exec_net(image, request_id=0)
92 |
93 | # Get the output of inference
94 | if infer_network.wait(request_id=0) == 0:
95 | result = infer_network.get_output(request_id=0)
96 | for box in result[0][0]: # Output shape is 1x1x100x7
97 | conf = box[2]
98 | if conf >= 0.5:
99 | xmin = int(box[3] * fw)
100 | ymin = int(box[4] * fh)
101 | xmax = int(box[5] * fw)
102 | ymax = int(box[6] * fh)
103 | cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (0, 0, 255), 3)
104 |
105 | cv2.imshow('frame', frame)
106 | if cv2.waitKey(1) & 0xFF == ord('q'):
107 | break
108 |
109 | #Break if escape key pressed
110 | if key_pressed == 27:
111 | break
112 |
113 | fps.update()
114 |
115 | # Release the out writer, capture, and destroy any OpenCV windows
116 | cap.release()
117 |
118 | cv2.destroyAllWindows()
119 |
120 | fps.stop()
121 |
122 | print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
123 |
124 | def main():
125 | """
126 | Load the network and parse the output.
127 | :return: None
128 | """
129 | # Set log to INFO
130 | log.basicConfig(level=log.CRITICAL)
131 |
132 | # Grab command line args
133 | args = build_argparser().parse_args()
134 |
135 | # Perform inference on the input stream
136 | infer_on_stream(args)
137 |
138 | if __name__ == '__main__':
139 | main()
--------------------------------------------------------------------------------
/openvino_analysis/public/yolo-v3/main.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | """
3 | Copyright (C) 2018-2020 Intel Corporation
4 |
5 | Licensed under the Apache License, Version 2.0 (the "License");
6 | you may not use this file except in compliance with the License.
7 | You may obtain a copy of the License at
8 |
9 | http://www.apache.org/licenses/LICENSE-2.0
10 |
11 | Unless required by applicable law or agreed to in writing, software
12 | distributed under the License is distributed on an "AS IS" BASIS,
13 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | See the License for the specific language governing permissions and
15 | limitations under the License.
16 | """
17 |
18 | import logging
19 | import threading
20 | import os
21 | import sys
22 | from collections import deque
23 | from argparse import ArgumentParser, SUPPRESS
24 | from math import exp as exp
25 | from time import perf_counter
26 | from enum import Enum
27 |
28 | import cv2
29 | import numpy as np
30 | from openvino.inference_engine import IECore
31 |
32 | sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'common'))
33 | # import monitors
34 |
35 |
36 | logging.basicConfig(format="[ %(levelname)s ] %(message)s", level=logging.INFO, stream=sys.stdout)
37 | log = logging.getLogger()
38 |
39 | def build_argparser():
40 | parser = ArgumentParser(add_help=False)
41 | args = parser.add_argument_group('Options')
42 | args.add_argument('-h', '--help', action='help', default=SUPPRESS, help='Show this help message and exit.')
43 | args.add_argument("-m", "--model", help="Required. Path to an .xml file with a trained model.",
44 | required=True, type=str)
45 | args.add_argument("-i", "--input", help="Required. Path to an image/video file. (Specify 'cam' to work with "
46 | "camera)", required=True, type=str)
47 | args.add_argument("-l", "--cpu_extension",
48 | help="Optional. Required for CPU custom layers. Absolute path to a shared library with "
49 | "the kernels implementations.", type=str, default=None)
50 | args.add_argument("-d", "--device",
51 | help="Optional. Specify the target device to infer on; CPU, GPU, FPGA, HDDL or MYRIAD is"
52 | " acceptable. The sample will look for a suitable plugin for device specified. "
53 | "Default value is CPU", default="CPU", type=str)
54 | args.add_argument("--labels", help="Optional. Labels mapping file", default=None, type=str)
55 | args.add_argument("-t", "--prob_threshold", help="Optional. Probability threshold for detections filtering",
56 | default=0.5, type=float)
57 | args.add_argument("-iout", "--iou_threshold", help="Optional. Intersection over union threshold for overlapping "
58 | "detections filtering", default=0.4, type=float)
59 | args.add_argument("-r", "--raw_output_message", help="Optional. Output inference results raw values showing",
60 | default=False, action="store_true")
61 | args.add_argument("-nireq", "--num_infer_requests", help="Optional. Number of infer requests",
62 | default=1, type=int)
63 | args.add_argument("-nstreams", "--num_streams",
64 | help="Optional. Number of streams to use for inference on the CPU or/and GPU in throughput mode "
65 | "(for HETERO and MULTI device cases use format :,: "
66 | "or just )",
67 | default="", type=str)
68 | args.add_argument("-nthreads", "--number_threads",
69 | help="Optional. Number of threads to use for inference on CPU (including HETERO cases)",
70 | default=None, type=int)
71 | args.add_argument("-loop_input", "--loop_input", help="Optional. Iterate over input infinitely",
72 | action='store_true')
73 | args.add_argument("-no_show", "--no_show", help="Optional. Don't show output", action='store_true')
74 | args.add_argument('-u', '--utilization_monitors', default='', type=str,
75 | help='Optional. List of monitors to show initially.')
76 | args.add_argument("--keep_aspect_ratio", action="store_true", default=False,
77 | help='Optional. Keeps aspect ratio on resize.')
78 | return parser
79 |
80 |
81 | class YoloParams:
82 | # ------------------------------------------- Extracting layer parameters ------------------------------------------
83 | # Magic numbers are copied from yolo samples
84 | def __init__(self, param, side):
85 | self.num = 3 if 'num' not in param else int(param['num'])
86 | self.coords = 4 if 'coords' not in param else int(param['coords'])
87 | self.classes = 80 if 'classes' not in param else int(param['classes'])
88 | self.side = side
89 | self.anchors = [10.0, 13.0, 16.0, 30.0, 33.0, 23.0, 30.0, 61.0, 62.0, 45.0, 59.0, 119.0, 116.0, 90.0, 156.0,
90 | 198.0,
91 | 373.0, 326.0] if 'anchors' not in param else [float(a) for a in param['anchors'].split(',')]
92 |
93 | self.isYoloV3 = False
94 |
95 | if param.get('mask'):
96 | mask = [int(idx) for idx in param['mask'].split(',')]
97 | self.num = len(mask)
98 |
99 | maskedAnchors = []
100 | for idx in mask:
101 | maskedAnchors += [self.anchors[idx * 2], self.anchors[idx * 2 + 1]]
102 | self.anchors = maskedAnchors
103 |
104 | self.isYoloV3 = True # Weak way to determine but the only one.
105 |
106 |
107 | class Modes(Enum):
108 | USER_SPECIFIED = 0
109 | MIN_LATENCY = 1
110 |
111 |
112 | class Mode():
113 | def __init__(self, value):
114 | self.current = value
115 |
116 | def next(self):
117 | if self.current.value + 1 < len(Modes):
118 | self.current = Modes(self.current.value + 1)
119 | else:
120 | self.current = Modes(0)
121 |
122 |
123 | class ModeInfo():
124 | def __init__(self):
125 | self.last_start_time = perf_counter()
126 | self.last_end_time = None
127 | self.frames_count = 0
128 | self.latency_sum = 0
129 |
130 |
131 | def scale_bbox(x, y, height, width, class_id, confidence, im_h, im_w, is_proportional):
132 | if is_proportional:
133 | scale = np.array([min(im_w/im_h, 1), min(im_h/im_w, 1)])
134 | offset = 0.5*(np.ones(2) - scale)
135 | x, y = (np.array([x, y]) - offset) / scale
136 | width, height = np.array([width, height]) / scale
137 | xmin = int((x - width / 2) * im_w)
138 | ymin = int((y - height / 2) * im_h)
139 | xmax = int(xmin + width * im_w)
140 | ymax = int(ymin + height * im_h)
141 | # Method item() used here to convert NumPy types to native types for compatibility with functions, which don't
142 | # support Numpy types (e.g., cv2.rectangle doesn't support int64 in color parameter)
143 | return dict(xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax, class_id=class_id.item(), confidence=confidence.item())
144 |
145 |
146 | def parse_yolo_region(predictions, resized_image_shape, original_im_shape, params, threshold, is_proportional):
147 | # ------------------------------------------ Validating output parameters ------------------------------------------
148 | _, _, out_blob_h, out_blob_w = predictions.shape
149 | assert out_blob_w == out_blob_h, "Invalid size of output blob. It sould be in NCHW layout and height should " \
150 | "be equal to width. Current height = {}, current width = {}" \
151 | "".format(out_blob_h, out_blob_w)
152 |
153 | # ------------------------------------------ Extracting layer parameters -------------------------------------------
154 | orig_im_h, orig_im_w = original_im_shape
155 | resized_image_h, resized_image_w = resized_image_shape
156 | objects = list()
157 | size_normalizer = (resized_image_w, resized_image_h) if params.isYoloV3 else (params.side, params.side)
158 | bbox_size = params.coords + 1 + params.classes
159 | # ------------------------------------------- Parsing YOLO Region output -------------------------------------------
160 | for row, col, n in np.ndindex(params.side, params.side, params.num):
161 | # Getting raw values for each detection bounding box
162 | bbox = predictions[0, n*bbox_size:(n+1)*bbox_size, row, col]
163 | x, y, width, height, object_probability = bbox[:5]
164 | class_probabilities = bbox[5:]
165 | if object_probability < threshold:
166 | continue
167 | # Process raw value
168 | x = (col + x) / params.side
169 | y = (row + y) / params.side
170 | # Value for exp is very big number in some cases so following construction is using here
171 | try:
172 | width = exp(width)
173 | height = exp(height)
174 | except OverflowError:
175 | continue
176 | # Depends on topology we need to normalize sizes by feature maps (up to YOLOv3) or by input shape (YOLOv3)
177 | width = width * params.anchors[2 * n] / size_normalizer[0]
178 | height = height * params.anchors[2 * n + 1] / size_normalizer[1]
179 |
180 | class_id = np.argmax(class_probabilities)
181 | confidence = class_probabilities[class_id]*object_probability
182 | if confidence < threshold:
183 | continue
184 | objects.append(scale_bbox(x=x, y=y, height=height, width=width, class_id=class_id, confidence=confidence,
185 | im_h=orig_im_h, im_w=orig_im_w, is_proportional=is_proportional))
186 | return objects
187 |
188 |
189 | def intersection_over_union(box_1, box_2):
190 | width_of_overlap_area = min(box_1['xmax'], box_2['xmax']) - max(box_1['xmin'], box_2['xmin'])
191 | height_of_overlap_area = min(box_1['ymax'], box_2['ymax']) - max(box_1['ymin'], box_2['ymin'])
192 | if width_of_overlap_area < 0 or height_of_overlap_area < 0:
193 | area_of_overlap = 0
194 | else:
195 | area_of_overlap = width_of_overlap_area * height_of_overlap_area
196 | box_1_area = (box_1['ymax'] - box_1['ymin']) * (box_1['xmax'] - box_1['xmin'])
197 | box_2_area = (box_2['ymax'] - box_2['ymin']) * (box_2['xmax'] - box_2['xmin'])
198 | area_of_union = box_1_area + box_2_area - area_of_overlap
199 | if area_of_union == 0:
200 | return 0
201 | return area_of_overlap / area_of_union
202 |
203 |
204 | def resize(image, size, keep_aspect_ratio, interpolation=cv2.INTER_LINEAR):
205 | if not keep_aspect_ratio:
206 | return cv2.resize(image, size, interpolation=interpolation)
207 |
208 | iw, ih = image.shape[0:2][::-1]
209 | w, h = size
210 | scale = min(w/iw, h/ih)
211 | nw = int(iw*scale)
212 | nh = int(ih*scale)
213 | image = cv2.resize(image, (nw, nh), interpolation=interpolation)
214 | new_image = np.full((size[1], size[0], 3), 128, dtype=np.uint8)
215 | dx = (w-nw)//2
216 | dy = (h-nh)//2
217 | new_image[dy:dy+nh, dx:dx+nw, :] = image
218 | return new_image
219 |
220 |
221 | def preprocess_frame(frame, input_height, input_width, nchw_shape, keep_aspect_ratio):
222 | in_frame = resize(frame, (input_width, input_height), keep_aspect_ratio)
223 | if nchw_shape:
224 | in_frame = in_frame.transpose((2, 0, 1)) # Change data layout from HWC to CHW
225 | in_frame = np.expand_dims(in_frame, axis=0)
226 | return in_frame
227 |
228 |
229 | def get_objects(output, net, new_frame_height_width, source_height_width, prob_threshold, is_proportional):
230 | objects = list()
231 |
232 | for layer_name, out_blob in output.items():
233 | out_blob = out_blob.buffer.reshape(net.layers[net.layers[layer_name].parents[0]].out_data[0].shape)
234 | layer_params = YoloParams(net.layers[layer_name].params, out_blob.shape[2])
235 | objects += parse_yolo_region(out_blob, new_frame_height_width, source_height_width, layer_params,
236 | prob_threshold, is_proportional)
237 |
238 | return objects
239 |
240 |
241 | def filter_objects(objects, iou_threshold, prob_threshold):
242 | # Filtering overlapping boxes with respect to the --iou_threshold CLI parameter
243 | objects = sorted(objects, key=lambda obj : obj['confidence'], reverse=True)
244 | for i in range(len(objects)):
245 | if objects[i]['confidence'] == 0:
246 | continue
247 | for j in range(i + 1, len(objects)):
248 | if intersection_over_union(objects[i], objects[j]) > iou_threshold:
249 | objects[j]['confidence'] = 0
250 |
251 | return tuple(obj for obj in objects if obj['confidence'] >= prob_threshold)
252 |
253 |
254 | def async_callback(status, callback_args):
255 | request, frame_id, frame_mode, frame, start_time, completed_request_results, empty_requests, \
256 | mode, event, callback_exceptions = callback_args
257 |
258 | try:
259 | if status != 0:
260 | raise RuntimeError('Infer Request has returned status code {}'.format(status))
261 |
262 | completed_request_results[frame_id] = (frame, request.output_blobs, start_time, frame_mode == mode.current)
263 |
264 | if mode.current == frame_mode:
265 | empty_requests.append(request)
266 | except Exception as e:
267 | callback_exceptions.append(e)
268 |
269 | event.set()
270 |
271 |
272 | def put_highlighted_text(frame, message, position, font_face, font_scale, color, thickness):
273 | cv2.putText(frame, message, position, font_face, font_scale, (255, 255, 255), thickness + 1) # white border
274 | cv2.putText(frame, message, position, font_face, font_scale, color, thickness)
275 |
276 |
277 | def await_requests_completion(requests):
278 | for request in requests:
279 | request.wait()
280 |
281 |
282 | def main():
283 | args = build_argparser().parse_args()
284 |
285 | # ------------- 1. Plugin initialization for specified device and load extensions library if specified -------------
286 | log.info("Creating Inference Engine...")
287 | ie = IECore()
288 |
289 | config_user_specified = {}
290 | config_min_latency = {}
291 |
292 | devices_nstreams = {}
293 | if args.num_streams:
294 | devices_nstreams = {device: args.num_streams for device in ['CPU', 'GPU'] if device in args.device} \
295 | if args.num_streams.isdigit() \
296 | else dict([device.split(':') for device in args.num_streams.split(',')])
297 |
298 | if 'CPU' in args.device:
299 | if args.cpu_extension:
300 | ie.add_extension(args.cpu_extension, 'CPU')
301 | if args.number_threads is not None:
302 | config_user_specified['CPU_THREADS_NUM'] = str(args.number_threads)
303 | if 'CPU' in devices_nstreams:
304 | config_user_specified['CPU_THROUGHPUT_STREAMS'] = devices_nstreams['CPU'] \
305 | if int(devices_nstreams['CPU']) > 0 \
306 | else 'CPU_THROUGHPUT_AUTO'
307 |
308 | config_min_latency['CPU_THROUGHPUT_STREAMS'] = '1'
309 |
310 | if 'GPU' in args.device:
311 | if 'GPU' in devices_nstreams:
312 | config_user_specified['GPU_THROUGHPUT_STREAMS'] = devices_nstreams['GPU'] \
313 | if int(devices_nstreams['GPU']) > 0 \
314 | else 'GPU_THROUGHPUT_AUTO'
315 |
316 | config_min_latency['GPU_THROUGHPUT_STREAMS'] = '1'
317 |
318 | # -------------------- 2. Reading the IR generated by the Model Optimizer (.xml and .bin files) --------------------
319 | log.info("Loading network")
320 | net = ie.read_network(args.model, os.path.splitext(args.model)[0] + ".bin")
321 |
322 | # ---------------------------------- 3. Load CPU extension for support specific layer ------------------------------
323 | if "CPU" in args.device:
324 | supported_layers = ie.query_network(net, "CPU")
325 | not_supported_layers = [l for l in net.layers.keys() if l not in supported_layers]
326 | if len(not_supported_layers) != 0:
327 | log.error("Following layers are not supported by the plugin for specified device {}:\n {}".
328 | format(args.device, ', '.join(not_supported_layers)))
329 | log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l "
330 | "or --cpu_extension command line argument")
331 | sys.exit(1)
332 |
333 | assert len(net.input_info) == 1, "Sample supports only YOLO V3 based single input topologies"
334 |
335 | # ---------------------------------------------- 4. Preparing inputs -----------------------------------------------
336 | log.info("Preparing inputs")
337 | input_blob = next(iter(net.input_info))
338 |
339 | # Read and pre-process input images
340 | if net.input_info[input_blob].input_data.shape[1] == 3:
341 | input_height, input_width = net.input_info[input_blob].input_data.shape[2:]
342 | nchw_shape = True
343 | else:
344 | input_height, input_width = net.input_info[input_blob].input_data.shape[1:3]
345 | nchw_shape = False
346 |
347 | if args.labels:
348 | with open(args.labels, 'r') as f:
349 | labels_map = [x.strip() for x in f]
350 | else:
351 | labels_map = None
352 |
353 | input_stream = 0 if args.input == "cam" else args.input
354 |
355 | mode = Mode(Modes.USER_SPECIFIED)
356 | cap = cv2.VideoCapture(input_stream)
357 | wait_key_time = 1
358 |
359 | # ----------------------------------------- 5. Loading model to the plugin -----------------------------------------
360 | log.info("Loading model to the plugin")
361 | exec_nets = {}
362 |
363 | exec_nets[Modes.USER_SPECIFIED] = ie.load_network(network=net, device_name=args.device,
364 | config=config_user_specified,
365 | num_requests=args.num_infer_requests)
366 | exec_nets[Modes.MIN_LATENCY] = ie.load_network(network=net, device_name=args.device.split(":")[-1].split(",")[0],
367 | config=config_min_latency,
368 | num_requests=1)
369 |
370 | empty_requests = deque(exec_nets[mode.current].requests)
371 | completed_request_results = {}
372 | next_frame_id = 0
373 | next_frame_id_to_show = 0
374 | mode_info = { mode.current: ModeInfo() }
375 | event = threading.Event()
376 | callback_exceptions = []
377 |
378 | # ----------------------------------------------- 6. Doing inference -----------------------------------------------
379 | log.info("Starting inference...")
380 | print("To close the application, press 'CTRL+C' here or switch to the output window and press ESC key")
381 | print("To switch between min_latency/user_specified modes, press TAB key in the output window")
382 |
383 | # presenter = monitors.Presenter(args.utilization_monitors, 55,
384 | # (round(cap.get(cv2.CAP_PROP_FRAME_WIDTH) / 4), round(cap.get(cv2.CAP_PROP_FRAME_HEIGHT) / 8)))
385 |
386 | while (cap.isOpened() \
387 | or completed_request_results \
388 | or len(empty_requests) < len(exec_nets[mode.current].requests)) \
389 | and not callback_exceptions:
390 | if next_frame_id_to_show in completed_request_results:
391 | frame, output, start_time, is_same_mode = completed_request_results.pop(next_frame_id_to_show)
392 |
393 | next_frame_id_to_show += 1
394 | if is_same_mode:
395 | mode_info[mode.current].frames_count += 1
396 |
397 | objects = get_objects(output, net, (input_height, input_width), frame.shape[:-1], args.prob_threshold,
398 | args.keep_aspect_ratio)
399 | objects = filter_objects(objects, args.iou_threshold, args.prob_threshold)
400 |
401 | if len(objects) and args.raw_output_message:
402 | log.info(" Class ID | Confidence | XMIN | YMIN | XMAX | YMAX | COLOR ")
403 |
404 | origin_im_size = frame.shape[:-1]
405 | # presenter.drawGraphs(frame)
406 | for obj in objects:
407 | # Validation bbox of detected object
408 | obj['xmax'] = min(obj['xmax'], origin_im_size[1])
409 | obj['ymax'] = min(obj['ymax'], origin_im_size[0])
410 | obj['xmin'] = max(obj['xmin'], 0)
411 | obj['ymin'] = max(obj['ymin'], 0)
412 | color = (min(obj['class_id'] * 12.5, 255),
413 | min(obj['class_id'] * 7, 255),
414 | min(obj['class_id'] * 5, 255))
415 | det_label = labels_map[obj['class_id']] if labels_map and len(labels_map) >= obj['class_id'] else \
416 | str(obj['class_id'])
417 |
418 | if args.raw_output_message:
419 | log.info(
420 | "{:^9} | {:10f} | {:4} | {:4} | {:4} | {:4} | {} ".format(det_label, obj['confidence'],
421 | obj['xmin'], obj['ymin'], obj['xmax'],
422 | obj['ymax'],
423 | color))
424 |
425 | cv2.rectangle(frame, (obj['xmin'], obj['ymin']), (obj['xmax'], obj['ymax']), color, 2)
426 | cv2.putText(frame,
427 | "#" + det_label + ' ' + str(round(obj['confidence'] * 100, 1)) + ' %',
428 | (obj['xmin'], obj['ymin'] - 7), cv2.FONT_HERSHEY_COMPLEX, 0.6, color, 1)
429 |
430 | # Draw performance stats over frame
431 | if mode_info[mode.current].frames_count != 0:
432 | fps_message = "FPS: {:.1f}".format(mode_info[mode.current].frames_count / \
433 | (perf_counter() - mode_info[mode.current].last_start_time))
434 | mode_info[mode.current].latency_sum += perf_counter() - start_time
435 | latency_message = "Latency: {:.1f} ms".format((mode_info[mode.current].latency_sum / \
436 | mode_info[mode.current].frames_count) * 1e3)
437 |
438 | put_highlighted_text(frame, fps_message, (15, 20), cv2.FONT_HERSHEY_COMPLEX, 0.75, (200, 10, 10), 2)
439 | put_highlighted_text(frame, latency_message, (15, 50), cv2.FONT_HERSHEY_COMPLEX, 0.75, (200, 10, 10), 2)
440 |
441 | mode_message = "{} mode".format(mode.current.name)
442 | put_highlighted_text(frame, mode_message, (10, int(origin_im_size[0] - 20)),
443 | cv2.FONT_HERSHEY_COMPLEX, 0.75, (10, 10, 200), 2)
444 |
445 | if not args.no_show:
446 | cv2.imshow("Detection Results", frame)
447 | key = cv2.waitKey(wait_key_time)
448 |
449 | if key in {ord("q"), ord("Q"), 27}: # ESC key
450 | break
451 | if key == 9: # Tab key
452 | prev_mode = mode.current
453 | mode.next()
454 |
455 | await_requests_completion(exec_nets[prev_mode].requests)
456 | empty_requests.clear()
457 | empty_requests.extend(exec_nets[mode.current].requests)
458 |
459 | mode_info[prev_mode].last_end_time = perf_counter()
460 | mode_info[mode.current] = ModeInfo()
461 | # else:
462 | # presenter.handleKey(key)
463 |
464 | elif empty_requests and cap.isOpened():
465 | start_time = perf_counter()
466 | ret, frame = cap.read()
467 | if not ret:
468 | if args.loop_input:
469 | cap.open(input_stream)
470 | else:
471 | cap.release()
472 | continue
473 |
474 | request = empty_requests.popleft()
475 |
476 | # resize input_frame to network size
477 | in_frame = preprocess_frame(frame, input_height, input_width, nchw_shape, args.keep_aspect_ratio)
478 |
479 | # Start inference
480 | request.set_completion_callback(py_callback=async_callback,
481 | py_data=(request,
482 | next_frame_id,
483 | mode.current,
484 | frame,
485 | start_time,
486 | completed_request_results,
487 | empty_requests,
488 | mode,
489 | event,
490 | callback_exceptions))
491 | request.async_infer(inputs={input_blob: in_frame})
492 | next_frame_id += 1
493 |
494 | else:
495 | event.wait()
496 |
497 | if callback_exceptions:
498 | raise callback_exceptions[0]
499 |
500 | for mode_value in mode_info.keys():
501 | log.info("")
502 | log.info("Mode: {}".format(mode_value.name))
503 |
504 | end_time = mode_info[mode_value].last_end_time if mode_value in mode_info \
505 | and mode_info[mode_value].last_end_time is not None \
506 | else perf_counter()
507 | log.info("FPS: {:.1f}".format(mode_info[mode_value].frames_count / \
508 | (end_time - mode_info[mode_value].last_start_time)))
509 | log.info("Latency: {:.1f} ms".format((mode_info[mode_value].latency_sum / \
510 | mode_info[mode_value].frames_count) * 1e3))
511 | # print(presenter.reportMeans())
512 |
513 | for exec_net in exec_nets.values():
514 | await_requests_completion(exec_net.requests)
515 |
516 |
517 | if __name__ == '__main__':
518 | sys.exit(main() or 0)
519 |
--------------------------------------------------------------------------------
/report/OpenCV_Spatial_AI_Competition.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/nullbyte91/deepEye/b4001ee26c59a426921abe0fe65b41cb86cfcc9f/report/OpenCV_Spatial_AI_Competition.pdf
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | # depthai dep
2 | opencv-python==4.2.0.34; platform_machine != "armv7l"
3 | opencv-python==4.1.0.25; platform_machine == "armv7l"
4 | concurrent-log-handler
5 | scipy
6 |
7 | # txt2speech
8 | google-speech
--------------------------------------------------------------------------------
/scripts/inference_engine_native_myriad.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | # A script to build inference engine for ARM arch - Tested on Jetson Nano - 4.3 Jetpack
3 |
4 | # System update and Dep Install
5 | sudo apt update && sudo apt upgrade -y
6 | sudo apt install build-essential
7 |
8 | # Update cmake from source
9 | cd ~/
10 | wget https://github.com/Kitware/CMake/releases/download/v3.14.4/cmake-3.14.4.tar.gz
11 | tar xvzf cmake-3.14.4.tar.gz
12 | cd ~/cmake-3.14.4
13 |
14 | ## Configure and install
15 | ./bootstrap
16 | make -j4
17 | sudo make install
18 |
19 | # Install OpenCV
20 | cd ~/
21 | git clone https://github.com/opencv/opencv.git
22 | cd opencv && mkdir build && cd build
23 | cmake –DCMAKE_BUILD_TYPE=Release –DCMAKE_INSTALL_PREFIX=/usr/local ..
24 | make -j4
25 | sudo make install
26 |
27 | # Install OpenVino Inference Engine
28 | cd ~/
29 | git clone https://github.com/openvinotoolkit/openvino.git
30 |
31 | cd ~/openvino/inference-engine
32 | git submodule update --init --recursive
33 |
34 | ## Dep install
35 | cd ~/openvino
36 | sh ./install_dependencies.sh
37 |
38 | ## Build
39 | export OpenCV_DIR=/usr/local/lib
40 |
41 | cd ~/openvino
42 | mkdir build && cd build
43 | cmake -DCMAKE_BUILD_TYPE=Release \
44 | -DENABLE_MKL_DNN=OFF \
45 | -DENABLE_CLDNN=ON \
46 | -DENABLE_GNA=OFF \
47 | -DENABLE_SSE42=OFF \
48 | -DTHREADING=SEQ \
49 | -DENABLE_SAMPLES=ON \
50 | -DENABLE_PYTHON=ON \
51 | -DPYTHON_EXECUTABLE=`which python3.6` \
52 | -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.6m.so \
53 | -DPYTHON_INCLUDE_DIR=/usr/include/python3.6
54 | ..
55 | make -j4
56 |
57 | # Update env
58 | echo "export PYTHONPATH=$PYTHONPATH:~/openvino/bin/aarch64/Release/lib/python_api/python3.6/" >> ~/.bashrc
59 | echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/openvino/bin/aarch64/Release/lib/" >> ~/.bashrc
--------------------------------------------------------------------------------
/scripts/model_intel.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | # Use this only when you have issue with Windows download env
3 | # Otherwise use download.py from openvino toolkit
4 |
5 | dModel="./model/"
6 |
7 | # Main starts here
8 | mkdir -p ${dModel}
9 | # pedestrian-detection-adas-0002
10 | pushd ${dModel}
11 | mkdir -p pedestrian-detection-adas-0002
12 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-detection-adas-0002/FP32/pedestrian-detection-adas-0002.xml -P \
13 | pedestrian-detection-adas-0002
14 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-detection-adas-0002/FP32/pedestrian-detection-adas-0002.bin -P \
15 | pedestrian-detection-adas-0002
16 |
17 | # pedestrian-detection-adas-binary-0001
18 | mkdir -p pedestrian-detection-adas-binary-0001
19 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-detection-adas-binary-0001/FP32-INT1/pedestrian-detection-adas-binary-0001.xml -P \
20 | pedestrian-detection-adas-binary-0001
21 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-detection-adas-binary-0001/FP32-INT1/pedestrian-detection-adas-binary-0001.bin -P \
22 | pedestrian-detection-adas-binary-0001
23 |
24 | # pedestrian-and-vehicle-detector-adas-0001
25 | mkdir -p pedestrian-and-vehicle-detector-adas-0001
26 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-and-vehicle-detector-adas-0001/FP32/pedestrian-and-vehicle-detector-adas-0001.xml -P \
27 | pedestrian-and-vehicle-detector-adas-0001
28 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/pedestrian-and-vehicle-detector-adas-0001/FP32/pedestrian-and-vehicle-detector-adas-0001.bin -P \
29 | pedestrian-and-vehicle-detector-adas-0001
30 |
31 | # vehicle-detection-adas-0002
32 | mkdir -p vehicle-detection-adas-0002
33 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/vehicle-detection-adas-0002/FP32/vehicle-detection-adas-0002.xml -P \
34 | vehicle-detection-adas-0002
35 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/vehicle-detection-adas-0002/FP32/vehicle-detection-adas-0002.bin -P \
36 | vehicle-detection-adas-0002
37 |
38 | # vehicle-detection-adas-binary-0001
39 | mkdir -p vehicle-detection-adas-binary-0001
40 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/vehicle-detection-adas-binary-0001/FP32-INT1/vehicle-detection-adas-binary-0001.xml -P \
41 | vehicle-detection-adas-binary-0001
42 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/vehicle-detection-adas-binary-0001/FP32-INT1/vehicle-detection-adas-binary-0001.bin -P \
43 | vehicle-detection-adas-binary-0001
44 |
45 | # road-segmentation-adas-0001
46 | mkdir -p road-segmentation-adas-0001
47 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/road-segmentation-adas-0001/FP32/road-segmentation-adas-0001.xml -P \
48 | road-segmentation-adas-0001
49 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/road-segmentation-adas-0001/FP32/road-segmentation-adas-0001.bin -P \
50 | road-segmentation-adas-0001
51 |
52 | # semantic-segmentation-adas-0001
53 | mkdir -p semantic-segmentation-adas-0001
54 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/semantic-segmentation-adas-0001/FP32/semantic-segmentation-adas-0001.xml -P \
55 | semantic-segmentation-adas-0001
56 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/semantic-segmentation-adas-0001/FP32/semantic-segmentation-adas-0001.bin -P \
57 | semantic-segmentation-adas-0001
58 |
59 | pushd
--------------------------------------------------------------------------------
/scripts/opencv.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | set -e
4 |
5 | # Main start from here
6 | chip_id=$(cat /sys/module/tegra_fuse/parameters/tegra_chip_id)
7 | case ${chip_id} in
8 | "33" ) # Nano and TX1
9 | cuda_compute=5.3
10 | ;;
11 | "24" ) # TX2
12 | cuda_compute=6.2
13 | ;;
14 | "25" ) # AGX Xavier
15 | cuda_compute=7.2
16 | ;;
17 | * ) # default
18 | cuda_compute=5.3,6.2,7.2
19 | ;;
20 | esac
21 |
22 | py3_ver=$(python3 -c "import sys; print(sys.version_info[1])")
23 |
24 | folder=${HOME}/src
25 | mkdir -p $folder
26 |
27 | echo "** Purge old opencv installation"
28 | sudo apt-get purge -y libopencv*
29 |
30 | echo "** Install requirements"
31 | sudo apt-get update
32 | sudo apt-get install -y build-essential make cmake cmake-curses-gui git g++ pkg-config curl
33 | sudo apt-get install -y libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libeigen3-dev libglew-dev libgtk2.0-dev
34 | sudo apt-get install -y libtbb2 libtbb-dev libv4l-dev v4l-utils qv4l2 v4l2ucp
35 | sudo apt-get install -y libdc1394-22-dev libxine2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev
36 | # sudo apt-get install -y libjasper-dev
37 | sudo apt-get install -y libjpeg8-dev libjpeg-turbo8-dev libtiff-dev libpng-dev
38 | sudo apt-get install -y libxvidcore-dev libx264-dev libgtk-3-dev
39 | sudo apt-get install -y libatlas-base-dev libopenblas-dev liblapack-dev liblapacke-dev gfortran
40 | sudo apt-get install -y qt5-default
41 |
42 | sudo apt-get install -y python3-dev python3-testresources
43 | rm -f $folder/get-pip.py
44 | wget https://bootstrap.pypa.io/get-pip.py -O $folder/get-pip.py
45 | sudo python3 $folder/get-pip.py
46 | sudo pip3 install protobuf
47 | sudo pip3 install -U numpy matplotlib
48 |
49 | if [ ! -f /usr/local/cuda/include/cuda_gl_interop.h.bak ]; then
50 | sudo cp /usr/local/cuda/include/cuda_gl_interop.h /usr/local/cuda/include/cuda_gl_interop.h.bak
51 | fi
52 | sudo patch -N -r - /usr/local/cuda/include/cuda_gl_interop.h < opencv/cuda_gl_interop.h.patch && echo "** '/usr/local/cuda/include/cuda_gl_interop.h' appears to be patched already. Continue..."
53 |
54 | echo "** Download opencv-4.1.2"
55 | cd $folder
56 | wget -O opencv.zip https://github.com/opencv/opencv/archive/4.1.2.zip
57 | wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.1.2.zip
58 | unzip opencv.zip
59 | unzip opencv_contrib.zip
60 | mv opencv-4.1.2 opencv
61 | mv opencv_contrib-4.1.2 opencv_contrib
62 | cd opencv
63 | mkdir build
64 | cd build
65 | cmake -D CMAKE_BUILD_TYPE=RELEASE \
66 | -D CMAKE_INSTALL_PREFIX=/usr/local \
67 | -D WITH_CUDA=ON \
68 | -D CUDA_ARCH_PTX="" \
69 | -D CUDA_ARCH_BIN=${cuda_compute} \
70 | -D WITH_CUBLAS=ON \
71 | -D WITH_LIBV4L=ON \
72 | -D ENABLE_FAST_MATH=ON \
73 | -D CUDA_FAST_MATH=ON \
74 | -D EIGEN_INCLUDE_PATH=/usr/include/eigen3 \
75 | -D BUILD_opencv_python3=ON \
76 | -D BUILD_opencv_python2=OFF \
77 | -D BUILD_opencv_java=OFF \
78 | -D WITH_GSTREAMER=ON \
79 | -D WITH_GTK=ON \
80 | -D BUILD_TESTS=OFF \
81 | -D BUILD_PERF_TESTS=OFF \
82 | -D BUILD_EXAMPLES=OFF \
83 | -D OPENCV_ENABLE_NONFREE=ON \
84 | -D OPENCV_EXTRA_MODULES_PATH=$folder/opencv_contrib/modules ..
85 | make -j3
86 | sudo make install
87 | sudo ldconfig
88 |
89 | pushd ~/depthai_v1/lib/python3.6/site-packages/
90 | ln -s /usr/local/lib/python3.6/site-packages/cv2/python-3.6/cv2.cpython-36m-aarch64-linux-gnu.so
91 | popd
92 |
93 | python3 -c 'import cv2; print("python3 cv2 version: %s" % cv2.__version__)'
--------------------------------------------------------------------------------
/scripts/rpi_openvino_install-2020_1.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | # A script to install OpenVino toolchain 2020 1 version
4 |
5 | #Main starts from here
6 | # Downlaod openvino tool chain
7 | wget https://download.01.org/opencv/2020/openvinotoolkit/2020.1/l_openvino_toolkit_runtime_raspbian_p_2020.1.023.tgz
8 |
9 | # Create openvino install path
10 | sudo mkdir -p /opt/intel/openvino
11 |
12 | # Unzip the toolchain
13 | sudo tar -xvf l_openvino_toolkit_runtime_raspbian_p_2020.1.023.tgz --strip 1 -C /opt/intel/openvino
14 |
15 | # Install cmake
16 | sudo apt install cmake
17 |
18 | # Export a path
19 | echo "source /opt/intel/openvino/bin/setupvars.sh" >> ~/.bashrc
20 |
21 | # Add use for USB
22 | sudo usermod -a -G users "$(whoami)"
23 |
--------------------------------------------------------------------------------