├── Jupyter ├── README.md ├── inference.py ├── shopper_gaze_monitor_jupyter.ipynb └── shopper_gaze_monitor_jupyter.py ├── LICENSE ├── README.md ├── application ├── inference.py └── shopper_gaze_monitor.py ├── docs └── images │ ├── architectural-diagram.png │ ├── jupyter.png │ ├── jupyter_code.png │ └── output.png ├── resources └── config.json └── setup.sh /Jupyter/README.md: -------------------------------------------------------------------------------- 1 | # Shopper Gaze Monitor 2 | 3 | | Details | | 4 | |-----------------------|---------------| 5 | | Target OS: | Ubuntu\* 18.04 LTS | 6 | | Programming Language: | Python* 3.5 | 7 | | Time to Complete: | 30 min | 8 | 9 | 12 | ![images](../docs/images/output.png) 13 | 14 | ## What it does 15 | 16 | This shopper gaze monitor application is designed for a retail shelf mounted camera system that counts the number of passers-by and the number of people who look towards the display. It is intended to provide real-world marketing statistics for in-store shelf-space advertising. 17 | 18 | ## Requirements 19 | ### Hardware 20 | 21 | * 6th to 8th generation Intel® Core™ processor with Iris® Pro graphics or Intel® HD Graphics 22 | 23 | ### Software 24 | 25 | * Ubuntu 16.04 26 | 27 | * OpenCL™ Runtime Package 28 | 29 | **Note:** We recommend using a 4.14+ kernel to use this software. Run the following command to determine your kernel version: 30 | 31 | uname -a 32 | 33 | * Intel® Distribution of OpenVINO™ toolkit 2020 R3 Release 34 | * Jupyter* Notebook v5.7.0 35 | 36 | ## How It works 37 | 38 | The application uses the Inference Engine included in the Intel Distribution of OpenVINO toolkit and the Intel Deep Learning Deployment Toolkit. It uses a video source, such as a camera, to grab frames and then uses two different Deep Neural Networks (DNNs) to process the data. The first network looks for faces and then if successful is counted as a "Shopper" 39 | 40 | A second neural network is then used to determine the head pose detection for each detected face. If the person's head is facing towards the camera, it is counted as a "Looker" 41 | 42 | The shopper and looker data are sent to a local web server using the Paho* MQTT C client libraries. 43 | 44 | The program creates two threads for concurrency: 45 | 46 | * Main thread that performs the video i/o, processes video frames using the trained neural network. 47 | * Worker thread that publishes MQTT messages. 48 | 49 | ![images](../docs/images/architectural-diagram.png) 50 | 51 | **Architectural Diagram** 52 | 53 | ## Setup 54 | ### Get the code 55 | 56 | Clone the reference implementation: 57 | ``` 58 | sudo apt-get update && sudo apt-get install git 59 | git clone https://gitlab.devtools.intel.com/reference-implementations/shopper-gaze-monitor-python.git 60 | ``` 61 | 62 | ### Install Intel® Distribution of OpenVINO™ toolkit 63 | 64 | Refer to https://software.intel.com/en-us/articles/OpenVINO-Install-Linux for more information about how to install and setup the Intel® Distribution of OpenVINO™ toolkit. 65 | 66 | You will need the OpenCL™ Runtime package if you plan to run inference on the GPU. It is not mandatory for CPU inference. 67 | 68 | ### Other dependencies 69 | #### Mosquitto* 70 | Mosquitto is an open source message broker that implements the MQTT protocol. The MQTT protocol provides a lightweight method of carrying out messaging using a publish/subscribe model. 71 | 72 | ### Which model to use 73 | 74 | This application uses the [face-detection-adas-0001](https://docs.openvinotoolkit.org/2020.3/_models_intel_face_detection_adas_0001_description_face_detection_adas_0001.html) and [head-pose-estimation-adas-0001](https://docs.openvinotoolkit.org/2020.3/_models_intel_human_pose_estimation_0001_description_human_pose_estimation_0001.html) Intel® model, that can be downloaded using the **model downloader**. The **model downloader** downloads the __.xml__ and __.bin__ files that is used by the application. 75 | 76 | To install the dependencies of the RI and to download the models Intel® model, run the following command: 77 | 78 | cd 79 | ./setup.sh 80 | 81 | The models will be downloaded inside the following directories: 82 | 83 | /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/ 84 | /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/ 85 | 86 | ### The Config File 87 | 88 | The _resources/config.json_ contains the path to the videos that will be used by the application. 89 | The _config.json_ file is of the form name/value pair, `video: ` 90 | 91 | Example of the _config.json_ file: 92 | 93 | ``` 94 | { 95 | 96 | "inputs": [ 97 | { 98 | "video": "videos/video1.mp4" 99 | } 100 | ] 101 | } 102 | ``` 103 | 104 | ### Which Input video to use 105 | 106 | The application works with any input video. Find sample videos [here](https://github.com/intel-iot-devkit/sample-videos/). 107 | 108 | For first-use, we recommend using the [face-demographics-walking-and-pause](https://github.com/intel-iot-devkit/sample-videos/blob/master/face-demographics-walking-and-pause.mp4) video.The video is automatically downloaded to the `resources/` folder. 109 | For example:
110 | The config.json would be: 111 | 112 | ``` 113 | { 114 | 115 | "inputs": [ 116 | { 117 | "video": "sample-videos/face-demographics-walking-and-pause.mp4" 118 | } 119 | ] 120 | } 121 | ``` 122 | To use any other video, specify the path in config.json file 123 | 124 | ### Using the Camera instead of video 125 | 126 | Replace the path/to/video in the _resources/config.json_ file with the camera ID, where the ID is taken from the video device (the number X in /dev/videoX). 127 | 128 | For example: 129 | 130 | ``` 131 | { 132 | 133 | "inputs": [ 134 | { 135 | "video": "0" 136 | } 137 | ] 138 | } 139 | ``` 140 | 141 | On Ubuntu, list all available video devices with the following command: 142 | 143 | ``` 144 | ls /dev/video* 145 | ``` 146 | 147 | ## Setup the environment 148 | You must configure the environment to use the Intel® Distribution of OpenVINO™ toolkit one time per session by running the following command: 149 | 150 | source /opt/intel/openvino/bin/setupvars.sh 151 | 152 | __Note__: This command needs to be executed only once in the terminal where the application will be executed. If the terminal is closed, the command needs to be executed again. 153 | 154 | ### Run the Application on Jupyter* 155 | 156 | * Go to the _shopper-gaze-monitor-python_ and open the Jupyter notebook by running the following commands: 157 | 158 | ``` 159 | cd /Jupyter 160 | jupyter notebook 161 | ``` 162 | 163 | 193 | 194 | 1. Go to the _shopper-gaze-monitor-python_ directory and open the Jupyter notebook by running the following command: 195 | 196 | cd /Jupyter 197 | 198 | jupyter notebook 199 | 200 | ![images](../docs/images/jupyter.png) 201 | 202 | 2. Click on **New** button on the right side of the Jupyter window. 203 | 204 | 3. Click on **Python 3** option from the drop down list. 205 | 206 | 4. In the first cell type **import os** and press **Shift+Enter** from the keyboard. 207 | 208 | 5. Export the below environment variables in second cell of Jupyter and press **Shift+Enter**. 209 | 210 | %env DEVICE = CPU 211 | %env MODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP32/face-detection-adas-0001.xml 212 | %env POSEMODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP32/head-pose-estimation-adas-0001.xml 213 | 214 | 6. User can set threshold for the detection (CONFIDENCE). Export the environment variables as given below if required else skip this step. If user skips this step, these values are set to default values. 215 | 216 | %env CONFIDENCE = 0.5 217 | 218 | To run the application on sync mode, export the environment variable **%env FLAG = sync**. By default, the application runs on async mode. 219 | 220 | 7. Copy the code from **shopper_gaze_monitor_jupyter.py** and paste it in the next cell and press **Shift+Enter**. 221 | 222 | 8. Alternatively, code can be run in the following way. 223 | 224 | i. Click on the **shopper_gaze_monitor_jupyter.ipynb** file in the Jupyter notebook window. 225 | 226 | ii. Click on the **Kernel** menu and then select **Restart & Run All** from the drop down list. 227 | 228 | iii. Click on Restart and Run All Cells. 229 | 230 | ![images](../docs/images/jupyter_code.png) 231 | 232 | **NOTE:** 233 | 234 | 1. To run the application on **GPU**: 235 | * With the floating point precision 32 (FP32), change the **%env DEVICE = CPU** to **%env DEVICE = GPU**
236 | **FP32**: FP32 is single-precision floating-point arithmetic uses 32 bits to represent numbers. 8 bits for the magnitude and 23 bits for the precision. For more information, [click here](https://en.wikipedia.org/wiki/Single-precision_floating-point_format)
237 | 238 | * With the floating point precision 16 (FP16), change the environment variables as given below:
239 | 240 | %env DEVICE = GPU 241 | %env MODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP16/face-detection-adas-0001.xml 242 | %env POSEMODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP16/head-pose-estimation-adas-0001.xml 243 | 244 | **FP16**: FP16 is half-precision floating-point arithmetic uses 16 bits. 5 bits for the magnitude and 10 bits for the precision. For more information, [click here](https://en.wikipedia.org/wiki/Half-precision_floating-point_format) 245 | * **CPU_EXTENSION** environment variable is not required.
246 | 247 | 2. To run the application on **Intel® Neural Compute Stick**: 248 | * Change the **%env DEVICE = CPU** to **%env DEVICE = MYRIAD** 249 | * The Intel® Neural Compute Stick can only run FP16 models. Hence change the environment variable for the model as shown below.
250 | 251 | %env MODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP16/face-detection-adas-0001.xml 252 | %env POSEMODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP16/head-pose-estimation-adas-0001.xml 253 | * **CPU_EXTENSION** environment variable is not required.
254 | 255 | 3. To run the application on **Intel® Movidius™ VPU**: 256 | * Change the **%env DEVICE = CPU** to **%env DEVICE = HDDL** 257 | * The Intel® Movidius™ VPU can only run FP16 models. Change the environment variable for the model as shown below and the models that are passed to the application must be of data type FP16.
258 | 259 | %env MODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP16/face-detection-adas-0001.xml 260 | %env POSEMODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP16/head-pose-estimation-adas-0001.xml 261 | 262 | * **CPU_EXTENSION** environment variable is not required.
263 | 264 | 265 | 271 | 272 | 273 | 5. To run the application on multiple devices:
274 | For example: 275 | * Change the **%env DEVICE = CPU** to **%env DEVICE = MULTI:CPU,GPU,MYRIAD** 276 | * The Intel® Neural Compute Stick can only run FP16 models. Hence change the environment variable for the model as shown below.
277 | 278 | %env MODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP16/face-detection-adas-0001.xml 279 | %env POSEMODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP16/head-pose-estimation-adas-0001.xml 280 | 281 | 282 | 283 | ## Machine to machine messaging with MQTT 284 | 285 | If you wish to use a MQTT server to publish data, you should set the following environment variables on the terminal before opening the jupyter notebook. 286 | 287 | export MQTT_SERVER=localhost:1883 288 | export MQTT_CLIENT_ID=cvservice 289 | 290 | Change the MQTT_SERVER to a value that matches the MQTT server you are connecting to. 291 | 292 | You should change the MQTT_CLIENT_ID to a unique value for each monitoring station, so you can track the data for individual locations. For example: 293 | 294 | export MQTT_CLIENT_ID=zone1337 295 | 296 | If you want to monitor the MQTT messages sent to your local server, and you have the mosquitto client utilities installed, you can run the following command in new terminal while executing the code: 297 | 298 | mosquitto_sub -h localhost -t shopper_gaze_monitor 299 | -------------------------------------------------------------------------------- /Jupyter/inference.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | Copyright (c) 2018 Intel Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining 6 | a copy of this software and associated documentation files (the 7 | "Software"), to deal in the Software without restriction, including 8 | without limitation the rights to use, copy, modify, merge, publish, 9 | distribute, sublicense, and/or sell copies of the Software, and to 10 | permit persons to whom the Software is furnished to do so, subject to 11 | the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be 14 | included in all copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 19 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 20 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 21 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 22 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 23 | """ 24 | 25 | import os 26 | import sys 27 | import logging as log 28 | from openvino.inference_engine import IENetwork, IECore 29 | 30 | 31 | class Network: 32 | """ 33 | Load and configure inference plugins for the specified target devices 34 | and performs synchronous and asynchronous modes for the specified infer requests. 35 | """ 36 | 37 | def __init__(self): 38 | self.net = None 39 | self.plugin = None 40 | self.input_blob = None 41 | self.out_blob = None 42 | self.net_plugin = None 43 | self.infer_request_handle = None 44 | 45 | def load_model(self, model, device, input_size, output_size, num_requests, cpu_extension=None, plugin=None): 46 | """ 47 | Loads a network and an image to the Inference Engine plugin. 48 | :param model: .xml file of pre trained model 49 | :param cpu_extension: extension for the CPU device 50 | :param device: Target device 51 | :param input_size: Number of input layers 52 | :param output_size: Number of output layers 53 | :param num_requests: Index of Infer request value. Limited to device capabilities. 54 | :param plugin: Plugin for specified device 55 | :return: Shape of input layer 56 | """ 57 | 58 | model_xml = model 59 | model_bin = os.path.splitext(model_xml)[0] + ".bin" 60 | # Plugin initialization for specified device 61 | # and load extensions library if specified 62 | if not plugin: 63 | log.info("Initializing plugin for {} device...".format(device)) 64 | self.plugin = IECore() 65 | else: 66 | self.plugin = plugin 67 | 68 | if cpu_extension and 'CPU' in device: 69 | self.plugin.add_extension(cpu_extension, "CPU") 70 | 71 | # Read IR 72 | log.info("Reading IR...") 73 | self.net = self.plugin.read_network(model=model_xml, weights=model_bin) 74 | log.info("Loading IR to the plugin...") 75 | 76 | if "CPU" in device: 77 | supported_layers = self.plugin.query_network(self.net, "CPU") 78 | not_supported_layers = \ 79 | [l for l in self.net.layers.keys() if l not in supported_layers] 80 | if len(not_supported_layers) != 0: 81 | log.error("Following layers are not supported by " 82 | "the plugin for specified device {}:\n {}". 83 | format(device, 84 | ', '.join(not_supported_layers))) 85 | log.error("Please try to specify cpu extensions library path" 86 | " in command line parameters using -l " 87 | "or --cpu_extension command line argument") 88 | sys.exit(1) 89 | 90 | if num_requests == 0: 91 | # Loads network read from IR to the plugin 92 | self.net_plugin = self.plugin.load_network(network=self.net, device_name=device) 93 | else: 94 | self.net_plugin = self.plugin.load_network(network=self.net, num_requests=num_requests, device_name=device) 95 | 96 | self.input_blob = next(iter(self.net.inputs)) 97 | self.out_blob = next(iter(self.net.outputs)) 98 | assert len(self.net.inputs.keys()) == input_size, \ 99 | "Supports only {} input topologies".format(len(self.net.inputs)) 100 | assert len(self.net.outputs) == output_size, \ 101 | "Supports only {} output topologies".format(len(self.net.outputs)) 102 | 103 | return self.plugin, self.get_input_shape() 104 | 105 | def get_input_shape(self): 106 | """ 107 | Gives the shape of the input layer of the network. 108 | :return: None 109 | """ 110 | return self.net.inputs[self.input_blob].shape 111 | 112 | def performance_counter(self, request_id): 113 | """ 114 | Queries performance measures per layer to get feedback of what is the 115 | most time consuming layer. 116 | :param request_id: Index of Infer request value. Limited to device capabilities 117 | :return: Performance of the layer 118 | """ 119 | perf_count = self.net_plugin.requests[request_id].get_perf_counts() 120 | return perf_count 121 | 122 | def exec_net(self, request_id, frame): 123 | """ 124 | Starts asynchronous inference for specified request. 125 | :param request_id: Index of Infer request value. Limited to device capabilities. 126 | :param frame: Input image 127 | :return: Instance of Executable Network class 128 | """ 129 | self.infer_request_handle = self.net_plugin.start_async( 130 | request_id=request_id, inputs={self.input_blob: frame}) 131 | return self.net_plugin 132 | 133 | def wait(self, request_id): 134 | """ 135 | Waits for the result to become available. 136 | :param request_id: Index of Infer request value. Limited to device capabilities. 137 | :return: Timeout value 138 | """ 139 | wait_process = self.net_plugin.requests[request_id].wait(-1) 140 | return wait_process 141 | 142 | def get_output(self, request_id, output=None): 143 | """ 144 | Gives a list of results for the output layer of the network. 145 | :param request_id: Index of Infer request value. Limited to device capabilities. 146 | :param output: Name of the output layer 147 | :return: Results for the specified request 148 | """ 149 | if output: 150 | res = self.infer_request_handle.outputs[output] 151 | else: 152 | res = self.net_plugin.requests[request_id].outputs[self.out_blob] 153 | return res 154 | 155 | def clean(self): 156 | """ 157 | Deletes all the instances 158 | :return: None 159 | """ 160 | del self.net_plugin 161 | del self.plugin 162 | del self.net 163 | -------------------------------------------------------------------------------- /Jupyter/shopper_gaze_monitor_jupyter.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 2, 15 | "metadata": {}, 16 | "outputs": [ 17 | { 18 | "name": "stdout", 19 | "output_type": "stream", 20 | "text": [ 21 | "env: DEVICE=CPU\n", 22 | "env: POSEMODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP32/head-pose-estimation-adas-0001.xml\n", 23 | "env: MODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP32/face-detection-adas-0001.xml\n" 24 | ] 25 | } 26 | ], 27 | "source": [ 28 | "%env DEVICE = CPU\n", 29 | "%env POSEMODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP32/head-pose-estimation-adas-0001.xml\n", 30 | "%env MODEL=/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP32/face-detection-adas-0001.xml" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": 3, 36 | "metadata": {}, 37 | "outputs": [ 38 | { 39 | "name": "stdout", 40 | "output_type": "stream", 41 | "text": [ 42 | "[ INFO ] Initializing plugin for CPU device...\n", 43 | "[ INFO ] Reading IR...\n", 44 | "[ INFO ] Loading IR to the plugin...\n", 45 | "[ INFO ] Reading IR...\n", 46 | "[ INFO ] Loading IR to the plugin...\n", 47 | "Application running in async mode...\n", 48 | "Attempting to stop background threads\n" 49 | ] 50 | } 51 | ], 52 | "source": [ 53 | "\"\"\"Shopper Gaze Monitor.\"\"\"\n", 54 | "\n", 55 | "\"\"\"\n", 56 | " Copyright (c) 2018 Intel Corporation.\n", 57 | "\n", 58 | " Permission is hereby granted, free of charge, to any person obtaining\n", 59 | " a copy of this software and associated documentation files (the\n", 60 | " \"Software\"), to deal in the Software without restriction, including\n", 61 | " without limitation the rights to use, copy, modify, merge, publish,\n", 62 | " distribute, sublicense, and/or sell copies of the Software, and to\n", 63 | " permit person to whom the Software is furnished to do so, subject to\n", 64 | " the following conditions:\n", 65 | "\n", 66 | " The above copyright notice and this permission notice shall be\n", 67 | " included in all copies or substantial portions of the Software.\n", 68 | "\n", 69 | " THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\n", 70 | " EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\n", 71 | " MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND\n", 72 | " NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE\n", 73 | " LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\n", 74 | " OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION\n", 75 | " WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n", 76 | "\n", 77 | "\"\"\"\n", 78 | "\n", 79 | "import os\n", 80 | "import sys\n", 81 | "import json\n", 82 | "import time\n", 83 | "import cv2\n", 84 | "\n", 85 | "import logging as log\n", 86 | "import paho.mqtt.client as mqtt\n", 87 | "\n", 88 | "from inference import Network\n", 89 | "from threading import Thread\n", 90 | "from collections import namedtuple\n", 91 | "\n", 92 | "# shoppingInfo contains statistics for the shopping information\n", 93 | "MyStruct = namedtuple(\"shoppingInfo\", \"shopper, looker\")\n", 94 | "INFO = MyStruct(0, 0)\n", 95 | "\n", 96 | "POSE_CHECKED = False\n", 97 | "\n", 98 | "# MQTT server environment variables\n", 99 | "TOPIC = \"shopper_gaze_monitor\"\n", 100 | "MQTT_HOST = \"localhost\"\n", 101 | "MQTT_PORT = 1883\n", 102 | "MQTT_KEEPALIVE_INTERVAL = 60\n", 103 | "\n", 104 | "# Global variables\n", 105 | "TARGET_DEVICE = 'CPU'\n", 106 | "accepted_devices = ['CPU', 'GPU', 'MYRIAD', 'HETERO:FPGA,CPU', 'HDDL']\n", 107 | "is_async_mode = True\n", 108 | "CONFIG_FILE = '../resources/config.json'\n", 109 | "\n", 110 | "# Flag to control background thread\n", 111 | "KEEP_RUNNING = True\n", 112 | "\n", 113 | "DELAY = 5\n", 114 | "\n", 115 | "\n", 116 | "def face_detection(res, initial_wh):\n", 117 | " \"\"\"\n", 118 | " Parse Face detection output.\n", 119 | "\n", 120 | " :param res: Detection results\n", 121 | " :param initial_wh: Initial width and height of the FRAME\n", 122 | " :return: Co-ordinates of the detected face\n", 123 | " \"\"\"\n", 124 | " global INFO\n", 125 | " faces = []\n", 126 | " INFO = INFO._replace(shopper=0)\n", 127 | "\n", 128 | " for obj in res[0][0]:\n", 129 | " # Draw only objects when probability more than specified threshold\n", 130 | " if obj[2] > CONFIDENCE:\n", 131 | " if obj[3] < 0:\n", 132 | " obj[3] = -obj[3]\n", 133 | " if obj[4] < 0:\n", 134 | " obj[4] = -obj[4]\n", 135 | " xmin = int(obj[3] * initial_wh[0])\n", 136 | " ymin = int(obj[4] * initial_wh[1])\n", 137 | " xmax = int(obj[5] * initial_wh[0])\n", 138 | " ymax = int(obj[6] * initial_wh[1])\n", 139 | " faces.append([xmin, ymin, xmax, ymax])\n", 140 | " INFO = INFO._replace(shopper=len(faces))\n", 141 | " return faces\n", 142 | "\n", 143 | "\n", 144 | "def message_runner():\n", 145 | " \"\"\"\n", 146 | " Publish worker status to MQTT topic.\n", 147 | "\n", 148 | " Pauses for rate second(s) between updates\n", 149 | " :return: None\n", 150 | " \"\"\"\n", 151 | " while KEEP_RUNNING:\n", 152 | " payload = json.dumps({\"Shopper\": INFO.shopper, \"Looker\": INFO.looker})\n", 153 | " time.sleep(1)\n", 154 | " CLIENT.publish(TOPIC, payload=payload)\n", 155 | "\n", 156 | "\n", 157 | "def main():\n", 158 | " \"\"\"\n", 159 | " Load the network and parse the output.\n", 160 | "\n", 161 | " :return: None\n", 162 | " \"\"\"\n", 163 | " global INFO\n", 164 | " global DELAY\n", 165 | " global CLIENT\n", 166 | " global KEEP_RUNNING\n", 167 | " global POSE_CHECKED\n", 168 | " global CONFIDENCE\n", 169 | " global TARGET_DEVICE\n", 170 | " global is_async_mode\n", 171 | "\n", 172 | " CLIENT = mqtt.Client()\n", 173 | " CLIENT.connect(MQTT_HOST, MQTT_PORT, MQTT_KEEPALIVE_INTERVAL)\n", 174 | "\n", 175 | " model = os.environ[\"MODEL\"]\n", 176 | " posemodel = os.environ[\"POSEMODEL\"]\n", 177 | "\n", 178 | " try:\n", 179 | " CONFIDENCE = float(os.environ['CONFIDENCE'])\n", 180 | " except:\n", 181 | " CONFIDENCE = 0.5\n", 182 | "\n", 183 | " if 'DEVICE' in os.environ.keys():\n", 184 | " TARGET_DEVICE = os.environ['DEVICE']\n", 185 | " if 'MULTI' not in TARGET_DEVICE and TARGET_DEVICE not in accepted_devices:\n", 186 | " print(\"Unsupported device: \" + TARGET_DEVICE)\n", 187 | " sys.exit(1)\n", 188 | " elif 'MULTI' in TARGET_DEVICE:\n", 189 | " target_devices = TARGET_DEVICE.split(':')[1].split(',')\n", 190 | " for multi_device in target_devices:\n", 191 | " if multi_device not in accepted_devices:\n", 192 | " print(\"Unsupported device: \" + TARGET_DEVICE)\n", 193 | " sys.exit(1)\n", 194 | " cpu_extension = os.environ['CPU_EXTENSION'] if 'CPU_EXTENSION' in os.environ.keys() else None\n", 195 | "\n", 196 | " if 'FLAG' in os.environ.keys():\n", 197 | " async_mode = os.environ['FLAG']\n", 198 | " if async_mode == \"sync\":\n", 199 | " is_async_mode = False\n", 200 | " else:\n", 201 | " is_async_mode = True\n", 202 | "\n", 203 | " log.basicConfig(format=\"[ %(levelname)s ] %(message)s\",\n", 204 | " level=log.INFO, stream=sys.stdout)\n", 205 | " logger = log.getLogger()\n", 206 | "\n", 207 | " assert os.path.isfile(CONFIG_FILE), \"{} file doesn't exist\".format(CONFIG_FILE)\n", 208 | " config = json.loads(open(CONFIG_FILE).read())\n", 209 | "\n", 210 | " for idx, item in enumerate(config['inputs']):\n", 211 | " if item['video'].isdigit():\n", 212 | " input_stream = int(item['video'])\n", 213 | " else:\n", 214 | " input_stream = item['video']\n", 215 | "\n", 216 | " cap = cv2.VideoCapture(input_stream)\n", 217 | "\n", 218 | " if not cap.isOpened():\n", 219 | " logger.error(\"ERROR! Unable to open video source\")\n", 220 | " return\n", 221 | "\n", 222 | " if input_stream:\n", 223 | " cap.open(input_stream)\n", 224 | " # Adjust DELAY to match the number of FPS of the video file\n", 225 | " DELAY = 1000 / cap.get(cv2.CAP_PROP_FPS)\n", 226 | "\n", 227 | " # Init inference request IDs\n", 228 | " cur_request_id = 0\n", 229 | " next_request_id = 1\n", 230 | "\n", 231 | " # Initialise the class\n", 232 | " infer_network = Network()\n", 233 | " infer_network_pose = Network()\n", 234 | " # Load the network to IE plugin to get shape of input layer\n", 235 | " plugin, (n_fd, c_fd, h_fd, w_fd) = infer_network.load_model(model, TARGET_DEVICE,\n", 236 | " 1, 1, 2, cpu_extension)\n", 237 | " n_hp, c_hp, h_hp, w_hp = infer_network_pose.load_model(posemodel,\n", 238 | " TARGET_DEVICE, 1, 3, 2,\n", 239 | " cpu_extension, plugin)[1]\n", 240 | "\n", 241 | " message_thread = Thread(target=message_runner)\n", 242 | " message_thread.setDaemon(True)\n", 243 | " message_thread.start()\n", 244 | "\n", 245 | " if is_async_mode:\n", 246 | " print(\"Application running in async mode...\")\n", 247 | " else:\n", 248 | " print(\"Application running in sync mode...\")\n", 249 | " det_time_fd = 0\n", 250 | " ret, frame = cap.read()\n", 251 | " while ret:\n", 252 | "\n", 253 | " looking = 0\n", 254 | " ret, frame = cap.read()\n", 255 | " if not ret:\n", 256 | " KEEP_RUNNING = False\n", 257 | " break\n", 258 | "\n", 259 | " initial_wh = [cap.get(3), cap.get(4)]\n", 260 | " in_frame_fd = cv2.resize(frame, (w_fd, h_fd))\n", 261 | " # Change data layout from HWC to CHW\n", 262 | " in_frame_fd = in_frame_fd.transpose((2, 0, 1))\n", 263 | " in_frame_fd = in_frame_fd.reshape((n_fd, c_fd, h_fd, w_fd))\n", 264 | "\n", 265 | " if frame is None:\n", 266 | " KEEP_RUNNING = False\n", 267 | " log.error(\"ERROR! blank FRAME grabbed\")\n", 268 | " break\n", 269 | "\n", 270 | " key_pressed = cv2.waitKey(int(DELAY))\n", 271 | "\n", 272 | " # Start asynchronous inference for specified request\n", 273 | " inf_start_fd = time.time()\n", 274 | " if is_async_mode:\n", 275 | " # Async enabled and only one video capture\n", 276 | " infer_network.exec_net(next_request_id, in_frame_fd)\n", 277 | " else:\n", 278 | " # Async disabled\n", 279 | " infer_network.exec_net(cur_request_id, in_frame_fd) # Wait for the result\n", 280 | " if infer_network.wait(cur_request_id) == 0:\n", 281 | " det_time_fd = time.time() - inf_start_fd\n", 282 | "\n", 283 | " # Results of the output layer of the network\n", 284 | " res = infer_network.get_output(cur_request_id)\n", 285 | "\n", 286 | " # Parse face detection output\n", 287 | " faces = face_detection(res, initial_wh)\n", 288 | "\n", 289 | " if len(faces) != 0:\n", 290 | " # Look for poses\n", 291 | " for res_hp in faces:\n", 292 | " xmin, ymin, xmax, ymax = res_hp\n", 293 | " head_pose = frame[ymin:ymax, xmin:xmax]\n", 294 | " in_frame_hp = cv2.resize(head_pose, (w_hp, h_hp))\n", 295 | " in_frame_hp = in_frame_hp.transpose((2, 0, 1))\n", 296 | " in_frame_hp = in_frame_hp.reshape((n_hp, c_hp, h_hp, w_hp))\n", 297 | "\n", 298 | " inf_start_hp = time.time()\n", 299 | " infer_network_pose.exec_net(0, in_frame_hp)\n", 300 | " infer_network_pose.wait(0)\n", 301 | " det_time_hp = time.time() - inf_start_hp\n", 302 | "\n", 303 | " # Parse head pose detection results\n", 304 | " angle_p_fc = infer_network_pose.get_output(0, \"angle_p_fc\")\n", 305 | " angle_y_fc = infer_network_pose.get_output(0, \"angle_y_fc\")\n", 306 | " if ((angle_y_fc > -22.5) & (angle_y_fc < 22.5) & (angle_p_fc > -22.5) &\n", 307 | " (angle_p_fc < 22.5)):\n", 308 | " looking += 1\n", 309 | " POSE_CHECKED = True\n", 310 | " INFO = INFO._replace(looker=looking)\n", 311 | " else:\n", 312 | " INFO = INFO._replace(looker=looking)\n", 313 | " else:\n", 314 | " INFO = INFO._replace(looker=0)\n", 315 | "\n", 316 | " # Draw performance stats\n", 317 | " inf_time_message = \"Face Inference time: N\\A for async mode\" if is_async_mode else \\\n", 318 | " \"Inference time: {:.3f} ms\".format(det_time_fd * 1000)\n", 319 | "\n", 320 | " if POSE_CHECKED:\n", 321 | " head_inf_time_message = \"Head pose Inference time: N\\A for async mode\" if is_async_mode else \\\n", 322 | " \"Inference time: {:.3f} ms\".format(det_time_hp * 1000)\n", 323 | " cv2.putText(frame, head_inf_time_message, (0, 55),\n", 324 | " cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1)\n", 325 | " log_message = \"Async mode is on.\" if is_async_mode else \\\n", 326 | " \"Async mode is off.\"\n", 327 | " cv2.putText(frame, log_message, (0, 15), cv2.FONT_HERSHEY_SIMPLEX,\n", 328 | " 0.5, (255, 255, 255), 1)\n", 329 | " cv2.putText(frame, inf_time_message, (0, 35), cv2.FONT_HERSHEY_SIMPLEX,\n", 330 | " 0.5, (255, 255, 255), 1)\n", 331 | " cv2.putText(frame, \"Shopper: {}\".format(INFO.shopper), (0, 90),\n", 332 | " cv2.FONT_HERSHEY_SIMPLEX,\n", 333 | " 0.5, (255, 255, 255), 1)\n", 334 | " cv2.putText(frame, \"Looker: {}\".format(INFO.looker), (0, 110),\n", 335 | " cv2.FONT_HERSHEY_SIMPLEX,\n", 336 | " 0.5, (255, 255, 255), 1)\n", 337 | "\n", 338 | " cv2.imshow(\"Shopper Gaze Monitor\", frame)\n", 339 | "\n", 340 | " if key_pressed == 27:\n", 341 | " print(\"Attempting to stop background threads\")\n", 342 | " KEEP_RUNNING = False\n", 343 | " break\n", 344 | " if key_pressed == 9:\n", 345 | " is_async_mode = not is_async_mode\n", 346 | " print(\"Switched to {} mode\".format(\"async\" if is_async_mode else \"sync\"))\n", 347 | "\n", 348 | " if is_async_mode:\n", 349 | " # Swap infer request IDs\n", 350 | " cur_request_id, next_request_id = next_request_id, cur_request_id\n", 351 | "\n", 352 | " infer_network.clean()\n", 353 | " infer_network_pose.clean()\n", 354 | " message_thread.join()\n", 355 | " cap.release()\n", 356 | " cv2.destroyAllWindows()\n", 357 | " CLIENT.disconnect()\n", 358 | "\n", 359 | "\n", 360 | "if __name__ == '__main__':\n", 361 | " main()" 362 | ] 363 | }, 364 | { 365 | "cell_type": "code", 366 | "execution_count": null, 367 | "metadata": {}, 368 | "outputs": [], 369 | "source": [] 370 | } 371 | ], 372 | "metadata": { 373 | "kernelspec": { 374 | "display_name": "Python 3", 375 | "language": "python", 376 | "name": "python3" 377 | }, 378 | "language_info": { 379 | "codemirror_mode": { 380 | "name": "ipython", 381 | "version": 3 382 | }, 383 | "file_extension": ".py", 384 | "mimetype": "text/x-python", 385 | "name": "python", 386 | "nbconvert_exporter": "python", 387 | "pygments_lexer": "ipython3", 388 | "version": "3.6.9" 389 | } 390 | }, 391 | "nbformat": 4, 392 | "nbformat_minor": 2 393 | } 394 | -------------------------------------------------------------------------------- /Jupyter/shopper_gaze_monitor_jupyter.py: -------------------------------------------------------------------------------- 1 | """Shopper Gaze Monitor.""" 2 | 3 | """ 4 | Copyright (c) 2018 Intel Corporation. 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining 7 | a copy of this software and associated documentation files (the 8 | "Software"), to deal in the Software without restriction, including 9 | without limitation the rights to use, copy, modify, merge, publish, 10 | distribute, sublicense, and/or sell copies of the Software, and to 11 | permit person to whom the Software is furnished to do so, subject to 12 | the following conditions: 13 | 14 | The above copyright notice and this permission notice shall be 15 | included in all copies or substantial portions of the Software. 16 | 17 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 18 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 19 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 20 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 21 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 22 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 23 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 24 | 25 | """ 26 | 27 | import os 28 | import sys 29 | import json 30 | import time 31 | import cv2 32 | 33 | import logging as log 34 | import paho.mqtt.client as mqtt 35 | 36 | from inference import Network 37 | from threading import Thread 38 | from collections import namedtuple 39 | 40 | # shoppingInfo contains statistics for the shopping information 41 | MyStruct = namedtuple("shoppingInfo", "shopper, looker") 42 | INFO = MyStruct(0, 0) 43 | 44 | POSE_CHECKED = False 45 | 46 | # MQTT server environment variables 47 | TOPIC = "shopper_gaze_monitor" 48 | MQTT_HOST = "localhost" 49 | MQTT_PORT = 1883 50 | MQTT_KEEPALIVE_INTERVAL = 60 51 | 52 | # Global variables 53 | TARGET_DEVICE = 'CPU' 54 | accepted_devices = ['CPU', 'GPU', 'MYRIAD', 'HETERO:FPGA,CPU', 'HDDL'] 55 | is_async_mode = True 56 | CONFIG_FILE = '../resources/config.json' 57 | 58 | # Flag to control background thread 59 | KEEP_RUNNING = True 60 | 61 | DELAY = 5 62 | 63 | 64 | def face_detection(res, initial_wh): 65 | """ 66 | Parse Face detection output. 67 | 68 | :param res: Detection results 69 | :param initial_wh: Initial width and height of the FRAME 70 | :return: Co-ordinates of the detected face 71 | """ 72 | global INFO 73 | faces = [] 74 | INFO = INFO._replace(shopper=0) 75 | 76 | for obj in res[0][0]: 77 | # Draw only objects when probability more than specified threshold 78 | if obj[2] > CONFIDENCE: 79 | if obj[3] < 0: 80 | obj[3] = -obj[3] 81 | if obj[4] < 0: 82 | obj[4] = -obj[4] 83 | xmin = int(obj[3] * initial_wh[0]) 84 | ymin = int(obj[4] * initial_wh[1]) 85 | xmax = int(obj[5] * initial_wh[0]) 86 | ymax = int(obj[6] * initial_wh[1]) 87 | faces.append([xmin, ymin, xmax, ymax]) 88 | INFO = INFO._replace(shopper=len(faces)) 89 | return faces 90 | 91 | 92 | def message_runner(): 93 | """ 94 | Publish worker status to MQTT topic. 95 | 96 | Pauses for rate second(s) between updates 97 | :return: None 98 | """ 99 | while KEEP_RUNNING: 100 | payload = json.dumps({"Shopper": INFO.shopper, "Looker": INFO.looker}) 101 | time.sleep(1) 102 | CLIENT.publish(TOPIC, payload=payload) 103 | 104 | 105 | def main(): 106 | """ 107 | Load the network and parse the output. 108 | 109 | :return: None 110 | """ 111 | global INFO 112 | global DELAY 113 | global CLIENT 114 | global KEEP_RUNNING 115 | global POSE_CHECKED 116 | global CONFIDENCE 117 | global TARGET_DEVICE 118 | global is_async_mode 119 | 120 | CLIENT = mqtt.Client() 121 | CLIENT.connect(MQTT_HOST, MQTT_PORT, MQTT_KEEPALIVE_INTERVAL) 122 | 123 | model = os.environ["MODEL"] 124 | posemodel = os.environ["POSEMODEL"] 125 | 126 | try: 127 | CONFIDENCE = float(os.environ['CONFIDENCE']) 128 | except: 129 | CONFIDENCE = 0.5 130 | 131 | if 'DEVICE' in os.environ.keys(): 132 | TARGET_DEVICE = os.environ['DEVICE'] 133 | if 'MULTI' not in TARGET_DEVICE and TARGET_DEVICE not in accepted_devices: 134 | print("Unsupported device: " + TARGET_DEVICE) 135 | sys.exit(1) 136 | elif 'MULTI' in TARGET_DEVICE: 137 | target_devices = TARGET_DEVICE.split(':')[1].split(',') 138 | for multi_device in target_devices: 139 | if multi_device not in accepted_devices: 140 | print("Unsupported device: " + TARGET_DEVICE) 141 | sys.exit(1) 142 | cpu_extension = os.environ['CPU_EXTENSION'] if 'CPU_EXTENSION' in os.environ.keys() else None 143 | 144 | if 'FLAG' in os.environ.keys(): 145 | async_mode = os.environ['FLAG'] 146 | if async_mode == "sync": 147 | is_async_mode = False 148 | else: 149 | is_async_mode = True 150 | 151 | log.basicConfig(format="[ %(levelname)s ] %(message)s", 152 | level=log.INFO, stream=sys.stdout) 153 | logger = log.getLogger() 154 | 155 | assert os.path.isfile(CONFIG_FILE), "{} file doesn't exist".format(CONFIG_FILE) 156 | config = json.loads(open(CONFIG_FILE).read()) 157 | 158 | for idx, item in enumerate(config['inputs']): 159 | if item['video'].isdigit(): 160 | input_stream = int(item['video']) 161 | else: 162 | input_stream = item['video'] 163 | 164 | cap = cv2.VideoCapture(input_stream) 165 | 166 | if not cap.isOpened(): 167 | logger.error("ERROR! Unable to open video source") 168 | return 169 | 170 | if input_stream: 171 | cap.open(input_stream) 172 | # Adjust DELAY to match the number of FPS of the video file 173 | DELAY = 1000 / cap.get(cv2.CAP_PROP_FPS) 174 | 175 | # Init inference request IDs 176 | cur_request_id = 0 177 | next_request_id = 1 178 | 179 | # Initialise the class 180 | infer_network = Network() 181 | infer_network_pose = Network() 182 | # Load the network to IE plugin to get shape of input layer 183 | plugin, (n_fd, c_fd, h_fd, w_fd) = infer_network.load_model(model, TARGET_DEVICE, 184 | 1, 1, 2, cpu_extension) 185 | n_hp, c_hp, h_hp, w_hp = infer_network_pose.load_model(posemodel, 186 | TARGET_DEVICE, 1, 3, 2, 187 | cpu_extension, plugin)[1] 188 | 189 | message_thread = Thread(target=message_runner) 190 | message_thread.setDaemon(True) 191 | message_thread.start() 192 | 193 | if is_async_mode: 194 | print("Application running in async mode...") 195 | else: 196 | print("Application running in sync mode...") 197 | det_time_fd = 0 198 | ret, frame = cap.read() 199 | while ret: 200 | 201 | looking = 0 202 | ret, frame = cap.read() 203 | if not ret: 204 | KEEP_RUNNING = False 205 | break 206 | 207 | initial_wh = [cap.get(3), cap.get(4)] 208 | in_frame_fd = cv2.resize(frame, (w_fd, h_fd)) 209 | # Change data layout from HWC to CHW 210 | in_frame_fd = in_frame_fd.transpose((2, 0, 1)) 211 | in_frame_fd = in_frame_fd.reshape((n_fd, c_fd, h_fd, w_fd)) 212 | 213 | if frame is None: 214 | KEEP_RUNNING = False 215 | log.error("ERROR! blank FRAME grabbed") 216 | break 217 | 218 | key_pressed = cv2.waitKey(int(DELAY)) 219 | 220 | # Start asynchronous inference for specified request 221 | inf_start_fd = time.time() 222 | if is_async_mode: 223 | # Async enabled and only one video capture 224 | infer_network.exec_net(next_request_id, in_frame_fd) 225 | else: 226 | # Async disabled 227 | infer_network.exec_net(cur_request_id, in_frame_fd) # Wait for the result 228 | if infer_network.wait(cur_request_id) == 0: 229 | det_time_fd = time.time() - inf_start_fd 230 | 231 | # Results of the output layer of the network 232 | res = infer_network.get_output(cur_request_id) 233 | 234 | # Parse face detection output 235 | faces = face_detection(res, initial_wh) 236 | 237 | if len(faces) != 0: 238 | # Look for poses 239 | for res_hp in faces: 240 | xmin, ymin, xmax, ymax = res_hp 241 | head_pose = frame[ymin:ymax, xmin:xmax] 242 | in_frame_hp = cv2.resize(head_pose, (w_hp, h_hp)) 243 | in_frame_hp = in_frame_hp.transpose((2, 0, 1)) 244 | in_frame_hp = in_frame_hp.reshape((n_hp, c_hp, h_hp, w_hp)) 245 | 246 | inf_start_hp = time.time() 247 | infer_network_pose.exec_net(0, in_frame_hp) 248 | infer_network_pose.wait(0) 249 | det_time_hp = time.time() - inf_start_hp 250 | 251 | # Parse head pose detection results 252 | angle_p_fc = infer_network_pose.get_output(0, "angle_p_fc") 253 | angle_y_fc = infer_network_pose.get_output(0, "angle_y_fc") 254 | if ((angle_y_fc > -22.5) & (angle_y_fc < 22.5) & (angle_p_fc > -22.5) & 255 | (angle_p_fc < 22.5)): 256 | looking += 1 257 | POSE_CHECKED = True 258 | INFO = INFO._replace(looker=looking) 259 | else: 260 | INFO = INFO._replace(looker=looking) 261 | else: 262 | INFO = INFO._replace(looker=0) 263 | 264 | # Draw performance stats 265 | inf_time_message = "Face Inference time: N\A for async mode" if is_async_mode else \ 266 | "Inference time: {:.3f} ms".format(det_time_fd * 1000) 267 | 268 | if POSE_CHECKED: 269 | head_inf_time_message = "Head pose Inference time: N\A for async mode" if is_async_mode else \ 270 | "Inference time: {:.3f} ms".format(det_time_hp * 1000) 271 | cv2.putText(frame, head_inf_time_message, (0, 55), 272 | cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1) 273 | log_message = "Async mode is on." if is_async_mode else \ 274 | "Async mode is off." 275 | cv2.putText(frame, log_message, (0, 15), cv2.FONT_HERSHEY_SIMPLEX, 276 | 0.5, (255, 255, 255), 1) 277 | cv2.putText(frame, inf_time_message, (0, 35), cv2.FONT_HERSHEY_SIMPLEX, 278 | 0.5, (255, 255, 255), 1) 279 | cv2.putText(frame, "Shopper: {}".format(INFO.shopper), (0, 90), 280 | cv2.FONT_HERSHEY_SIMPLEX, 281 | 0.5, (255, 255, 255), 1) 282 | cv2.putText(frame, "Looker: {}".format(INFO.looker), (0, 110), 283 | cv2.FONT_HERSHEY_SIMPLEX, 284 | 0.5, (255, 255, 255), 1) 285 | 286 | cv2.imshow("Shopper Gaze Monitor", frame) 287 | 288 | if key_pressed == 27: 289 | print("Attempting to stop background threads") 290 | KEEP_RUNNING = False 291 | break 292 | if key_pressed == 9: 293 | is_async_mode = not is_async_mode 294 | print("Switched to {} mode".format("async" if is_async_mode else "sync")) 295 | 296 | if is_async_mode: 297 | # Swap infer request IDs 298 | cur_request_id, next_request_id = next_request_id, cur_request_id 299 | 300 | infer_network.clean() 301 | infer_network_pose.clean() 302 | message_thread.join() 303 | cap.release() 304 | cv2.destroyAllWindows() 305 | CLIENT.disconnect() 306 | 307 | 308 | if __name__ == '__main__': 309 | main() 310 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2021, Intel Corporation 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Shopper Gaze Monitor 2 | 3 | | Details | | 4 | |-----------------------|---------------| 5 | | Target OS: | Ubuntu\* 18.04 LTS | 6 | | Programming Language: | Python* 3.5 | 7 | | Time to Complete: | 30 min | 8 | 9 | 12 | ![images](./docs/images/output.png) 13 | 14 | ## What it does 15 | 16 | This shopper gaze monitor application is designed for a retail shelf mounted camera system that counts the number of passers-by and the number of people who look towards the display. It is intended to provide real-world marketing statistics for in-store shelf-space advertising. 17 | 18 | ## Requirements 19 | ### Hardware 20 | 21 | * 6th to 8th generation Intel® Core™ processor with Iris® Pro graphics or Intel® HD Graphics 22 | 23 | ### Software 24 | 25 | * Ubuntu 18.04 26 | 27 | * OpenCL™ Runtime Package 28 | 29 | **Note:** We recommend using a 4.14+ kernel to use this software. Run the following command to determine your kernel version: 30 | 31 | uname -a 32 | 33 | * Intel® Distribution of OpenVINO™ toolkit 2020 R3 Release 34 | 35 | ## How It works 36 | 37 | The application uses the Inference Engine included in the Intel Distribution of OpenVINO toolkit and the Intel Deep Learning Deployment Toolkit. It uses a video source, such as a camera, to grab frames and then uses two different Deep Neural Networks (DNNs) to process the data. The first network looks for faces and then if successful is counted as a "Shopper" 38 | 39 | A second neural network is then used to determine the head pose detection for each detected face. If the person's head is facing towards the camera, it is counted as a "Looker" 40 | 41 | The shopper and looker data are sent to a local web server using the Paho* MQTT C client libraries. 42 | 43 | The program creates two threads for concurrency: 44 | 45 | * Main thread that performs the video i/o, processes video frames using the trained neural network. 46 | * Worker thread that publishes MQTT messages. 47 | 48 | ![images](./docs/images/architectural-diagram.png) 49 | 50 | **Architectural Diagram** 51 | 52 | ## Setup 53 | ### Get the code 54 | 55 | Clone the reference implementation: 56 | ``` 57 | sudo apt-get update && sudo apt-get install git 58 | git clone https://github.com/intel-iot-devkit/shopper-gaze-monitor-python.git 59 | ``` 60 | 61 | ### Install Intel® Distribution of OpenVINO™ toolkit 62 | 63 | Refer to https://software.intel.com/en-us/articles/OpenVINO-Install-Linux for more information about how to install and setup the Intel® Distribution of OpenVINO™ toolkit. 64 | 65 | You will need the OpenCL™ Runtime package if you plan to run inference on the GPU. It is not mandatory for CPU inference. 66 | 67 | ### Other dependencies 68 | #### Mosquitto* 69 | Mosquitto is an open source message broker that implements the MQTT protocol. The MQTT protocol provides a lightweight method of carrying out messaging using a publish/subscribe model. 70 | 71 | ### Which model to use 72 | 73 | This application uses the [face-detection-adas-0001](https://docs.openvinotoolkit.org/2020.3/_models_intel_face_detection_adas_0001_description_face_detection_adas_0001.html) and [head-pose-estimation-adas-0001](https://docs.openvinotoolkit.org/2020.3/_models_intel_human_pose_estimation_0001_description_human_pose_estimation_0001.html) Intel® model, that can be downloaded using the **model downloader**. The **model downloader** downloads the __.xml__ and __.bin__ files that is used by the application. 74 | 75 | To install the dependencies of the RI and to download the models Intel® model, run the following command: 76 | 77 | cd 78 | ./setup.sh 79 | 80 | The models will be downloaded inside the following directories: 81 | 82 | /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/ 83 | /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/ 84 | 85 | ### The Config File 86 | 87 | The _resources/config.json_ contains the path to the videos that will be used by the application. 88 | The _config.json_ file is of the form name/value pair, `video: ` 89 | 90 | Example of the _config.json_ file: 91 | 92 | ``` 93 | { 94 | 95 | "inputs": [ 96 | { 97 | "video": "videos/video1.mp4" 98 | } 99 | ] 100 | } 101 | ``` 102 | 103 | ### Which Input video to use 104 | 105 | The application works with any input video. Find sample videos [here](https://github.com/intel-iot-devkit/sample-videos/). 106 | 107 | For first-use, we recommend using the [face-demographics-walking-and-pause](https://github.com/intel-iot-devkit/sample-videos/blob/master/face-demographics-walking-and-pause.mp4) video.The video is automatically downloaded to the `resources/` folder. 108 | For example:
109 | The config.json would be: 110 | 111 | ``` 112 | { 113 | 114 | "inputs": [ 115 | { 116 | "video": "sample-videos/face-demographics-walking-and-pause.mp4" 117 | } 118 | ] 119 | } 120 | ``` 121 | To use any other video, specify the path in config.json file 122 | 123 | ### Using the Camera instead of video 124 | 125 | Replace the path/to/video in the _resources/config.json_ file with the camera ID, where the ID is taken from the video device (the number X in /dev/videoX). 126 | 127 | For example: 128 | 129 | ``` 130 | { 131 | 132 | "inputs": [ 133 | { 134 | "video": "0" 135 | } 136 | ] 137 | } 138 | ``` 139 | 140 | On Ubuntu, list all available video devices with the following command: 141 | 142 | ``` 143 | ls /dev/video* 144 | ``` 145 | 146 | ## Setup the environment 147 | You must configure the environment to use the Intel® Distribution of OpenVINO™ toolkit one time per session by running the following command: 148 | 149 | source /opt/intel/openvino/bin/setupvars.sh 150 | 151 | __Note__: This command needs to be executed only once in the terminal where the application will be executed. If the terminal is closed, the command needs to be executed again. 152 | 153 | ## Run the application 154 | 155 | Change the current directory to the git-cloned application code location on your system: 156 | 157 | cd /application 158 | 159 | To see a list of the various options: 160 | 161 | python3 shopper_gaze_monitor.py --help 162 | 163 | A user can specify a target device to run on by using the device command-line argument `-d` followed by one of the values `CPU`, `GPU`,`MYRIAD`, `HDDL` or `HETERO:FPGA,CPU`.
164 | To run with multiple devices use -d MULTI:device1,device2. For example: `-d MULTI:CPU,GPU,MYRIAD` 165 | 166 | ### Running on the CPU 167 | 168 | Though by default the application runs on CPU, this can also be explicitly specified by ```-d CPU``` command-line argument: 169 | 170 | python3 shopper_gaze_monitor.py -m /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP32/face-detection-adas-0001.xml -pm /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP32/head-pose-estimation-adas-0001.xml -d CPU 171 | 172 | ### Running on the GPU 173 | 174 | * To run on the integrated Intel® GPU with floating point precision 32 (FP32), use the `-d GPU` command-line argument: 175 | 176 | python3 shopper_gaze_monitor.py -m /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP32/face-detection-adas-0001.xml -pm /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP32/head-pose-estimation-adas-0001.xml -d GPU 177 | 178 | **FP32**: FP32 is single-precision floating-point arithmetic uses 32 bits to represent numbers. 8 bits for the magnitude and 23 bits for the precision. For more information, [click here](https://en.wikipedia.org/wiki/Single-precision_floating-point_format)
179 | 180 | * To run on the integrated Intel® GPU with floating point precision 16 (FP16): 181 | 182 | python3 shopper_gaze_monitor.py -m /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP16/face-detection-adas-0001.xml -pm /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP16/head-pose-estimation-adas-0001.xml -d GPU 183 | 184 | **FP16**: FP16 is half-precision floating-point arithmetic uses 16 bits. 5 bits for the magnitude and 10 bits for the precision. For more information, [click here](https://en.wikipedia.org/wiki/Half-precision_floating-point_format) 185 | 186 | ### Running on the Intel® Neural Compute Stick 187 | To run on the Intel® Neural Compute Stick, use the ```-d MYRIAD``` command-line argument: 188 | 189 | python3 shopper_gaze_monitor.py -d MYRIAD -m /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP16/face-detection-adas-0001.xml -pm /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP16/head-pose-estimation-adas-0001.xml 190 | 191 | **Note:** The Intel® Neural Compute Stick can only run FP16 models. The model that is passed to the application, through the `-m ` command-line argument, must be of data type FP16. 192 | 193 | ### Running on the Intel® Movidius™ VPU 194 | To run on the Intel® Movidius™ VPU, use the ```-d HDDL``` command-line argument: 195 | 196 | python3 shopper_gaze_monitor.py -m /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/face-detection-adas-0001/FP16/face-detection-adas-0001.xml -pm /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/intel/head-pose-estimation-adas-0001/FP16/head-pose-estimation-adas-0001.xml -d HDDL 197 | 198 | **Note:** The Intel® Movidius™ VPU can only run FP16 models. The model that is passed to the application, through the `-m ` command-line argument, must be of data type FP16. 199 | 200 | # Machine to machine messaging with MQTT 201 | 202 | If you wish to use a MQTT server to publish data, you should set the following environment variables on the terminal before running the program. 203 | 204 | export MQTT_SERVER=localhost:1883 205 | export MQTT_CLIENT_ID=cvservice 206 | 207 | Change the MQTT_SERVER to a value that matches the MQTT server you are connecting to. 208 | 209 | You should change the MQTT_CLIENT_ID to a unique value for each monitoring station, so you can track the data for individual locations. For example: 210 | 211 | export MQTT_CLIENT_ID=zone1337 212 | 213 | If you want to monitor the MQTT messages sent to your local server, and you have the mosquitto client utilities installed, you can run the following command in new terminal while executing the code: 214 | 215 | mosquitto_sub -h localhost -t shopper_gaze_monitor 216 | -------------------------------------------------------------------------------- /application/inference.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | Copyright (c) 2018 Intel Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining 6 | a copy of this software and associated documentation files (the 7 | "Software"), to deal in the Software without restriction, including 8 | without limitation the rights to use, copy, modify, merge, publish, 9 | distribute, sublicense, and/or sell copies of the Software, and to 10 | permit persons to whom the Software is furnished to do so, subject to 11 | the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be 14 | included in all copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 19 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 20 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 21 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 22 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 23 | """ 24 | 25 | import os 26 | import sys 27 | import logging as log 28 | from openvino.inference_engine import IENetwork, IECore 29 | 30 | 31 | class Network: 32 | """ 33 | Load and configure inference plugins for the specified target devices 34 | and performs synchronous and asynchronous modes for the specified infer requests. 35 | """ 36 | 37 | def __init__(self): 38 | self.net = None 39 | self.plugin = None 40 | self.input_blob = None 41 | self.out_blob = None 42 | self.net_plugin = None 43 | self.infer_request_handle = None 44 | 45 | def load_model(self, model, device, input_size, output_size, num_requests, cpu_extension=None, plugin=None): 46 | """ 47 | Loads a network and an image to the Inference Engine plugin. 48 | :param model: .xml file of pre trained model 49 | :param cpu_extension: extension for the CPU device 50 | :param device: Target device 51 | :param input_size: Number of input layers 52 | :param output_size: Number of output layers 53 | :param num_requests: Index of Infer request value. Limited to device capabilities. 54 | :param plugin: Plugin for specified device 55 | :return: Shape of input layer 56 | """ 57 | 58 | model_xml = model 59 | model_bin = os.path.splitext(model_xml)[0] + ".bin" 60 | # Plugin initialization for specified device 61 | # and load extensions library if specified 62 | if not plugin: 63 | log.info("Initializing plugin for {} device...".format(device)) 64 | self.plugin = IECore() 65 | else: 66 | self.plugin = plugin 67 | 68 | if cpu_extension and 'CPU' in device: 69 | self.plugin.add_extension(cpu_extension, "CPU") 70 | 71 | # Read IR 72 | log.info("Reading IR...") 73 | self.net = self.plugin.read_network(model=model_xml, weights=model_bin) 74 | log.info("Loading IR to the plugin...") 75 | 76 | if "CPU" in device: 77 | supported_layers = self.plugin.query_network(self.net, "CPU") 78 | not_supported_layers = \ 79 | [l for l in self.net.layers.keys() if l not in supported_layers] 80 | if len(not_supported_layers) != 0: 81 | log.error("Following layers are not supported by " 82 | "the plugin for specified device {}:\n {}". 83 | format(device, 84 | ', '.join(not_supported_layers))) 85 | log.error("Please try to specify cpu extensions library path" 86 | " in command line parameters using -l " 87 | "or --cpu_extension command line argument") 88 | sys.exit(1) 89 | 90 | if num_requests == 0: 91 | # Loads network read from IR to the plugin 92 | self.net_plugin = self.plugin.load_network(network=self.net, device_name=device) 93 | else: 94 | self.net_plugin = self.plugin.load_network(network=self.net, num_requests=num_requests, device_name=device) 95 | 96 | self.input_blob = next(iter(self.net.inputs)) 97 | self.out_blob = next(iter(self.net.outputs)) 98 | assert len(self.net.inputs.keys()) == input_size, \ 99 | "Supports only {} input topologies".format(len(self.net.inputs)) 100 | assert len(self.net.outputs) == output_size, \ 101 | "Supports only {} output topologies".format(len(self.net.outputs)) 102 | 103 | return self.plugin, self.get_input_shape() 104 | 105 | def get_input_shape(self): 106 | """ 107 | Gives the shape of the input layer of the network. 108 | :return: None 109 | """ 110 | return self.net.inputs[self.input_blob].shape 111 | 112 | def performance_counter(self, request_id): 113 | """ 114 | Queries performance measures per layer to get feedback of what is the 115 | most time consuming layer. 116 | :param request_id: Index of Infer request value. Limited to device capabilities 117 | :return: Performance of the layer 118 | """ 119 | perf_count = self.net_plugin.requests[request_id].get_perf_counts() 120 | return perf_count 121 | 122 | def exec_net(self, request_id, frame): 123 | """ 124 | Starts asynchronous inference for specified request. 125 | :param request_id: Index of Infer request value. Limited to device capabilities. 126 | :param frame: Input image 127 | :return: Instance of Executable Network class 128 | """ 129 | self.infer_request_handle = self.net_plugin.start_async( 130 | request_id=request_id, inputs={self.input_blob: frame}) 131 | return self.net_plugin 132 | 133 | def wait(self, request_id): 134 | """ 135 | Waits for the result to become available. 136 | :param request_id: Index of Infer request value. Limited to device capabilities. 137 | :return: Timeout value 138 | """ 139 | wait_process = self.net_plugin.requests[request_id].wait(-1) 140 | return wait_process 141 | 142 | def get_output(self, request_id, output=None): 143 | """ 144 | Gives a list of results for the output layer of the network. 145 | :param request_id: Index of Infer request value. Limited to device capabilities. 146 | :param output: Name of the output layer 147 | :return: Results for the specified request 148 | """ 149 | if output: 150 | res = self.infer_request_handle.outputs[output] 151 | else: 152 | res = self.net_plugin.requests[request_id].outputs[self.out_blob] 153 | return res 154 | 155 | def clean(self): 156 | """ 157 | Deletes all the instances 158 | :return: None 159 | """ 160 | del self.net_plugin 161 | del self.plugin 162 | del self.net 163 | -------------------------------------------------------------------------------- /application/shopper_gaze_monitor.py: -------------------------------------------------------------------------------- 1 | """Shopper Gaze Monitor.""" 2 | 3 | """ 4 | Copyright (c) 2018 Intel Corporation. 5 | Permission is hereby granted, free of charge, to any person obtaining 6 | a copy of this software and associated documentation files (the 7 | "Software"), to deal in the Software without restriction, including 8 | without limitation the rights to use, copy, modify, merge, publish, 9 | distribute, sublicense, and/or sell copies of the Software, and to 10 | permit person to whom the Software is furnished to do so, subject to 11 | the following conditions: 12 | The above copyright notice and this permission notice shall be 13 | included in all copies or substantial portions of the Software. 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | """ 22 | 23 | import os 24 | import sys 25 | import json 26 | import time 27 | import cv2 28 | 29 | import logging as log 30 | import paho.mqtt.client as mqtt 31 | 32 | from threading import Thread 33 | from collections import namedtuple 34 | from argparse import ArgumentParser 35 | from inference import Network 36 | 37 | # shoppingInfo contains statistics for the shopping information 38 | MyStruct = namedtuple("shoppingInfo", "shopper, looker") 39 | INFO = MyStruct(0, 0) 40 | 41 | POSE_CHECKED = False 42 | 43 | # MQTT server environment variables 44 | TOPIC = "shopper_gaze_monitor" 45 | MQTT_HOST = "localhost" 46 | MQTT_PORT = 1883 47 | MQTT_KEEPALIVE_INTERVAL = 60 48 | 49 | # Global variables 50 | TARGET_DEVICE = 'CPU' 51 | accepted_devices = ['CPU', 'GPU', 'MYRIAD', 'HETERO:FPGA,CPU', 'HDDL'] 52 | is_async_mode = True 53 | CONFIG_FILE = '../resources/config.json' 54 | 55 | # Flag to control background thread 56 | KEEP_RUNNING = True 57 | 58 | DELAY = 5 59 | 60 | 61 | def args_parser(): 62 | """ 63 | Parse command line arguments. 64 | :return: Command line arguments 65 | """ 66 | parser = ArgumentParser() 67 | parser.add_argument("-m", "--model", required=True, 68 | help="Path to an .xml file with a pre-trained" 69 | "face detection model") 70 | parser.add_argument("-pm", "--posemodel", required=True, 71 | help="Path to an .xml file with a pre-trained model" 72 | "head pose model") 73 | parser.add_argument("-l", "--cpu_extension", type=str, default=None, 74 | help="MKLDNN (CPU)-targeted custom layers. Absolute " 75 | "path to a shared library with the kernels impl.") 76 | parser.add_argument("-d", "--device", default="CPU", type=str, 77 | help="Specify the target device to infer on; " 78 | "CPU, GPU, FPGA, HDDL or MYRIAD is acceptable. To run with multiple devices use" 79 | " MULTI:,,etc. Application " 80 | "will look for a suitable plugin for device specified" 81 | "(CPU by default)") 82 | parser.add_argument("-c", "--confidence", default=0.5, type=float, 83 | help="Probability threshold for detections filtering") 84 | parser.add_argument("-f", "--flag", help="sync or async", default="async", type=str) 85 | 86 | global TARGET_DEVICE, is_async_mode 87 | args = parser.parse_args() 88 | if args.device: 89 | TARGET_DEVICE = args.device 90 | if args.flag == "sync": 91 | is_async_mode = False 92 | else: 93 | is_async_mode = True 94 | return parser 95 | 96 | 97 | def check_args(): 98 | # ArgumentParser checks the device 99 | 100 | global TARGET_DEVICE 101 | if 'MULTI' not in TARGET_DEVICE and TARGET_DEVICE not in accepted_devices: 102 | print("Unsupported device: " + TARGET_DEVICE) 103 | sys.exit(1) 104 | elif 'MULTI' in TARGET_DEVICE: 105 | target_devices = TARGET_DEVICE.split(':')[1].split(',') 106 | for multi_device in target_devices: 107 | if multi_device not in accepted_devices: 108 | print("Unsupported device: " + TARGET_DEVICE) 109 | sys.exit(1) 110 | 111 | 112 | def face_detection(res, args, initial_wh): 113 | """ 114 | Parse Face detection output. 115 | :param res: Detection results 116 | :param args: Parsed arguments 117 | :param initial_wh: Initial width and height of the FRAME 118 | :return: Co-ordinates of the detected face 119 | """ 120 | global INFO 121 | faces = [] 122 | INFO = INFO._replace(shopper=0) 123 | 124 | for obj in res[0][0]: 125 | # Draw only objects when probability more than specified threshold 126 | if obj[2] > args.confidence: 127 | if obj[3] < 0: 128 | obj[3] = -obj[3] 129 | if obj[4] < 0: 130 | obj[4] = -obj[4] 131 | xmin = int(obj[3] * initial_wh[0]) 132 | ymin = int(obj[4] * initial_wh[1]) 133 | xmax = int(obj[5] * initial_wh[0]) 134 | ymax = int(obj[6] * initial_wh[1]) 135 | faces.append([xmin, ymin, xmax, ymax]) 136 | INFO = INFO._replace(shopper=len(faces)) 137 | return faces 138 | 139 | 140 | def message_runner(): 141 | """ 142 | Publish worker status to MQTT topic. 143 | Pauses for rate second(s) between updates 144 | :return: None 145 | """ 146 | while KEEP_RUNNING: 147 | payload = json.dumps({"Shopper": INFO.shopper, "Looker": INFO.looker}) 148 | time.sleep(1) 149 | CLIENT.publish(TOPIC, payload=payload) 150 | 151 | 152 | def main(): 153 | """ 154 | Load the network and parse the output. 155 | :return: None 156 | """ 157 | global INFO 158 | global DELAY 159 | global CLIENT 160 | global KEEP_RUNNING 161 | global POSE_CHECKED 162 | global TARGET_DEVICE 163 | global is_async_mode 164 | CLIENT = mqtt.Client() 165 | CLIENT.connect(MQTT_HOST, MQTT_PORT, MQTT_KEEPALIVE_INTERVAL) 166 | 167 | log.basicConfig(format="[ %(levelname)s ] %(message)s", 168 | level=log.INFO, stream=sys.stdout) 169 | args = args_parser().parse_args() 170 | logger = log.getLogger() 171 | check_args() 172 | 173 | assert os.path.isfile(CONFIG_FILE), "{} file doesn't exist".format(CONFIG_FILE) 174 | config = json.loads(open(CONFIG_FILE).read()) 175 | 176 | for idx, item in enumerate(config['inputs']): 177 | if item['video'].isdigit(): 178 | input_stream = int(item['video']) 179 | else: 180 | input_stream = item['video'] 181 | 182 | cap = cv2.VideoCapture(input_stream) 183 | 184 | if not cap.isOpened(): 185 | logger.error("ERROR! Unable to open video source") 186 | return 187 | 188 | if input_stream: 189 | cap.open(input_stream) 190 | # Adjust DELAY to match the number of FPS of the video file 191 | DELAY = 1000 / cap.get(cv2.CAP_PROP_FPS) 192 | 193 | # Init inference request IDs 194 | cur_request_id = 0 195 | next_request_id = 1 196 | 197 | # Initialise the class 198 | infer_network = Network() 199 | infer_network_pose = Network() 200 | # Load the network to IE plugin to get shape of input layer 201 | plugin, (n_fd, c_fd, h_fd, w_fd) = infer_network.load_model(args.model, TARGET_DEVICE, 1, 1, 2, 202 | args.cpu_extension) 203 | n_hp, c_hp, h_hp, w_hp = infer_network_pose.load_model(args.posemodel, 204 | TARGET_DEVICE, 1, 205 | 3, 2, 206 | args.cpu_extension, plugin)[1] 207 | message_thread = Thread(target=message_runner) 208 | message_thread.setDaemon(True) 209 | message_thread.start() 210 | 211 | if is_async_mode: 212 | print("Application running in async mode...") 213 | else: 214 | print("Application running in sync mode...") 215 | det_time_fd = 0 216 | ret, frame = cap.read() 217 | while ret: 218 | 219 | looking = 0 220 | ret, frame = cap.read() 221 | if not ret: 222 | KEEP_RUNNING = False 223 | break 224 | 225 | if frame is None: 226 | KEEP_RUNNING = False 227 | log.error("ERROR! blank FRAME grabbed") 228 | break 229 | 230 | initial_wh = [cap.get(3), cap.get(4)] 231 | in_frame_fd = cv2.resize(frame, (w_fd, h_fd)) 232 | # Change data layout from HWC to CHW 233 | in_frame_fd = in_frame_fd.transpose((2, 0, 1)) 234 | in_frame_fd = in_frame_fd.reshape((n_fd, c_fd, h_fd, w_fd)) 235 | 236 | key_pressed = cv2.waitKey(int(DELAY)) 237 | 238 | # Start asynchronous inference for specified request 239 | inf_start_fd = time.time() 240 | if is_async_mode: 241 | # Async enabled and only one video capture 242 | infer_network.exec_net(next_request_id, in_frame_fd) 243 | else: 244 | # Async disabled 245 | infer_network.exec_net(cur_request_id, in_frame_fd) 246 | # Wait for the result 247 | if infer_network.wait(cur_request_id) == 0: 248 | det_time_fd = time.time() - inf_start_fd 249 | 250 | # Results of the output layer of the network 251 | res = infer_network.get_output(cur_request_id) 252 | # Parse face detection output 253 | faces = face_detection(res, args, initial_wh) 254 | if len(faces) != 0: 255 | # Look for poses 256 | for res_hp in faces: 257 | xmin, ymin, xmax, ymax = res_hp 258 | head_pose = frame[ymin:ymax, xmin:xmax] 259 | in_frame_hp = cv2.resize(head_pose, (w_hp, h_hp)) 260 | in_frame_hp = in_frame_hp.transpose((2, 0, 1)) 261 | in_frame_hp = in_frame_hp.reshape((n_hp, c_hp, h_hp, w_hp)) 262 | 263 | inf_start_hp = time.time() 264 | infer_network_pose.exec_net(cur_request_id, in_frame_hp) 265 | infer_network_pose.wait(cur_request_id) 266 | det_time_hp = time.time() - inf_start_hp 267 | 268 | # Parse head pose detection results 269 | angle_p_fc = infer_network_pose.get_output(0, "angle_p_fc") 270 | angle_y_fc = infer_network_pose.get_output(0, "angle_y_fc") 271 | if ((angle_y_fc > -22.5) & (angle_y_fc < 22.5) & (angle_p_fc > -22.5) & 272 | (angle_p_fc < 22.5)): 273 | looking += 1 274 | POSE_CHECKED = True 275 | INFO = INFO._replace(looker=looking) 276 | else: 277 | INFO = INFO._replace(looker=looking) 278 | else: 279 | INFO = INFO._replace(looker=0) 280 | 281 | # Draw performance stats 282 | inf_time_message = "Face Inference time: N\A for async mode" if is_async_mode else \ 283 | "Inference time: {:.3f} ms".format(det_time_fd * 1000) 284 | 285 | if POSE_CHECKED: 286 | head_inf_time_message = "Head pose Inference time: N\A for async mode" if is_async_mode else \ 287 | "Inference time: {:.3f} ms".format(det_time_hp * 1000) 288 | cv2.putText(frame, head_inf_time_message, (0, 55), 289 | cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1) 290 | log_message = "Async mode is on." if is_async_mode else \ 291 | "Async mode is off." 292 | cv2.putText(frame, log_message, (0, 15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1) 293 | cv2.putText(frame, inf_time_message, (0, 35), cv2.FONT_HERSHEY_SIMPLEX, 294 | 0.5, (255, 255, 255), 1) 295 | cv2.putText(frame, "Shopper: {}".format(INFO.shopper), (0, 90), cv2.FONT_HERSHEY_SIMPLEX, 296 | 0.5, (255, 255, 255), 1) 297 | cv2.putText(frame, "Looker: {}".format(INFO.looker), (0, 110), cv2.FONT_HERSHEY_SIMPLEX, 298 | 0.5, (255, 255, 255), 1) 299 | 300 | cv2.imshow("Shopper Gaze Monitor", frame) 301 | 302 | if key_pressed == 27: 303 | print("Attempting to stop background threads") 304 | KEEP_RUNNING = False 305 | break 306 | if key_pressed == 9: 307 | is_async_mode = not is_async_mode 308 | print("Switched to {} mode".format("async" if is_async_mode else "sync")) 309 | 310 | if is_async_mode: 311 | # Swap infer request IDs 312 | cur_request_id, next_request_id = next_request_id, cur_request_id 313 | 314 | infer_network.clean() 315 | infer_network_pose.clean() 316 | message_thread.join() 317 | cap.release() 318 | cv2.destroyAllWindows() 319 | CLIENT.disconnect() 320 | 321 | 322 | if __name__ == '__main__': 323 | main() 324 | sys.exit() 325 | -------------------------------------------------------------------------------- /docs/images/architectural-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/intel-iot-devkit/shopper-gaze-monitor-python/74a14de33436de6a52f1ae878e0f1c4c9c125d75/docs/images/architectural-diagram.png -------------------------------------------------------------------------------- /docs/images/jupyter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/intel-iot-devkit/shopper-gaze-monitor-python/74a14de33436de6a52f1ae878e0f1c4c9c125d75/docs/images/jupyter.png -------------------------------------------------------------------------------- /docs/images/jupyter_code.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/intel-iot-devkit/shopper-gaze-monitor-python/74a14de33436de6a52f1ae878e0f1c4c9c125d75/docs/images/jupyter_code.png -------------------------------------------------------------------------------- /docs/images/output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/intel-iot-devkit/shopper-gaze-monitor-python/74a14de33436de6a52f1ae878e0f1c4c9c125d75/docs/images/output.png -------------------------------------------------------------------------------- /resources/config.json: -------------------------------------------------------------------------------- 1 | { 2 | 3 | "inputs": [ 4 | 5 | { 6 | "video": "../resources/face-demographics-walking-and-pause.mp4" 7 | } 8 | ] 9 | 10 | } -------------------------------------------------------------------------------- /setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Copyright (c) 2018 Intel Corporation. 4 | # Permission is hereby granted, free of charge, to any person obtaining 5 | # a copy of this software and associated documentation files (the 6 | # "Software"), to deal in the Software without restriction, including 7 | # without limitation the rights to use, copy, modify, merge, publish, 8 | # distribute, sublicense, and/or sell copies of the Software, and to 9 | # permit persons to whom the Software is furnished to do so, subject to 10 | # the following conditions: 11 | # 12 | # The above copyright notice and this permission notice shall be 13 | # included in all copies or substantial portions of the Software. 14 | # 15 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 16 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 18 | # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 19 | # LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 20 | # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 21 | # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 22 | 23 | #Install the dependencies 24 | sudo apt-get update 25 | sudo apt-get install python3-pip 26 | sudo apt-get install mosquitto mosquitto-clients 27 | sudo pip3 install numpy jupyter paho-mqtt 28 | 29 | #Download the video 30 | cd resources 31 | wget -O face-demographics-walking-and-pause.mp4 https://github.com/intel-iot-devkit/sample-videos/raw/master/face-demographics-walking-and-pause.mp4 32 | 33 | #Download the model 34 | cd /opt/intel/openvino/deployment_tools/tools/model_downloader 35 | sudo ./downloader.py --name face-detection-adas-0001 36 | sudo ./downloader.py --name head-pose-estimation-adas-0001 37 | --------------------------------------------------------------------------------