├── README.md ├── TF-Pi-Image-OD.py ├── TF-Pi-Video-OD.py ├── TF-PiCamera-OD.py ├── doc ├── .gitignore ├── Camera Interface.png ├── Raspi vid.png ├── Thumbnail.png ├── Thumbnail2.png ├── demo.png ├── directory.png └── modelzoo.png ├── get-prerequisites.sh └── install-object-detection-api.sh /README.md: -------------------------------------------------------------------------------- 1 | # Object-Detection-on-Raspberry-Pi 2 | [![TensorFlow 2.2](https://img.shields.io/badge/TensorFlow-2.2-FF6F00?logo=tensorflow)](https://github.com/tensorflow/tensorflow/releases/tag/v2.2.0) 3 | ### This Tutorial Covers How to deploy the New TensorFlow 2 Object Detection Models and Custom Object Detection Models on the Raspberry Pi 4 |

5 | 6 |

7 | 8 | ***Note: TensorFlow Lite is much more popular on smaller devices such as the Raspberry Pi, but with the recent release of the TensorFlow 2 Custom Object Detection API and TensorFlow saved_model format, TensorFlow Lite has become quite error-prone with these newer models. Upon testing, converted tflite models weren't very stable nor compatible with the tflite_runtime module. However, this is sure to change with time to come! So be sure to stay tuned and alert for new guides coming soon!*** 9 | 10 | ## Introduction 11 | 12 | Continuing with my tutorial on the TensorFlow 2 Object Detection API, what better way to deploy an Object Detection Model than on the Raspberry Pi? This guide will contain step-by-step instructions to do exactly so. To make everything as easy as possible for you guys, I have simplified all the commands into a few shellscripts compressing tons of commands into only a few! I've also provided three object detection scripts for images, video, and real-time object detection with the Pi Camera! Many thanks to my friend Gareth who helped me out with testing and refining my instructions! 13 | 14 | **I will soon make a YouTube Tutorial which will be posted [here](https://www.youtube.com/watch?v=PWMQQAL0PCM), and an extremely import step [here](https://www.youtube.com/channel/UCT9t2Bug62RDUfSBcPt0Bzg?sub_confirmation=1)!** 15 | 16 | ## Table of Contents 17 | 1. [Setting up the Raspberry Pi and Getting Updates](https://github.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/blob/master/README.md#step-1-setting-up-the-raspberry-pi-and-getting-updates) 18 | 2. [Organizing our Workspace and Virtual Environment](https://github.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi#step-2-organizing-our-workspace-and-virtual-environment) 19 | 3. [Installing TensorFlow, OpenCV, and other Prerequisites](https://github.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/blob/master/README.md#step-3-installing-tensorflow-opencv-and-other-prerequisites) 20 | 4. [Preparing our Object Detection Model](https://github.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/blob/master/README.md#step-4-preparing-our-object-detection-model) 21 | 5. [Running Object Detection on Image, Video, or Pi Camera](https://github.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/blob/master/README.md#step-5-running-object-detection-on-image-video-or-pi-camera) 22 | 23 | ## Step 1: Setting up the Raspberry Pi and Getting Updates 24 | Before we can get started, we must have access to the Raspberry Pi's Desktop Interface. This can be done with VNC Viewer or the standard Monitor and HDMI. I made a more detailed video which can be found below 25 | 26 | [![Link to my vid](https://github.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/blob/master/doc/Raspi%20vid.png)](https://www.youtube.com/watch?v=jVzMRlCNO3U) 27 | 28 | Once you have access to the Desktop Interface, either remote or physical, open up a terminal. Retrieve updates for the Raspberry Pi with 29 | 30 | ``` 31 | sudo apt-get update 32 | sudo apt-get dist-upgrade 33 | ``` 34 | 35 | Depending on how recently you setup or updated your Pi, this can be instantaneous or lengthy. After your Raspberry Pi is up-to-date, we should make sure our Camera is enabled. First to open up the System Interface, use 36 | 37 | ``` 38 | sudo raspi-config 39 | ``` 40 | 41 | Then navigate to Interfacing Options -> Camera and make sure it is enabled. Then hit Finish and reboot if necessary. 42 | 43 |

44 | 45 |

46 | 47 | ## Step 2: Organizing our Workspace and Virtual Environment 48 | 49 | Then, your going to want to clone this repository with 50 | 51 | ``` 52 | git clone https://github.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi.git 53 | ``` 54 | 55 | This name is a bit long so let's trim it down with 56 | 57 | ``` 58 | mv Object-Detection-on-Raspberry-Pi tensorflow 59 | ``` 60 | 61 | We are now going to create a Virtual Environment to avoid version conflicts with previously installed packages on the Raspberry Pi. First, let's install virtual env with 62 | 63 | ``` 64 | sudo pip3 install virtualenv 65 | ``` 66 | 67 | Now, we can create our ```tensorflow``` virtual environment with 68 | 69 | ``` 70 | python3 -m venv tensorflow 71 | ``` 72 | 73 | There should now be a ```bin``` folder inside of our ```tensorflow``` directory. So let's change directories with 74 | 75 | ``` 76 | cd tensorflow 77 | ``` 78 | 79 | We can then activate our Virtual Envvironment with 80 | 81 | ``` 82 | source bin/activate 83 | ``` 84 | 85 | **Note: Now that we have a virtual environment, everytime you start a new terminal, you will no longer be in the virtual environment. You can reactivate it manually or issue ```echo "source tensorflow/bin/activate" >> ~/.bashrc```. This basically activates our Virtual Environment as soon as we open a new terminal. You can tell if the Virtual Environment is active by the name showing up in parenthesis next to the working directory.** 86 | 87 | When you issue ```ls```, your ```tensorflow``` directory should now look something like this 88 | 89 |

90 | 91 |

92 | 93 | ## Step 3: Installing TensorFlow, OpenCV, and other Prerequisites 94 | To make this step as user-friendly as possible, I condensed the installation process into 2 shell scripts. 95 | 96 | - ```get-prerequisites.sh```: This script installs OpenCV, TensorFlow 2.2.0, and matplotlib along with the dependencies for each module 97 | - ```install-object-detection-api.sh```: This script clones the tensorflow/models repo, compiles the protos, and installs the Object Detection API through an Environment Variable 98 | 99 | To install all the prerequisites needed, use 100 | 101 | ``` 102 | bash get-prerequisites.sh 103 | ``` 104 | This took me around 5-10 minutes, so you can sit back and relax for a bit! Once finished running, the following message should be printed 105 | 106 | ``` 107 | Prerequisites Downloaded Successfully 108 | ``` 109 | 110 | You can test your installation by entering 111 | 112 | ``` 113 | python 114 | Python 3.7.3 (default, Jul 25 2020, 13:03:44) 115 | [GCC 8.3.0] on linux 116 | Type "help", "copyright", "credits" or "license" for more information. 117 | >>> import tensorflow as tf 118 | >>> print (tf.__version__) 119 | ``` 120 | 121 | If everything was installed properly, you should get ```2.2.0```. This means we can now setup the Object Detection API with 122 | 123 | ``` 124 | source ./install-object-detection-api.sh 125 | ``` 126 | 127 | You should a similar success message looking like this 128 | 129 | ``` 130 | TensorFlow Object Detection API Setup Successfully! 131 | ``` 132 | 133 | To test out this installation, another similar step looking like this 134 | 135 | ``` 136 | python 137 | Python 3.7.3 (default, Jul 25 2020, 13:03:44) 138 | [GCC 8.3.0] on linux 139 | Type "help", "copyright", "credits" or "license" for more information. 140 | >>> import object_detection 141 | ``` 142 | 143 | If everything went according to plan, the object_detection module should import without any errors. 144 | 145 | **Note: Similar to the Virtual Environment, everytime you start a new terminal, the $PYTHONPATH variable set by the shell script will no longer be active. This means you will not be able to import the object_detection module. You can reactivate it manually with ```export PYTHONPATH=$PYTHONPATH:/home/pi/tensorflow/models/research:/home/pi/tensorflow/models/research/slim``` everytime you open a new terminal or issue ```echo "export PYTHONPATH=$PYTHONPATH:/home/pi/tensorflow/models/research:/home/pi/tensorflow/models/research/slim" >> ~/.bashrc```. This sets the system variable upon opening a new terminal.** 146 | 147 | ## Step 4: Preparing our Object Detection Model 148 | 149 | For this step, there are two options. You can use one of the TensorFlow Pre-Trained Object Detection Models which can be found in the [TensorFlow 2 Model Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md). Or you can train your own Custom Object Detector with the TensorFlow 2 Custom Object Detection API. Later on, I will cover both of these options a bit more extensively. First let's create a directory to store our models. Since we already have a folder named ```models```, let's call it ```od-models```. 150 | 151 | ``` 152 | mkdir od-models 153 | ``` 154 | 155 | Then let's cd into it with 156 | 157 | ``` 158 | cd od-models 159 | ``` 160 | 161 | Now, let's cover both options with more detail. 162 | 163 | ### Option 1: Using a TensorFlow 2 Pre-Trained Model 164 | For this guide, I will be using this option. TensorFlow's pre-trained models are trained on the [2017 COCO Dataset](https://cocodataset.org/#home) containing a variety of common, everyday, objects. TensorFlow 2's new ```saved_model``` format consists of a ```saved_model.pb``` and a ```variables``` directory. These two hold weights and the actual inference graph for object detection. The TensorFlow 2 Model Zoo can be found [here](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md). 165 | 166 |

167 | 168 |

169 | 170 | As you can see, there's tons of models to choose from! However, you probably noticed that I circled Speed in red. Since we are running on a Raspberry Pi, we're going to have to use one of the faster models. I'd recommend sticking to models with speeds under 40 ms. For this guide, I'll be using the SSD MobileNet v2 320x320 model. This is the fastest model, but there will be a small drop in accuracy. Let's download the model to our Raspberry Pi with 171 | 172 | ``` 173 | wget http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_320x320_coco17_tpu-8.tar.gz 174 | ``` 175 | If you plan to use a different model, right-click the name of the model and copy the download link. Then use that after wget instead of the link I provided. We can then extract the contents of the tar.gz file with 176 | 177 | ``` 178 | tar -xvf ssd_mobilenet_v2_320x320_coco17_tpu-8.tar.gz 179 | ``` 180 | 181 | This name is a bit long and confusing to work with so let's rename it with 182 | 183 | ``` 184 | mv ssd_mobilenet_v2_320x320_coco17_tpu-8 my_mobilenet_model 185 | ``` 186 | 187 | Once done so, our model should be ready for testing! 188 | 189 | ### Option 2: Using a TensorFlow Custom Object Detector 190 | If you wanted to detect an object that's not in the COCO Dataset, this is the option for you! Recently, I made a video with more details on training a Custom Object Detector with TensorFlow 2. 191 | 192 | [![Link to my vid](https://github.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/blob/master/doc/Thumbnail2.png)](https://www.youtube.com/watch?v=oqd54apcgGE) 193 | 194 | After you've followed all the steps mentioned in the video, you should end up with a ```labelmap.pbtxt``` file and a ```saved_model``` folder. You'll need to transfer these two files to our ```od-models``` directory on the Raspberry Pi. I usually use an SFTP Client such as [WinSCP](https://winscp.net/eng/index.php) to transfer files, but you can use whatever you want. Once your ```od-models``` directory contains your ```labelmap.pbtxt``` and ```saved_model```, you are ready to test! 195 | 196 | ## Step 5: Running Object Detection on Image, Video, or Pi Camera 197 | 198 | Once your model is ready, cd into the ```tensorflow``` directory with 199 | 200 | ``` 201 | cd ~/tensorflow 202 | ``` 203 | 204 | If you used a pre-trained model, the default programs should work. Let's run the Pi Camera Script with 205 | 206 | ``` 207 | python TF-PiCamera-OD.py 208 | ``` 209 | 210 | If you are using a Custom Object Detection Model, the usage for the Pi Camera Script looks like 211 | 212 | ``` 213 | usage: TF-PiCamera-OD.py [-h] [--model MODEL] [--labels LABELS] 214 | [--threshold THRESHOLD] 215 | 216 | optional arguments: 217 | -h, --help show this help message and exit 218 | --model MODEL Folder that the Saved Model is Located In 219 | --labels LABELS Where the Labelmap is Located 220 | --threshold THRESHOLD 221 | Minimum confidence threshold for displaying detected 222 | objects 223 | ``` 224 | 225 | If you were wondering about the arguments taken by the other programs, just use -h or --help after the command. An example command would look like 226 | 227 | ``` 228 | python TF-PiCamera-OD.py --model od-models --labels od-models/labelmap.pbtxt 229 | ``` 230 | 231 | It takes about 3 minutes for the model to load, but a window with results should open up. Your ouputs should look something like this 232 | 233 |

234 | 236 | 237 | Congratulations! This means we're successfully performing real-time object detection on the Raspberry Pi! Now that you've tried out the Pi Camera, why not one of the other scripts? Over the next weeks I'll continue to add on to this repo and tinker with the programs to make them better than ever! If you find something cool, feel free to share it, as others can also learn! And if you have any errors, just raise an issue and I'll be happy to take a look at it. Great work, and until next time, bye! 238 | -------------------------------------------------------------------------------- /TF-Pi-Image-OD.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | """ 4 | Object Detection (On Image) From TF2 Saved Model 5 | ===================================== 6 | """ 7 | 8 | import os 9 | os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' # Suppress TensorFlow logging (1) 10 | import pathlib 11 | import tensorflow as tf 12 | import cv2 13 | import argparse 14 | 15 | tf.get_logger().setLevel('ERROR') # Suppress TensorFlow logging (2) 16 | 17 | parser = argparse.ArgumentParser() 18 | parser.add_argument('--model', help='Folder that the Saved Model is Located In', 19 | default='od-models/my_mobilenet_model') 20 | parser.add_argument('--labels', help='Where the Labelmap is Located', 21 | default='models/research/object_detection/data/mscoco_label_map.pbtxt') 22 | parser.add_argument('--image', help='Name of the single image to perform detection on', 23 | default='test.png') 24 | parser.add_argument('--threshold', help='Minimum confidence threshold for displaying detected objects', 25 | default=0.5) 26 | 27 | args = parser.parse_args() 28 | # Enable GPU dynamic memory allocation 29 | gpus = tf.config.experimental.list_physical_devices('GPU') 30 | for gpu in gpus: 31 | tf.config.experimental.set_memory_growth(gpu, True) 32 | 33 | # PROVIDE PATH TO IMAGE DIRECTORY 34 | IMAGE_PATHS = args.image 35 | 36 | 37 | # PROVIDE PATH TO MODEL DIRECTORY 38 | PATH_TO_MODEL_DIR = args.model 39 | 40 | # PROVIDE PATH TO LABEL MAP 41 | PATH_TO_LABELS = args.labels 42 | 43 | # PROVIDE THE MINIMUM CONFIDENCE THRESHOLD 44 | MIN_CONF_THRESH = float(args.threshold) 45 | 46 | # LOAD THE MODEL 47 | 48 | import time 49 | from object_detection.utils import label_map_util 50 | from object_detection.utils import visualization_utils as viz_utils 51 | 52 | PATH_TO_SAVED_MODEL = PATH_TO_MODEL_DIR + "/saved_model" 53 | 54 | print('Loading model...', end='') 55 | start_time = time.time() 56 | 57 | # LOAD SAVED MODEL AND BUILD DETECTION FUNCTION 58 | detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL) 59 | 60 | end_time = time.time() 61 | elapsed_time = end_time - start_time 62 | print('Done! Took {} seconds'.format(elapsed_time)) 63 | 64 | # LOAD LABEL MAP DATA FOR PLOTTING 65 | 66 | category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, 67 | use_display_name=True) 68 | import numpy as np 69 | import matplotlib.pyplot as plt 70 | import warnings 71 | warnings.filterwarnings('ignore') # Suppress Matplotlib warnings 72 | 73 | 74 | print('Running inference for {}... '.format(IMAGE_PATHS), end='') 75 | 76 | image = cv2.imread(IMAGE_PATHS) 77 | image_rgb = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) 78 | imH, imW, _ = image.shape 79 | image_expanded = np.expand_dims(image_rgb, axis=0) 80 | 81 | # The input needs to be a tensor, convert it using `tf.convert_to_tensor`. 82 | input_tensor = tf.convert_to_tensor(image) 83 | # The model expects a batch of images, so add an axis with `tf.newaxis`. 84 | input_tensor = input_tensor[tf.newaxis, ...] 85 | 86 | # input_tensor = np.expand_dims(image_np, 0) 87 | detections = detect_fn(input_tensor) 88 | 89 | # All outputs are batches tensors. 90 | # Convert to numpy arrays, and take index [0] to remove the batch dimension. 91 | # We're only interested in the first num_detections. 92 | num_detections = int(detections.pop('num_detections')) 93 | detections = {key: value[0, :num_detections].numpy() 94 | for key, value in detections.items()} 95 | detections['num_detections'] = num_detections 96 | 97 | # detection_classes should be ints. 98 | detections['detection_classes'] = detections['detection_classes'].astype(np.int64) 99 | scores = detections['detection_scores'] 100 | boxes = detections['detection_boxes'] 101 | classes = detections['detection_classes'] 102 | count = 0 103 | for i in range(len(scores)): 104 | if ((scores[i] > MIN_CONF_THRESH) and (scores[i] <= 1.0)): 105 | #increase count 106 | count += 1 107 | # Get bounding box coordinates and draw box 108 | # Interpreter can return coordinates that are outside of image dimensions, need to force them to be within image using max() and min() 109 | ymin = int(max(1,(boxes[i][0] * imH))) 110 | xmin = int(max(1,(boxes[i][1] * imW))) 111 | ymax = int(min(imH,(boxes[i][2] * imH))) 112 | xmax = int(min(imW,(boxes[i][3] * imW))) 113 | 114 | cv2.rectangle(image, (xmin,ymin), (xmax,ymax), (10, 255, 0), 2) 115 | # Draw label 116 | object_name = category_index[int(classes[i])]['name'] # Look up object name from "labels" array using class index 117 | label = '%s: %d%%' % (object_name, int(scores[i]*100)) # Example: 'person: 72%' 118 | labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2) # Get font size 119 | label_ymin = max(ymin, labelSize[1] + 10) # Make sure not to draw label too close to top of window 120 | cv2.rectangle(image, (xmin, label_ymin-labelSize[1]-10), (xmin+labelSize[0], label_ymin+baseLine-10), (255, 255, 255), cv2.FILLED) # Draw white box to put label text in 121 | cv2.putText(image, label, (xmin, label_ymin-7), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2) # Draw label text 122 | 123 | 124 | cv2.putText (image,'Objects Detected : ' + str(count),(10,25),cv2.FONT_HERSHEY_SIMPLEX,1,(70,235,52),2,cv2.LINE_AA) 125 | print('Done') 126 | # DISPLAYS OUTPUT IMAGE 127 | cv2.imshow('Object Counter', image) 128 | # CLOSES WINDOW ONCE KEY IS PRESSED 129 | cv2.waitKey(0) 130 | # CLEANUP 131 | cv2.destroyAllWindows() 132 | 133 | -------------------------------------------------------------------------------- /TF-Pi-Video-OD.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | """ 4 | Object Detection (On Video) From TF2 Saved Model 5 | ===================================== 6 | """ 7 | 8 | import os 9 | os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' # Suppress TensorFlow logging (1) 10 | import pathlib 11 | import tensorflow as tf 12 | import cv2 13 | import argparse 14 | 15 | tf.get_logger().setLevel('ERROR') # Suppress TensorFlow logging (2) 16 | 17 | parser = argparse.ArgumentParser() 18 | parser.add_argument('--model', help='Folder that the Saved Model is Located In', 19 | default='od-models/my_mobilenet_model') 20 | parser.add_argument('--labels', help='Where the Labelmap is Located', 21 | default='models/research/object_detection/data/mscoco_label_map.pbtxt') 22 | parser.add_argument('--video', help='Name of the video to perform detection on', 23 | default='test.mp4') 24 | parser.add_argument('--threshold', help='Minimum confidence threshold for displaying detected objects', 25 | default=0.5) 26 | 27 | args = parser.parse_args() 28 | # Enable GPU dynamic memory allocation 29 | gpus = tf.config.experimental.list_physical_devices('GPU') 30 | for gpu in gpus: 31 | tf.config.experimental.set_memory_growth(gpu, True) 32 | 33 | # PROVIDE PATH TO IMAGE DIRECTORY 34 | VIDEO_PATHS = args.video 35 | 36 | 37 | # PROVIDE PATH TO MODEL DIRECTORY 38 | PATH_TO_MODEL_DIR = args.model 39 | 40 | # PROVIDE PATH TO LABEL MAP 41 | PATH_TO_LABELS = args.labels 42 | 43 | # PROVIDE THE MINIMUM CONFIDENCE THRESHOLD 44 | MIN_CONF_THRESH = float(args.threshold) 45 | 46 | # Load the model 47 | # ~~~~~~~~~~~~~~ 48 | import time 49 | from object_detection.utils import label_map_util 50 | from object_detection.utils import visualization_utils as viz_utils 51 | 52 | PATH_TO_SAVED_MODEL = PATH_TO_MODEL_DIR + "/saved_model" 53 | 54 | print('Loading model...', end='') 55 | start_time = time.time() 56 | 57 | # Load saved model and build the detection function 58 | detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL) 59 | 60 | end_time = time.time() 61 | elapsed_time = end_time - start_time 62 | print('Done! Took {} seconds'.format(elapsed_time)) 63 | 64 | # Load label map data (for plotting) 65 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 66 | 67 | 68 | category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, 69 | use_display_name=True) 70 | 71 | import numpy as np 72 | from PIL import Image 73 | import matplotlib.pyplot as plt 74 | import warnings 75 | warnings.filterwarnings('ignore') # Suppress Matplotlib warnings 76 | 77 | def load_image_into_numpy_array(path): 78 | """Load an image from file into a numpy array. 79 | Puts image into numpy array to feed into tensorflow graph. 80 | Note that by convention we put it into a numpy array with shape 81 | (height, width, channels), where channels=3 for RGB. 82 | Args: 83 | path: the file path to the image 84 | Returns: 85 | uint8 numpy array with shape (img_height, img_width, 3) 86 | """ 87 | return np.array(Image.open(path)) 88 | 89 | 90 | 91 | 92 | print('Running inference for {}... '.format(VIDEO_PATHS), end='') 93 | 94 | video = cv2.VideoCapture(VIDEO_PATHS) 95 | while(video.isOpened()): 96 | 97 | # Acquire frame and expand frame dimensions to have shape: [1, None, None, 3] 98 | # i.e. a single-column array, where each item in the column has the pixel RGB value 99 | ret, frame = video.read() 100 | frame_rgb = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR) 101 | frame_expanded = np.expand_dims(frame_rgb, axis=0) 102 | imH, imW, _ = frame.shape 103 | 104 | # The input needs to be a tensor, convert it using `tf.convert_to_tensor`. 105 | input_tensor = tf.convert_to_tensor(frame) 106 | # The model expects a batch of images, so add an axis with `tf.newaxis`. 107 | input_tensor = input_tensor[tf.newaxis, ...] 108 | 109 | # input_tensor = np.expand_dims(image_np, 0) 110 | detections = detect_fn(input_tensor) 111 | 112 | # All outputs are batches tensors. 113 | # Convert to numpy arrays, and take index [0] to remove the batch dimension. 114 | # We're only interested in the first num_detections. 115 | num_detections = int(detections.pop('num_detections')) 116 | detections = {key: value[0, :num_detections].numpy() 117 | for key, value in detections.items()} 118 | detections['num_detections'] = num_detections 119 | 120 | # detection_classes should be ints. 121 | detections['detection_classes'] = detections['detection_classes'].astype(np.int64) 122 | 123 | 124 | # SET MIN SCORE THRESH TO MINIMUM THRESHOLD FOR DETECTIONS 125 | 126 | detections['detection_classes'] = detections['detection_classes'].astype(np.int64) 127 | scores = detections['detection_scores'] 128 | boxes = detections['detection_boxes'] 129 | classes = detections['detection_classes'] 130 | count = 0 131 | for i in range(len(scores)): 132 | if ((scores[i] > MIN_CONF_THRESH) and (scores[i] <= 1.0)): 133 | #increase count 134 | count += 1 135 | # Get bounding box coordinates and draw box 136 | # Interpreter can return coordinates that are outside of image dimensions, need to force them to be within image using max() and min() 137 | ymin = int(max(1,(boxes[i][0] * imH))) 138 | xmin = int(max(1,(boxes[i][1] * imW))) 139 | ymax = int(min(imH,(boxes[i][2] * imH))) 140 | xmax = int(min(imW,(boxes[i][3] * imW))) 141 | 142 | cv2.rectangle(frame, (xmin,ymin), (xmax,ymax), (10, 255, 0), 2) 143 | # Draw label 144 | object_name = category_index[int(classes[i])]['name'] # Look up object name from "labels" array using class index 145 | label = '%s: %d%%' % (object_name, int(scores[i]*100)) # Example: 'person: 72%' 146 | labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2) # Get font size 147 | label_ymin = max(ymin, labelSize[1] + 10) # Make sure not to draw label too close to top of window 148 | cv2.rectangle(frame, (xmin, label_ymin-labelSize[1]-10), (xmin+labelSize[0], label_ymin+baseLine-10), (255, 255, 255), cv2.FILLED) # Draw white box to put label text in 149 | cv2.putText(frame, label, (xmin, label_ymin-7), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2) # Draw label text 150 | 151 | 152 | cv2.putText (frame,'Objects Detected : ' + str(count),(10,25),cv2.FONT_HERSHEY_SIMPLEX,1,(70,235,52),2,cv2.LINE_AA) 153 | cv2.imshow('Object Detector', frame) 154 | 155 | if cv2.waitKey(1) == ord('q'): 156 | break 157 | 158 | cv2.destroyAllWindows() 159 | print("Done") 160 | -------------------------------------------------------------------------------- /TF-PiCamera-OD.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | """ 4 | Object Detection (On Pi Camera) From TF2 Saved Model 5 | ===================================== 6 | """ 7 | 8 | import os 9 | os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' # Suppress TensorFlow logging (1) 10 | import pathlib 11 | import tensorflow as tf 12 | import cv2 13 | import argparse 14 | from threading import Thread 15 | 16 | tf.get_logger().setLevel('ERROR') # Suppress TensorFlow logging (2) 17 | class VideoStream: 18 | """Camera object that controls video streaming from the Picamera""" 19 | def __init__(self,resolution=(640,480),framerate=30): 20 | # Initialize the PiCamera and the camera image stream 21 | self.stream = cv2.VideoCapture(0) 22 | ret = self.stream.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*'MJPG')) 23 | ret = self.stream.set(3,resolution[0]) 24 | ret = self.stream.set(4,resolution[1]) 25 | 26 | # Read first frame from the stream 27 | (self.grabbed, self.frame) = self.stream.read() 28 | 29 | # Variable to control when the camera is stopped 30 | self.stopped = False 31 | 32 | def start(self): 33 | # Start the thread that reads frames from the video stream 34 | Thread(target=self.update,args=()).start() 35 | return self 36 | 37 | def update(self): 38 | # Keep looping indefinitely until the thread is stopped 39 | while True: 40 | # If the camera is stopped, stop the thread 41 | if self.stopped: 42 | # Close camera resources 43 | self.stream.release() 44 | return 45 | 46 | # Otherwise, grab the next frame from the stream 47 | (self.grabbed, self.frame) = self.stream.read() 48 | 49 | def read(self): 50 | # Return the most recent frame 51 | return self.frame 52 | 53 | def stop(self): 54 | # Indicate that the camera and thread should be stopped 55 | self.stopped = True 56 | 57 | 58 | parser = argparse.ArgumentParser() 59 | parser.add_argument('--model', help='Folder that the Saved Model is Located In', 60 | default='od-models/my_mobilenet_model') 61 | parser.add_argument('--labels', help='Where the Labelmap is Located', 62 | default='models/research/object_detection/data/mscoco_label_map.pbtxt') 63 | parser.add_argument('--threshold', help='Minimum confidence threshold for displaying detected objects', 64 | default=0.5) 65 | 66 | args = parser.parse_args() 67 | 68 | 69 | # PROVIDE PATH TO MODEL DIRECTORY 70 | PATH_TO_MODEL_DIR = args.model 71 | 72 | # PROVIDE PATH TO LABEL MAP 73 | PATH_TO_LABELS = args.labels 74 | 75 | # PROVIDE THE MINIMUM CONFIDENCE THRESHOLD 76 | MIN_CONF_THRESH = float(args.threshold) 77 | 78 | # Load the model 79 | # ~~~~~~~~~~~~~~ 80 | import time 81 | from object_detection.utils import label_map_util 82 | from object_detection.utils import visualization_utils as viz_utils 83 | 84 | PATH_TO_SAVED_MODEL = PATH_TO_MODEL_DIR + "/saved_model" 85 | 86 | print('Loading model...', end='') 87 | start_time = time.time() 88 | 89 | # Load saved model and build the detection function 90 | detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL) 91 | 92 | end_time = time.time() 93 | elapsed_time = end_time - start_time 94 | print('Done! Took {} seconds'.format(elapsed_time)) 95 | 96 | # Load label map data (for plotting) 97 | # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 98 | 99 | 100 | category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, 101 | use_display_name=True) 102 | 103 | import numpy as np 104 | import matplotlib.pyplot as plt 105 | import warnings 106 | warnings.filterwarnings('ignore') # Suppress Matplotlib warnings 107 | 108 | print('Running inference for PiCamera') 109 | videostream = VideoStream(resolution=(640,480),framerate=30).start() 110 | while True: 111 | 112 | # Acquire frame and expand frame dimensions to have shape: [1, None, None, 3] 113 | # i.e. a single-column array, where each item in the column has the pixel RGB value 114 | frame = videostream.read() 115 | frame_rgb = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR) 116 | frame_expanded = np.expand_dims(frame_rgb, axis=0) 117 | imH, imW, _ = frame.shape 118 | 119 | # The input needs to be a tensor, convert it using `tf.convert_to_tensor`. 120 | input_tensor = tf.convert_to_tensor(frame) 121 | # The model expects a batch of images, so add an axis with `tf.newaxis`. 122 | input_tensor = input_tensor[tf.newaxis, ...] 123 | 124 | # input_tensor = np.expand_dims(image_np, 0) 125 | detections = detect_fn(input_tensor) 126 | 127 | # All outputs are batches tensors. 128 | # Convert to numpy arrays, and take index [0] to remove the batch dimension. 129 | # We're only interested in the first num_detections. 130 | num_detections = int(detections.pop('num_detections')) 131 | detections = {key: value[0, :num_detections].numpy() 132 | for key, value in detections.items()} 133 | detections['num_detections'] = num_detections 134 | 135 | # detection_classes should be ints. 136 | detections['detection_classes'] = detections['detection_classes'].astype(np.int64) 137 | 138 | 139 | # SET MIN SCORE THRESH TO MINIMUM THRESHOLD FOR DETECTIONS 140 | 141 | detections['detection_classes'] = detections['detection_classes'].astype(np.int64) 142 | scores = detections['detection_scores'] 143 | boxes = detections['detection_boxes'] 144 | classes = detections['detection_classes'] 145 | count = 0 146 | for i in range(len(scores)): 147 | if ((scores[i] > MIN_CONF_THRESH) and (scores[i] <= 1.0)): 148 | #increase count 149 | count += 1 150 | # Get bounding box coordinates and draw box 151 | # Interpreter can return coordinates that are outside of image dimensions, need to force them to be within image using max() and min() 152 | ymin = int(max(1,(boxes[i][0] * imH))) 153 | xmin = int(max(1,(boxes[i][1] * imW))) 154 | ymax = int(min(imH,(boxes[i][2] * imH))) 155 | xmax = int(min(imW,(boxes[i][3] * imW))) 156 | 157 | cv2.rectangle(frame, (xmin,ymin), (xmax,ymax), (10, 255, 0), 2) 158 | # Draw label 159 | object_name = category_index[int(classes[i])]['name'] # Look up object name from "labels" array using class index 160 | label = '%s: %d%%' % (object_name, int(scores[i]*100)) # Example: 'person: 72%' 161 | labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2) # Get font size 162 | label_ymin = max(ymin, labelSize[1] + 10) # Make sure not to draw label too close to top of window 163 | cv2.rectangle(frame, (xmin, label_ymin-labelSize[1]-10), (xmin+labelSize[0], label_ymin+baseLine-10), (255, 255, 255), cv2.FILLED) # Draw white box to put label text in 164 | cv2.putText(frame, label, (xmin, label_ymin-7), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2) # Draw label text 165 | 166 | 167 | cv2.putText (frame,'Objects Detected : ' + str(count),(10,25),cv2.FONT_HERSHEY_SIMPLEX,1,(70,235,52),2,cv2.LINE_AA) 168 | cv2.imshow('Object Detector', frame) 169 | 170 | if cv2.waitKey(1) == ord('q'): 171 | break 172 | 173 | cv2.destroyAllWindows() 174 | print("Done") 175 | -------------------------------------------------------------------------------- /doc/.gitignore: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /doc/Camera Interface.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/7f977cd306230f01773e6d771a26b55004d42e2b/doc/Camera Interface.png -------------------------------------------------------------------------------- /doc/Raspi vid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/7f977cd306230f01773e6d771a26b55004d42e2b/doc/Raspi vid.png -------------------------------------------------------------------------------- /doc/Thumbnail.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/7f977cd306230f01773e6d771a26b55004d42e2b/doc/Thumbnail.png -------------------------------------------------------------------------------- /doc/Thumbnail2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/7f977cd306230f01773e6d771a26b55004d42e2b/doc/Thumbnail2.png -------------------------------------------------------------------------------- /doc/demo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/7f977cd306230f01773e6d771a26b55004d42e2b/doc/demo.png -------------------------------------------------------------------------------- /doc/directory.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/7f977cd306230f01773e6d771a26b55004d42e2b/doc/directory.png -------------------------------------------------------------------------------- /doc/modelzoo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/armaanpriyadarshan/Object-Detection-on-Raspberry-Pi/7f977cd306230f01773e6d771a26b55004d42e2b/doc/modelzoo.png -------------------------------------------------------------------------------- /get-prerequisites.sh: -------------------------------------------------------------------------------- 1 | # Installs Prerequisites for OpenCV 2 | 3 | sudo apt-get install libhdf5-dev libhdf5-serial-dev libhdf5-103 4 | sudo apt-get install libqtgui4 libqtwebkit4 libqt4-test python3-pyqt5 5 | sudo apt-get install libatlas-base-dev 6 | sudo apt-get install libjasper-dev 7 | 8 | # Installs OpenCV pip package 9 | 10 | pip install opencv-python==4.1.0.25 11 | 12 | # Install prerequisites for TensorFlow 2.2.0 13 | 14 | sudo apt-get install gfortran 15 | sudo apt-get install libhdf5-dev libc-ares-dev libeigen3-dev 16 | sudo apt-get install libatlas-base-dev libopenblas-dev libblas-dev 17 | sudo apt-get install liblapack-dev cython 18 | sudo pip3 install pybind11 19 | sudo pip3 install h5py 20 | sudo pip3 install --upgrade setuptools 21 | pip install gdown 22 | sudo cp /home/pi/.local/bin/gdown /usr/local/bin/gdown 23 | gdown https://drive.google.com/uc?id=11mujzVaFqa7R1_lB7q0kVPW22Ol51MPg 24 | 25 | # Install the Downloaded Wheel 26 | 27 | pip install tensorflow-2.2.0-cp37-cp37m-linux_armv7l.whl 28 | 29 | # A Few more Prerequisites for the TensorFlow Object Detection API and Testing 30 | 31 | pip install matplotlib 32 | sudo apt-get install protobuf-compiler 33 | 34 | # Print Success Message 35 | 36 | echo Prerequisites Downloaded Successfully 37 | -------------------------------------------------------------------------------- /install-object-detection-api.sh: -------------------------------------------------------------------------------- 1 | # Clone the TensorFlow Models Repository, Compile the Protos, Add Object Detection Module to Path 2 | 3 | git clone https://github.com/tensorflow/models.git 4 | cd models/research 5 | protoc object_detection/protos/*.proto --python_out=. 6 | export PYTHONPATH=$PYTHONPATH:/home/pi/tensorflow/models/research:/home/pi/tensorflow/models/research/slim 7 | 8 | # Return to our Directory 9 | cd ../.. 10 | 11 | # Echo Success Message 12 | 13 | echo TensorFlow Object Detection API Setup Successfully! 14 | --------------------------------------------------------------------------------