├── README.md ├── images ├── 00238.jpg ├── bird-camera.jpg ├── bird.gif ├── birdcamera.jpeg ├── birdduel.gif └── jetson.png ├── join-birds.py ├── richierich.py ├── stl ├── README.md ├── cameramountPI2.stl └── pimount2.stl └── training └── extractFromTimePoints.py /README.md: -------------------------------------------------------------------------------- 1 | # Rich Man's Deep Learning Camera 2 | Building a Self Contained Deep Learning Camera with the NVIDIA Jetson and Python 3 | 4 | ![NVIDIA Jetson Deep Learning Camera](https://github.com/burningion/rich-mans-deep-learning-camera/raw/master/images/bird-camera.jpg) 5 | 6 | ## Installation 7 | 8 | You'll need to have OpenCV 3 and Darkflow installed on the NVIDIA Jetson TX1 or TX2. You can use my [fork](https://github.com/burningion/buildOpenCVTX1) of JetsonHacks' build script to get Python 3 support built on the Jetson platform. 9 | 10 | Once this is installed, you'll also need to get the Darkflow Tiny Model downloaded and running on the TX1/TX2. With this, you'll just need to install the `imutils` library, and you should be ready to go. 11 | 12 | ## Architecture 13 | 14 | ![Deep Learning Camera Architecture](https://github.com/burningion/rich-mans-deep-learning-camera/raw/master/images/jetson.png) 15 | 16 | We use an external hard disk, and a USB webcam to take in and store our images. Once we've got a stream of images coming in from the webcam, we run a detection on every other N frames. This allows us to build a pre-buffer, where we can store a tiny movie of the bird before it's detected. 17 | 18 | Once it's detected, we create a new thread to continue recording from the webcam, while the original process writes out this pre-buffer to disk. 19 | 20 | The inference keeps running until the process itself is killed (usually via CTRL+c). 21 | 22 | ## Blog Post 23 | 24 | The blog post accompanying this repo is at [Make Art with Python](https://www.makeartwithpython.com/blog/rich-mans-deep-learning-camera/). 25 | 26 | ## Detected Bird Videos 27 | 28 | I've included some example outputs from detected birds in this repo. Good Luck! 29 | 30 | ![Single Bird](https://github.com/burningion/rich-mans-deep-learning-camera/raw/master/images/bird.gif) 31 | ![Bird Duel](https://github.com/burningion/rich-mans-deep-learning-camera/raw/master/images/birdduel.gif) 32 | ![HD Bird](https://github.com/burningion/rich-mans-deep-learning-camera/raw/master/images/00238.jpg) 33 | 34 | # Changing What Gets Detected / Recorded, and How Long the Videos Are 35 | 36 | Just change the `detectLabel` variable to somethinng out of the Cocos dataset ("person", for example), and then change the `beforeFrames` and `afterFrames`, in order to match your webcam's FPS and the length you'd like. (By default, it's 120, for 4 seconds of 30fps.) 37 | 38 | # Turning Image Sequences into Videos 39 | 40 | You can turn any of the images into videos using ffmpeg: 41 | 42 | ```bash 43 | $ ffmpeg -i %05d.jpg -profile:v high -level 4.0 -strict -2 out.mp4 44 | ``` 45 | 46 | I also added a script `joinImages.py`, that will create a directory and fill it with all the recorded image sequences. Just run it, and it should grab every bird directory sequentially, and then spit out a new directory with all of the images in order, ready to run the `ffmpeg` command above. 47 | 48 | # Getting Images from Videos for Training 49 | 50 | Once you've got a day's worth of video, you can quickly run through it for specific events you'd like to add to your dataset. 51 | 52 | Save these time points in a file called `timepoints.txt`, with timecodes like the following in hh:mm:ss format: 53 | 54 | ``` 55 | 00:00:01 56 | 00:04:03 57 | 00:06:06 58 | 00:17:25 59 | 00:18:35 60 | 00:18:50 61 | 00:20:17 62 | 00:23:30 63 | 00:34:30 64 | ``` 65 | 66 | Run the `extractTimePoints.py` Python script to extract images from these timepoints using `ffmpeg`: 67 | 68 | ``` 69 | $ python3 extractFromTimePoints.py -f .mp4 70 | ``` 71 | 72 | This will create a new directory with images from the timepoints you've selected. You can then label and train these images in something like [labelImg](https://github.com/tzutalin/labelImg) for training. 73 | -------------------------------------------------------------------------------- /images/00238.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/burningion/rich-mans-deep-learning-camera/1c163e835230106b2275b00327981bf459ae6676/images/00238.jpg -------------------------------------------------------------------------------- /images/bird-camera.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/burningion/rich-mans-deep-learning-camera/1c163e835230106b2275b00327981bf459ae6676/images/bird-camera.jpg -------------------------------------------------------------------------------- /images/bird.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/burningion/rich-mans-deep-learning-camera/1c163e835230106b2275b00327981bf459ae6676/images/bird.gif -------------------------------------------------------------------------------- /images/birdcamera.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/burningion/rich-mans-deep-learning-camera/1c163e835230106b2275b00327981bf459ae6676/images/birdcamera.jpeg -------------------------------------------------------------------------------- /images/birdduel.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/burningion/rich-mans-deep-learning-camera/1c163e835230106b2275b00327981bf459ae6676/images/birdduel.gif -------------------------------------------------------------------------------- /images/jetson.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/burningion/rich-mans-deep-learning-camera/1c163e835230106b2275b00327981bf459ae6676/images/jetson.png -------------------------------------------------------------------------------- /join-birds.py: -------------------------------------------------------------------------------- 1 | # This script will convert the captured birds into a video for the day, 2 | # at every end of day. If it finishes succesfully, it will delete source 3 | # images. Later, it will also break the video out to thumbnails 4 | # for viewing online. 5 | 6 | # To install and run at midnight every day, run crontab -e and add the line: 7 | # 0 23 * * * cd /home/nvidia/rich-mans-deep-learning-camera/ && python3 /home/nvidia/rich-mans-deep-learning-camera/join-birds.py 8 | # Replace the above line with your home directory, and where this project is running. 9 | 10 | # First, kill the running python3 process to free up memory 11 | # Terrible hack checks for the python3 process using most memory and kills that 12 | # If detector isn't running, kills itself 13 | import subprocess, datetime 14 | import psutil 15 | import glob 16 | 17 | import json 18 | 19 | try: 20 | print("Killing existing detector process to free up memory") 21 | python_processes = [p.info for p in psutil.process_iter(attrs=['pid', 'name', 'memory_info']) if 'python' in p.info['name']] 22 | if len(python_processes) < 2: 23 | print("Detector doesn't seem to have been running. Exiting") 24 | exit() 25 | command = "kill %i" % python_processes[0]['pid'] 26 | subprocess.check_call(command.split()) 27 | except subprocess.CalledProcessError: 28 | print("Couldn't kill an existing python3 process. Did you run the detector using python3?") 29 | exit() 30 | 31 | # next, create list of detected images for ffmpeg to compress into daily video 32 | # and create an output directory for each portrait and metadat of detection 33 | import glob 34 | 35 | today = str(datetime.date.today()) 36 | numBirdSeqs = len(glob.glob('bird*')) 37 | 38 | create_dir = "mkdir %s" % today 39 | subprocess.check_call(create_dir.split()) 40 | print("Creating new directory for day") 41 | with open('image-list.txt', 'w') as imageList: 42 | for i in range(numBirdSeqs): 43 | # if this isn't our current date, it's old laying around not deleted 44 | with open("bird%i/metadata.json" % (i + 1)) as meta: 45 | metadata = json.load(meta) 46 | detection_date = datetime.datetime.strptime(metadata['detection_time'], "%c") 47 | if detection_date.date() != datetime.datetime.now().date(): 48 | print("Skipping folder bird%i as it's older than today" % (i + 1)) 49 | break 50 | numCurrFiles = len(glob.glob('bird%i/00*.jpg' % (i + 1))) 51 | for j in range(numCurrFiles): 52 | imageList.write('file \'bird%i/%05d.jpg\'\n' % (i + 1, j)) 53 | copy_metadata = "cp bird%i/metadata.json ./%s/" % (i + 1, today) 54 | subprocess.check_call(copy_metadata.split()) 55 | copy_portrait = "cp bird%i/portrait.jpg ./%s/portrait%05d.jpg" % (i + 1, today, i + 1) 56 | subprocess.check_call(copy_portrait.split()) 57 | 58 | # create video file with current date with ffmpeg 59 | print("Running ffmpeg, this could take a while") 60 | try: 61 | command = "ffmpeg -r 24 -f concat -i image-list.txt -profile:v high -level 4.1 -pix_fmt yuv420p %s.mp4" % today 62 | subprocess.check_call(command.split()) 63 | except subprocess.CalledProcessError: 64 | print("Couldn't create the video summary of the day.") 65 | exit() 66 | 67 | # delete the original bird images 68 | print("ffmpeg ran successfully, deleting original images") 69 | try: 70 | command = "rm -rf bird*" 71 | subprocess.check_call(command.split()) 72 | except subprocess.CalledProcessError: 73 | print("Couldn't delete old detected bird images.") 74 | exit() 75 | 76 | # and restart the python detection process once more 77 | print("Restarting python detection process") 78 | command = "nohup python3 /home/nvidia/rich-mans-deep-learning-camera/richierich.py -u http://10.0.0.3:5000/image.jpg" 79 | subprocess.call(command.split()) 80 | 81 | -------------------------------------------------------------------------------- /richierich.py: -------------------------------------------------------------------------------- 1 | from darkflow.net.build import TFNet 2 | import cv2 3 | 4 | from imutils.video import VideoStream 5 | from imutils import resize 6 | 7 | import numpy as np 8 | 9 | import os 10 | import threading 11 | import time 12 | 13 | import subprocess 14 | import json 15 | 16 | import argparse 17 | 18 | parser = argparse.ArgumentParser() 19 | parser.add_argument('-u', "--url", type=str, help="Url to hit to grab another image perspective") 20 | parsed = parser.parse_args() 21 | 22 | vs = VideoStream(resolution=(1280,720), framerate=30) 23 | vs.stream.stream.set(3, 1280) 24 | vs.stream.stream.set(4, 720) 25 | 26 | theCam = vs.start() 27 | 28 | options = {"model": "cfg/tiny-yolo-voc.cfg", "load": "bin/tiny-yolo-voc.weights", "threshold": 0.1, "gpu": 0.2} 29 | 30 | tfnet = TFNet(options) 31 | 32 | beforeFrames = 30 33 | afterFrames = 240 34 | 35 | # skip frames, we'll check n/30 times per second for bird 36 | skipFrames = 10 37 | 38 | # label to try detecting 39 | detectLabel = "bird" 40 | 41 | birdDetected = False 42 | 43 | birdFrames = 0 44 | 45 | birdsSeen = 0 46 | 47 | frame = theCam.read() 48 | #frame = resize(frame, width=512) 49 | 50 | 51 | theBuffer = np.zeros((frame.shape[0], frame.shape[1], frame.shape[2], beforeFrames), dtype='uint8') 52 | 53 | # prefill buffer with frames 54 | def prefillBuffer(): 55 | for i in range(beforeFrames): 56 | frame = theCam.read() 57 | #frame = resize(frame, width=512) 58 | theBuffer[:,:,:,i] = frame 59 | 60 | def getHighRes(detectLabel, birdsSeen, url): 61 | # gets a high res image from raspberry pi camera v2 server 62 | # https://github.com/burningion/poor-mans-deep-learning-camera 63 | # (optional!) 64 | return subprocess.Popen(['wget', '-O', '%s%i/portrait.jpg' % (detectLabel, birdsSeen), url]) 65 | 66 | 67 | prefillBuffer() 68 | 69 | currentFrame = 0 70 | 71 | def getFramesAfterDetection(fileName, frameBegin, frameLength): 72 | for i in range(frameLength): 73 | frame = theCam.read() 74 | #frame = resize(frame, width=512) 75 | cv2.imwrite('%s%i/%05d.jpg' % (detectLabel, fileName, frameBegin + i), frame) 76 | # add this sleep as a hack so we don't write the same frame 77 | # more than once. the tx1 can write faster than 30 fps to disk 78 | # on my ssd 79 | time.sleep(.01) 80 | 81 | print('getframes thread finished') 82 | 83 | while True: 84 | # this is the numpy implementation of our circular buffer 85 | theBuffer = np.roll(theBuffer, -1, axis=3) 86 | frame = theCam.read() 87 | #frame = resize(frame, width=512) 88 | 89 | theBuffer[:,:,:,-1] = frame 90 | 91 | if not birdDetected: 92 | currentFrame += 1 93 | if currentFrame % skipFrames == 0 and currentFrame > 0: 94 | frame = resize(frame, width=512) 95 | result = tfnet.return_predict(frame) 96 | for detection in result: 97 | if detection['label'] == detectLabel: 98 | birdDetected = True 99 | birdsSeen += 1 100 | print("%s seen!" % detectLabel) 101 | if not os.path.exists('%s%i' % (detectLabel, birdsSeen)): 102 | os.makedirs('%s%i' % (detectLabel, birdsSeen)) 103 | 104 | # spawn a new thread to start capturing directly from webcam while we save preroll 105 | afterT = threading.Thread(target=getFramesAfterDetection, args=(birdsSeen, beforeFrames, afterFrames)) 106 | afterT.start() 107 | 108 | # save prebuffer to disk on main thread 109 | for i in range(beforeFrames): 110 | birdFrames += 1 111 | print('writing preframes') 112 | cv2.imwrite('%s%i/%05d.jpg' % (detectLabel, birdsSeen, i), theBuffer[:,:,:,i]) 113 | currentFrame = 0 114 | print("preframes %i written" % birdFrames) 115 | birdDetected = False 116 | birdFrames = 0 117 | if parsed.url: 118 | getHighRes(detectLabel, birdsSeen, parsed.url) 119 | while afterT.is_alive(): 120 | time.sleep(0) 121 | print("done with thread") 122 | with open('%s%i/metadata.json' % (detectLabel, birdsSeen), 'w') as metadata: 123 | det = {'detections': result, 'detection_time': time.ctime()} 124 | for detection in det['detections']: 125 | detection['confidence'] = float(detection['confidence']) 126 | json.dump(det, metadata) 127 | prefillBuffer() 128 | 129 | break 130 | 131 | 132 | theCam.stop() 133 | -------------------------------------------------------------------------------- /stl/README.md: -------------------------------------------------------------------------------- 1 | # STL Files for 3D Printing Camera Mounts 2 | 3 | In this directory there are STL files for 3D printing the camera mounts. 4 | 5 | I used Blender to create the files in this folder, and fair warning, I have zero idea what I'm doing when it comes to 3D modeling. 6 | 7 | I had to scale this model file, to 41% for 3D printing on my Prusa i3 MK2. I used PLA, and hot glued velcro on to the side where I mounted my camera. 8 | 9 | This works for both my Raspberry PI v2 camera, and my Logitech HD webcam. 10 | 11 | ## Mounting onto Your Camera 12 | 13 | ![NVIDIA Jetson Deep Learning Camera](https://github.com/burningion/rich-mans-deep-learning-camera/raw/master/images/birdcamera.jpeg) 14 | 15 | The idea with this design is that we have 1" holes we can 3D print modules for. 16 | 17 | Each of these modules then fits into the 1" hole, and expands the capabilities of the deep learning camera. 18 | 19 | For now there's just a camera mount, but later versions will include food dispensers and interactive buttons for birds/crows. 20 | 21 | 22 | -------------------------------------------------------------------------------- /stl/cameramountPI2.stl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/burningion/rich-mans-deep-learning-camera/1c163e835230106b2275b00327981bf459ae6676/stl/cameramountPI2.stl -------------------------------------------------------------------------------- /stl/pimount2.stl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/burningion/rich-mans-deep-learning-camera/1c163e835230106b2275b00327981bf459ae6676/stl/pimount2.stl -------------------------------------------------------------------------------- /training/extractFromTimePoints.py: -------------------------------------------------------------------------------- 1 | import os 2 | import argparse 3 | import subprocess 4 | 5 | parser = argparse.ArgumentParser() 6 | parser.add_argument('-f', '--filename', help="Input video filename", required=True) 7 | parsed = parser.parse_args() 8 | 9 | counter = 0 10 | 11 | video_date = parsed.filename.split('.')[0] 12 | if not os.path.exists(video_date): 13 | os.makedirs(video_date) 14 | 15 | with open('timepoints.txt') as timepoints: 16 | for timepoint in timepoints: 17 | counter += 1 18 | print("Extracting timepoint %s" % timepoint) 19 | command = "ffmpeg -ss %s -i %s -vframes: 40 -q:v 2 -r 2 %s/%i-%%05d.png" % (timepoint, parsed.filename, video_date, counter) 20 | subprocess.check_call(command.split()) 21 | --------------------------------------------------------------------------------