├── .gitignore ├── LICENSE ├── MANIFEST.in ├── Overview.png ├── README.md ├── dist ├── pythonic-cv-1.2.2.tar.gz └── pythonic_cv-1.2.2-py3-none-any.whl ├── pyproject.toml ├── setup.cfg └── src └── pcv ├── __init__.py ├── examples ├── __init__.py ├── cam_video_switch.py ├── hand_write.py ├── names.txt └── something_fishy.py ├── interact.py ├── process.py ├── screen.py ├── source.py └── vidIO.py /.gitignore: -------------------------------------------------------------------------------- 1 | # compiled bytecode 2 | __pycache__ 3 | build/* 4 | **/*.egg-info 5 | 6 | # database stuff 7 | *.db -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 ES-Alexander 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include src/pcv/examples/names.txt -------------------------------------------------------------------------------- /Overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ES-Alexander/pythonic-cv/fca76c582df549e102266f38e85ba1e3010edc1b/Overview.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | _________________________________ 2 | Version: 1.2.2 3 | Author: ES Alexander 4 | Release Date: 26/May/2021 5 | _________________________________ 6 | 7 | # About 8 | OpenCV is a fantastic tool for computer vision, with significant Python support 9 | through automatically generated bindings. Unfortunately some basic functionality 10 | is frustrating to use, and documentation is sparse and fragmented as to how best to 11 | approach even simple tasks such as efficiently processing a webcam feed. 12 | 13 | This library aims to address frustrations in the OpenCV Python api that can be 14 | fixed using pythonic constructs and methodologies. Solutions are not guaranteed to 15 | be optimal, but every effort has been made to make the code as performant as 16 | possible while ensuring ease of use and helpful errors/documentation. 17 | 18 | # Requirements 19 | This library requires an existing version of `OpenCV` with Python bindings to be 20 | installed (e.g. `python3 -m pip install opencv-python`). Some features (mainly 21 | property access helpers) may not work for versions of OpenCV earlier than 4.2.0. 22 | The library was tested using Python 3.8.5, and is expected to work down to at least 23 | Python 3.4 (although the integrated advanced features example uses matmul (@) for 24 | some processing, which was introduced in Python 3.5). 25 | 26 | `Numpy` is also used throughout, so a recent version is suggested (tested with 1.19.0). 27 | `mss` is used for cross-platform screen-capturing functionality (requires >= 3.0.0) 28 | 29 | # Installation 30 | The library can be installed from pip, with `python3 -m pip install pythonic-cv`. 31 | 32 | # Usage 33 | New functionality is provided in the `pcv` module, as described below. All other 34 | opencv functionality should be accessed through the standard `cv2` import. 35 | 36 | ## Main Functionality 37 | The main implemented functionality is handling video through a context manager, 38 | while also enabling iteration over the input stream. While iterating, key-bindings 39 | have been set up for play/pause (`SPACE`) and stopping playback (`q`). A dictionary 40 | of pause_effects can be passed in to add additional key-bindings while paused without 41 | needing to create a subclass. In addition, video playback can be sped up with `w`, 42 | slowed down with `s`, and if enabled allows rewinding with `a` and returning to 43 | forwards playback with `d`. Forwards playback at 1x speed can be restored with `r`. 44 | While paused, video can be stepped backwards and forwards using `a` and `d`. All 45 | default key-bindings can be overwritten using the play_commands and pause_effects 46 | dictionaries and the quit and play_pause variables on initialisation. 47 | 48 | ## Video I/O Details 49 | For reading and writing video files, the `VideoReader` and `VideoWriter` classes should 50 | be used. For streaming, the classes `Camera`, `SlowCamera`, and `LockedCamera` are 51 | provided. The simplest of these is `SlowCamera`, which has slow iteration because image 52 | grabbing is performed synchronously, with a blocking call while reading each frame. 53 | `Camera` extends `SlowCamera` with additional logic to perform repeated grabbing in a 54 | separate thread, so processing and image grabbing can occur concurrently. `LockedCamera` 55 | sits between the two, providing thread based I/O but with more control over when each 56 | image is taken. Analogues for all three camera classes are provided in the `pcv.screen` 57 | module - `SlowScreen`, `Screen`, and `LockedScreen`. The interface is the same as that 58 | for the corresponding camera class, except that a monitor index (-1 for all) or screen 59 | region is passed in instead of a camera id. 60 | 61 | `Camera` is most useful for applications with processing speeds that require the most 62 | up to date information possible and don't want to waste time decoding frames that are 63 | grabbed too early to be processed (frame grabbing occurs in a separate thread, and only 64 | the latest frame is retrieved (decoded) on read). `SlowCamera` should only be used where 65 | power consumption or overall CPU usage are more important than fast processing, or in 66 | hardware that is only capable of single-thread execution, in which case the 67 | separate image-grabbing thread will only serve to slow things down. 68 | 69 | `LockedCamera` is intended to work asynchronously like `Camera`, but with more control. 70 | It allows the user to specify when the next image should be taken, which leads to less 71 | wasted CPU and power usage on grabbing frames that aren't used, but with time for the 72 | image to be grabbed and decoded before the next iteration needs to start. The locking 73 | protocol adds a small amount of additional syntax, and starting the image 74 | grabbing process too late in an iteration can result in waits similar to those in 75 | `SlowCamera`, while starting the process too early can result in images being somewhat 76 | out of date. Tuning can be done using the 'preprocess' and 'process' keyword arguments, 77 | with an in-depth usage example provided in `something_fishy.py`. When used correctly 78 | `LockedCamera` has the fastest iteration times, or if delays are used to slow down the 79 | process it can have CPU and power usage similar to that of `SlowCamera`. 80 | 81 | If using a video file to simulate a live camera stream, use `SlowCamera` or 82 | `LockedCamera` - `Camera` will skip frames. 83 | 84 | There is also a `GuaranteedVideoWriter` class which guarantees the output framerate by 85 | repeating frames when given input too slowly, and skipping frames when input is too 86 | fast. 87 | 88 | ## Overview 89 | ![Overview of classes diagram](https://github.com/ES-Alexander/pythonic-cv/blob/master/Overview.png) 90 | 91 | ## Examples 92 | ### Basic Camera Stream 93 | ```python 94 | from pcv.vidIO import Camera 95 | from pcv.process import channel_options, downsize 96 | 97 | # start streaming camera 0 (generally laptop webcam/primary camera), and destroy 'frame' 98 | # window (default streaming window) when finished. 99 | # Auto-initialised to have 1ms waitKey between iterations, breaking on 'q' key-press, 100 | # and play/pause using the spacebar. 101 | with Camera(0) as cam: 102 | cam.stream() 103 | 104 | # stream camera 0 on window 'channels', downsized and showing all available channels. 105 | with LockedCamera(0, display='channels', 106 | process=lambda img: channel_options(downsize(img, 4))) as cam: 107 | cam.stream() 108 | ``` 109 | 110 | ### Stream and Record 111 | ```python 112 | from pcv.vidIO import Camera 113 | 114 | with Camera(0) as cam: 115 | print("press 'q' to quit and stop recording.") 116 | cam.record_stream('me.mp4') 117 | ``` 118 | 119 | ### Screen Functionality 120 | #### Main Monitor 121 | ```python 122 | import cv2 123 | from pcv.screen import LockedScreen 124 | 125 | # stream monitor 0, and record to 'screen-record.mp4' file. 126 | with LockedScreen(0, process=lambda img: \ 127 | cv2.cvtColor(img, cv2.COLOR_BGRA2BGR) as screen: 128 | # video-recording requires 3-channel (BGR) or single-channel 129 | # (greyscale, isColor=False) to work 130 | screen.record_stream('screen-record.mp4') 131 | ``` 132 | #### Screen Region 133 | ```python 134 | from pcv.Screen import Screen 135 | 136 | with Screen({'left': -10, 'top': 50, 'width': 100, 'height': 200}) as screen: 137 | screen.stream() 138 | ``` 139 | 140 | ### VideoReader 141 | ```python 142 | from pcv.vidIO import VideoReader 143 | from pcv.process import downsize 144 | 145 | # just play (simple) 146 | # Press 'b' to jump playback back to the beginning (only works if pressed 147 | # before playback is finished) 148 | with VideoReader('my_vid.mp4') as vid: 149 | vid.stream() 150 | 151 | # start 15 seconds in, end at 1:32, downsize the video by a factor of 4 152 | with VideoReader('my_vid.mp4', start='15', end='1:32', 153 | preprocess=lambda img: downsize(img, 4)) as vid: 154 | vid.stream() 155 | 156 | # enable rewinding and super fast playback 157 | # Press 'a' to rewind, 'd' to go forwards, 'w' to speed up, 's' to slow down 158 | # and 'r' to reset to forwards at 1x speed. 159 | with VideoReader('my_vid.mp4', skip_frames=0) as vid: 160 | vid.stream() 161 | 162 | # headless mode (no display), operating on every 10th frame 163 | with VideoReader('my_vid.mp4', auto_delay=False, skip_frames=10, 164 | process=my_processing_func) as vid: 165 | vid.headless_stream() 166 | ``` 167 | 168 | ### Mouse Events 169 | ```python 170 | # Example courtesy of @MatusGasparik in Issue#15 171 | import cv2 172 | from pcv.vidIO import VideoReader 173 | from pcv.interact import MouseCallback 174 | 175 | def on_mouse_click(event, x, y, flags, params=None): 176 | if event == cv2.EVENT_LBUTTONDOWN: 177 | print('Click', x, y) 178 | 179 | # create a window beforehand so a mouse callback can be assigned [REQUIRED] 180 | window = 'foo' 181 | cv2.namedWindow(window) 182 | with VideoReader('my_vid.mp4', display=window) as vid: 183 | vid.stream(mouse_handler=MouseCallback(vid.display, on_mouse_click)) 184 | ``` 185 | 186 | ### Advanced Examples 187 | Check the `pcv/examples` folder for some examples of full programs using this library. 188 | 189 | #### Something Fishy 190 | This example is a relatively basic augmented reality example, which creates a tank of 191 | fish that swim around on top of a video/webbcam feed. You can catch the fish (click and 192 | drag a 'net' over them with your mouse), or tickle them (move around in your webcam feed). 193 | There are several generally useful processing techniques included, so take a look 194 | through the code and find the functionality that's most interesting to you to explore 195 | and modify. 196 | 197 | To run the example use `python3 -m pcv.examples.something_fishy`, or optionally specify 198 | the path to a newline separated text file of names (of your friends and family for 199 | example), and run with `python3 -m pcv.examples.something_fishy path/to/names.txt`. 200 | 201 | #### Video Switcher 202 | This was made in response to a question from `u/guillerubio` on reddit, to show how to 203 | efficiently read multiple videos simultaneously while only displaying one, and switching 204 | between videos based on what's happening in a webcam feed. 205 | 206 | The current video is switched out when the camera is covered/uncovered. Switching is 207 | performed intelligently by only actually reading frames from a single 'active' video 208 | at a time. The `VideoSwitcher` class allows tracking along all the videos simultaneously 209 | based on how many frames have occurred since they were last active, as well as just 210 | resuming from where each one left off when it was last active. When a video ends it gets 211 | started again (in an infinite loop). 212 | 213 | To run the example use `python3 -m pcv.examples.cam_video_switch` while in a directory 214 | with the `.mp4` files you want to switch between. 215 | 216 | #### Hand Writer 217 | This uses Google's `mediapipe` library for hand detection, to show a relatively simple 218 | AR workflow, where a pointed right hand index finger can be used to write/draw on a 219 | camera feed. Once you've installed the library (`python3 -m pip install mediapipe`), 220 | run with `python3 -m pcv.examples.hand_write` and start drawing. 221 | 222 | The pointing detection algorithm is quite rudimentary, so may require some angling of 223 | your hand before it detects that the pointer finger is straight and the other fingers 224 | are closed. It currently intentionally only detects right hands, so your left hand can 225 | also be in the frame without causing issues. The side detection assumes a front-facing 226 | (selfie) camera, as is commonly the case with webcams and similar. 227 | 228 | # Contributors 💜 229 | 230 | - MatusGasparik 231 | -------------------------------------------------------------------------------- /dist/pythonic-cv-1.2.2.tar.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ES-Alexander/pythonic-cv/fca76c582df549e102266f38e85ba1e3010edc1b/dist/pythonic-cv-1.2.2.tar.gz -------------------------------------------------------------------------------- /dist/pythonic_cv-1.2.2-py3-none-any.whl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ES-Alexander/pythonic-cv/fca76c582df549e102266f38e85ba1e3010edc1b/dist/pythonic_cv-1.2.2-py3-none-any.whl -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = [ 3 | "setuptools>=42", 4 | "wheel" 5 | ] 6 | build-backend = "setuptools.build_meta" -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = pythonic-cv 3 | version = 1.2.2 4 | author = ES-Alexander 5 | author_email = sandman.esalexander@gmail.com 6 | description = Performant pythonic wrapper of unnecessarily painful opencv functionality 7 | long_description = file: README.md 8 | long_description_content_type = text/markdown 9 | url = https://github.com/ES-Alexander/pythonic-cv 10 | classifiers = 11 | Programming Language :: Python :: 3 12 | License :: OSI Approved :: MIT License 13 | Operating System :: OS Independent 14 | 15 | [options] 16 | package_dir = 17 | = src 18 | packages = find: 19 | python_requires = >=3.6 20 | 21 | [options.packages.find] 22 | where = src -------------------------------------------------------------------------------- /src/pcv/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ES-Alexander/pythonic-cv/fca76c582df549e102266f38e85ba1e3010edc1b/src/pcv/__init__.py -------------------------------------------------------------------------------- /src/pcv/examples/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ES-Alexander/pythonic-cv/fca76c582df549e102266f38e85ba1e3010edc1b/src/pcv/examples/__init__.py -------------------------------------------------------------------------------- /src/pcv/examples/cam_video_switch.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from pcv.vidIO import VideoReader, OutOfFrames 4 | import cv2 5 | 6 | class VideoSwitcher: 7 | ''' A class for switching between multiple videos. ''' 8 | def __init__(self, *filenames, track_along=True): 9 | ''' Opens multiple videos for switching between. 10 | 11 | When iterating, if a video completes it is re-started. 12 | 13 | 'track_along' is a boolean specifying if all the videos track along 14 | together. Defaults to True, so when swapping to the next video it 15 | gets jumped ahead to the number of frames that were covered since 16 | it was last active. If set to False, swapping back to a video 17 | resumes where it left off. 18 | 19 | ''' 20 | self.filenames = filenames 21 | self.track_along = track_along 22 | self.readers = [self._initialise(video) for video in filenames] 23 | self._counts = [0 for video in filenames] # position of each video 24 | self._count = 0 # total frames processed 25 | self.active = 0 # current active video index 26 | 27 | def _initialise(self, video): 28 | ''' Initialise a video and its iterator. ''' 29 | reader = VideoReader(video, destroy=None, verbose=False) 30 | iter(reader) 31 | return reader 32 | 33 | @property 34 | def active(self): 35 | return self._active 36 | 37 | @active.setter 38 | def active(self, value): 39 | ''' Set a new reader as active. Track frames as required. ''' 40 | self._active = value 41 | new_reader = self.readers[self._active] 42 | if self.track_along: 43 | extra = self._count - self._counts[self._active] 44 | new_frame = (new_reader.frame + extra) % new_reader._end 45 | new_reader.set_frame(new_frame) 46 | new_reader.reset_delay() # avoid pause jump/negative delay 47 | 48 | @property 49 | def video(self): 50 | return self.readers[self.active] 51 | 52 | @video.setter 53 | def video(self, replacement): 54 | self.readers[self.active] = replacement 55 | 56 | def __enter__(self): 57 | ''' Ensure all VideoReaders are ''' 58 | for reader in self.readers: 59 | reader.__enter__() 60 | 61 | return self 62 | 63 | def __exit__(self, *args): 64 | ''' Clean up all the VideoReaders at context end. ''' 65 | for reader in self.readers: 66 | try: 67 | reader.__exit__(*args) 68 | except: 69 | continue 70 | 71 | def __iter__(self): 72 | return self 73 | 74 | def __next__(self): 75 | ''' Get the next video frame from the active video. 76 | 77 | Restarts the active video first if it was finished. 78 | 79 | ''' 80 | self._count += 1 81 | self._counts[self.active] += 1 82 | try: 83 | return next(self.video) 84 | except OutOfFrames: 85 | return self._reinitialise_active_video() 86 | 87 | def _reinitialise_active_video(self): 88 | ''' Close and re-open the currently active video. ''' 89 | self.video.__exit__(None, None, None) 90 | filename = self.filenames[self.active] 91 | self.video = self._initialise(filename) 92 | return next(self.video) 93 | 94 | def next_video(self): 95 | ''' Switch to the next video (circular buffer). ''' 96 | self.active = (self.active + 1) % len(self.readers) 97 | 98 | def prev_video(self): 99 | ''' Switch to the previous video (circular buffer). ''' 100 | self.active = (self.active - 1) % len(self.readers) 101 | 102 | def set_active(self, index): 103 | ''' Switch to a specified video index, modulo the number of videos. ''' 104 | self.active = index % len(self.readers) 105 | 106 | 107 | if __name__ == '__main__': 108 | from os import listdir 109 | from pcv.vidIO import Camera 110 | 111 | class CoverAnalyser: 112 | ''' Analyses a camera feed and switches videos when un/covered. ''' 113 | def __init__(self, video_switcher): 114 | ''' Cycles through 'video_switcher' videos based on frame data. ''' 115 | self._video_switcher = video_switcher 116 | self._covered = False 117 | 118 | def analyse(self, frame): 119 | ''' Switches to the next video if the camera is un/covered. 120 | 121 | Switches on both camera cover and un-cover events. 122 | 123 | Mean values of 40 and 60 were relevant for my camera when testing, 124 | and may need to be different depending on camera and lighting. 125 | 126 | ''' 127 | mean = frame.mean() 128 | if (mean < 40 and not self._covered) \ 129 | or (mean > 50 and self._covered): 130 | # switch to next video and toggle state 131 | self._video_switcher.next_video() 132 | self._covered ^= True # toggle state using boolean xor 133 | 134 | videos = (file for file in listdir() if file.endswith('.mp4')) 135 | 136 | print("press space to pause, 'q' to quit") 137 | with VideoSwitcher(*videos) as video_switcher: 138 | analyser = CoverAnalyser(video_switcher) 139 | with Camera(0) as cam: 140 | # zip together the videos and camera to progress in sync 141 | for (_, v_frame), (_, c_frame) in zip(video_switcher, cam): 142 | analyser.analyse(c_frame) 143 | cv2.imshow('video', v_frame) 144 | -------------------------------------------------------------------------------- /src/pcv/examples/hand_write.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import numpy as np 3 | from pcv.vidIO import LockedCamera 4 | import mediapipe as mp 5 | mp_hands = mp.solutions.hands 6 | mp_drawing = mp.solutions.drawing_utils 7 | 8 | class HandWriter: 9 | def __init__(self, **kwargs): 10 | self._hands = mp_hands.Hands(**kwargs) 11 | self._drawing = None 12 | self._points = [] 13 | 14 | def __enter__(self): 15 | self._hands.__enter__() 16 | return self 17 | 18 | def __exit__(self, *args, **kwargs): 19 | return self._hands.__exit__(*args, **kwargs) 20 | 21 | def __call__(self, frame): 22 | flipped = cv2.flip(frame, 1) 23 | if self._drawing is None: 24 | self._drawing = np.zeros(flipped.shape[:2], np.uint8) 25 | image = cv2.cvtColor(flipped, cv2.COLOR_BGR2RGB) 26 | image.flags.writeable = False 27 | results = self._hands.process(image) 28 | height, width = image.shape[:2] 29 | num_points = len(self._points) 30 | if results.multi_hand_landmarks: 31 | for hand_landmarks, handedness in zip(results.multi_hand_landmarks, 32 | results.multi_handedness): 33 | 34 | if handedness.classification[0].label != 'Right': 35 | continue # only interested in right hands 36 | 37 | tips = {} 38 | pips = {} 39 | dips = {} 40 | bases = {} 41 | INDEX = 'index_finger' 42 | MIDDLE = 'middle_finger' 43 | for finger in (INDEX, MIDDLE): 44 | FINGER = finger.upper() 45 | for PART, d in (('TIP', tips), ('PIP', pips), 46 | ('DIP', dips), ('MCP', bases)): 47 | segment = hand_landmarks.landmark[ 48 | getattr(mp_hands.HandLandmark, 49 | f'{FINGER}_{PART}')] 50 | d[finger] = self.land2coord(segment, width, height) 51 | 52 | index_angle = self.finger_angle(INDEX, tips, pips, dips, bases) 53 | 54 | # only draw if index finger is open, and middle finger is 55 | # closed (assumes ring finger and pinky also closed) 56 | if index_angle < 0.3: 57 | middle_angle = self.finger_angle(MIDDLE, tips, pips, dips, 58 | bases) 59 | if middle_angle > 0.5: 60 | self._points.append(np.int32(tips[INDEX][:2])) 61 | 62 | mp_drawing.draw_landmarks(flipped, hand_landmarks, 63 | mp_hands.HAND_CONNECTIONS) 64 | break # only allow first detected right hand to draw 65 | 66 | # if there are no new points, or two points 67 | if (latest := len(self._points)) in (num_points, 2): 68 | if latest == 2: 69 | # draw the latest line on the drawing image 70 | # (means only need to keep track of one line at a time) 71 | self._drawing = cv2.line(self._drawing, self._points[0], 72 | self._points[1], 255, 3) 73 | if latest != 0: 74 | self._points.pop(0) # remove oldest point 75 | 76 | # saturate red channel at drawn locations 77 | flipped[:,:,2] |= self._drawing 78 | 79 | return flipped 80 | 81 | @staticmethod 82 | def land2coord(landmark, width, height): 83 | return np.array([landmark.x * width, landmark.y * height, 84 | landmark.z * width]) 85 | 86 | @staticmethod 87 | def finger_angle(segment, tips, pips, dips, bases): 88 | segment_first = tips[segment] - pips[segment] 89 | segment_base = dips[segment] - bases[segment] 90 | usf = segment_first / np.linalg.norm(segment_first) 91 | usb = segment_base / np.linalg.norm(segment_base) 92 | return np.arccos(np.clip(np.dot(usf, usb), -1.0, 1.0)) 93 | 94 | 95 | def main(filename, **kwargs): 96 | defaults = dict(min_detection_confidence=0.5, min_tracking_confidence=0.5, 97 | max_num_hands=2) 98 | defaults.update(kwargs) 99 | 100 | with HandWriter(**defaults) as hand_writer, \ 101 | LockedCamera(0, process=hand_writer) as cam: 102 | cam.record_stream(filename) 103 | 104 | 105 | if __name__ == '__main__': 106 | import sys 107 | filename = 'handwriting.mp4' if len(sys.argv) == 1 else sys.argv[1] 108 | main(filename) 109 | -------------------------------------------------------------------------------- /src/pcv/examples/names.txt: -------------------------------------------------------------------------------- 1 | Mohamed 2 | Youssef 3 | Pierre 4 | Petros 5 | Manuel 6 | Ahmed 7 | Mamadou 8 | Hamza 9 | Melokuhle 10 | Mehdi 11 | Fatima 12 | Reem 13 | Mary 14 | Bridget 15 | Helen 16 | Isabel 17 | Aya 18 | Fatoumata 19 | Salma 20 | Amahle 21 | Shayma 22 | Santiago 23 | Daniel 24 | Eliot 25 | Enzo 26 | Francisco 27 | Noah 28 | William 29 | Agustin 30 | Samuel 31 | Oliver 32 | Pepe 33 | Stanley 34 | Jayden 35 | Mateo 36 | Ramon 37 | Luis 38 | Dylan 39 | Liam 40 | James 41 | Bruno 42 | David 43 | Sofia 44 | Alysha 45 | Maria 46 | Ana 47 | Olivia 48 | Emma 49 | Isidora 50 | Valentina 51 | Widelene 52 | Gabrielle 53 | Ximena 54 | Elizabeth 55 | Camila 56 | Victoria 57 | Taylar 58 | Avril 59 | Ali 60 | Adam 61 | Cheng 62 | Wei 63 | Jie 64 | Hao 65 | Aarav 66 | Hossein 67 | Abbas 68 | Omar 69 | Ori 70 | Noam 71 | Abed 72 | Elias 73 | Rani 74 | Hinata 75 | Yerasyl 76 | Faisal 77 | Charbel 78 | Ethan 79 | Sukhbaatar 80 | Krishna 81 | Bilal 82 | Nathaniel 83 | Turki 84 | Joo-won 85 | Wonoo 86 | Chien-hung 87 | Yusuf 88 | Somchai 89 | Abdullah 90 | Maryam 91 | Fang 92 | Xiaoyan 93 | Diya 94 | Hasti 95 | Tom 96 | Sakineh 97 | Tamar 98 | Ariel 99 | Aline 100 | Maya 101 | Eden 102 | Sakura 103 | Riko 104 | Rimas 105 | Inzhu 106 | Hussa 107 | Zeinab 108 | Siti 109 | Odval 110 | Rabina 111 | Bismah 112 | Samantha 113 | Seo-yun 114 | Latifa 115 | Mei-ling 116 | Oisha 117 | Noor 118 | Alban 119 | Marc 120 | Davit 121 | Lukas 122 | Raul 123 | Elchin 124 | Mikhail 125 | Arthur 126 | Wout 127 | Hugo 128 | Rayan 129 | Nathan 130 | Vedad 131 | Marko 132 | Martin 133 | Luka 134 | Christos 135 | Jan 136 | Malthe 137 | Jack 138 | Harry 139 | Benjamin 140 | Leo 141 | Max 142 | Gabriel 143 | Giorgi 144 | Ben 145 | Dimitrios 146 | Aputsiaq 147 | Lewis 148 | Levente 149 | Viktor 150 | Connor 151 | Sean 152 | Lorenzo 153 | Gustavs 154 | Raphael 155 | Matas 156 | Dragan 157 | Andrej 158 | Jorgen 159 | Petar 160 | Zachary 161 | Artiom 162 | Bogdan 163 | Aaron 164 | Zoran 165 | Lazar 166 | Jesse 167 | Charlie 168 | Filip 169 | Aleksander 170 | Rodrigo 171 | Stefan 172 | Ivan 173 | Alexey 174 | Logan 175 | Nikola 176 | Jovan 177 | Matej 178 | Tim 179 | Pablo 180 | Julen 181 | Biel 182 | Liam 183 | Leon 184 | Mustafa 185 | Osman 186 | Mykhailo 187 | Oscar 188 | Amelia 189 | Laia 190 | Nareh 191 | Anna 192 | Zahra 193 | Sevinj 194 | Laylani 195 | Polina 196 | Elise 197 | Rita 198 | Julie 199 | Manon 200 | Lina 201 | Sanne 202 | Jasmine 203 | Merjem 204 | Teodora 205 | Daria 206 | Petra 207 | Eleni 208 | Natalie 209 | Alma 210 | Lily 211 | Laura 212 | Mia 213 | Elsa 214 | Eevi 215 | Alva 216 | Chloe 217 | Anastasia 218 | Lena 219 | Vasiliki 220 | Paninnguaq 221 | Jessica 222 | Hanna 223 | Eva 224 | Ivanna 225 | Aino 226 | Saga 227 | Jade 228 | Barbare 229 | Angeliki 230 | Daisy 231 | Ella 232 | Grace 233 | Abbie 234 | Ginevra 235 | Marta 236 | Noemi 237 | Vilte 238 | Amy 239 | Biljana 240 | Jovana 241 | Jana 242 | Catherine 243 | Evelina 244 | Giulia 245 | Milica 246 | Dunja 247 | Zoe 248 | Tess 249 | Aoife 250 | Ingrid 251 | Alicja 252 | Beatriz 253 | Andreea 254 | Yelizaveta 255 | Yekaterina 256 | Charlotte 257 | Dragana 258 | Shweta 259 | Katarina 260 | Nela 261 | Lara 262 | Laura 263 | Kate 264 | Sheridan 265 | Megan 266 | Paula 267 | June 268 | Martina 269 | Tegan 270 | Skylar 271 | Alicia 272 | Emilia 273 | Elif 274 | Emine 275 | Polina 276 | Janet 277 | Evie 278 | Vivienne 279 | Gloria 280 | Ruben 281 | Mitch 282 | Ethan 283 | Neil 284 | James 285 | Emma 286 | Julia 287 | Sarah 288 | Shaun 289 | Manaia 290 | Aidan 291 | Hiro 292 | Ava 293 | Ruby 294 | Rachel 295 | Aria 296 | Maeva 297 | Elon's Musk 298 | Roberto Down-June 299 | Nikola Tesla 300 | Eddie Red-Main 301 | Stephen Hawking 302 | Xzibit -------------------------------------------------------------------------------- /src/pcv/examples/something_fishy.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import numpy as np 4 | import cv2 5 | from os import mkdir 6 | from os.path import isfile, isdir 7 | from pcv.vidIO import LockedCamera 8 | from pcv.interact import MouseCallback 9 | 10 | 11 | class Fish: 12 | ''' A fish that floats, and can be tickled and/or caught. ''' 13 | def __init__(self, name, position, velocity, tank_dims, depth): 14 | ''' Create a fish with specified attributes. 15 | 16 | 'name' should be a string, and determines the internal properties 17 | of the fish, such as size and colour. 18 | 'position' and 'velocity' should be complex numbers with the real 19 | axis pointing down the image, and the imaginary axis pointing 20 | across to the right. The origin is the top left corner. 21 | This helps facilitate easy flipping at the sides (complex 22 | conjugate), and rotation at the top/bottom. 23 | 'tank_dims' is a tuple of the dimensions of the tank, as 24 | (height, width) coordinates. 25 | 'depth' is this fish's location relative to the viewer/other fish. 26 | It should generally be between 1 and 50, and will likely raise 27 | an Exception if below -5 or above 95 (it is used to determine 28 | water cover). 29 | 30 | ''' 31 | self.name = name 32 | self.position = position 33 | self.velocity = velocity 34 | self.x_min = self.y_min = 0 35 | self.y_max, self.x_max = tank_dims 36 | self.depth = depth 37 | self.is_caught = 0 38 | self._changed = [False, False] 39 | self.determine_appearance() 40 | self.update_angle() 41 | 42 | @property 43 | def position_tuple(self): 44 | ''' The x, y position coordinates. ''' 45 | return self.position.imag, self.position.real 46 | 47 | @property 48 | def velocity_tuple(self): 49 | ''' The x, y velocity coordinates. ''' 50 | return self.velocity.imag, self.velocity.real 51 | 52 | @property 53 | def bbox(self): 54 | ''' A rough bounding box of the fish. ''' 55 | long = self.axes[0] 56 | return np.array(list(self.position_tuple)*2).reshape(2,2) \ 57 | - np.array([[long], [-long]]) 58 | 59 | def determine_appearance(self): 60 | ''' Use self.name to determine size and colour. ''' 61 | # convert name to (0,1] numbers, use only lowercase for good range 62 | min_char, max_char = ord('a')-1, ord('z') 63 | range_ = max_char - min_char 64 | numbers = [(ord(c.lower())-min_char)/range_ 65 | for c in self.name if c not in " -'"] 66 | self.colours = np.array([255*np.array([numbers[i], numbers[-i%3]]) 67 | for i in range(-3,0)]) 68 | # add blue for depth (at least 5% behind water) 69 | alpha = self.depth / 100 + 0.05 70 | self.colours = self.add_water(self.colours, alpha).T 71 | 72 | # determine size and shape 73 | self.size = 3*(sum(numbers) + len(numbers)) / (sum(numbers[:2])/2) 74 | self.axes = (self.size / (numbers[1]+2), 75 | self.size / (numbers[2]+3.2)) 76 | 77 | # eye properties 78 | self._eye_offset = np.array([3*self.axes[0]/5, self.axes[1]/4]) 79 | self._eye_size = int(self.axes[0] / 7) 80 | self._pupil_size = int(self.axes[1] / 8) 81 | 82 | # fin properties 83 | self._fin_points = np.array([[0,0],[-1,numbers[2]],[-1,-numbers[1]]]) \ 84 | * self.axes[1] 85 | 86 | # tail properties 87 | self._tail_points = np.array( 88 | [[-self._eye_offset[0], 0], 89 | -self._eye_offset * [2, 3], 90 | self._eye_offset * [-(1.2 + 2 * numbers[0] / 3), 1]] * 2) 91 | self._tail_points = [self._tail_points[:3], self._tail_points[3:]] 92 | self._tail_points[1][1,1] *= -1 93 | self._tail_points[1][2,1] *= -1 94 | 95 | @staticmethod 96 | def add_water(colour, alpha): 97 | ''' Add (100*alpha)% water cover to the specified colour. ''' 98 | beta = 1 - alpha 99 | colour *= beta 100 | colour[0] += 200 * alpha 101 | colour[1] += 40 * alpha 102 | return colour 103 | 104 | def update_angle(self): 105 | ''' Compute the updated angle and rotation based on the velocity. ''' 106 | angle = np.arctan2(*self.velocity_tuple[::-1]) 107 | self._rotate_fin = np.array([np.cos(angle), np.sin(angle)]) 108 | self._rotate = np.array([self._rotate_fin, 109 | (self._rotate_fin * [1,-1])[::-1]]) 110 | self.angle = np.degrees(angle) 111 | 112 | def draw(self, img, acceleration): 113 | ''' Draw self and update state according to acceleration. 114 | 115 | If self.is_caught, draws a fading ellipse for 50 frames before 116 | skipping drawing altogether. 117 | 118 | 'img' is the image to draw onto, and should be an 8-bit colour image. 119 | 'acceleration' should be a single-channel image with the same width 120 | and height as 'img', mapping position to acceleration. 121 | 122 | ''' 123 | if self.is_caught < 50: 124 | colour = self.colours[0].copy() 125 | thickness = -1 126 | if self.is_caught: 127 | alpha = self.is_caught / 50 128 | colour = tuple(self.add_water(colour, alpha)) 129 | pos = tuple(int(dim) for dim in self.position_tuple) 130 | self.is_caught += 1 131 | thickness = 1 132 | 133 | cv2.ellipse(img, (self.position_tuple, 134 | tuple(2*dim for dim in self.axes), 135 | self.angle), colour, thickness) 136 | 137 | if not self.is_caught: 138 | for draw_func in (self.draw_eye, self.draw_tail, self.draw_fin): 139 | draw_func(img) 140 | 141 | self.update_state(acceleration) 142 | 143 | def draw_eye(self, img): 144 | ''' Draw eye on 'img'. ''' 145 | eye_offset = self._eye_offset * [1, -np.sign(self.velocity.imag)] 146 | eye_offset = eye_offset @ self._rotate 147 | pupil_offset = eye_offset * 1.05 148 | eye_pos = tuple(int(dim) for dim in eye_offset + self.position_tuple) 149 | pupil_pos = tuple(int(dim) for dim in 150 | pupil_offset + self.position_tuple) 151 | for pos, size, colour in [[eye_pos, self._eye_size, (244,212,204)], 152 | [pupil_pos, self._pupil_size, (40,8,0)]]: 153 | cv2.circle(img, pos, size, colour, -1) 154 | 155 | def draw_tail(self, img): 156 | ''' Draw tail on 'img'. ''' 157 | colour = tuple(int(channel) for channel in self.colours[1]) 158 | for half in self._tail_points: 159 | half_points = half @ self._rotate + self.position_tuple 160 | cv2.fillConvexPoly(img, np.int32(half_points), colour) 161 | 162 | def draw_fin(self, img): 163 | ''' Draw fin on 'img'. ''' 164 | colour = tuple(int(channel) for channel in self.colours[1]) 165 | fin_points = self._rotate_fin * self._fin_points + self.position_tuple 166 | cv2.fillConvexPoly(img, np.int32(fin_points), colour) 167 | 168 | def update_state(self, acceleration): 169 | ''' Update the fish position/velocity. ''' 170 | # update position 171 | self.position += self.velocity 172 | 173 | # update velocity 174 | self.velocity *= 0.995 # add a touch of damping to avoid crazy speeds 175 | 176 | x,y = np.int32(np.round(self.position_tuple)) 177 | x_accel, y_accel = acceleration 178 | try: 179 | self.velocity += (x_accel[y,x] + 1j * y_accel[y,x]) 180 | except IndexError: 181 | pass # outside the tank 182 | 183 | # update relevant velocity component if outside the tank 184 | for index, (min_, max_) in enumerate([[self.x_min, self.x_max], 185 | [self.y_min, self.y_max]]): 186 | val = self.position_tuple[index] 187 | left = val < min_ 188 | if (left or val > max_): 189 | if not self._changed[index]: 190 | if index == 0: # mirror if hitting side 191 | self.velocity = self.velocity.conjugate() 192 | else: # rotate 90 degrees if hitting top/bottom 193 | direc = -2 * (left - 0.5) 194 | self.velocity *= \ 195 | direc * np.sign(self.velocity.imag) * 1j 196 | self._changed[index] = True 197 | elif self._changed[index]: 198 | # back in the tank 199 | self._changed[index] = False 200 | 201 | self.update_angle() 202 | 203 | def catch(self): 204 | self.is_caught = 1 205 | print(f'Caught {self}!') 206 | 207 | def __str__(self): 208 | return f'{self.name}: {self.size/100:.3f}kg' 209 | 210 | 211 | class FishTank: 212 | ''' A tank for storing and tracking fish. ''' 213 | def __init__(self, tank_dims, max_fish=40, name_file='names.txt'): 214 | ''' Create a new fish tank as specified. 215 | 216 | 'tank_dims' should be a tuple of (height, width) pixels 217 | 'max_fish' is the largest number of fish that can be generated on 218 | initialisation (random number generated between 1 and max_fish). 219 | Defaults to 40. 220 | 'name_file' is the filename of a newline separated file containing 221 | possible names for the fish in the tank. Defaults to 'names.txt' 222 | which on distribution contains a few hundred popular names from 223 | around the world. Feel free to change it to a set of names of your 224 | family and friends! 225 | 226 | ''' 227 | # store/initialise input parameters 228 | self.dims = np.int32(tank_dims) 229 | self.max_fish = max_fish 230 | with open(name_file) as names: 231 | self.names = [name.strip() for name in names] 232 | 233 | self._initialise_fish() 234 | self._initialise_stats() 235 | self._initialise_instructions() 236 | self._setup_mouse_control() 237 | 238 | def _initialise_fish(self): 239 | ''' initialise fish, including storage and trackers ''' 240 | # create a fast random number generator 241 | self.rng = np.random.default_rng() 242 | 243 | self.caught_fish = [] 244 | self.fish = [] 245 | self._num_fish = self.rng.integers(2, self.max_fish) 246 | for i in range(self._num_fish): 247 | self.fish.append(self.random_fish(i)) 248 | 249 | def random_fish(self, depth): 250 | ''' Create a random fish instance. ''' 251 | return Fish(self.random_name(), self.random_position(), 252 | self.random_velocity(), self.dims, depth) 253 | 254 | def random_name(self): 255 | ''' Randomly choose a name for a fish. ''' 256 | return self.names[self.rng.integers(len(self.names))] 257 | 258 | def random_position(self): 259 | ''' Determine a valid random position for a fish. ''' 260 | offset = 40 261 | return complex(self.rng.integers(offset, self.dims[0]-offset), 262 | self.rng.integers(offset, self.dims[1]-offset)) 263 | 264 | def random_velocity(self): 265 | ''' Create a random initial velocity for a fish. ''' 266 | max_x = self.dims[1] // 100 267 | max_y = self.dims[0] // 100 268 | return complex(self.rng.integers(-max_y, max_y), 269 | self.rng.integers(-max_x, max_x)) 270 | 271 | def _initialise_stats(self): 272 | ''' Intialise stats and tracking parameters. ''' 273 | self._prev = np.zeros(tuple(self.dims), dtype=np.int32) 274 | self._precision = 0 275 | self._gradient = False 276 | self._attempts = 0 277 | self._t = 1 278 | 279 | def _initialise_instructions(self): 280 | ''' Create some helpful instructions to display at the start. ''' 281 | self._instructions_visible = True 282 | 283 | scale = 0.6 284 | thickness = 1 285 | height, width = self.dims 286 | font = cv2.FONT_HERSHEY_SIMPLEX 287 | self._instructions = np.zeros(tuple((height, width, 3)), dtype=np.uint8) 288 | 289 | instructions = ( 290 | "Who lives in a pineapple with OpenCV?", 291 | '', 292 | "Press 'i' to toggle instructions on/off", 293 | "Press 'g' to toggle the image gradient used for acceleration", 294 | "Press 'q' to quit, and SPACE to pause/resume", 295 | '', 296 | "Catch fish by dragging your 'net' over them with the mouse", 297 | "(if your box is too big or small they'll escape).", 298 | "Caught fish will have their image, with name and size", 299 | "displayed in the 'gallery' folder.", 300 | '', 301 | "'Hit rate' is the percentage of attempts you've caught a fish in.", 302 | "'Avg size ratio' is the ratio of your box size over the fish size", 303 | "for each of your successful catches - smaller is more skillful.", 304 | '', 305 | "Some fish might escape the tank and not come back, and that's ok.", 306 | ) 307 | 308 | # add instructions to an empty image, for merging later 309 | num_instructions = len(instructions) 310 | text_height = cv2.getTextSize(' ', font, scale, thickness)[0][1] 311 | spacing = 2 * text_height / 3 312 | tot_y = num_instructions * text_height + spacing * (num_instructions - 2) 313 | 314 | y_offset = (height - tot_y) // 2 315 | 316 | for index, line in enumerate(instructions): 317 | x,y = cv2.getTextSize(line, font, scale, thickness)[0] 318 | x_pos = int((width - x) / 2) 319 | y_pos = int(y_offset + (y + spacing) * index) 320 | cv2.putText(self._instructions, line, (x_pos, y_pos), font, 321 | scale, (255,255,255), thickness) 322 | 323 | 324 | def _setup_mouse_control(self): 325 | ''' Specify mouse control functions. ''' 326 | self._start_point = self._end_point = None 327 | self._catch_bindings = { 328 | cv2.EVENT_LBUTTONDOWN : self.start_catching, 329 | cv2.EVENT_MOUSEMOVE : self.catch_to, 330 | cv2.EVENT_LBUTTONUP : self.end_catch, 331 | } 332 | 333 | def mouse_handler(self, event, *args): 334 | self._catch_bindings.get(event, lambda *args: None)(*args) 335 | 336 | def start_catching(self, x, y, *args): 337 | ''' Start the creation of a net for catching fish. ''' 338 | self._start_point = self._end_point = x,y 339 | 340 | def catch_to(self, x, y, *args): 341 | ''' Draw the net live as it resizes. ''' 342 | if self._start_point: 343 | self._end_point = x,y 344 | 345 | def end_catch(self, x, y, *args): 346 | ''' Register a catch attempt and check for catches. ''' 347 | self._catch_img = self._img 348 | self._catch_fish() 349 | self._start_point = self._end_point = None 350 | self._attempts += 1 351 | 352 | def _catch_fish(self): 353 | ''' Check if any fish were caught in the last attempt. ''' 354 | # get current fish bounding boxes 355 | try: 356 | min_pts, max_pts = self._fish_bboxes 357 | except ValueError: 358 | return # no more fish to catch 359 | min_pts = min_pts.reshape(-1,2) 360 | max_pts = max_pts.reshape(-1,2) 361 | 362 | min_pt, max_pt = self._get_net_extent() 363 | caught = self._find_caught_fish(min_pts, max_pts, 364 | min_pt, max_pt) 365 | self._register_catches(min_pt, max_pt, caught) 366 | 367 | def _get_net_extent(self): 368 | ''' Returns the min_pt, max_pt of the net extent. ''' 369 | pts = [] 370 | for i in range(2): 371 | p1 = self._start_point[i] 372 | p2 = self._end_point[i] 373 | if p1 < p2: 374 | pts.append([p1,p2]) 375 | else: 376 | pts.append([p2,p1]) 377 | return np.array(pts).T 378 | 379 | def _find_caught_fish(self, min_pts, max_pts, min_pt, max_pt): 380 | ''' Returns an index array of caught fish. ''' 381 | min_diff = min_pts - min_pt 382 | max_diff = max_pt - max_pts 383 | box_size = (max_pt - min_pt).sum() 384 | size_ratio = box_size / (max_pts - min_pts).sum(axis=1) 385 | caught, = np.nonzero((size_ratio < 4) & 386 | (((min_diff > 0) & (max_diff > 0)).sum(axis=1) == 2)) 387 | self._precision += size_ratio[caught].sum() 388 | return caught 389 | 390 | def _register_catches(self, min_pt, max_pt, caught): 391 | ''' Register catches and track which fish are free. ''' 392 | free_fish = [] 393 | caught_fish = [] 394 | for index, fish in enumerate(self.fish): 395 | if index in caught: 396 | caught_fish.append(fish) 397 | fish.catch() 398 | else: 399 | free_fish.append(fish) 400 | 401 | # save image for caught fish 402 | if len(caught): 403 | # create the gallery if it doesn't already exist 404 | if not isdir('gallery'): 405 | mkdir('gallery') 406 | 407 | # determine relevant image filename 408 | fish = '-'.join(f'{fish.name}_{fish.size/100:.3f}kg' 409 | for fish in caught_fish) 410 | pre, extension = 'gallery/caught_', '.png' 411 | filename = pre + fish + extension 412 | 413 | # put a count at the end if the fish has already been caught 414 | count = 0 415 | while isfile(filename): 416 | count += 1 417 | filename = f'{pre}{fish}({count}){extension}' 418 | 419 | # ensure image is within frame 420 | min_pt[min_pt < 0] = 0 421 | max_pt[0] = min(max_pt[0], self.dims[1]) 422 | max_pt[1] = min(max_pt[1], self.dims[0]) 423 | 424 | # write to file 425 | cv2.imwrite(filename, self._catch_img[min_pt[1]:max_pt[1], 426 | min_pt[0]:max_pt[0]]) 427 | 428 | self.caught_fish.extend(caught_fish) 429 | self.fish = free_fish 430 | 431 | @property 432 | def _fish_bboxes(self): 433 | ''' Returns an array of the min_pts and max_pts of each fish. ''' 434 | return np.c_[tuple(fish.bbox for fish in self.fish)] 435 | 436 | def toggle_gradient(self, vid=None): 437 | ''' Toggle gradient display mode on or off. ''' 438 | self._gradient ^= True 439 | 440 | def toggle_instructions(self, vid=None): 441 | ''' Toggle the instructions display on or off. ''' 442 | self._instructions_visible ^= True 443 | 444 | def preprocess(self, img): 445 | ''' Light preprocessing. ''' 446 | self._t += 1 447 | max_accel = 30 448 | 449 | blur = cv2.GaussianBlur(img, (7,7), 0) 450 | flipped = cv2.flip(blur, 1) # mirror webcam 451 | grey = np.float32(cv2.cvtColor(flipped, cv2.COLOR_BGR2GRAY)) / 255 452 | 453 | # calculate acceleration from difference to previous image 454 | diff = (grey - self._prev + 1) / 2 455 | x_accel = cv2.Sobel(diff, cv2.CV_64F, 1, 0, ksize=5) 456 | x_accel /= (1e-10+x_accel.max()-x_accel.min()) / max_accel 457 | y_accel = cv2.Sobel(diff, cv2.CV_64F, 0, 1, ksize=5) 458 | y_accel /= (1e-10+y_accel.max()-y_accel.min()) / max_accel 459 | self._acceleration = x_accel, y_accel 460 | self._prev = grey 461 | 462 | return flipped 463 | 464 | def __call__(self, flipped): 465 | ''' Main processing, while waiting for next image. ''' 466 | if self._gradient: 467 | x_accel, y_accel = self._acceleration 468 | max_val = np.max([x_accel.max(), y_accel.max()]) 469 | min_val = np.min([x_accel.min(), y_accel.min()]) 470 | range_ = max_val - min_val 471 | x_norm = (x_accel - min_val) / range_ 472 | y_norm = (y_accel - min_val) / range_ 473 | gradient = cv2.addWeighted(x_norm, 0.5, y_norm, 0.5, 0.0) 474 | flipped = cv2.merge([np.uint8(255 * gradient)]*3) 475 | else: 476 | self._draw_water(flipped) 477 | 478 | self._draw_fish(flipped, self._acceleration) 479 | self._img = flipped 480 | self._text_overlay(flipped) 481 | self._draw_net(flipped) 482 | 483 | return flipped 484 | 485 | def _draw_water(self, img): 486 | # make some blue and green that varies a bit with time 487 | blue = np.zeros(img.shape, dtype=img.dtype) 488 | mag = 30 * np.sin(self._t/100) 489 | blue[:,:,0] = 200 + mag * np.sin(np.arange(img.shape[1])/(mag/6+50)) 490 | blue[:,:,1] = 40 + mag * np.sin(np.arange(img.shape[0])[:,None] \ 491 | / (mag/6+50)) 492 | 493 | # blend with the background image 494 | alpha = 0.45 495 | beta = 1 - alpha 496 | cv2.addWeighted(img, alpha, blue, beta, 0.0, img) 497 | 498 | def _draw_fish(self, img, acceleration): 499 | ''' Draw in all the free and caught fish. ''' 500 | for fish in self.fish: 501 | fish.draw(img, acceleration) 502 | for fish in self.caught_fish: 503 | fish.draw(img, None) 504 | 505 | def _text_overlay(self, img): 506 | ''' Show instructions or how many fish have been caught + stats. ''' 507 | if self._instructions_visible: 508 | cv2.addWeighted(img, 0.3, self._instructions, 0.7, 0, img) 509 | return 510 | 511 | caught_fish = len(self.caught_fish) 512 | texts = [f'Caught {caught_fish}/{self._num_fish}'] 513 | if self._attempts: 514 | texts.append(f'Hit rate: {100 * caught_fish / self._attempts:.2f}%') 515 | if caught_fish: 516 | texts.append(f'Avg size ratio: {self._precision / caught_fish:.3f}') 517 | 518 | for index, text in enumerate(texts): 519 | cv2.putText(img, text, (10, 20*(index+1)), cv2.FONT_HERSHEY_SIMPLEX, 520 | 0.6, (255,255,255), 1) 521 | 522 | def _draw_net(self, img): 523 | ''' Draws the 'catch' net if one is in progress. ''' 524 | if self._end_point: 525 | # make thicker lines for larger net 526 | thickness = 1 + \ 527 | sum(abs(self._end_point[index] - self._start_point[index]) 528 | for index in range(2)) // 90 529 | cv2.rectangle(img, self._start_point, self._end_point, (0,0,100), 530 | thickness) 531 | 532 | if __name__ == '__main__': 533 | import sys, pathlib 534 | 535 | if len(sys.argv) > 1: 536 | name_file = sys.argv[1] 537 | else: 538 | name_file = pathlib.Path(__file__).parent.absolute() / 'names.txt' 539 | 540 | 541 | tank = FishTank((720,1280), name_file=name_file) 542 | window = 'Fish Tank' 543 | with LockedCamera(0, preprocess=tank.preprocess, process=tank, 544 | display=window, play_commands={ 545 | ord('g'):tank.toggle_gradient, 546 | ord('i'):tank.toggle_instructions 547 | }) as cam: 548 | cam.record_stream('fish.mp4', mouse_handler = 549 | MouseCallback(window, tank.mouse_handler)) 550 | -------------------------------------------------------------------------------- /src/pcv/interact.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import cv2 4 | 5 | 6 | waitKey = lambda ms : cv2.waitKey(ms) & 0xFF 7 | 8 | 9 | class DoNothing: 10 | ''' A context manager that does nothing. ''' 11 | def __init__(self): pass 12 | def __enter__(self): return self 13 | def __exit__(self, *args): pass 14 | 15 | 16 | class MouseCallback: 17 | ''' A context manager for temporary mouse callbacks. ''' 18 | def __init__(self, window, handler, param=None, 19 | restore=lambda *args: None, restore_param=None): 20 | ''' Initialise with the window, handler, and restoration command. 21 | 22 | 'window' is the name of the window to set the callback for. 23 | 'handler' is the function for handling the callback, which should take 24 | x, y, flags ('&'ed EVENT_FLAG bits), and an optional param passed 25 | in from the callback handler. 26 | 'param' is any Python object that should get passed to the handler 27 | on each call - useful for tracking state. 28 | 'restore' is the function to restore as the handler on context exit. 29 | 'restore_param' is the handler param to restore on context exit. 30 | 31 | ''' 32 | self.window = window 33 | self.handler = handler 34 | self.param = param 35 | self.restore = restore 36 | self.restore_param = restore_param 37 | 38 | def __enter__(self): 39 | cv2.setMouseCallback(self.window, self.handler, self.param) 40 | return self 41 | 42 | def __exit__(self, *args): 43 | cv2.setMouseCallback(self.window, self.restore, self.restore_param) 44 | -------------------------------------------------------------------------------- /src/pcv/process.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import cv2 4 | 5 | def downsize(img, ratio): 6 | ''' downsize 'img' by 'ratio'. ''' 7 | return cv2.resize(img, 8 | tuple(dim // ratio for dim in reversed(img.shape[:2])), 9 | interpolation = cv2.INTER_AREA) 10 | 11 | def channel_options(img): 12 | ''' Create a composite image of img in all of opencv's colour channels 13 | 14 | |img| -> | blue | green | red | 15 | | hue | saturation | value | 16 | | hue2 | luminosity | saturation2 | 17 | | lightness | green-red | blue-yellow | 18 | | lightness2 | u | v | 19 | 20 | ''' 21 | B,G,R = cv2.split(img) 22 | H,S,V = cv2.split(cv2.cvtColor(img, cv2.COLOR_BGR2HSV)) 23 | H2,L2,S2 = cv2.split(cv2.cvtColor(img, cv2.COLOR_BGR2HLS)) 24 | L,a,b = cv2.split(cv2.cvtColor(img, cv2.COLOR_BGR2LAB)) 25 | L3,u,v = cv2.split(cv2.cvtColor(img, cv2.COLOR_BGR2LUV)) 26 | channels = (((B, 'blue'), (G, 'green'), (R, 'red')), 27 | ((H, 'hue'), (S, 'saturation'), (V, 'value')), 28 | ((H2, 'hue2'), (L2, 'luminosity'), (S2, 'saturation2')), 29 | ((L, 'lightness'), (a, 'green-red'), (b, 'blue-yellow')), 30 | ((L3,'lightness2'), (u, 'u'), (v, 'v'))) 31 | out = [] 32 | for row in channels: 33 | img_row = [] 34 | for img, name in row: 35 | cv2.putText(img, name, (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 36 | 0.6, 255, 1) 37 | img_row.append(img) 38 | out.append(cv2.hconcat(img_row)) 39 | return cv2.vconcat(out) 40 | -------------------------------------------------------------------------------- /src/pcv/screen.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import mss 4 | import platform 5 | import numpy as np 6 | from pcv.vidIO import ContextualVideoCapture, SlowCamera, Camera, LockedCamera 7 | 8 | Mss = mss.mss().__class__ # handle different operating systems 9 | DARWIN = platform.system() == 'Darwin' 10 | 11 | 12 | class ScreenSource(Mss): 13 | ''' A class to provide mss as a VideoSource backend. ''' 14 | def __init__(self, monitor, *args, **kwargs): 15 | ''' 16 | 'monitor' can be an integer index, -1 for all monitors together, a 17 | PIL.grab-style tuple of (left, top, right, bottom), or a dictionary 18 | with keys 'left', 'top', 'width', and 'height'. 19 | NOTE: macOS values should be half of the actual screen resolution. 20 | 21 | ''' 22 | self.__image = None 23 | super().__init__(*args, **kwargs) 24 | self.open(monitor) 25 | 26 | def get(self, property): 27 | return getattr(self, property) 28 | 29 | def set(self, property, value): 30 | return setattr(self, property, value) 31 | 32 | def open(self, monitor, api_preference=None): 33 | # more efficient to use dictionary form 34 | if isinstance(monitor, tuple): 35 | monitor = { 36 | 'left' : monitor[0], 37 | 'top' : monitor[1], 38 | 'width' : monitor[2] - monitor[0], 39 | 'height': monitor[3] - monitor[1] 40 | } 41 | elif isinstance(monitor, int): 42 | monitor = self.monitors[monitor+1] 43 | 44 | scale = 2 if DARWIN else 1 # handle macOS pixel doubling 45 | self.width = monitor['width'] * scale 46 | self.height = monitor['height'] * scale 47 | self._monitor = monitor 48 | self._open = True 49 | 50 | def isOpened(self): 51 | return self._open 52 | 53 | def grab(self): 54 | try: 55 | self.__image = super().grab(self._monitor) 56 | return True 57 | except mss.exception.ScreenShotError: 58 | self.__image = None 59 | return False 60 | 61 | def retrieve(self, image=None, *args): 62 | if self.__image is None: 63 | return False, None 64 | 65 | if image is not None: 66 | image[:] = np.array(self.__image) 67 | else: 68 | image = np.array(self.__image) 69 | 70 | return True, image 71 | 72 | def read(self, image=None): 73 | self.grab() 74 | return self.retrieve(image) 75 | 76 | def release(self): 77 | self.close() 78 | self._open = False 79 | 80 | 81 | class ScreenWrap: 82 | ''' A VideoSource mixin to use ScreenSource instead of OpenCVSource. ''' 83 | def __init__(self, monitor, *args, **kwargs): 84 | super().__init__(monitor, *args, source=ScreenSource, **kwargs) 85 | 86 | def __repr__(self): 87 | monitor = self._monitor 88 | return f'{self.__class__.__name__}({monitor=})' 89 | 90 | 91 | class SlowScreen(ScreenWrap, SlowCamera): pass 92 | class Screen(ScreenWrap, Camera): pass 93 | class LockedScreen(ScreenWrap, LockedCamera): pass 94 | 95 | 96 | if __name__ == '__main__': 97 | import cv2 98 | with LockedScreen(0, process=lambda img: \ 99 | cv2.cvtColor(img, cv2.COLOR_BGRA2BGR)) as screen: 100 | # video-recording requires 3-channel (BGR) or single-channel 101 | # (greyscale, isColor=False) to work 102 | screen.record_stream('screen-record.mp4') 103 | -------------------------------------------------------------------------------- /src/pcv/source.py: -------------------------------------------------------------------------------- 1 | class VideoSource: 2 | ''' A generic video source with swappable back-end. ''' 3 | def __init__(self, source, *args, **kwargs): 4 | self.__source = source(*args, **kwargs) 5 | 6 | def get(self, property): 7 | ''' Return the value of 'property' if it exists, else 0.0. ''' 8 | return self.__source.get(property) 9 | 10 | def set(self, property, value): 11 | ''' Attempts to set 'property' to 'value', returning success. ''' 12 | return self.__source.set(property, value) 13 | 14 | def read(self, image=None): 15 | ''' Returns success, frame of reading the next frame. 16 | 17 | 'image' an optional array to store the output frame in. 18 | 19 | ''' 20 | return self.__source.read(image) 21 | 22 | def __getattr__(self, key): 23 | ''' On failure to find an attribute in this instance, check source. ''' 24 | return getattr(self.__source, key) 25 | -------------------------------------------------------------------------------- /src/pcv/vidIO.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import cv2 4 | import numpy as np 5 | from time import perf_counter, sleep 6 | from queue import Queue 7 | from threading import Thread, Event 8 | from pcv.interact import DoNothing, waitKey 9 | from pcv.source import VideoSource 10 | 11 | 12 | class BlockingVideoWriter(cv2.VideoWriter): 13 | ''' A cv2.VideoWriter with a context manager for releasing. 14 | 15 | Generally suggested to use the non-blocking, threaded VideoWriter class 16 | instead, unless your application requires no wait time on completion 17 | but permits performance reduction throughout to write frames. If that's 18 | the case, try VideoWriter anyway, and come back to this if a notable 19 | backlog occurs (it will tell you). 20 | 21 | ''' 22 | properties = { 23 | 'quality' : cv2.VIDEOWRITER_PROP_QUALITY, 24 | 'framebytes' : cv2.VIDEOWRITER_PROP_FRAMEBYTES, 25 | 'nstripes' : cv2.VIDEOWRITER_PROP_NSTRIPES, 26 | } 27 | 28 | # functioning combinations are often hard to find - these hopefully work 29 | SUGGESTED_CODECS = { 30 | 'avi' : ['H264','X264','XVID','MJPG'], 31 | 'mp4' : ['avc1','mp4v'], 32 | 'mov' : ['avc1','mp4v'], 33 | 'mkv' : ['H264'], 34 | } 35 | 36 | def __init__(self, filename, fourcc, fps, frameSize, isColor=True, 37 | apiPreference=None): 38 | ''' Initialise a BlockingVideoWriter with the given parameters. 39 | 40 | 'filename' The video file to write to. 41 | 'fourcc' the "four character code" representing the writing codec. 42 | Can be a four character string, or an int as returned by 43 | cv2.VideoWriter_fourcc. As functioning combinations of file 44 | extension + codec can be difficult to find, the helper method 45 | VideoWriter.suggested_codec is provided, accepting a filename 46 | (or file extension) and a list of previously tried codecs that 47 | didn't work and should be excluded. Suggested codecs are populated 48 | from VideoWriter.SUGGESTED_CODECS, if you wish to view the 49 | suggested options directly. 50 | 'fps' is the framerate (frames per second) to save as. It is a constant 51 | float, and can only be set on initialisation. To have a video that 52 | plays back faster than the recording stream, set the framerate to 53 | higher than the input framerate. The VideoWriter.from_camera 54 | factory function is provided to create a video-writer directly 55 | from a camera instance, and allows measuring of the input framerate 56 | for accurate output results if desired. 57 | 'frameSize' is the size of the input frames as a tuple of (rows, cols). 58 | 'isColor' is a boolean specifying if the saved images are coloured. 59 | Defaults to True. Set False for greyscale input streams. 60 | 'apiPreference' allows specifying which API backend to use. It can be 61 | used to enforce a specific reader implementation if multiple are 62 | available (e.g. cv2.CAP_FFMPEG or cv2.CAP_GSTREAMER). Generally 63 | this is not required, and if left as None it is ignored. 64 | 65 | ''' 66 | self.filename = filename 67 | self.fps = fps 68 | self.is_color = isColor 69 | self.frame_size = frameSize 70 | self.api_preference = apiPreference 71 | self.set_fourcc(fourcc) 72 | 73 | super().__init__(*self._construct_open_args()) 74 | 75 | def set_fourcc(self, fourcc): 76 | ''' Set fourcc code as an integer or an iterable of 4 chars. ''' 77 | self.fourcc = fourcc # save for checking value 78 | if not isinstance(fourcc, int): 79 | # assume iterable of 4 chars 80 | fourcc = cv2.VideoWriter_fourcc(*fourcc) 81 | self._fourcc = fourcc 82 | 83 | def _construct_open_args(self): 84 | args = [self.filename, self._fourcc, self.fps, self.frame_size, 85 | self.is_color] 86 | if self.api_preference is not None: 87 | args = [args[0], self.api_preference, *args[1:]] 88 | return args 89 | 90 | def __enter__(self): 91 | ''' Re-entrant ''' 92 | if not self.isOpened(): 93 | self.open(*self._construct_open_args()) 94 | return self 95 | 96 | def __exit__(self, *args): 97 | self.release() 98 | 99 | def get(self, property): 100 | ''' Returns 'property' value, or 0 if not supported by the backend. 101 | 102 | 'property' can be a string key for the VideoWriter.properties 103 | dictionary or an integer from cv2.VIDEOWRITER_PROP_* 104 | 105 | self.get(str/int) -> float 106 | 107 | ''' 108 | try: 109 | return super().get(self.properties[property.lower()]) 110 | except AttributeError: 111 | return super().get(property) 112 | 113 | def set(self, property, value): 114 | ''' Attempts to set the specified property value. 115 | Returns True if the property is supported by the backend in use. 116 | 117 | 'property' can be a string key for the VideoWriter.properties 118 | dictionary or an integer from cv2.VIDEOWRITER_PROP_* 119 | 'value' should be a float 120 | 121 | self.set(str/int, float) -> bool 122 | 123 | ''' 124 | try: 125 | return super().set(self.properties[property.lower()], value) 126 | except AttributeError: 127 | return super().set(property, value) 128 | 129 | @classmethod 130 | def suggested_codec(cls, filename, exclude=[]): 131 | extension = filename.split('.')[-1] 132 | try: 133 | return [codec for codec in cls.SUGGESTED_CODECS[extension.lower()] 134 | if codec not in exclude][0] 135 | except IndexError: 136 | raise Exception('No codecs available, try a different extension') 137 | 138 | @classmethod 139 | def from_camera(cls, filename, camera, fourcc=None, isColor=True, 140 | apiPreference=None, fps=-3, frameSize=None, **kwargs): 141 | ''' Returns a VideoWriter based on the properties of the input camera. 142 | 143 | 'filename' is the name of the file to save to. 144 | 'camera' is the SlowCamera instance (or any of its subclasses). 145 | 'fourcc' is the codec four-character code. If left as None is 146 | determined automatically from filename. 147 | 'isColor' specifies if the video stream is colour or greyscale. 148 | 'fps' can be set as a float, 'camera' to ask the camera for the value, 149 | or a negative integer to measure over that number of frames. 150 | If no processing is occurring, 'camera' is suggested, otherwise 151 | it is generally best to measure the frame output. 152 | Defaults to -3, to measure over 3 frames. 153 | 'frameSize' is an integer tuple of (width, height)/(cols, rows). 154 | If left as None, uses `camera.get` to retrieve width and height. 155 | 'kwargs' are any additional keyword arguments for initialisation. 156 | 157 | ''' 158 | if fourcc is None: 159 | fourcc = cls.suggested_codec(filename) 160 | if frameSize is None: 161 | frameSize = tuple(int(camera.get(dim)) 162 | for dim in ('width', 'height')) 163 | 164 | if fps == 'camera': 165 | fps = camera.get('fps') 166 | elif fps < 0: 167 | fps = camera.measure_framerate(-fps) 168 | 169 | return cls(filename, fourcc, fps, frameSize, isColor, apiPreference, 170 | **kwargs) 171 | 172 | def __repr__(self): 173 | return (f'{self.__class__.__name__}(filename={repr(self.filename)}, ' 174 | f'fourcc={repr(self.fourcc)}, fps={self.fps}, ' 175 | f'frameSize={self.frame_size}, isColor={self.is_colour}, ' 176 | f'apiPreference={self.api_preference})') 177 | 178 | 179 | class VideoWriter(BlockingVideoWriter): 180 | ''' A non-blocking thread-based video writer, using a queue. ''' 181 | def __init__(self, *args, maxsize=0, verbose_exit=True, **kwargs): 182 | ''' Initialise the video writer. 183 | 184 | 'maxsize' is the maximum allowed frame buildup before adding frames 185 | blocks execution. Defaults to 0 (no maximum). Set a meaningful 186 | number if you have fast processing, limited memory, and can't 187 | afford the time required to wait at the end once you've finished 188 | recording. Setting a number for this is helpful in early testing 189 | to get notified of cases where writing to disk is a bottleneck 190 | (you may get processing freezes from time to time as a result). 191 | Consistently slow write times may indicate a need for a more 192 | efficient file format, memory type, or just lower resolution in 193 | time or space (ie fewer fps or smaller images). 194 | 'verbose_exit' is a boolean indicating if the writer should notify 195 | you on exit if a backlog wait will be required, and if so once it 196 | completes and how long it took. Defaults to True. 197 | 198 | *args and **kwargs are the same as those for BlockingVideoWriter. 199 | 200 | ''' 201 | super().__init__(*args, **kwargs) 202 | self._initialise_writer(maxsize) 203 | self._verbose_exit = verbose_exit 204 | 205 | def _initialise_writer(self, maxsize): 206 | ''' Start the Thread for writing images from the queue. ''' 207 | self.max_queue_size = maxsize 208 | self._write_queue = Queue(maxsize=maxsize) 209 | self._image_writer = Thread(name='writer', target=self._writer, 210 | daemon=True) 211 | self._image_writer.start() 212 | 213 | def _writer(self): 214 | ''' Write frames forever, until ''' 215 | while "not finished": 216 | # retrieve an image, wait indefinitely if necessary 217 | img = self._write_queue.get() 218 | # write the image to file ('where' is specified outside) 219 | super().write(img) 220 | # inform the queue that a frame has been written 221 | self._write_queue.task_done() 222 | 223 | def write(self, img): 224 | ''' Send 'img' to the write queue. ''' 225 | self._write_queue.put(img) 226 | 227 | def __exit__(self, *args): 228 | ''' Wait for writing to complete, and release writer. ''' 229 | # assume not waiting 230 | waited = False 231 | 232 | # check if waiting required 233 | if self._verbose_exit and not self._write_queue.empty(): 234 | print(f'Writing {self._write_queue.qsize()} remaining frames.') 235 | print('Force quitting may result in a corrupted video file.') 236 | waited = perf_counter() 237 | 238 | # finish writing all frames 239 | self._write_queue.join() 240 | 241 | # cleanup as normal 242 | super().__exit__(*args) 243 | 244 | # if wait occurred, inform of completion 245 | if waited and self._verbose_exit: 246 | print(f'Writing complete in {perf_counter()-waited:.3f}s.') 247 | 248 | 249 | class GuaranteedVideoWriter(VideoWriter): 250 | ''' A VideoWriter with guaranteed output FPS. 251 | 252 | Repeats frames when input too slow, and skips frames when input too fast. 253 | 254 | ''' 255 | def _initialise_writer(self, maxsize): 256 | ''' Start the write-queue putter and getter threads. ''' 257 | super()._initialise_writer(maxsize) 258 | self._period = 1 / self.fps 259 | self.latest = None 260 | self._finished = Event() 261 | self._looper = Thread(name='looper', target=self._write_loop) 262 | self._looper.start() 263 | 264 | def _write_loop(self): 265 | ''' Write the latest frame to the queue, at self.fps. 266 | 267 | Repeats frames when input too slow, and skips frames when input too fast. 268 | 269 | ''' 270 | # wait until first image set, or early finish 271 | while self.latest is None and not self._finished.is_set(): 272 | sleep(self._period / 2) 273 | prev = perf_counter() 274 | self._error = 0 275 | delay = self._period - 5e-3 276 | 277 | # write frames at specified rate until told to stop 278 | while not self._finished.is_set(): 279 | super().write(self.latest) 280 | new = perf_counter() 281 | self._error += self._period - (new - prev) 282 | delay -= self._error 283 | delay = max(delay, 0) # can't go back in time 284 | sleep(delay) 285 | prev = new 286 | 287 | def write(self, img): 288 | ''' Set the latest image. ''' 289 | self.latest = img 290 | 291 | def __exit__(self, *args): 292 | self._finished.set() 293 | self._looper.join() 294 | if self._verbose_exit: 295 | print(f'Net timing error = {self._error * 1e3:.3f}ms') 296 | super().__exit__(*args) 297 | 298 | 299 | class OutOfFrames(StopIteration): 300 | def __init__(msg='Out of video frames', *args, **kwargs): 301 | super().__init__(msg, *args, **kwargs) 302 | 303 | 304 | class UserQuit(StopIteration): 305 | def __init__(msg='User quit manually', *args, **kwargs): 306 | super().__init__(msg, *args, **kwargs) 307 | 308 | 309 | class OpenCVSource(cv2.VideoCapture): 310 | ''' A class to provide opencv's VideoCapture as a VideoSource backend. ''' 311 | # more properties + descriptions can be found in the docs: 312 | # https://docs.opencv.org/3.4/d4/d15/group__videoio__flags__base.html#gaeb8dd9c89c10a5c63c139bf7c4f5704d 313 | properties = { 314 | 'fps' : cv2.CAP_PROP_FPS, 315 | 'mode' : cv2.CAP_PROP_MODE, 316 | 'width' : cv2.CAP_PROP_FRAME_WIDTH, 317 | 'height' : cv2.CAP_PROP_FRAME_HEIGHT, 318 | 'backend' : cv2.CAP_PROP_BACKEND, 319 | } 320 | 321 | def get(self, property): 322 | try: 323 | return super().get(self.properties.get(property, property)) 324 | except TypeError: # property must be an unknown string 325 | return super().get(getattr(cv2, 'CAP_PROP_' + property.upper())) 326 | 327 | def set(self, property, value): 328 | try: 329 | return super().set(self.properties.get(property, property), value) 330 | except TypeError: # 'property' must be an unknown string 331 | return super().set(getattr(cv2, 'CAP_PROP_' + property.upper()), value) 332 | 333 | 334 | class ContextualVideoCapture(VideoSource): 335 | ''' A video-capturing class with a context manager for releasing. ''' 336 | 337 | def __init__(self, id, *args, display='frame', delay=None, quit=ord('q'), 338 | play_pause=ord(' '), pause_effects={}, play_commands={}, 339 | destroy=-1, source=OpenCVSource, **kwargs): 340 | ''' A pausable, quitable, iterable video-capture object 341 | with context management. 342 | 343 | 'id' is the id that gets passed to the underlying VideoCapture object. 344 | it can be an integer to select a connected camera, or a filename 345 | to open a video. 346 | 'display' is used as the default window name when streaming. Defaults 347 | to 'frame'. 348 | 'delay' is the integer millisecond delay applied between each iteration 349 | to enable windows to update. If set to None, this is skipped and 350 | the user must manually call waitKey to update windows. 351 | Default is None, which allows headless operation without 352 | unnecessary waiting. 353 | 'quit' is an integer ordinal corresponding to a key which can be used 354 | to stop the iteration loop. Only applies if delay is not None. 355 | Default is ord('q'), so press the 'q' key to quit when iterating. 356 | 'play_pause' is an integer ordinal corresponding to a key which can be 357 | used to pause and resume the iteration loop. Only applies if delay 358 | is not None. Default is ord(' '), so press space-bar to pause/ 359 | resume when iterating. 360 | 'pause_effects' is a dictionary of key ordinals and corresponding 361 | handler functions. The handler will be passed self as its only 362 | argument, which gives it access to the 'get' and 'set' methods, 363 | as well as the 'status' and 'image' properties from the last 'read' 364 | call. This can be useful for logging, selecting images for 365 | labelling, or temporary manual control of the event/read loop. 366 | Note that this is only used while paused, and does not get passed 367 | quit or play_pause key events. 368 | 'play_commands' is the same as 'pause_effects' but operates instead 369 | while playback/streaming is occurring. For live processing, 370 | this can be used to change playback modes, or more generally for 371 | similar scenarios as 'pause_effects'. 372 | 'destroy' destroys any specified windows on context exit. Can be 'all' 373 | to destroy all active opencv windows, a string of a specific window 374 | name, or a list of window names to close. If left as -1, destroys 375 | the window specified in 'display'. 376 | 'source' is a class which acts like a video source, by implementing at 377 | minimum the methods 'get', 'set', 'read', 'grab', 'retrieve', 378 | 'open', 'isOpened', and 'release'. 379 | 380 | ''' 381 | super().__init__(source, id, *args, **kwargs) 382 | self._id = id 383 | self.display = display 384 | self._delay = delay 385 | self._quit = quit 386 | self._play_pause = play_pause 387 | self._pause_effects = pause_effects 388 | self._play_commands = play_commands 389 | self._destroy = destroy 390 | 391 | self._api_preference = kwargs.get('apiPreference', None) 392 | 393 | def __enter__(self, force=False): 394 | ''' Enter a re-entrant context for this camera. ''' 395 | if force or not self.isOpened(): 396 | if self._api_preference: 397 | self.open(self._id, self._api_preference) 398 | else: 399 | self.open(self._id) 400 | 401 | return self 402 | 403 | def __exit__(self, exc_type, exc_value, exc_traceback): 404 | ''' Clean up on context exit. 405 | 406 | Releases the internal VideoCapture object, and destroys any windows 407 | specified at initialisation. 408 | 409 | ''' 410 | # release VideoCapture object 411 | self.release() 412 | 413 | # clean up window(s) if specified on initialisation 414 | destroy = self._destroy 415 | try: 416 | if destroy == -1: 417 | cv2.destroyWindow(self.display) 418 | elif destroy == 'all': 419 | cv2.destroyAllWindows() 420 | elif isinstance(destroy, str): 421 | # a single window name 422 | cv2.destroyWindow(destroy) 423 | elif destroy is not None: 424 | # assume an iterable of multiple windows 425 | for window in destroy: cv2.destroyWindow(window) 426 | else: 427 | return # destroy is None 428 | except cv2.error as e: 429 | print('Failed to destroy window(s)', e) 430 | 431 | waitKey(3) # allow the GUI manager to update 432 | 433 | def __iter__(self): 434 | return self 435 | 436 | def __next__(self): 437 | # check if doing automatic waits 438 | if self._delay is not None: 439 | key = waitKey(self._delay) 440 | 441 | if key == self._quit: 442 | raise UserQuit 443 | elif key == self._play_pause: 444 | self._handle_pause() 445 | else: 446 | # pass self to a triggered user-defined key handler, or nothing 447 | self._play_commands.get(key, lambda cap: None)(self) 448 | 449 | # wait completed, get next frame if possible 450 | if self.isOpened(): 451 | return self.read() 452 | raise OutOfFrames 453 | 454 | def _handle_pause(self): 455 | ''' Handle event loop and key-presses while paused. ''' 456 | while "paused": 457 | key = waitKey(1) 458 | if key == self._quit: 459 | raise UserQuit 460 | if key == self._play_pause: 461 | break 462 | # pass self to a triggered user-defined key handler, or do nothing 463 | self._pause_effects.get(key, lambda cap: None)(self) 464 | 465 | def stream(self, mouse_handler=DoNothing()): 466 | ''' Capture and display stream on window specified at initialisation. 467 | 468 | 'mouse_handler' is an optional MouseCallback instance determining 469 | the effects of mouse clicks and moves during the stream. Defaults 470 | to DoNothing. 471 | 472 | ''' 473 | # ensure mouse_handler has something to bind to 474 | cv2.namedWindow(self.display) 475 | # engage mouse_handler and start the stream 476 | with mouse_handler: 477 | for read_success, frame in self: 478 | if read_success: 479 | cv2.imshow(self.display, frame) 480 | else: 481 | break # camera disconnected 482 | 483 | def headless_stream(self): 484 | ''' Capture and process stream without display. ''' 485 | for read_success, frame in self: 486 | if not read_success: break # camera disconnected 487 | 488 | def record_stream(self, filename, show=True, mouse_handler=DoNothing(), 489 | writer=VideoWriter, **kwargs): 490 | ''' Capture and record stream, with optional display. 491 | 492 | 'filename' is the file to save to. 493 | 'show' is a boolean specifying if the result is displayed (on the 494 | window specified at initialisation). 495 | 'mouse_handler' is an optional MouseCallback instance determining 496 | the effects of mouse clicks and moves during the stream. It is only 497 | useful if 'show' is set to True. Defaults to DoNothing. 498 | 'writer' is a subclass of VideoWriter. Defaults to VideoWriter. 499 | Set to GuaranteedVideoWriter to allow repeated and skipped frames 500 | to better ensure a consistent output framerate. 501 | 502 | **kwargs are passed to the 'writer's from_camera method (e.g. can be used 503 | to indicate the 'frameSize' or a greyscale output (isColor=False)). 504 | 505 | ''' 506 | if show: 507 | # ensure mouse_handler has something to bind to 508 | cv2.namedWindow(self.display) 509 | # create writer, engage mouse_handler, and start the stream 510 | with writer.from_camera(filename, self, **kwargs) as writer, \ 511 | mouse_handler: 512 | for read_success, frame in self: 513 | if read_success: 514 | if show: 515 | cv2.imshow(self.display, frame) 516 | writer.write(frame) 517 | else: 518 | break # camera disconnected 519 | 520 | def read(self, image=None): 521 | if image is not None: 522 | status, image = super().read(image) 523 | else: 524 | status, image = super().read() 525 | self.status, self.image = status, image 526 | return status, image 527 | 528 | 529 | class SlowCamera(ContextualVideoCapture): 530 | ''' A basic, slow camera class for processing frames relatively far apart. 531 | 532 | Use 'Camera' instead unless you need to reduce power/CPU usage and the time 533 | to read an image is insignificant in your processing pipeline. 534 | 535 | ''' 536 | def __init__(self, camera_id=0, *args, delay=1, **kwargs): 537 | ''' Create a camera capture instance with the given id. 538 | 539 | Arguments are the same as ContextualVideoCapture, but 'id' is replaced 540 | with 'camera_id', and 'delay' is set to 1 by default instead of 541 | None. 542 | 543 | ''' 544 | super().__init__(camera_id, *args, delay=delay, **kwargs) 545 | 546 | def measure_framerate(self, frames): 547 | ''' Measure framerate for specified number of frames. ''' 548 | count = 0 549 | for read_success, frame in self: 550 | if self.display: 551 | cv2.imshow(self.display, frame) 552 | count += 1 553 | if count == 1: 554 | start = perf_counter() # avoid timing opening the window 555 | if count > frames: 556 | # desired frames reached, set fps as average framerate 557 | return count / (perf_counter() - start) 558 | 559 | def __repr__(self): 560 | return f"{self.__class__.__name__}(camera_id={self._id!r})" 561 | 562 | 563 | class Camera(SlowCamera): 564 | ''' A camera for always capturing the latest frame, fast. 565 | 566 | Use this instead of 'SlowCamera', unless you need to reduce power/CPU 567 | usage, and the time to read an image is insignificant in your processing 568 | pipeline. 569 | 570 | ''' 571 | def __init__(self, *args, **kwargs): 572 | super().__init__(*args, **kwargs) 573 | self._initialise_grabber() 574 | 575 | def _initialise_grabber(self): 576 | ''' Start the Thread for grabbing images. ''' 577 | self._finished = Event() 578 | self._image_grabber = Thread(name='grabber', target=self._grabber, 579 | daemon=True) # auto-kill when finished 580 | self._image_grabber.start() 581 | self._wait_for_grabber_start() 582 | 583 | def _grabber(self): 584 | ''' Grab images as fast as possible - only latest gets processed. ''' 585 | while not self._finished.is_set(): 586 | self.grab() 587 | 588 | def _wait_for_grabber_start(self): 589 | ''' Waits for a successful retrieve. Raises Exception after 50 attempts. ''' 590 | for check in range(50): 591 | if self.retrieve()[0]: break 592 | sleep(0.1) 593 | else: 594 | raise Exception(f'Failed to start {self.__class__.__name__}') 595 | 596 | def __exit__(self, *args): 597 | self._finished.set() 598 | self._image_grabber.join() 599 | super().__exit__(*args) 600 | 601 | def read(self, image=None): 602 | ''' Read and return the latest available image. ''' 603 | if image is not None: 604 | status, image = self.retrieve(image) 605 | else: 606 | status, image = self.retrieve() 607 | self.status, self.image = status, image 608 | return status, image 609 | 610 | 611 | class LockedCamera(Camera): 612 | ''' A camera for semi-synchronously capturing a single image at a time. 613 | 614 | Like 'Camera' but uses less power+CPU by only capturing images on request. 615 | Allows specifying when each image should start being captured, then doing 616 | some processing while the image is being grabbed and decoded (and 617 | optionally pre-processed), before using it. 618 | 619 | Images may be less recent than achieved with Camera, depending on when the 620 | user starts the capture process within their processing pipeline, but can 621 | also be more recent if started near the end of the pipeline (at the risk of 622 | having to wait for the capturing to complete). 623 | 624 | ''' 625 | def __init__(self, *args, preprocess=lambda img:img, 626 | process=lambda img:img, **kwargs): 627 | ''' Create a camera capture instance with the given id. 628 | 629 | 'preprocess' is an optional function which takes an image and returns 630 | a modified image, which gets applied to each frame on read. 631 | Defaults to no preprocessing. 632 | 'process' is an optional function which takes an image and returns 633 | a modified image, which gets applied to each preprocessed frame 634 | after the next frame has been requested. Defaults to no processing. 635 | 636 | *args and **kwargs are the same as for Camera. 637 | 638 | ''' 639 | super().__init__(*args, **kwargs) 640 | self._preprocess = preprocess 641 | self._process = process 642 | self._get_latest_image() # start getting the first image 643 | 644 | def _initialise_grabber(self): 645 | ''' Create locks and start the grabber thread. ''' 646 | self._image_desired = Event() 647 | self._image_ready = Event() 648 | super()._initialise_grabber() 649 | 650 | def _grabber(self): 651 | ''' Grab and preprocess images on demand, ready for later usage ''' 652 | while not self._finished.is_set(): 653 | self._wait_until_needed() 654 | # read the latest frame 655 | read_success, frame = super(ContextualVideoCapture, self).read() 656 | 657 | # apply any desired pre-processing and store for main thread 658 | self._preprocessed = self._preprocess(frame) if read_success \ 659 | else None 660 | # inform that image is ready for access/main processing 661 | self._inform_image_ready() 662 | 663 | def _wait_for_grabber_start(self): 664 | ''' Not used - done automatically with Events. ''' 665 | pass 666 | 667 | def _wait_until_needed(self): 668 | ''' Wait for main to request the next image. ''' 669 | self._image_desired.wait() 670 | self._image_desired.clear() 671 | 672 | def _inform_image_ready(self): 673 | ''' Inform main that next image is available. ''' 674 | self._image_ready.set() 675 | 676 | def _get_latest_image(self): 677 | ''' Ask camera handler for next image. ''' 678 | self._image_desired.set() 679 | 680 | def _wait_for_camera_image(self): 681 | ''' Wait until next image is available. ''' 682 | self._image_ready.wait() 683 | self._image_ready.clear() 684 | 685 | def __exit__(self, *args): 686 | self._finished.set() 687 | self._image_desired.set() # allow thread to reach finished check 688 | super().__exit__(*args) 689 | 690 | def read(self, image=None): 691 | ''' For optimal usage, tune _process to take the same amount of time 692 | as getting the next frame. 693 | ''' 694 | self._wait_for_camera_image() 695 | preprocessed = self._preprocessed 696 | self._get_latest_image() 697 | if preprocessed is None: 698 | self.status, self.image = False, None 699 | else: 700 | self.image = self._process(preprocessed) 701 | if image is not None: 702 | image = self.image 703 | self.status = True 704 | return self.status, self.image 705 | 706 | 707 | class VideoReader(LockedCamera): 708 | ''' A class for reading video files. ''' 709 | properties = { 710 | **OpenCVSource.properties, 711 | 'frame' : cv2.CAP_PROP_POS_FRAMES, 712 | 'codec' : cv2.CAP_PROP_FOURCC, 713 | 'timestamp' : cv2.CAP_PROP_POS_MSEC, 714 | 'num_frames' : cv2.CAP_PROP_FRAME_COUNT, 715 | 'proportion' : cv2.CAP_PROP_POS_AVI_RATIO, 716 | } 717 | 718 | FASTER, SLOWER, REWIND, FORWARD, RESET, RESTART = \ 719 | (ord(key) for key in 'wsadrb') 720 | FORWARD_DIRECTION, REVERSE_DIRECTION = 1, -1 721 | MIN_DELAY = 1 # integer milliseconds 722 | 723 | def __init__(self, filename, *args, start=None, end=None, auto_delay=True, 724 | fps=None, skip_frames=None, verbose=True, display='video', 725 | **kwargs): 726 | ''' Initialise a video reader from the given file. 727 | 728 | For default key-bindings see 'auto_delay' details. 729 | 730 | 'filename' is the string path of a video file. Depending on the file 731 | format some features may not be available. 732 | 'start' and 'end' denote the respective times of playback, according 733 | to the specified fps. They can be integers of milliseconds, or 734 | strings of 'hours:minutes:seconds' (larger amounts can be left off 735 | if 0, e.g. '5:10.35' for no hours). If left as None, the video 736 | starts and ends at the first and last frames respectively. 737 | It is expected that 'start' < 'end', or playback ends immediately. 738 | 'auto_delay' is a boolean specifying if the delay between frames should 739 | be automatically adjusted during playback to match the specified 740 | fps. Set to False if operating headless (not viewing the video), or 741 | if manual control is desired while iterating over the video. 742 | If set to False, sets 'destroy' to None if not otherwise set. 743 | If True enables playback control with 'w' increasing playback 744 | speed, 's' slowing it down, 'a' rewinding (only possible if 745 | 'skip_frames' is True), and 'd' returning to forwards playback. 746 | The 'r' key can be pressed to reset to 1x speed and forwards 747 | direction playback. 'a' and 'd' can be used while paused to step 748 | back and forwards, regardless of skip_frames. 'b' can be used while 749 | playing or paused to jump the video back to its starting point. 750 | These defaults can be overridden using the 'play_commands' and 751 | 'pause_effects' keyword arguments, supplying a dictionary of key 752 | ordinals that sets the desired behaviour. Note that the defaults 753 | are set internally, so to turn them off the dictionary must be 754 | used, with e.g. play_commands={ord('a'):lambda vid:None} to disable 755 | rewinding. 756 | 'fps' is a float specifying the desired frames per second for playback. 757 | If left as None the fps is read from file, or if that fails is set 758 | to 25 by default. Value is ignored if 'auto_delay' is False. 759 | 'skip_frames' allows frames to be manually set, as required by reverse 760 | or high speed playback. If left as None this is disallowed. If 761 | 'auto_delay' is True, any integer value can be set (suggested 0), 762 | and the number of frames to skip at each iteration is determined 763 | as part of the delay tuning. If 'auto_delay' is False, an integer 764 | can be used as a consistent number of frames to skip at each 765 | iteration (e.g. only read every 10th frame). Note that enabling 766 | frame skipping can make playback jerky on devices and/or file 767 | formats with slow video frame setting times, and inconsistent 768 | skipping amounts with 'auto_delay' may cause issues with 769 | time-dependent processing. 770 | 'verbose' is a boolean determining if status updates (e.g. initial fps, 771 | and playback speed and direction changes) are printed. Defaults to 772 | True. 773 | 774 | *args and **kwargs get passed up the inheritance chain, with notable 775 | keywords including the 'preprocess' and 'process' functions which 776 | take an image and return a processed result (see LockedCamera), 777 | the 'quit' and 'play_pause' key ordinals which are checked if 778 | 'auto_delay' is True, and the 'play_commands' and 'pause_effects' 779 | dictionaries mapping key ordinals to desired functionality while 780 | playing and paused (see ContextualVideoCapture documentation for 781 | details). 782 | 783 | ''' 784 | super().__init__(filename, *args, display=display, **kwargs) 785 | self.filename = filename 786 | self._fps = fps or self.fps or 25 # user-specified or auto-retrieved 787 | self._period = 1e3 / self._fps 788 | self._verbose = verbose 789 | self.status = True 790 | self._initialise_delay(auto_delay) 791 | self._initialise_playback(start, end, skip_frames) 792 | 793 | def _initialise_delay(self, auto_delay): 794 | ''' Determines the delay automatically, or leaves as None. ''' 795 | if auto_delay: 796 | if self._fps == 0 or self._fps >= 1e3: 797 | self.verbose_print('failed to determine fps, setting to 25') 798 | self._period = 1e3 / 25 799 | # set a bit low to allow image read times 800 | self._delay = self._period - 5 801 | else: 802 | self._delay = int(self._period) 803 | self.verbose_print('delay set automatically to', 804 | f'{self._delay}ms from fps={self._fps}') 805 | else: 806 | self._delay = None 807 | if self._destroy == -1: 808 | self._destroy = None 809 | 810 | def _initialise_playback(self, start, end, skip_frames): 811 | ''' Set up playback settings as specified. ''' 812 | self._wait_for_camera_image() # don't set frame while grabber running 813 | 814 | self._set_start(start) 815 | self._set_end(end) 816 | 817 | self._skip_frames = skip_frames 818 | self._direction = self.FORWARD_DIRECTION 819 | self._speed = 1 820 | self._adjusted_period = self._period 821 | self._calculate_frames() 822 | 823 | self._play_commands = { 824 | self.FASTER : self._speed_up, 825 | self.SLOWER : self._slow_down, 826 | self.REWIND : self._go_back, 827 | self.FORWARD : self._go_forward, 828 | self.RESET : self._reset, 829 | self.RESTART : self.restart, 830 | **self._play_commands 831 | } 832 | 833 | # add step back and forward functionality if keys not already used 834 | self._pause_effects = { 835 | self.REWIND : self.step_back, 836 | self.FORWARD : self.step_forward, 837 | self.RESTART : self.restart, 838 | **self._pause_effects 839 | } 840 | 841 | # ensure time between frames is ignored while paused 842 | class LogDict(dict): 843 | def get(this, *args, **kwargs): 844 | self.reset_delay() 845 | return dict.get(this, *args, **kwargs) 846 | 847 | self._pause_effects = LogDict(self._pause_effects) 848 | 849 | self._get_latest_image() # re-initialise as ready 850 | 851 | def get(self, property): 852 | return super().get(self.properties.get(property, property)) 853 | 854 | def set(self, property, value): 855 | return super().set(self.properties.get(property, property), value) 856 | 857 | def _set_start(self, start): 858 | ''' Set the start of the video to user specification, if possible. ''' 859 | self._frame = 0 860 | if start is not None: 861 | if self.set_timestamp(start): 862 | self.verbose_print(f'starting at {start}') 863 | else: 864 | self.verbose_print('start specification failed, ' 865 | 'starting at 0:00') 866 | self._start = self._frame 867 | 868 | def _set_end(self, end): 869 | ''' Set playback to end where specified by user. ''' 870 | if end is not None: 871 | if isinstance(end, str): 872 | self._end = self.timestamp_to_ms(end) 873 | else: 874 | self._end = end 875 | self._end /= self._period # convert to number of frames 876 | else: 877 | self._end = self.get('num_frames') or np.inf 878 | 879 | def verbose_print(self, *args, **kwargs): 880 | if self._verbose: 881 | print(*args, **kwargs) 882 | 883 | # NOTE: key callbacks set as static methods for clarity/ease of reference 884 | # VideoReader to be modified gets passed in (so that external functions 885 | # can be used), so also having a reference to self would be confusing. 886 | 887 | @staticmethod 888 | def _speed_up(vid): 889 | ''' Increase the speed by 10% of the initial value. ''' 890 | vid._speed += 0.1 891 | vid._register_speed_change() 892 | 893 | @staticmethod 894 | def _slow_down(vid): 895 | ''' Reduce the speed by 10% of the initial value. ''' 896 | vid._speed -= 0.1 897 | vid._register_speed_change() 898 | 899 | def _register_speed_change(self): 900 | ''' Update internals and print new speed. ''' 901 | self._calculate_period() 902 | self.verbose_print(f'speed set to {self._speed:.1f}x starting fps') 903 | 904 | def _calculate_period(self): 905 | ''' Determine the adjusted period given the speed. ''' 906 | self._adjusted_period = self._period / self._speed 907 | self._calculate_timestep() 908 | 909 | def _calculate_timestep(self): 910 | ''' Determine the desired timestep of each iteration. ''' 911 | self._timestep = self._adjusted_period * self._frames 912 | 913 | def _calculate_frames(self): 914 | ''' Determine the number of frames to increment each iteration. ''' 915 | self._frames = (1 + self._skip_frames 916 | if self._skip_frames is not None 917 | else 1) 918 | self._calculate_timestep() 919 | 920 | def reset_delay(self): 921 | ''' Resets the delay between frames. 922 | 923 | Use to avoid fast playback/frame skipping after pauses. 924 | 925 | ''' 926 | self._prev = perf_counter() - (self._period - self.MIN_DELAY) / 1e3 927 | 928 | @staticmethod 929 | def _go_back(vid): 930 | ''' Set playback to backwards. ''' 931 | if vid._skip_frames is not None: 932 | vid._direction = vid.REVERSE_DIRECTION 933 | vid.verbose_print('Rewinding') 934 | else: 935 | vid.verbose_print('Cannot go backwards without skip_frames=True') 936 | 937 | @staticmethod 938 | def _go_forward(vid): 939 | ''' Set playback to go forwards. ''' 940 | vid._direction = vid.FORWARD_DIRECTION 941 | vid.verbose_print('Going forwards') 942 | 943 | @staticmethod 944 | def _reset(vid): 945 | ''' Restore playback to 1x speed and forwards. ''' 946 | vid._speed = 1 947 | vid._direction = vid.FORWARD_DIRECTION 948 | vid._calculate_period() 949 | vid.verbose_print('Going forwards with speed set to 1x starting fps ' 950 | f'({vid._fps:.2f})') 951 | 952 | @staticmethod 953 | def step_back(vid): 954 | ''' Take a step backwards. ''' 955 | # store existing state 956 | old_state = (vid._skip_frames, vid._direction, vid._verbose) 957 | 958 | # enable back-stepping if not currently permitted 959 | vid._skip_frames = 0 960 | # make sure no unnecessary prints trigger from playback keys 961 | vid._verbose = False 962 | 963 | # go back a step 964 | vid._direction = vid.REVERSE_DIRECTION 965 | next(vid) 966 | 967 | # restore state 968 | vid._skip_frames, vid._direction, vid._verbose = old_state 969 | 970 | @staticmethod 971 | def step_forward(vid): 972 | ''' Take a step forwards. ''' 973 | # store existing state 974 | old_state = (vid._direction, vid._verbose) 975 | 976 | # make sure no unnecessary prints trigger from playback keys 977 | vid._verbose = False 978 | 979 | # go forwards a step 980 | vid._direction = vid.FORWARD_DIRECTION 981 | next(vid) 982 | 983 | # restore state 984 | vid._direction, vid._verbose = old_state 985 | 986 | @staticmethod 987 | def restart(vid): 988 | ''' Attempts to continue playback from the start of the video. 989 | 990 | Respects user-defined start-point from initialisation. 991 | 992 | ''' 993 | vid.set_frame(vid._start) 994 | 995 | @property 996 | def fps(self): 997 | ''' The constant FPS assumed of the video file. ''' 998 | return self.get('fps') 999 | 1000 | @property 1001 | def frame(self): 1002 | ''' Retrieve the current video frame. ''' 1003 | self._frame = int(self.get('frame')) 1004 | return self._frame 1005 | 1006 | def set_frame(self, frame): 1007 | ''' Attempts to set the frame number, returns success. 1008 | 1009 | 'frame' is an integer >= 0. Setting past the last frame 1010 | either has no effect or ends the playback. 1011 | 1012 | self.set_frame(int) -> bool 1013 | 1014 | ''' 1015 | if self.set('frame', frame): 1016 | self._frame = frame 1017 | return True 1018 | return False 1019 | 1020 | @property 1021 | def timestamp(self): 1022 | ''' Returns the video timestamp if possible, else 0.0. 1023 | 1024 | Returns a human-readable time string, as hours:minutes:seconds, 1025 | or minutes:seconds if there are no hours. 1026 | For the numerical ms value use self.get('timestamp') instead. 1027 | 1028 | self.timestamp -> str 1029 | 1030 | ''' 1031 | # cv2.VideoCapture returns ms timestamp -> convert to meaningful time 1032 | seconds = self.get('timestamp') / 1000 1033 | minutes, seconds = divmod(seconds, 60) 1034 | hours, minutes = divmod(round(minutes), 60) 1035 | if hours: 1036 | return f'{hours:02d}:{minutes:02d}:{seconds:06.3f}' 1037 | return f'{minutes:02d}:{seconds:06.3f}' 1038 | 1039 | @property 1040 | def iso_timestamp(self): 1041 | ''' Returns the video timestamp if possible, else 0.0. 1042 | 1043 | Timestamp is in ISO 8601 duration format: PThh:mm:ss.sss 1044 | For a human-readable timestamp use self.timestamp. 1045 | For a numerical ms value use self.get('timestamp') instead. 1046 | 1047 | self.iso_timestamp -> str 1048 | 1049 | ''' 1050 | timestamp = self.timestamp 1051 | # check if hours not specified 1052 | if len(timestamp) == 9: 1053 | timestamp = '00:' + timestamp 1054 | return f'PT{timestamp}' 1055 | 1056 | def set_timestamp(self, timestamp): 1057 | ''' Attempts to set the timestamp as specified, returns success. 1058 | 1059 | 'timestamp' can be a float/integer of milliseconds, or a string 1060 | of 'hours:minutes:seconds', 'minutes:seconds', or 'seconds', 1061 | where all values can be integers or floats. 1062 | 1063 | self.set_timestamp(str/float/int) -> bool 1064 | 1065 | ''' 1066 | ms = self.timestamp_to_ms(timestamp) if isinstance(timestamp, str) \ 1067 | else timestamp 1068 | fps = self._fps 1069 | if fps == 0: 1070 | # fps couldn't be determined - set ms directly and hope 1071 | return self.set('timestamp', ms) 1072 | return self.set_frame(int(ms * fps / 1e3)) 1073 | 1074 | @staticmethod 1075 | def timestamp_to_ms(timestamp): 1076 | ''' Converts a string timestamp of hours:minutes:seconds to ms.''' 1077 | return 1000 * sum(60 ** index * float(period) for index, period \ 1078 | in enumerate(reversed(timestamp.split(':')))) 1079 | 1080 | def __iter__(self): 1081 | if self._delay is not None: 1082 | self._prev = perf_counter() 1083 | self._error = 0 1084 | self._delay = 1 1085 | return self 1086 | 1087 | def __next__(self): 1088 | if self._delay is not None: 1089 | # auto-adjust to get closer to desired fps 1090 | now = perf_counter() 1091 | diff = 1e3 * (now - self._prev) # s to ms 1092 | self._error += diff - self._timestep 1093 | 1094 | self._update_playback_settings() 1095 | 1096 | self._prev = now 1097 | 1098 | self._update_frame_tracking() 1099 | read_success, frame = super().__next__() 1100 | if not read_success: 1101 | raise OutOfFrames 1102 | return read_success, frame 1103 | 1104 | def _update_playback_settings(self): 1105 | ''' Adjusts delay/frame skipping if error is sufficiently large. ''' 1106 | error_magnitude = abs(self._error) 1107 | if error_magnitude > self.MIN_DELAY: 1108 | # determine distribution of change 1109 | if self._skip_frames is not None: 1110 | # can only skip full frames, rest left to delay 1111 | skip_frames_change, delay_change = \ 1112 | divmod(error_magnitude, self._adjusted_period) 1113 | else: 1114 | delay_change = error_magnitude 1115 | # can only delay in MIN_DELAY increments, remainder is error 1116 | delay_change, new_error_mag = \ 1117 | divmod(delay_change, self.MIN_DELAY) 1118 | 1119 | # determine if going too slowly (+) or too fast (-) 1120 | sign = np.sign(self._error) 1121 | # implement delay (and skip frames) change 1122 | # reducing delay increases speed 1123 | self._delay -= int(sign * delay_change) 1124 | if self._skip_frames is not None: 1125 | # skipping additional frames increases speed 1126 | self._skip_frames += int(sign * skip_frames_change) 1127 | self._calculate_frames() # update internals 1128 | 1129 | self._error = sign * new_error_mag 1130 | if self._delay < self.MIN_DELAY: 1131 | self._error += self.MIN_DELAY - self._delay 1132 | self._delay = self.MIN_DELAY 1133 | 1134 | def _update_frame_tracking(self): 1135 | # frame skip with no auto-delay allows continual frame skipping 1136 | # only set frame if necessary (moving one frame ahead isn't helpful) 1137 | if self._skip_frames is not None and \ 1138 | (self._direction == -1 or self._frames != 1): 1139 | self._image_ready.wait() 1140 | self.set_frame(self._frame + self._frames * self._direction) 1141 | else: 1142 | self._frame += 1 1143 | 1144 | if self.status == False or self._frame > self._end \ 1145 | or self._frame < self._start: 1146 | raise OutOfFrames 1147 | 1148 | def __repr__(self): 1149 | return f"{self.__class__.__name__}(filename={repr(self.filename)})" 1150 | 1151 | 1152 | if __name__ == '__main__': 1153 | with Camera(0) as cam: 1154 | cam.stream() 1155 | 1156 | --------------------------------------------------------------------------------