├── .gitignore ├── LICENSE ├── MANIFEST.in ├── README.md ├── docs ├── example-explained.md ├── example1.jpg ├── example2.jpg ├── example3.jpg ├── example4.jpg ├── eyes.jpg ├── group.jpg ├── group_photo.jpg ├── portrait.jpg ├── portrait_exif.jpg ├── portrait_fl.jpg ├── recolored.jpg └── tutorial.md ├── fdlite ├── __init__.py ├── data │ ├── camdb.csv │ ├── camdb.txt │ ├── face_detection_back.tflite │ ├── face_detection_front.tflite │ ├── face_detection_full_range.tflite │ ├── face_detection_full_range_sparse.tflite │ ├── face_detection_short_range.tflite │ ├── face_landmark.tflite │ └── iris_landmark.tflite ├── errors.py ├── examples │ ├── __init__.py │ └── iris_recoloring.py ├── exif.py ├── face_detection.py ├── face_landmark.py ├── iris_landmark.py ├── nms.py ├── render.py ├── transform.py └── types.py ├── mypy.ini ├── pyproject.toml ├── setup.cfg └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | .vscode/ 2 | **/__pycache__ 3 | **/.mypy_cache 4 | build/ 5 | dist/ 6 | *.egg-info/ 7 | .eggs/ 8 | .mypy_cache 9 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | Copyright © 2021,2024 Patrick Levin 3 | 4 | Permission is hereby granted, free of charge, to any person obtaining a copy of 5 | this software and associated documentation files (the "Software"), to deal in 6 | the Software without restriction, including without limitation the rights to 7 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 8 | of the Software, and to permit persons to whom the Software is furnished to do 9 | so, subject to the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be included in all 12 | copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 15 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 16 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 17 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 18 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 19 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 20 | SOFTWARE. 21 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include *.py 2 | include *.in 3 | include *.md 4 | include *.toml 5 | include mypy.ini 6 | include LICENSE 7 | 8 | graft docs 9 | graft fdlite 10 | 11 | recursive-include fdlite *.tflite 12 | recursive-include fdlite *.csv 13 | recursive-include fdlite *.txt 14 | 15 | exclude .editorconfig 16 | global-exclude .git* 17 | global-exclude *.pyc 18 | global-exclude *.so 19 | global-exclude *.dll 20 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Face Detection For Python 2 | 3 | This package implements parts of Google®'s [**MediaPipe**](https://mediapipe.dev/#!) models in pure Python (with a little help from Numpy and PIL) without `Protobuf` graphs and with minimal dependencies (just [**TF Lite**](https://www.tensorflow.org/lite/api_docs) and [**Pillow**](https://python-pillow.org/)). 4 | 5 | ## Models and Examples 6 | 7 | The package provides the following models: 8 | 9 | * Face Detection 10 | 11 | ![Face detection example](https://raw.githubusercontent.com/patlevin/face-detection-tflite/main/docs/group_photo.jpg) 12 | 13 | * Face Landmark Detection 14 | 15 | ![Face landmark example](https://raw.githubusercontent.com/patlevin/face-detection-tflite/main/docs/portrait_fl.jpg) 16 | 17 | * Iris Landmark Detection 18 | 19 | ![Iris landmark example](https://raw.githubusercontent.com/patlevin/face-detection-tflite/main/docs/eyes.jpg) 20 | 21 | * Iris recoloring example 22 | 23 | ![Iris recoloring example](https://raw.githubusercontent.com/patlevin/face-detection-tflite/main/docs/recolored.jpg) 24 | 25 | ## Motivation 26 | 27 | The package doesn't use the graph approach implemented by **MediaPipe** and 28 | is therefore not as flexible. It is, however, somewhat easier to use and 29 | understand and more accessible to recreational programming and experimenting 30 | with the pretrained ML models than the rather complex **MediaPipe** framework. 31 | 32 | Here's how face detection works and an image like shown above can be produced: 33 | 34 | ```python 35 | from fdlite import FaceDetection, FaceDetectionModel 36 | from fdlite.render import Colors, detections_to_render_data, render_to_image 37 | from PIL import Image 38 | 39 | image = Image.open('group.jpg') 40 | detect_faces = FaceDetection(model_type=FaceDetectionModel.BACK_CAMERA) 41 | faces = detect_faces(image) 42 | if not len(faces): 43 | print('no faces detected :(') 44 | else: 45 | render_data = detections_to_render_data(faces, bounds_color=Colors.GREEN) 46 | render_to_image(render_data, image).show() 47 | ``` 48 | 49 | While this example isn't that much simpler than the **MediaPipe** equivalent, 50 | some models (e.g. iris detection) aren't available in the Python API. 51 | 52 | Note that the package ships with five models: 53 | 54 | * `FaceDetectionModel.FRONT_CAMERA` - a smaller model optimised for 55 | selfies and close-up portraits; this is the default model used 56 | * `FaceDetectionModel.BACK_CAMERA` - a larger model suitable for group 57 | images and wider shots with smaller faces 58 | * `FaceDetectionModel.SHORT` - a model best suited for short range images, 59 | i.e. faces are within 2 metres from the camera 60 | * `FaceDetectionModel.FULL` - a model best suited for mid range images, 61 | i.e. faces are within 5 metres from the camera 62 | * `FaceDetectionModel.FULL_SPARSE` - a model best suited for mid range images, 63 | i.e. faces are within 5 metres from the camera 64 | 65 | The `FaceDetectionModel.FULL` and `FaceDetectionModel.FULL_SPARSE` models are 66 | equivalent in terms of detection quality. They differ in that the full model 67 | is a dense model whereas the sparse model runs up to 30% faster on CPUs. On a 68 | GPU, both models exhibit similar runtime performance. In addition, the dense 69 | full model has slightly better [Recall](https://en.wikipedia.org/wiki/Precision_and_recall), 70 | whereas the sparse model features a higher [Precision](https://en.wikipedia.org/wiki/Precision_and_recall). 71 | 72 | If you don't know whether the image is a close-up portrait or you get no 73 | detections with the default model, try using the `BACK_CAMERA`-model instead. 74 | 75 | ## Installation 76 | 77 | The latest release version is available in [PyPI](https://pypi.org/project/face-detection-tflite/0.1.0/) 78 | and can be installed via: 79 | 80 | ```sh 81 | pip install -U face-detection-tflite 82 | ``` 83 | 84 | The package can be also installed from source by navigating to the folder 85 | containing `setup.py` and running 86 | 87 | ```sh 88 | pip install . 89 | ``` 90 | 91 | from a shell or command prompt. 92 | -------------------------------------------------------------------------------- /docs/example-explained.md: -------------------------------------------------------------------------------- 1 | # Iris Recoloring Example Explained 2 | 3 | For those interested in how the iris recoloring example is implemented, 4 | this document goes into detail on how it works. 5 | 6 | ## Objective and General Approach 7 | 8 | The goal of iris recoloring is to change the iris color from its original 9 | appearance to another one. Ideally, we want the pupils and any reflections 10 | to stay black and white respetively while changing the eye color to another 11 | shade: 12 | 13 | ![Example image of eye color changed from brown to green](example1.jpg) 14 | 15 | In order to keep things simple, we can isolate the iris, convert it to 16 | grayscale and use [Pillow's colorize](https://pillow.readthedocs.io/en/stable/reference/ImageOps.html#PIL.ImageOps.colorize) function to apply a new tint: 17 | 18 | ![Eye with original color, grayscaled and colorized next to each other](example2.jpg) 19 | 20 | We can then paste the recolored iris back into the image. 21 | The `paste` function allows us to pass a mask image so we can easily leave 22 | the sclera (the white part of the eyeball) untouched. 23 | 24 | An initial attempt might look something like this: 25 | 26 | ```python 27 | ... 28 | left_eye = detect_iris(image, left_eye_roi) 29 | # isolate the iris 30 | iris_location, iris_size = get_iris_position_and_size(left_eye, image) 31 | iris = image.transform(iris_size, Transform.EXTENT, data=iris_location) 32 | # grayscale 33 | iris = iris.convert('L') 34 | # create mask 35 | mask = get_iris_mask(iris_size) 36 | # apply color 37 | iris_new = ImageOps.colorize(iris, 'black', 'white', mid=(120, 210, 45)) 38 | # paste results into image 39 | image.paste(iris_new, iris_location, mask) 40 | ``` 41 | 42 | Getting the iris size- and position is straight forward. We already get the 43 | landmarks that mark the iris position and -size with the iris detection 44 | results: 45 | 46 | ```python 47 | def get_iris_position_and_size(iris_results, image): 48 | bbox = bbox_from_landmarks(iris_results.iris).absolute(image.size) 49 | l, t, r, b = bbox.as_tuple 50 | # we add one pixel to the right and bottom because bounds are inclusive 51 | iris_location = (int(l), int(t), int(r+1), int(b+1)) 52 | iris_size = (int(bbox.width+1), int(bbox.height+1)) 53 | return iris_location, iris_size 54 | ``` 55 | 56 | Creating the mask is simple as well if we know the size of the mask image: 57 | 58 | ```python 59 | def get_iris_mask(iris_size): 60 | mask = Image.new(mode='L', size=iris_size) 61 | draw = ImageDraw.Draw(mask) 62 | draw.ellipse((0, 0, mask.width, mask.height), fill=255) 63 | return mask 64 | ``` 65 | 66 | The result would be ok, but we clearly colorize regions covered by the eyelids: 67 | 68 | ![Recolored eye without restricting to eye contours](example4.jpg) 69 | 70 | ## Staying Within Boundaries 71 | 72 | Using the parameters obtained from the bounding box, we can create an image 73 | that represents a mask for colorizing. This would be great if we didn't have 74 | the eye's contours to take into account. Here's what we've got so far: 75 | 76 | ![Iris bounding box, iris ellipse, eye contour landmarks, and iris mask](example3.jpg) 77 | 78 | We need to adjust the mask such that the parts covered by the eyelids are 79 | also masked and not colorized. There are a few ways of achieving this. One 80 | idea is to draw the mask manually and checking each point whether it's 81 | within the eye contours. 82 | 83 | ## What's Our Vector, Victor? - Dealing With Contour Segments 84 | 85 | The contours of the eye are part of the iris detection results and consist 86 | of a series of (3d) points. We can sort the points by their x value and use 87 | a line sweep algorithm to determine line segments. 88 | The algorithm to create a mask image that respects the eye contours is quite 89 | simple. 90 | 91 | ```pseudocode 92 | ' ellipse center 93 | let cx, cy = mask_height/2, mask_width/2 94 | ' we use a and b (the ellipse radii) for clarity 95 | let a, b = mask_height/2, mask_width/2 96 | for y from 0 to *mask_height* 97 | ' find left and right ellipse boundary point at y 98 | ' x₁ is the left and x₂ is the right ellipse boundary 99 | let x₁, x₂ = cx±a•sqrt(b²-(y-cy)²/b) 100 | find contour line segment AB₁ that contains x₁ 101 | find contour line segment AB₂ that contains x₂ 102 | if y is above AB₁ and above AB₂ skip to next y 103 | if y is above AB₁ set x₁ = intersection of AB₁ and ellipse 104 | if y is above AB₂ set x₂ = intersection of AB₂ and ellipse 105 | mark points from (x₁, y) to (x₂, y) as visible part of iris 106 | ``` 107 | 108 | x₁ and x₂ are outside the mask if they are located below their matching segments 109 | if the y coordinate of the starting point is below the center of the eye. 110 | Another important detail is the fact that x might match two line segments - 111 | one above and one below it. We select the segment with the closest starting 112 | point in that case. 113 | 114 | That's just one way of doing it. The results won't be pixel perfect for 115 | various reasons. One improvement might be blending towards the edges of the 116 | mask or growing/shrinking the mask based on traditional edge detection. 117 | -------------------------------------------------------------------------------- /docs/example1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/example1.jpg -------------------------------------------------------------------------------- /docs/example2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/example2.jpg -------------------------------------------------------------------------------- /docs/example3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/example3.jpg -------------------------------------------------------------------------------- /docs/example4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/example4.jpg -------------------------------------------------------------------------------- /docs/eyes.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/eyes.jpg -------------------------------------------------------------------------------- /docs/group.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/group.jpg -------------------------------------------------------------------------------- /docs/group_photo.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/group_photo.jpg -------------------------------------------------------------------------------- /docs/portrait.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/portrait.jpg -------------------------------------------------------------------------------- /docs/portrait_exif.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/portrait_exif.jpg -------------------------------------------------------------------------------- /docs/portrait_fl.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/portrait_fl.jpg -------------------------------------------------------------------------------- /docs/recolored.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/docs/recolored.jpg -------------------------------------------------------------------------------- /docs/tutorial.md: -------------------------------------------------------------------------------- 1 | # Face Detection Package Tutorial 2 | 3 | This package is a port of some [**Google® MediaPipe**](https://google.github.io/mediapipe/) 4 | graphs to simple Python functions. The goal is to provide access to some of 5 | the pretrained models used in MediaPipe without the complexities of the 6 | graph concepts used. 7 | 8 | ## A Word on Coordinates 9 | 10 | The library uses custom types which by default only hold normalized 11 | coordinates. This means the values for x and y range from 0..1 and are 12 | relative to the input image. The reasoning behind this choice is 13 | flexibility. Normalized coordinates don't need to change if you scale 14 | the image (e.g.for displaying purposes). 15 | 16 | Most types contain a `scale()`-method that can be used to scale the 17 | coordinates to the requested (image-)size. 18 | 19 | ## Detecting Faces 20 | 21 | You can use the `face_detection` module to find faces within an image. 22 | The `FaceDetection` model will return a list of `Detection`s for each face 23 | found. These detections are *normalized*, meaning the coordinates range from 24 | 0..1 and are relative to the input image. 25 | 26 | Each model class is callable, meaning once instanciated you can call them 27 | just like a function. The example below shows how to detect faces and 28 | display results using `Pillow`: 29 | 30 | ```python 31 | from fdlite import FaceDetection, FaceDetectionModel 32 | from fdlite.render import Colors, detections_to_render_data, render_to_image 33 | from PIL import Image 34 | 35 | # load the model; select "back camera"-model for groups and smaller faces 36 | detect_faces = FaceDetection(model_type=FaceDetectionModel.BACK_CAMERA) 37 | # open an image 38 | img = Image.open('group.jpg') 39 | # detect faces 40 | detections = detect_faces(img) 41 | if len(detections): 42 | # convert results to render data; show bounding boxes only 43 | render_data = detections_to_render_data( 44 | detections, bounds_color=Colors.GREEN, line_width=4) 45 | # render to image and display results 46 | render_to_image(render_data, img).show() 47 | else: 48 | print('no faces found :(') 49 | ``` 50 | 51 | The result could look something like this: 52 | 53 | ![Group photo with face detections](group_photo.jpg) 54 | 55 | *image source: [pexels.com](https://www.pexels.com/photo/group-of-people-watching-on-laptop-1595385/)* 56 | 57 | ## Detecting Face Landmarks 58 | 59 | The face detection model only produces bounding boxes and crude keypoints. 60 | A detailed 3D face mesh with over 480 landmarks can be obtained by using the 61 | `FaceLandmark` model found in the `face-landmark` module. The recommended 62 | use of this model is to calculate a *region of interest* (ROI) from the 63 | output of the `FaceDetection` model and use it as an input: 64 | 65 | > **FaceDetection(image) ⇒ ROI from detection ⇒ FaceLandmarks(image, ROI)** 66 | 67 | Face landmark detection is fairly simple: 68 | 69 | ```python 70 | from fdlite import FaceDetection, FaceLandmark, face_detection_to_roi 71 | from fdlite.render import Colors, landmarks_to_render_data, render_to_image 72 | from PIL import Image 73 | 74 | # load detection models 75 | detect_faces = FaceDetection() 76 | detect_face_landmarks = FaceLandmark() 77 | 78 | # open image; by default, the "front camera"-model is used, which is smaller 79 | # and ideal for selfies, and close-up portraits 80 | img = Image.open('portrait.jpg') 81 | # detect face 82 | face_detections = detect_faces(img) 83 | if len(face_detections): 84 | # get ROI for the first face found 85 | face_roi = face_detection_to_roi(face_detections[0], img.size) 86 | # detect face landmarks 87 | face_landmarks = detect_face_landmarks(img, face_roi) 88 | # convert detections to render data 89 | render_data = landmarks_to_render_data( 90 | face_landmarks, [], landmark_color=Colors.PINK, thickness=3) 91 | # render and display landmarks (points only) 92 | render_to_image(render_data, img).show() 93 | else: 94 | print('no face detected :(') 95 | ``` 96 | 97 | The result could look something like this: 98 | 99 | ![Portrait with face landmarks](portrait_fl.jpg) 100 | 101 | *Photo by Andrea Piacquadio from [Pexels](https://www.pexels.com/photo/brown-freckles-on-face-3763152/)* 102 | 103 | ## Iris Detection 104 | 105 | The final model in this package is iris detection. Just like face landmark 106 | detection, this model is best used with an **ROI** obtained from face 107 | landmarks. The processing chain from image to iris landmarks looks like this: 108 | 109 | > **FaceDetection ⇒ ROI from detection ⇒ FaceLandmark ⇒ ROI from landmarks ⇒ IrisDetection** 110 | 111 | Iris detection results consist of two sets of landmarks: one contains the 112 | basic keypoints (eye boundary points and pupil center), while the other is a 113 | refined version of the landmarks returned by `FaceLandmark`. 114 | 115 | ```python 116 | # for face landmark detection 117 | from fdlite import FaceDetection, FaceLandmark, face_detection_to_roi 118 | # for iris landmark detection 119 | from fdlite import IrisLandmark, iris_roi_from_face_landmarks 120 | # for eye landmark rendering 121 | from fdlite import eye_landmarks_to_render_data 122 | # for rendering to image 123 | from fdlite.render import Colors, render_to_image 124 | # for finding just the eye region in the image 125 | from fdlite.transform import bbox_from_landmarks 126 | from PIL import Image 127 | 128 | # load detection models 129 | detect_faces = FaceDetection() 130 | detect_face_landmarks = FaceLandmark() 131 | detect_iris = IrisLandmark() 132 | 133 | # open image 134 | img = Image.open('portrait.jpg') 135 | # detect face 136 | face_detections = detect_faces(img) 137 | if len(face_detections): 138 | # get ROI for the first face found 139 | face_roi = face_detection_to_roi(face_detections[0], img.size) 140 | # detect face landmarks 141 | face_landmarks = detect_face_landmarks(img, face_roi) 142 | # get ROI for both eyes 143 | eye_roi = iris_roi_from_face_landmarks(face_landmarks, img.size) 144 | left_eye_roi, right_eye_roi = eye_roi 145 | # detect iris landmarks for both eyes 146 | left_eye_results = detect_iris(img, left_eye_roi) 147 | right_eye_results = detect_iris(img, right_eye_roi, is_right_eye=True) 148 | # convert landmarks to render data 149 | render_data = eye_landmarks_to_render_data(left_eye_results.contour, 150 | landmark_color=Colors.PINK, 151 | connection_color=Colors.GREEN) 152 | # add landmarks of the right eye 153 | _ = eye_landmarks_to_render_data(right_eye_results.contour, 154 | landmark_color=Colors.PINK, 155 | connection_color=Colors.GREEN, 156 | output=render_data) 157 | # render to image 158 | render_to_image(render_data, img) 159 | # get the bounds of just the eyes 160 | contours = left_eye_results.contour + right_eye_results.contour 161 | eye_box = bbox_from_landmarks(contours).absolute(img.size) 162 | # calculate a scaled-up version of the eye region 163 | new_size = (int(eye_box.width * 2), int(eye_box.height * 2)) 164 | # isolate the eyes, scale them up, and display the result 165 | img.crop(eye_box.as_tuple).resize(new_size).show() 166 | else: 167 | print('no face detected :(') 168 | ``` 169 | 170 | The result will look something like this: 171 | 172 | ![Eyes with detected contours marked](eyes.jpg) 173 | 174 | ## Fun with Iris Detection 175 | 176 | The package includes a rudimentary exmaple of iris recoloring using iris 177 | detection results. The function `recolor_iris` from the `examples` module 178 | accepts iris detections results and a tuple of `(red, green, blue)` for 179 | simple iris recoloring: 180 | 181 | ```python 182 | from fdlite import FaceDetection, FaceLandmark, face_detection_to_roi 183 | from fdlite import IrisLandmark, iris_roi_from_face_landmarks 184 | from fdlite.examples.iris_recoloring import recolor_iris 185 | from PIL import Image 186 | 187 | EXCITING_NEW_EYE_COLOR = (161, 52, 216) 188 | 189 | # load detection models 190 | detect_faces = FaceDetection() 191 | detect_face_landmarks = FaceLandmark() 192 | detect_iris = IrisLandmark() 193 | 194 | # open image 195 | img = Image.open('portrait.jpg') 196 | # detect face 197 | face_detections = detect_faces(img) 198 | if len(face_detections): 199 | # get ROI for the first face found 200 | face_roi = face_detection_to_roi(face_detections[0], img.size) 201 | # detect face landmarks 202 | face_landmarks = detect_face_landmarks(img, face_roi) 203 | # get ROI for both eyes 204 | eye_roi = iris_roi_from_face_landmarks(face_landmarks, img.size) 205 | left_eye_roi, right_eye_roi = eye_roi 206 | # detect iris landmarks for both eyes 207 | left_eye_results = detect_iris(img, left_eye_roi) 208 | right_eye_results = detect_iris(img, right_eye_roi, is_right_eye=True) 209 | # change the iris color 210 | recolor_iris(img, left_eye_results, iris_color=EXCITING_NEW_EYE_COLOR) 211 | recolor_iris(img, right_eye_results, iris_color=EXCITING_NEW_EYE_COLOR) 212 | img.show() 213 | else: 214 | print('no face detected :(') 215 | ``` 216 | 217 | The result will look similar to this: 218 | 219 | ![Recolored iris (cropped; with vignette effect)](recolored.jpg) 220 | 221 | (cropped with vignette applied to distract from the inaccurracies present in the output) 222 | 223 | ## Estimating Eye Distance from Camera 224 | 225 | The package also contains a function to estimate the distance of the eyes to 226 | the camera. This comes with several caveats, though. Firstly, the image must 227 | contain additional information about the lens and the sensor size. This data 228 | (EXIF) is usually stored by smartphones and professional photo cameras but 229 | might be missing from downloaded pictures. In this case, the information 230 | can be provided manually as well. 231 | 232 | The package contains a library of camera models to look up sensor size 233 | information if it is missing from the EXIF data (focal length is still 234 | required to be present). 235 | 236 | To get a distance estimation, first detect the iris data and then pass it to 237 | `iris_depth_in_mm_from_landmarks` along with the image: 238 | 239 | ```python 240 | from fdlite import FaceDetection, FaceLandmark, face_detection_to_roi 241 | from fdlite import IrisLandmark, iris_roi_from_face_landmarks 242 | from fdlite import iris_depth_in_mm_from_landmarks 243 | from PIL import Image 244 | 245 | # load detection models 246 | detect_faces = FaceDetection() 247 | detect_face_landmarks = FaceLandmark() 248 | detect_iris = IrisLandmark() 249 | 250 | # open image 251 | img = Image.open('portrait_exif.jpg') 252 | # detect face 253 | face_detections = detect_faces(img) 254 | if len(face_detections): 255 | # get ROI for the first face found 256 | face_roi = face_detection_to_roi(face_detections[0], img.size) 257 | # detect face landmarks 258 | face_landmarks = detect_face_landmarks(img, face_roi) 259 | # get ROI for both eyes 260 | eye_roi = iris_roi_from_face_landmarks(face_landmarks, img.size) 261 | left_eye_roi, right_eye_roi = eye_roi 262 | # detect iris landmarks for both eyes 263 | left_eye_results = detect_iris(img, left_eye_roi) 264 | right_eye_results = detect_iris(img, right_eye_roi, is_right_eye=True) 265 | # change the iris color 266 | dist_left_mm, dist_right_mm = iris_depth_in_mm_from_landmarks( 267 | img, left_eye_results, right_eye_results) 268 | print(f'Distance from camera appr. {dist_left_mm//10} cm to ' 269 | f'{dist_right_mm//10} cm') 270 | else: 271 | print('no face detected :(') 272 | ``` 273 | 274 | Given the image below 275 | ![Portrait with EXIF data](portrait_exif.jpg) 276 | 277 | (source: [pixnio.com](https://pixnio.com/people/female-women/portrait-people-girl-woman-fashion-face-shadow-person)) 278 | 279 | will result in the following output: 280 | 281 | > Distance from camera appr. 95.0 cm to 99.0 cm 282 | 283 | Please note that this feature works best with smartphone photos and can 284 | be very inaccurate with pictures taken using professional equipment. 285 | This is due to the various possibilities that these cameras provide (lenses, 286 | optical and digital zoom, etc.) Using values obtained from tools like 287 | `Exiftool` will often yield vastly different results. 288 | -------------------------------------------------------------------------------- /fdlite/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from .errors import ArgumentError, CoordinateRangeError # noqa:F401 3 | from .errors import InvalidEnumError, MissingExifDataError # noqa:F401 4 | from .errors import ModelDataError # noqa:F401 5 | from .face_detection import FaceDetection, FaceDetectionModel # noqa:F401 6 | from .face_detection import FaceIndex # noqa:F401 7 | from .face_landmark import FaceLandmark, face_detection_to_roi # noqa:F401 8 | from .face_landmark import face_landmarks_to_render_data # noqa:F401 9 | from .iris_landmark import IrisIndex, IrisLandmark, IrisResults # noqa:F401 10 | from .iris_landmark import eye_landmarks_to_render_data # noqa:F401 11 | from .iris_landmark import iris_depth_in_mm_from_landmarks # noqa:F401 12 | from .iris_landmark import iris_landmarks_to_render_data # noqa:F401 13 | from .iris_landmark import iris_roi_from_face_landmarks # noqa:F401 14 | 15 | 16 | __version__ = '0.6.0' 17 | -------------------------------------------------------------------------------- /fdlite/data/camdb.csv: -------------------------------------------------------------------------------- 1 | Mamiya 645,0.577 2 | Mamiya ZD,0.721 3 | P 25,0.577 4 | 2.8E,0.510 5 | HD2,6.4 6 | Hero3+ black,5.42 7 | HERO4 Silver,7.66 8 | HERO4 Black,5 9 | HERO5 Black,5 10 | Phantom Vision FC200,6 11 | Phantom 3 Pro FC300X,5.5 12 | AEE DV,6 13 | AEE MagiCam SD19 & compatibles,6 14 | Git2,5.57 15 | Mavic Pro FC220,5.64 16 | FC6310,2.73 17 | Canon PowerShot Pro1,3.933 18 | PowerShot Pro1,3.933 19 | Canon PowerShot Pro70,5.47 20 | PowerShot Pro70,5.47 21 | Canon PowerShot G6,4.843 22 | PowerShot G6,4.843 23 | Canon PowerShot G5,4.843 24 | PowerShot G5,4.843 25 | Canon PowerShot G3,4.843 26 | PowerShot G3,4.843 27 | Canon PowerShot G2,4.843 28 | PowerShot G2,4.843 29 | Canon PowerShot G1,4.843 30 | PowerShot G1,4.843 31 | Canon PowerShot G7,4.843 32 | PowerShot G7,4.843 33 | Canon PowerShot G5 X,2.72 34 | PowerShot G5 X (3:2),2.72 35 | Canon PowerShot G5 X 4:3,2.94 36 | PowerShot G5 X (4:3),2.94 37 | Canon PowerShot G5 X 16:9,2.85 38 | PowerShot G5 X (16:9),2.85 39 | Canon PowerShot G7 X,2.72 40 | PowerShot G7 X (3:2),2.72 41 | Canon PowerShot G7 X 4:3,2.94 42 | PowerShot G7 X (4:3),2.94 43 | Canon PowerShot G7 X 16:9,2.85 44 | PowerShot G7 X (16:9),2.85 45 | Canon PowerShot G7 X Mark II,2.72 46 | PowerShot G7 X Mark II (3:2),2.72 47 | Canon PowerShot G7 X Mark II 4:3,2.94 48 | PowerShot G7 X Mark II (4:3),2.94 49 | Canon PowerShot G7 X Mark II 16:9,2.85 50 | PowerShot G7 X Mark II (16:9),2.85 51 | Canon PowerShot G9,4.605 52 | PowerShot G9,4.605 53 | Canon Powershot G9 X,2.72 54 | PowerShot G9 X,2.72 55 | Canon PowerShot G10,4.605 56 | PowerShot G10,4.605 57 | Canon PowerShot G11,4.554 58 | PowerShot G11,4.554 59 | Canon PowerShot G12,4.63 60 | PowerShot G12,4.63 61 | Canon PowerShot G1 X,1.85 62 | PowerShot G1 X,1.85 63 | Canon PowerShot G1 X Mark II,1.93 64 | PowerShot G1 X Mark II,1.93 65 | Canon PowerShot G15,4.65 66 | PowerShot G15,4.65 67 | Canon PowerShot G16,4.67 68 | PowerShot G16,4.67 69 | Canon PowerShot SD550,4.8 70 | PowerShot SD550,4.8 71 | Canon PowerShot SD950 IS,4.7 72 | PowerShot SD950 IS,4.7 73 | Canon DIGITAL IXUS 750,4.8 74 | IXUS 750,4.8 75 | IXY Digital 700,4.8 76 | Canon PowerShot SD500,4.8 77 | PowerShot SD500,4.8 78 | Canon DIGITAL IXUS 700,4.8 79 | IXUS 700,4.8 80 | IXY Digital 600,4.8 81 | Canon PowerShot SD450,6.05 82 | PowerShot SD450,6.05 83 | Canon DIGITAL IXUS 55,6.05 84 | IXUS 55,6.05 85 | Canon PowerShot SD400,6.05 86 | PowerShot SD400,6.05 87 | Canon IXY DIGITAL 55,6.05 88 | IXY 55,6.05 89 | Canon DIGITAL IXUS 50,6.05 90 | IXUS 50,6.05 91 | Canon DIGITAL IXUS 70,6.05 92 | IXUS 70,6.05 93 | Canon PowerShot SD300,6.05 94 | PowerShot SD300,6.05 95 | Canon IXY DIGITAL 50,6.05 96 | IXY 50,6.05 97 | Canon DIGITAL IXUS 40,6.05 98 | IXUS 40,6.05 99 | Canon PowerShot SD200,6.05 100 | PowerShot SD200,6.05 101 | Canon IXY DIGITAL 40,6.05 102 | IXY 40,6.05 103 | Canon DIGITAL IXUS 30,6.05 104 | IXUS 30,6.05 105 | Canon PowerShot S200,6.5 106 | PowerShot S200,6.5 107 | Canon DIGITAL IXUS v2,6.5 108 | IXUS v2,6.5 109 | Canon IXY DIGITAL 200a,6.5 110 | IXY 200a,6.5 111 | Canon PowerShot SD110,6.5 112 | PowerShot SD110,6.5 113 | Canon DIGITAL IXUS II,6.5 114 | IXUS II,6.5 115 | Canon IXY DIGITAL 30,6.5 116 | IXY 30,6.5 117 | Canon PowerShot SD100,6.5 118 | PowerShot SD100,6.5 119 | Canon DIGITAL IXUS i,6.144 120 | IXUS i,6.144 121 | Canon PowerShot SD10,6.144 122 | PowerShot SD10,6.144 123 | Canon PowerShot S500,4.843 124 | PowerShot S500,4.843 125 | Canon DIGITAL IXUS 500,4.843 126 | IXUS 500,4.843 127 | Canon IXY DIGITAL 500,4.843 128 | IXY 500,4.843 129 | Canon PowerShot S410,4.843 130 | PowerShot S410,4.843 131 | Canon DIGITAL IXUS 430,4.843 132 | IXUS 430,4.843 133 | Canon IXY DIGITAL 450,4.843 134 | IXY 450,4.843 135 | Canon PowerShot S400,4.843 136 | PowerShot S400,4.843 137 | Canon DIGITAL IXUS 400,4.843 138 | IXUS 400,4.843 139 | Canon IXY DIGITAL 400,4.843 140 | IXY 400,4.843 141 | Canon PowerShot S80,4.843 142 | PowerShot S80,4.843 143 | Canon PowerShot S70,4.843 144 | PowerShot S70,4.843 145 | Canon PowerShot S60,4.843 146 | PowerShot S60,4.843 147 | Canon PowerShot S50,4.843 148 | PowerShot S50,4.843 149 | Canon PowerShot S45,4.843 150 | PowerShot S45,4.843 151 | Canon PowerShot S40,4.843 152 | PowerShot S40,4.843 153 | Canon PowerShot S30,4.843 154 | PowerShot S30,4.843 155 | Canon PowerShot A610,4.8 156 | PowerShot A610,4.8 157 | Canon PowerShot A620,4.8 158 | PowerShot A620,4.8 159 | Canon PowerShot A520,6.05 160 | PowerShot A520,6.05 161 | Canon PowerShot A510,6.05 162 | PowerShot A510,6.05 163 | Canon PowerShot A95,4.85 164 | PowerShot A95,4.85 165 | Canon PowerShot A80,4.85 166 | PowerShot A80,4.85 167 | Canon PowerShot A85,6.500 168 | PowerShot A85,6.500 169 | Canon PowerShot A75,6.500 170 | PowerShot A75,6.500 171 | Canon PowerShot A70,6.500 172 | PowerShot A70,6.500 173 | Canon PowerShot A60,6.500 174 | PowerShot A60,6.500 175 | Canon PowerShot A40,6.500 176 | PowerShot A40,6.500 177 | Canon PowerShot A30,6.500 178 | PowerShot A30,6.500 179 | Canon PowerShot A20,6.500 180 | PowerShot A20,6.500 181 | Canon PowerShot A10,6.500 182 | PowerShot A10,6.500 183 | Canon PowerShot S2 IS,6.0 184 | PowerShot S2 IS,6.0 185 | Canon PowerShot S1 IS,6.56 186 | PowerShot S1 IS,6.56 187 | Canon PowerShot S5 IS,6.0 188 | PowerShot S5 IS,6.0 189 | Canon PowerShot Pro90 IS,5.28 190 | PowerShot Pro90 IS,5.28 191 | Canon DIGITAL IXUS 80 IS,6.02 192 | IXUS 80 IS,6.02 193 | Canon PowerShot SX1 IS,5.5 194 | PowerShot SX1 IS,5.5 195 | Canon PowerShot S90,4.67 196 | PowerShot S90,4.67 197 | Canon PowerShot A650 IS,4.67712 198 | PowerShot A650 IS,4.67712 199 | Canon PowerShot SX150 IS,5.62 200 | PowerShot SX150 IS,5.62 201 | Canon PowerShot SX10 IS,5.6 202 | PowerShot SX10 IS,5.6 203 | Canon PowerShot SX130 IS,5.62 204 | PowerShot SX130 IS,5.62 205 | Canon PowerShot SX220 HS,5.6 206 | PowerShot SX220 HS,5.6 207 | Canon PowerShot SX230 HS,5.6 208 | PowerShot SX230 HS,5.6 209 | Canon PowerShot A720 IS,6.03 210 | PowerShot A720 IS,6.03 211 | Canon PowerShot S120,4.62 212 | PowerShot S120,4.62 213 | Canon PowerShot A640,4.79 214 | PowerShot A640,4.79 215 | Canon PowerShot SX260 HS,5.56 216 | PowerShot SX260 HS,5.56 217 | Canon IXUS 220 HS,5.58 218 | IXUS 220 HS,5.58 219 | Canon PowerShot S95,4.67 220 | PowerShot S95,4.67 221 | Canon PowerShot S100,4.62 222 | PowerShot S100,4.62 223 | Canon PowerShot A4000 IS,5.6 224 | PowerShot A4000 IS,5.6 225 | Canon PowerShot ELPH 110 HS,5.58 226 | Powershot ELPH 110 HS,5.58 227 | Canon IXUS 125 HS,5.58 228 | IXUS 125 HS,5.58 229 | Canon PowerShot SX510 HS,5.58 230 | PowerShot SX510 HS,5.58 231 | Canon PowerShot S110,4.62 232 | PowerShot S110,4.62 233 | Canon PowerShot SD1100 IS,6.02 234 | PowerShot SD1100 IS,6.02 235 | Canon PowerShot SX160 IS,5.6 236 | PowerShot SX160 IS,5.6 237 | Canon PowerShot A495,5.39 238 | PowerShot A495,5.39 239 | Canon PowerShot A490,5.39 240 | PowerShot A490,5.39 241 | Canon PowerShot SX30 IS,5.581 242 | PowerShot SX30 IS,5.581 243 | EX-Z750,4.8 244 | EX-P700,4.65 245 | EX-P600,4.65 246 | EX-Z4,6.05 247 | EX-Z3,6.05 248 | EX-Z55,6.05 249 | EX-Z40,6.05 250 | EX-Z30,6.05 251 | QV-4000,4.843 252 | QV-3500EX,4.8 253 | QV-3000EX,4.8 254 | EX-FH20,5.68 255 | FinePix F11,4.5 256 | FinePix F10,4.5 257 | FinePix S9000,4.48 258 | FinePix S9500,4.48 259 | FinePix S9600,4.48 260 | FinePix S5500,6.491 261 | FinePix S5100,6.491 262 | FinePix S7000,4.487 263 | FinePix S20Pro,4.487 264 | FinePix S602 ZOOM,4.487 265 | FinePix F601 ZOOM,4.487 266 | FinePix F810,4.9 267 | FinePix F710,4.5 268 | FinePix E550,4.5 269 | FinePix 3800,6.3333333 270 | FinePix S304,6.3333333 271 | FinePix S3000,6.3333333 272 | FinePix2800ZOOM,6.3333333 273 | FinePix F200EXR,4.341 274 | FinePix X100,1.523 275 | X100S,1.523 276 | X100T,1.523 277 | X10,3.93 278 | X70,1.5375 279 | X20,3.93 280 | X30,3.93 281 | FinePix HS20EXR,5.71 282 | FinePix F770EXR,5.43 283 | FinePix S5600,6.03 284 | XQ1,3.91 285 | FinePix A370,6.03 286 | X-S1,3.93 287 | Kodak DC120 ZOOM Digital Camera,1 288 | Kodak DC120,1 289 | Kodak Digital Science DC50 Zoom Camera,1 290 | Kodak DC50,1 291 | Kodak CX6330 Zoom Digital Camera,6.593 292 | Kodak CX6330,6.593 293 | DiMAGE Z6,6.05 294 | DiMAGE Z5,6.05 295 | DiMAGE Z3,6.05 296 | DiMAGE Z2,6.03 297 | DiMAGE Z1,6.545 298 | DiMAGE Z20,6.05 299 | DiMAGE Z10,6.05 300 | DiMAGE A200,3.933 301 | DiMAGE A2,3.933 302 | DiMAGE A1,3.933 303 | DiMAGE 7Hi,3.933 304 | DiMAGE 7i,3.933 305 | DiMAGE 7,3.933 306 | DiMAGE Xt,6.5 307 | DiMAGE Xi,6.5 308 | DiMAGE X,6.5 309 | DiMAGE G400,6.144 310 | Revio KD-420Z,6.144 311 | Digilux 2,4.45 312 | D-LUX2,4.45 313 | D-LUX 4,4.33 314 | Digilux 4,4.33 315 | D-LUX 3,4.33 316 | Digilux 3,2 317 | Leica Q (Typ 116),1 318 | Coolpix P60,5.62 319 | E8800,3.933 320 | Coolpix 8800,3.933 321 | E8700,3.933 322 | Coolpix 8700,3.933 323 | E5700,3.933 324 | Coolpix 5700,3.933 325 | E7900,4.86 326 | Coolpix 7900,4.86 327 | E7600,4.86 328 | Coolpix 7600,4.86 329 | E5900,4.86 330 | Coolpix 5900,4.86 331 | E5200,4.86 332 | Coolpix 5200,4.86 333 | E4200,4.86 334 | Coolpix 4200,4.86 335 | E5400,4.843 336 | Coolpix 5400,4.843 337 | E5000,3.933 338 | Coolpix 5000,3.933 339 | E4800,6 340 | Coolpix 4800,6 341 | E4500,4.843 342 | Coolpix 4500,4.843 343 | E995,4.843 344 | Coolpix 995,4.843 345 | E990,4.843 346 | Coolpix 990,4.843 347 | E950,5.408 348 | Coolpix 950,5.408 349 | Coolpix P330,4.706 350 | Coolpix S3300,5.65 351 | Coolpix P6000,4.6 352 | E8400,3.933 353 | Coolpix 8400,3.933 354 | Coolpix P7000,4.69 355 | Coolpix P7800,4.67 356 | Coolpix A,1.523 357 | C8080WZ,3.933 358 | C-8080 Wide Zoom,3.933 359 | "u-miniD,Stylus V",6.0 360 | µ-mini Digital,6.0 361 | Stylus Verve Digital,6.0 362 | "C70Z,C7000Z",4.8 363 | "C-70 Zoom, C-7000 Zoom",4.8 364 | "C4100Z,C4000Z",4.92 365 | "C-4000 Zoom, C-4100 Zoom",4.92 366 | "X-2,C-50Z",4.9 367 | X-2,4.9 368 | C-50 Zoom,4.9 369 | C750UZ,6.03 370 | C-750 Ultra Zoom,6.03 371 | C730UZ,6.52 372 | C-730 Ultra Zoom,6.52 373 | C700UZ,6.52 374 | C-700 Ultra Zoom,6.52 375 | C7070WZ,4.930 376 | C-7070 Wide Zoom,4.930 377 | C5060WZ,4.930 378 | C-5060 Wide Zoom,4.930 379 | C5050Z,4.930 380 | C-5050 Zoom,4.930 381 | C4040Z,4.930 382 | C-4040 Zoom,4.930 383 | C3040Z,4.930 384 | C-3040 Zoom,4.930 385 | C2040Z,4.930 386 | C-2040 Zoom,4.930 387 | "C860L,D360L",6.563 388 | C-860L,6.563 389 | D-360L,6.563 390 | mju-II,1 391 | µ-II,1 392 | Stylus Epic,1 393 | SP350,4.8 394 | SP-350,4.8 395 | SP500UZ,6.0 396 | SP-500 Ultra Zoom,6.0 397 | SP560UZ,5.8 398 | SP-560 Ultra Zoom,5.8 399 | XZ-1,4.68 400 | XZ-2,4.68 401 | Stylus1,4.67 402 | Stylus 1,4.67 403 | "Stylus1,1s",4.67 404 | "Stylus 1, 1s",4.67 405 | TG-1,5.56 406 | Stylus TG-1,5.56 407 | TG-2,5.56 408 | Stylus TG-2,5.56 409 | TG-3,5.56 410 | Stylus TG-3,5.56 411 | TG-4,5.56 412 | Stylus TG-4,5.56 413 | DMC-LX1,5.40 414 | DMC-LX3,5.32 415 | DMC-LX5,4.71 416 | DMC-LX7,5.1 417 | DMC-LX10,2.73 418 | DMC-LX15,2.73 419 | DMC-LC1,3.933 420 | DMC-LZ2,6.06 421 | DMC-LZ1,6.06 422 | DMC-FX9,6.05 423 | DMC-FX8,6.05 424 | DMC-FX7,6.05 425 | DMC-FX2,6.05 426 | DMC-FZ30,4.73 427 | DMC-FZ20,6.0 428 | DMC-FZ10,5.84 429 | DMC-FZ5,6.0 430 | DMC-FZ3,7.6 431 | DMC-FZ28,5.6 432 | DMC-FX150,4.7 433 | DMC-LX2,4.44 434 | DMC-FZ18,6.1 435 | DMC-FZ35,5.6 436 | DMC-FZ40,5.60 437 | DMC-FZ40 (3:2),5.81 438 | DMC-FZ45,5.60 439 | DMC-FZ45 (3:2),5.81 440 | DMC-FZ50,4.84 441 | DMC-FZ8,6.02 442 | DMC-FZ100,5.60 443 | DMC-FZ100 (3:2),5.81 444 | DMC-FZ200,5.56 445 | DMC-FZ300,5.56 446 | DMC-LX100,2.21 447 | DMC-FZ1000,2.73 448 | DMC-FZ150,5.56 449 | DMC-LF1,4.67 450 | DMC-TZ71,5.58 451 | DMC-TZ61,5.58 452 | DMC-TZ100,2.75 453 | DMC-ZS100,2.75 454 | Pentax Optio 43WR,6.5 455 | Optio 43WR,6.5 456 | Pentax Optio 750Z,4.843 457 | Optio 750Z,4.843 458 | Pentax Optio 555,4.843 459 | Optio 555,4.843 460 | Pentax Optio 550,4.843 461 | Optio 550,4.843 462 | Pentax Optio 450,4.843 463 | Optio 450,4.843 464 | Pentax Optio 33LF,6.563 465 | Optio 33LF,6.563 466 | Pentax Optio 33L,6.563 467 | Optio 33L,6.563 468 | Pentax Optio 230GS,6.563 469 | Optio 230GS,6.563 470 | Pentax Optio 330GS,6.563 471 | Optio 330GS,6.563 472 | Pentax Optio 430,4.85 473 | Optio 430,4.85 474 | GR Digital,4.8 475 | Caplio GX8,4.9 476 | Caplio GX,4.9 477 | Caplio RR30,6.4 478 | GR,1.523 479 | WB2000,5.69 480 | SM-G935F,6.19 481 | Galaxy S7,6.19 482 | Sigma DP1,1.739 483 | DP1,1.739 484 | Sigma DP1 Merrill,1.531 485 | DP1 Merrill,1.531 486 | Sigma DP1S,1.739 487 | DP1S,1.739 488 | Sigma DP1X,1.739 489 | DP1X,1.739 490 | Sigma DP2,1.739 491 | DP2,1.739 492 | Sigma DP2 Merrill,1.531 493 | DP2 Merrill,1.531 494 | Sigma DP2S,1.739 495 | DP2S,1.739 496 | Sigma DP2X,1.739 497 | DP2X,1.739 498 | Sigma DP3 Merrill,1.531 499 | DP3 Merrill,1.531 500 | DSC-R1,1.68 501 | DSC-H1,6.0 502 | DSC-F828,3.933 503 | Cybershot,4.843 504 | Cyber-shot,4.843 505 | DSC-P200,4.8 506 | DSC-P150,4.8 507 | DSC-P100,4.8 508 | DSC-P93,4.8 509 | DSC-W12,4.8 510 | DSC-W7,4.8 511 | DSC-W15,4.8 512 | DSC-W5,4.8 513 | DSC-W1,4.8 514 | DSC-V3,4.843 515 | DSC-V1,4.843 516 | DSC-T1,5.650 517 | DSC-S90,6.5 518 | DSC-ST80,6.5 519 | DSC-S80,6.5 520 | DSC-S60,6.5 521 | DSC-P73,6.5 522 | DSC-RX100,2.73 523 | RX100,2.73 524 | DSC-RX100M2,2.73 525 | RX100 II,2.73 526 | DSC-RX100M3,2.73 527 | RX100 III,2.73 528 | DSC-RX100M4,2.73 529 | RX100 IV,2.73 530 | DSC-RX100M5,2.73 531 | RX100 V,2.73 532 | DSC-HX300,5.58 533 | DSC-HX20V,5.62 534 | Xperia Z3,7.87 535 | DSC-RX10,2.73 536 | RX10,2.73 537 | DSC-RX10M2,2.73 538 | RX10 II,2.73 539 | DSC-RX10M3,2.73 540 | RX10 III,2.73 541 | DSC-RX10M4,2.73 542 | RX10 IV,2.73 543 | DSC-RX1RM2,1 544 | RX1 R II,1 545 | 35mm film: full frame,1 546 | Crop-factor 1.0 (Full Frame),1 547 | Crop-Faktor 1.0 (Kleinbildformat),1 548 | Crop-factor 1.1,1.1 549 | Crop-Faktor 1.1,1.1 550 | Crop-factor 1.3 (APS-H),1.267 551 | Crop-Faktor 1.3 (APS-H),1.267 552 | Crop-factor 1.5 (APS-C),1.53 553 | Crop-Faktor 1.5 (APS-C),1.53 554 | Crop-factor 1.6 (APS-C),1.611 555 | Crop-Faktor 1.6 (Canon APS-C),1.611 556 | Crop-factor 1.7,1.739 557 | Crop-Faktor 1.7 (Sigma),1.739 558 | Crop-factor 2.0 (Four-Thirds),2 559 | Crop-Faktor 2.0 (Four-Thirds),2 560 | Canon EOS M,1.613 561 | EOS M,1.613 562 | Canon EOS M2,1.613 563 | EOS M2,1.613 564 | Canon EOS M3,1.613 565 | EOS M3,1.613 566 | Canon EOS M5,1.613 567 | EOS M5,1.613 568 | Canon EOS M6,1.613 569 | EOS M6,1.613 570 | Canon EOS M10,1.613 571 | EOS M10,1.613 572 | Canon EOS M50,1.613 573 | EOS M50,1.613 574 | Canon EOS M100,1.613 575 | EOS M100,1.613 576 | X-Pro1,1.529 577 | X-Pro2,1.528 578 | X-E1,1.529 579 | X-E2,1.529 580 | X-E2S,1.529 581 | X-T1,1.529 582 | X-T2,1.528 583 | X-M1,1.529 584 | X-A1,1.529 585 | X-A2,1.529 586 | X-T10,1.529 587 | X-T20,1.529 588 | Nikon 1 S1,2.727 589 | 1 S1,2.727 590 | Nikon 1 V1,2.727 591 | 1 V1,2.727 592 | Nikon 1 J1,2.727 593 | 1 J1,2.727 594 | Nikon 1 AW1,2.727 595 | 1 AW1,2.727 596 | Nikon 1 S2,2.727 597 | 1 S2,2.727 598 | Nikon 1 V2,2.727 599 | 1 V2,2.727 600 | Nikon 1 J2,2.727 601 | 1 J2,2.727 602 | Nikon 1 V3,2.727 603 | 1 V3,2.727 604 | Nikon 1 J3,2.727 605 | 1 J3,2.727 606 | Nikon 1 J4,2.727 607 | 1 J4,2.727 608 | Nikon 1 J5,2.727 609 | 1 J5,2.727 610 | E-P1,2 611 | E-P2,2 612 | E-P3,2 613 | E-P5,2 614 | E-PL1,2 615 | E-PL1s,2 616 | E-PL2,2 617 | E-PL3,2 618 | E-PL5,2 619 | E-PL6,2 620 | E-PL7,2 621 | E-PL8,2 622 | E-PM1,2 623 | E-PM2,2 624 | E-M1,2 625 | E-M1MarkII,2 626 | E-M5,2 627 | E-M5MarkII,2 628 | E-M10,2 629 | E-M10MarkII,2 630 | E-M10 II,2 631 | E-M10 Mark III,2 632 | E-M10 III,2 633 | PEN-F,2 634 | DMC-G1,2 635 | DMC-GF1,2 636 | DMC-GH1,2 637 | DMC-GM1,2 638 | DMC-GX1,2 639 | DMC-G2,2 640 | DMC-GF2,2 641 | DMC-GH2,2 642 | DMC-G3,2 643 | DMC-GF3,2 644 | DMC-GH3,2 645 | DMC-GH4,2 646 | DC-GH5,2 647 | DMC-G5,2 648 | DMC-GF5,2 649 | DMC-G6,2 650 | DMC-GF6,2 651 | DMC-G7,2 652 | DMC-GF7,2 653 | DMC-GX7,2 654 | DMC-G10,2 655 | DMC-GM5,2 656 | DMC-GX8,2 657 | DMC-G80,2 658 | DMC-G81,2 659 | DMC-G85,2 660 | DMC-GX80,2 661 | DMC-GX85,2 662 | Pentax Q,5.53 663 | Pentax Q7,4.6 664 | Pentax Q10,5.53 665 | EK-GN120,1.531 666 | Galaxy NX,1.531 667 | NX10,1.525 668 | NX11,1.525 669 | NX100,1.525 670 | NX1000,1.525 671 | NX20,1.525 672 | NX200,1.525 673 | NX210,1.525 674 | NX300,1.531 675 | NX300M,1.531 676 | NX500,1.531 677 | NX1100,1.531 678 | NX2000,1.531 679 | NX3000,1.531 680 | NX1,1.531 681 | NX5,1.525 682 | NX30,1.531 683 | NX mini,2.727 684 | sd Quattro,1.523 685 | NEX-3,1.534 686 | NEX-C3,1.534 687 | NEX-F3,1.534 688 | NEX-3N,1.534 689 | NEX-5,1.534 690 | NEX-5N,1.534 691 | NEX-5R,1.534 692 | NEX-5T,1.534 693 | NEX-6,1.534 694 | NEX-7,1.534 695 | ILCE-6000,1.534 696 | Alpha 6000,1.534 697 | ILCE-3000,1.534 698 | Alpha 3000,1.534 699 | ILCE-5000,1.534 700 | Alpha 5000,1.534 701 | ILCE-5100,1.534 702 | Alpha 5100,1.534 703 | ILCE-7,1 704 | Alpha 7,1 705 | ILCE-7R,1 706 | Alpha 7R,1 707 | ILCE-7RM2,1 708 | Alpha 7R II,1 709 | ILCE-7S,1 710 | Alpha 7S,1 711 | ILCE-7M2,1 712 | Alpha 7 II,1 713 | ILCE-6300,1.534 714 | Alpha 6300,1.534 715 | ILCE-6500,1.534 716 | Alpha 6500,1.534 717 | ILCE-9,1 718 | Alpha 9,1 719 | Lumia 1020,3.93 720 | Lumia 950,6.02 721 | Lumia 950 XL,6.02 722 | R-D1,1.525 723 | LG-H815,6.34 724 | LG G4,6.34 725 | Lumia 1520,6.26 726 | Leica M (Typ 240),1 727 | Canon EOS 6D,1.005 728 | EOS 6D,1.005 729 | Canon EOS 6D Mark II,1 730 | EOS 6D Mark II,1 731 | Canon EOS 5D Mark IV,1 732 | EOS 5D Mark IV,1 733 | Canon EOS 5D Mark III,1 734 | EOS 5D Mark III,1 735 | Canon EOS 5D Mark II,1 736 | EOS 5D Mark II,1 737 | Canon EOS 5D,1 738 | EOS 5D,1 739 | Canon EOS-1Ds,1 740 | EOS-1Ds,1 741 | Canon EOS-1Ds Mark II,1 742 | EOS-1Ds Mark II,1 743 | Canon EOS-1D Mark II,1.255 744 | EOS-1D Mark II,1.255 745 | Canon EOS-1D Mark II N,1.255 746 | EOS-1D Mark II N,1.255 747 | Canon EOS-1Ds Mark III,1 748 | EOS-1Ds Mark III,1 749 | Canon EOS-1D Mark III,1.282 750 | EOS-1D Mark III,1.282 751 | Canon EOS-1D Mark IV,1.290 752 | EOS-1D Mark IV,1.290 753 | Canon EOS-1D,1.255 754 | EOS-1D,1.255 755 | Canon EOS-1D X,1 756 | EOS-1D X,1 757 | Canon EOS-1D X Mark II,1 758 | EOS-1D X Mark II,1 759 | Canon EOS 1000D,1.622 760 | EOS 1000D,1.622 761 | Canon EOS DIGITAL REBEL XS,1.622 762 | EOS Digital Rebel XS,1.622 763 | Canon EOS Kiss Digital F,1.622 764 | EOS Kiss Digital F,1.622 765 | Canon EOS 1100D,1.620 766 | EOS 1100D,1.620 767 | Canon EOS REBEL T3,1.620 768 | EOS Rebel T3,1.620 769 | Canon EOS Kiss X50,1.620 770 | EOS Kiss X50,1.620 771 | Canon EOS 300D DIGITAL,1.587 772 | EOS 300D,1.587 773 | Canon EOS DIGITAL REBEL,1.587 774 | EOS Digital REBEL,1.587 775 | Canon EOS Kiss Digital,1.587 776 | EOS Kiss Digital,1.587 777 | Canon EOS 350D DIGITAL,1.622 778 | EOS 350D,1.622 779 | Canon EOS DIGITAL REBEL XT,1.622 780 | EOS Digital Rebel XT,1.622 781 | Canon EOS Kiss Digital N,1.622 782 | EOS Kiss Digital N,1.622 783 | Canon EOS 400D DIGITAL,1.622 784 | EOS 400D,1.622 785 | Canon EOS DIGITAL REBEL XTi,1.622 786 | EOS Digital Rebel XTi,1.622 787 | Canon EOS Kiss Digital X,1.622 788 | EOS Kiss Digital X,1.622 789 | Canon EOS 450D,1.622 790 | EOS 450D,1.622 791 | Canon EOS DIGITAL REBEL XSi,1.622 792 | EOS Digital Rebel XSi,1.622 793 | Canon EOS Kiss Digital X2,1.622 794 | EOS Kiss Digital X2,1.622 795 | Canon EOS 500D,1.613 796 | EOS 500D,1.613 797 | Canon EOS REBEL T1i,1.613 798 | EOS Rebel T1i,1.613 799 | Canon EOS Kiss X3,1.613 800 | EOS Kiss X3,1.613 801 | Canon EOS 550D,1.613 802 | EOS 550D,1.613 803 | Canon EOS REBEL T2i,1.613 804 | EOS Rebel T2i,1.613 805 | Canon EOS Kiss X4,1.613 806 | EOS Kiss X4,1.613 807 | Canon EOS 600D,1.613 808 | EOS 600D,1.613 809 | Canon EOS REBEL T3i,1.613 810 | EOS Rebel T3i,1.613 811 | Canon EOS Kiss X5,1.613 812 | EOS Kiss X5,1.613 813 | Canon EOS 650D,1.613 814 | EOS 650D,1.613 815 | Canon EOS REBEL T4i,1.613 816 | EOS Rebel T4i,1.613 817 | Canon EOS Kiss X6i,1.613 818 | EOS Kiss X6i,1.613 819 | Canon EOS 700D,1.613 820 | EOS 700D,1.613 821 | Canon EOS REBEL T5i,1.613 822 | EOS Rebel T5i,1.613 823 | Canon EOS 800D,1.613 824 | EOS 800D,1.613 825 | Canon EOS REBEL T7i,1.613 826 | EOS Rebel T7i,1.613 827 | Canon EOS Kiss X7i,1.613 828 | EOS Kiss X7i,1.613 829 | Canon EOS 750D,1.613 830 | EOS 750D,1.613 831 | Canon EOS Rebel T6i,1.613 832 | EOS Rebel T6i,1.613 833 | Canon EOS 760D,1.613 834 | EOS 760D,1.613 835 | Canon EOS 1200D,1.620 836 | EOS 1200D,1.620 837 | Canon EOS REBEL T5,1.620 838 | EOS Rebel T5,1.620 839 | Canon EOS Kiss X70,1.620 840 | EOS Kiss X70,1.620 841 | Canon EOS 1300D,1.620 842 | EOS 1300D,1.620 843 | Canon EOS Rebel T6,1.620 844 | EOS Rebel T6,1.620 845 | Canon EOS 2000D,1.613 846 | EOS 2000D,1.613 847 | Canon EOS Kiss X90,1.613 848 | EOS Kiss X90,1.613 849 | Canon EOS Rebel T7,1.613 850 | EOS Rebel T7,1.613 851 | Canon EOS 4000D,1.613 852 | EOS 4000D,1.613 853 | Canon EOS Rebel T100,1.613 854 | EOS Rebel T100,1.613 855 | Canon EOS 100D,1.613 856 | EOS 100D,1.613 857 | Canon EOS REBEL SL1,1.613 858 | EOS Rebel SL1,1.613 859 | Canon EOS Kiss X7,1.613 860 | EOS Kiss X7,1.613 861 | Canon EOS 200D,1.613 862 | EOS 200D,1.613 863 | Canon EOS REBEL SL2,1.613 864 | EOS Rebel SL2,1.613 865 | Canon EOS Kiss X9,1.613 866 | EOS Kiss X9,1.613 867 | Canon EOS 7D,1.620 868 | EOS 7D,1.620 869 | Canon EOS 7D Mark II,1.605 870 | EOS 7D Mark II,1.605 871 | Canon EOS 80D,1.613 872 | EOS 80D,1.613 873 | Canon EOS 77D,1.613 874 | EOS 77D,1.613 875 | Canon EOS 70D,1.600 876 | EOS 70D,1.600 877 | Canon EOS 60D,1.620 878 | EOS 60D,1.620 879 | Canon EOS 50D,1.622 880 | EOS 50D,1.622 881 | Canon EOS 40D,1.622 882 | EOS 40D,1.622 883 | Canon EOS 30D,1.600 884 | EOS 30D,1.600 885 | Canon EOS 20D,1.600 886 | EOS 20D,1.600 887 | Canon EOS 10D,1.587 888 | EOS 10D,1.587 889 | Canon EOS D60,1.587 890 | EOS D60,1.587 891 | Canon EOS D30,1.587 892 | EOS D30,1.587 893 | DCS Pro SLR/c,1 894 | DCS520,1.593 895 | DCS 520,1.593 896 | EOS D2000,1.593 897 | Hasselblad 500 mech.,0.66 898 | Hasselblad H3D,0.72 899 | Maxxum 5D,1.531 900 | Dynax 5D,1.531 901 | Maxxum 7D,1.522 902 | Dynax 7D,1.522 903 | Nikon D40,1.525 904 | D40,1.525 905 | Nikon D40X,1.523 906 | D40X,1.523 907 | Nikon D50,1.525 908 | D50,1.525 909 | Nikon D60,1.523 910 | D60,1.523 911 | Nikon D70,1.525 912 | D70,1.525 913 | Nikon D70s,1.525 914 | D70s,1.525 915 | Nikon D80,1.523 916 | D80,1.523 917 | Nikon D90,1.523 918 | D90,1.523 919 | Nikon D100,1.525 920 | D100,1.525 921 | Nikon D200,1.523 922 | D200,1.523 923 | Nikon D300,1.523 924 | D300,1.523 925 | Nikon D300S,1.523 926 | D300S,1.523 927 | Nikon D500,1.531 928 | D500,1.531 929 | Nikon D600,1 930 | D600,1 931 | Nikon D610,1 932 | D610,1 933 | Nikon D700,1 934 | D700,1 935 | Nikon D750,1 936 | D750,1 937 | Nikon D800,1 938 | D800,1 939 | Nikon D800E,1 940 | D800E,1 941 | Nikon D810,1 942 | D810,1 943 | Nikon D850,1 944 | D850,1 945 | Nikon D3000,1.523 946 | D3000,1.523 947 | Nikon D3100,1.558 948 | D3100,1.558 949 | Nikon D3200,1.554 950 | D3200,1.554 951 | Nikon D3300,1.523 952 | D3300,1.523 953 | Nikon D3400,1.523 954 | D3400,1.523 955 | Nikon D5000,1.523 956 | D5000,1.523 957 | Nikon D5100,1.523 958 | D5100,1.523 959 | Nikon D5200,1.534 960 | D5200,1.534 961 | Nikon D5300,1.534 962 | D5300,1.534 963 | Nikon D5500,1.534 964 | D5500,1.534 965 | Nikon D5600,1.534 966 | D5600,1.534 967 | Nikon D7000,1.523 968 | D7000,1.523 969 | Nikon D7100,1.534 970 | D7100,1.534 971 | Nikon D7200,1.534 972 | D7200,1.534 973 | Nikon D7500,1.534 974 | D7500,1.534 975 | Nikon D1,1.525 976 | D1,1.525 977 | Nikon D1H,1.525 978 | D1H,1.525 979 | Nikon D1X,1.525 980 | D1X,1.525 981 | Nikon D2H,1.546 982 | D2H,1.546 983 | Nikon D2Hs,1.546 984 | D2Hs,1.546 985 | Nikon D2X,1.522 986 | D2X,1.522 987 | Nikon D2Xs,1.522 988 | D2Xs,1.522 989 | Nikon D3,1 990 | D3,1 991 | Nikon D3X,1 992 | D3X,1 993 | Nikon D3S,1 994 | D3S,1 995 | Nikon D4,1 996 | D4,1 997 | Nikon D4s,1 998 | D4s,1 999 | Nikon D5,1.003 1000 | D5,1.003 1001 | Nikon Df,1.001 1002 | Df,1.001 1003 | FinePixS1Pro,1.543 1004 | FinePix S1 Pro,1.543 1005 | FinePixS2Pro,1.543 1006 | FinePix S2 Pro,1.543 1007 | FinePix S3Pro,1.534 1008 | FinePix S3 Pro,1.534 1009 | FinePix S5Pro,1.560 1010 | FinePix S5 Pro,1.560 1011 | IS Pro,1.560 1012 | FinePix IS Pro,1.560 1013 | DCS Pro SLR/n,1 1014 | DCS Pro 14nx,1 1015 | DCS Pro 14N,1 1016 | E-300,2 1017 | E-330,2 1018 | E-400,2 1019 | E-410,2 1020 | E-420,2 1021 | E-450,2 1022 | E-500,2 1023 | E-510,2 1024 | E-520,2 1025 | E-600,2 1026 | E-620,2 1027 | E-10,3.933 1028 | "E-20,E-20N,E-20P",3.933 1029 | "E-20, E-20N, E-20P",3.933 1030 | E-30,2 1031 | E-1,2 1032 | E-3,2 1033 | E-5,2 1034 | DMC-L1,2 1035 | DMC-L10,2 1036 | Pentax *ist DL2,1.531 1037 | *ist DL2,1.531 1038 | Pentax *ist DL,1.531 1039 | *ist DL,1.531 1040 | Pentax *ist DS2,1.531 1041 | *ist DS2,1.531 1042 | Pentax *ist DS,1.531 1043 | *ist DS,1.531 1044 | Pentax *ist D,1.531 1045 | *ist D,1.531 1046 | Pentax K-m,1.531 1047 | K-m,1.531 1048 | Pentax K100D,1.531 1049 | K100D,1.531 1050 | Pentax K100D Super,1.531 1051 | K100D Super,1.531 1052 | Pentax K110D,1.531 1053 | K110D,1.531 1054 | Pentax K10D,1.531 1055 | K10D,1.531 1056 | Pentax K20D,1.538 1057 | K20D,1.538 1058 | Pentax K-30,1.526 1059 | K-30,1.526 1060 | Pentax K-50,1.526 1061 | K-50,1.526 1062 | Pentax K-500,1.534 1063 | K-500,1.534 1064 | Pentax K-7,1.538 1065 | K-7,1.538 1066 | Pentax K-5,1.526 1067 | K-5,1.526 1068 | Pentax K-5 II,1.526 1069 | K-5 II,1.526 1070 | Pentax K-5 II s,1.526 1071 | K-5 II s,1.526 1072 | Pentax K-3,1.534 1073 | K-3,1.534 1074 | Pentax K-3 II,1.534 1075 | K-3 II,1.534 1076 | Pentax K-1,1 1077 | K-1,1 1078 | Pentax K2000,1.531 1079 | K2000,1.531 1080 | Pentax K200D,1.531 1081 | K200D,1.531 1082 | Pentax K-x,1.531 1083 | K-x,1.531 1084 | Pentax K-r,1.523 1085 | K-r,1.523 1086 | Pentax K-S2,1.534 1087 | K-S2,1.534 1088 | Pentax K-S1,1.534 1089 | K-S1,1.534 1090 | Pentax K-70,1.534 1091 | K-70,1.534 1092 | Pentax K-01,1.522 1093 | K-01,1.522 1094 | Pentax KP,1.534 1095 | KP,1.534 1096 | SAMSUNG GX10,1.531 1097 | GX10,1.531 1098 | GX20,1.538 1099 | GX-1S,1.531 1100 | Sigma SD1,1.500 1101 | SD1,1.500 1102 | Sigma SD1 Merrill,1.500 1103 | SD1 Merrill,1.500 1104 | Sigma SD9,1.739 1105 | SD9,1.739 1106 | Sigma SD10,1.739 1107 | SD10,1.739 1108 | Sigma SD14,1.739 1109 | SD14,1.739 1110 | Sigma SD15,1.739 1111 | SD15,1.739 1112 | SLT-A33,1.523 1113 | Alpha 33,1.523 1114 | SLT-A35,1.523 1115 | Alpha 35,1.523 1116 | SLT-A37,1.523 1117 | Alpha 37,1.523 1118 | SLT-A55V,1.523 1119 | Alpha 55,1.523 1120 | SLT-A57,1.523 1121 | Alpha 57,1.523 1122 | SLT-A58,1.523 1123 | Alpha 58,1.523 1124 | SLT-A65V,1.523 1125 | Alpha 65,1.523 1126 | ILCA-68,1.534 1127 | Alpha 68,1.534 1128 | SLT-A77V,1.534 1129 | Alpha 77,1.534 1130 | ILCA-77M2,1.534 1131 | Alpha 77 II,1.534 1132 | SLT-A99,1.005 1133 | Alpha 99,1.005 1134 | SLT-A99V,1.005 1135 | Alpha 99V,1.005 1136 | DSLR-A100,1.523 1137 | Alpha 100,1.523 1138 | DSLR-A200,1.523 1139 | Alpha 200,1.523 1140 | DSLR-A230,1.523 1141 | Alpha 230,1.523 1142 | DSLR-A290,1.523 1143 | Alpha 290,1.523 1144 | DSLR-A300,1.534 1145 | Alpha 300,1.534 1146 | DSLR-A330,1.534 1147 | Alpha 330,1.534 1148 | DSLR-A350,1.534 1149 | Alpha 350,1.534 1150 | DSLR-A380,1.528 1151 | Alpha 380,1.528 1152 | DSLR-A390,1.523 1153 | Alpha 390,1.523 1154 | DSLR-A450,1.534 1155 | Alpha 450,1.534 1156 | DSLR-A500,1.534 1157 | Alpha 500,1.534 1158 | DSLR-A550,1.534 1159 | Alpha 550,1.534 1160 | DSLR-A560,1.523 1161 | Alpha 560,1.523 1162 | DSLR-A580,1.523 1163 | Alpha 580,1.523 1164 | DSLR-A700,1.534 1165 | Alpha 700,1.534 1166 | DSLR-A850,1 1167 | Alpha 850,1 1168 | DSLR-A900,1 1169 | Alpha 900,1 1170 | ILCA-99M2,1 1171 | Alpha 99 II,1 1172 | Zenit 122,1 1173 | Зенит 122,1 1174 | Zenit 122K,1 1175 | Зенит 122K,1 1176 | Zenit 212K,1 1177 | Зенит 212K,1 1178 | Zenit 312K,1 1179 | Зенит 312K,1 1180 | Zenit 412LS,1 1181 | Зенит 412LS,1 1182 | Zenit KM,1 1183 | Зенит KM,1 1184 | -------------------------------------------------------------------------------- /fdlite/data/camdb.txt: -------------------------------------------------------------------------------- 1 | Crop Factor database was extracted from 2 | https://github.com/lensfun/lensfun 3 | 4 | "model" and "cropfactor" where extracted from the XML files and stored as CSV. 5 | 6 | The lens database is licensed under the 7 | Creative Commons Attribution-Share Alike 3.0 license (CC BY-SA 3.0). 8 | -------------------------------------------------------------------------------- /fdlite/data/face_detection_back.tflite: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/fdlite/data/face_detection_back.tflite -------------------------------------------------------------------------------- /fdlite/data/face_detection_front.tflite: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/fdlite/data/face_detection_front.tflite -------------------------------------------------------------------------------- /fdlite/data/face_detection_full_range.tflite: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/fdlite/data/face_detection_full_range.tflite -------------------------------------------------------------------------------- /fdlite/data/face_detection_full_range_sparse.tflite: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/fdlite/data/face_detection_full_range_sparse.tflite -------------------------------------------------------------------------------- /fdlite/data/face_detection_short_range.tflite: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/fdlite/data/face_detection_short_range.tflite -------------------------------------------------------------------------------- /fdlite/data/face_landmark.tflite: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/fdlite/data/face_landmark.tflite -------------------------------------------------------------------------------- /fdlite/data/iris_landmark.tflite: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/fdlite/data/iris_landmark.tflite -------------------------------------------------------------------------------- /fdlite/errors.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright © 2021 Patrick Levin 3 | # SPDX-Identifier: MIT 4 | """This module contains all custom excpetions used by the library""" 5 | 6 | 7 | class InvalidEnumError(Exception): 8 | """Raised when a function was called with an invalid Enum value""" 9 | 10 | 11 | class ModelDataError(Exception): 12 | """Raised when a model returns data that is incompatible""" 13 | 14 | 15 | class CoordinateRangeError(Exception): 16 | """Raised when coordinates are expected to be in a different range""" 17 | 18 | 19 | class ArgumentError(Exception): 20 | """Raised when an argument is of the wrong type or malformed""" 21 | 22 | 23 | class MissingExifDataError(Exception): 24 | """Raised if required EXIF data is missing from an image""" 25 | -------------------------------------------------------------------------------- /fdlite/examples/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patlevin/face-detection-tflite/17e567dc0b7acc77acb857e7a3b08793062822a3/fdlite/examples/__init__.py -------------------------------------------------------------------------------- /fdlite/examples/iris_recoloring.py: -------------------------------------------------------------------------------- 1 | from bisect import bisect_left, bisect_right 2 | from os import PathLike 3 | import numpy as np 4 | from PIL import Image, ImageOps 5 | from PIL.Image import Image as PILImage 6 | import sys 7 | from typing import Sequence, Tuple, Union 8 | from fdlite.face_detection import FaceDetection, FaceDetectionModel 9 | from fdlite.face_landmark import FaceLandmark, face_detection_to_roi 10 | from fdlite.iris_landmark import IrisLandmark, IrisResults 11 | from fdlite.iris_landmark import iris_roi_from_face_landmarks 12 | from fdlite.transform import bbox_from_landmarks 13 | """Iris recoloring example based on face- and iris detection models. 14 | """ 15 | 16 | _Point = Tuple[int, int] 17 | _Size = Tuple[int, int] 18 | _Rect = Tuple[int, int, int, int] 19 | 20 | 21 | def recolor_iris( 22 | image: PILImage, 23 | iris_results: IrisResults, 24 | iris_color: Tuple[int, int, int] 25 | ) -> PILImage: 26 | """Colorize an eye. 27 | 28 | Args: 29 | image (Image): PIL image instance containing a face. 30 | 31 | iris_results (IrisResults): Iris detection results. 32 | 33 | iris_color (tuple): Tuple of `(red, green, blue)` representing the 34 | new iris color to apply. Color values must be integers in the 35 | range [0, 255]. 36 | 37 | Returns: 38 | (Image) The function returns the modified PIL image instance. 39 | """ 40 | iris_location, iris_size = _get_iris_location(iris_results, image.size) 41 | # nothing fancy - just grab the iris part as an Image and work with that 42 | eye_image = image.transform(iris_size, Image.EXTENT, data=iris_location) 43 | eye_image = eye_image.convert(mode='L') 44 | eye_image = ImageOps.colorize(eye_image, 'black', 'white', mid=iris_color) 45 | # build a mask for copying back into the original image 46 | # no fancy anti-aliasing or blending, though 47 | mask = _get_iris_mask(iris_results, iris_location, iris_size, image.size) 48 | image.paste(eye_image, iris_location, mask) 49 | return image 50 | 51 | 52 | def _get_iris_location( 53 | results: IrisResults, image_size: _Size 54 | ) -> Tuple[_Rect, _Size]: 55 | """Return iris location and -size""" 56 | bbox = bbox_from_landmarks(results.iris).absolute(image_size) 57 | width, height = int(bbox.width + 1), int(bbox.height + 1) 58 | size = (width, height) 59 | left, top = int(bbox.xmin), int(bbox.ymin) 60 | location = (left, top, left + width, top + height) 61 | return location, size 62 | 63 | 64 | def _get_iris_mask( 65 | results: IrisResults, 66 | iris_location: _Rect, 67 | iris_size: _Size, 68 | image_size: _Size 69 | ) -> PILImage: 70 | """Return a mask for the visible portion of the iris inside eye landmarks. 71 | """ 72 | left, top, _, bottom = iris_location 73 | iris_width, iris_height = iris_size 74 | img_width, img_height = image_size 75 | # sort lexicographically by x then y 76 | eyeball_sorted = sorted([(int(pt.x * img_width), int(pt.y * img_height)) 77 | for pt in results.eyeball_contour]) 78 | bbox = bbox_from_landmarks(results.eyeball_contour).absolute(image_size) 79 | x_ofs = left 80 | y_ofs = top 81 | y_start = int(max(bbox.ymin, top)) 82 | y_end = int(min(bbox.ymax, bottom)) 83 | mask = np.zeros((iris_height, iris_width), dtype=np.uint8) 84 | # iris ellipse radii (horizontal and vertical radius) 85 | a = iris_width // 2 86 | b = iris_height // 2 87 | # iris ellipse foci (horizontal and vertical) 88 | cx = left + a 89 | cy = top + b 90 | box_center_y = int(bbox.ymin + bbox.ymax) // 2 91 | b_sqr = b**2 92 | for y in range(y_start, y_end): 93 | # evaluate iris ellipse at y 94 | x = int(a * np.math.sqrt(b_sqr - (y-cy)**2) / b) 95 | x0, x1 = cx - x, cx + x 96 | A, B = _find_contour_segment(eyeball_sorted, (x0, y)) 97 | left_inside = _is_below_segment(A, B, (x0, y), box_center_y) 98 | C, D = _find_contour_segment(eyeball_sorted, (x1, y)) 99 | right_inside = _is_below_segment(C, D, (x1, y), box_center_y) 100 | if not (left_inside or right_inside): 101 | continue 102 | elif not left_inside: 103 | x0 = int(max((B[0] - A[0])/(B[1] - A[1]) * (y - A[1]) + A[0], x0)) 104 | elif not right_inside: 105 | x1 = int(min((D[0] - C[0])/(D[1] - C[1]) * (y - C[1]) + C[0], x1)) 106 | # mark ellipse row as visible 107 | mask[(y - y_ofs), int(x0 - x_ofs):int(x1 - x_ofs)] = 255 108 | return Image.fromarray(mask, mode='L') 109 | 110 | 111 | def _is_below_segment(A: _Point, B: _Point, C: _Point, mid: int) -> bool: 112 | """Return whether a point is below the line AB""" 113 | dx = B[0] - A[0] 114 | dy = B[1] - A[1] 115 | if not dx: 116 | # vertical line: check if the point is on the line 117 | return A[1] <= C[1] and B[1] >= C[1] 118 | # x -> [0, 1] 119 | x = (C[0] - A[0]) / dx 120 | m = dy / dx 121 | y = x * m + A[1] 122 | # flip the sign if the leftmost point is below the threshold 123 | sign = -1 if A[1] > mid else 1 124 | return sign * (C[1] - y) > 0 125 | 126 | 127 | def _find_contour_segment( 128 | contour: Sequence[_Point], point: _Point 129 | ) -> Tuple[_Point, _Point]: 130 | """Find contour segment (points A and B) that contains the point. 131 | (contour must be lexicographically sorted!) 132 | """ 133 | def distance(a: _Point, b: _Point) -> int: 134 | return (a[0] - b[0]) ** 2 + (a[1] - b[1]) ** 2 135 | 136 | MAX_IDX = len(contour)-1 137 | # contour point left of point 138 | left_idx = max(bisect_left(contour, point) - 1, 0) 139 | right_idx = min(bisect_right(contour, point), MAX_IDX) 140 | d = distance(point, contour[left_idx]) 141 | # try to find a closer left point 142 | while left_idx > 0 and d > distance(point, contour[left_idx - 1]): 143 | left_idx -= 1 144 | d = distance(point, contour[left_idx - 1]) 145 | # try to find a closer right point 146 | d = distance(point, contour[right_idx]) 147 | while right_idx < MAX_IDX and d > distance(point, contour[right_idx + 1]): 148 | right_idx += 1 149 | d = distance(point, contour[right_idx + 1]) 150 | return (contour[left_idx], contour[right_idx]) 151 | 152 | 153 | def main(image_file: Union[str, PathLike]) -> None: 154 | EXCITING_NEW_EYE_COLOR = (161, 52, 216) 155 | img = Image.open(image_file) 156 | # run the detection pipeline 157 | # Step 1: detect the face and get a proper ROI for further processing 158 | face_detection = FaceDetection(FaceDetectionModel.BACK_CAMERA) 159 | detections = face_detection(img) 160 | if not len(detections): 161 | print('No face detected :(') 162 | exit(0) 163 | face_roi = face_detection_to_roi(detections[0], img.size) 164 | 165 | # Step 2: detect face landmarks and extract per-eye ROIs from them 166 | face_landmarks = FaceLandmark() 167 | landmarks = face_landmarks(img, face_roi) 168 | eyes_roi = iris_roi_from_face_landmarks(landmarks, img.size) 169 | 170 | # Step 3: perform iris detection to get detailed landmarksfor each eye 171 | iris_landmarks = IrisLandmark() 172 | left_eye_roi, right_eye_roi = eyes_roi 173 | left_eye_results = iris_landmarks(img, left_eye_roi) 174 | right_eye_results = iris_landmarks(img, right_eye_roi, is_right_eye=True) 175 | 176 | # Step 4: apply new eye and exciting eye color 177 | recolor_iris(img, left_eye_results, iris_color=EXCITING_NEW_EYE_COLOR) 178 | recolor_iris(img, right_eye_results, iris_color=EXCITING_NEW_EYE_COLOR) 179 | 180 | # Step 5: Profit! ...or just show the result ¯\_(ツ)_/¯ 181 | img.show() 182 | 183 | 184 | if __name__ == '__main__': 185 | if len(sys.argv) != 2: 186 | print('pass an image name as argument') 187 | else: 188 | main(sys.argv[1]) 189 | -------------------------------------------------------------------------------- /fdlite/exif.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright © 2021 Patrick Levin 3 | # SPDX-Identifier: MIT 4 | u"""EXIF utilities for extracting focal length for distance calculation. 5 | 6 | The module contains the function `get_focal_length` to extract focal length 7 | data from image EXIF data. 8 | 9 | Missing 35mm focal length data (usually provided by smartphone cameras) will 10 | be calculated from the crop factor (or focal length multiplier). The crop 11 | factor is read from a database of camera models. The function loads the 12 | database lazily, e.g. if input data always contains the required information, 13 | no data will ever be loaded. 14 | """ 15 | import csv 16 | import os 17 | from enum import IntEnum 18 | from typing import Dict, Optional, Tuple 19 | from PIL.Image import Image 20 | 21 | 22 | DATABASE_NAME = 'camdb.csv' 23 | _MODEL_DATABASE: Dict[str, float] = {} 24 | 25 | 26 | class ExifTag(IntEnum): 27 | """EXIF tag indexes: see https://exiftool.org/TagNames/EXIF.html""" 28 | MODEL = 272 29 | ORIENTATION = 274 30 | FOCAL_LENGTH_IN_MM = 37386 31 | PIXEL_WIDTH = 40962 32 | PIXEL_HEIGHT = 40963 33 | FOCAL_LENGTH_35MM = 41989 34 | 35 | 36 | def get_focal_length(image: Image) -> Optional[Tuple[int, int, int, int]]: 37 | u"""Extract focal length data from EXIF data. 38 | 39 | The function will try to calculate missing data (e.g. focal length in 40 | 35mm) using a camera model database. The database will be loaded once 41 | it is first required. 42 | 43 | Args: 44 | image (Image): PIL image instance to get EXIF data from. 45 | 46 | Returns: 47 | (tuple) Tuple of `(focal_length_35mm, focal_length, width, height)` 48 | for use with `iris_depth_in_mm_from_landmarks`. `None` is returned 49 | if EXIF data is missing or couldn't be calculated (e.g. camera model 50 | missing or not in database). 51 | """ 52 | exif = image.getexif() 53 | if ExifTag.FOCAL_LENGTH_IN_MM not in exif: 54 | # no focal length 55 | return None 56 | if ExifTag.PIXEL_WIDTH not in exif or ExifTag.PIXEL_HEIGHT not in exif: 57 | width_px, height_px = image.size 58 | else: 59 | width_px = exif[ExifTag.PIXEL_WIDTH] 60 | height_px = exif[ExifTag.PIXEL_HEIGHT] 61 | # swap width and height if orientation is rotate 90° or 270° 62 | if ExifTag.ORIENTATION in exif and exif[ExifTag.ORIENTATION] > 4: 63 | width_px, height_px = height_px, width_px 64 | focal_length_in_mm = exif[ExifTag.FOCAL_LENGTH_IN_MM] 65 | if ExifTag.FOCAL_LENGTH_35MM in exif: 66 | focal_length_35mm = exif[ExifTag.FOCAL_LENGTH_35MM] 67 | else: 68 | model = exif[ExifTag.MODEL] if ExifTag.MODEL in exif else None 69 | if model is not None and len(_MODEL_DATABASE) == 0: 70 | _load_database() 71 | if model not in _MODEL_DATABASE: 72 | return None 73 | crop_factor = _MODEL_DATABASE[model] 74 | focal_length_35mm = round(focal_length_in_mm * crop_factor) 75 | return focal_length_35mm, focal_length_in_mm, width_px, height_px 76 | 77 | 78 | def _load_database(): 79 | """Load camera model names and associated crop factor""" 80 | base_path = os.path.dirname(os.path.abspath(__file__)) 81 | database_path = os.path.join(base_path, '../data', DATABASE_NAME) 82 | with open(database_path, 'r', encoding='utf8') as csv_file: 83 | reader = csv.reader(csv_file) 84 | for model, crop_factor in reader: 85 | _MODEL_DATABASE[model] = float(crop_factor) 86 | -------------------------------------------------------------------------------- /fdlite/face_detection.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright © 2021 Patrick Levin 3 | # SPDX-Identifier: MIT 4 | u"""BlazeFace face detection. 5 | 6 | Ported from Google® MediaPipe (https://google.github.io/mediapipe/). 7 | 8 | Model card: 9 | 10 | https://mediapipe.page.link/blazeface-mc 11 | 12 | Reference: 13 | 14 | V. Bazarevsky et al. BlazeFace: Sub-millisecond 15 | Neural Face Detection on Mobile GPUs. CVPR 16 | Workshop on Computer Vision for Augmented and 17 | Virtual Reality, Long Beach, CA, USA, 2019. 18 | """ 19 | import numpy as np 20 | import os 21 | import tensorflow as tf 22 | from enum import IntEnum 23 | from PIL.Image import Image 24 | from typing import List, Optional, Union 25 | from fdlite import InvalidEnumError 26 | from fdlite.nms import non_maximum_suppression 27 | from fdlite.transform import detection_letterbox_removal, image_to_tensor 28 | from fdlite.transform import sigmoid 29 | from fdlite.types import Detection, Rect 30 | 31 | MODEL_NAME_BACK = 'face_detection_back.tflite' 32 | MODEL_NAME_FRONT = 'face_detection_front.tflite' 33 | MODEL_NAME_SHORT = 'face_detection_short_range.tflite' 34 | MODEL_NAME_FULL = 'face_detection_full_range.tflite' 35 | MODEL_NAME_FULL_SPARSE = 'face_detection_full_range_sparse.tflite' 36 | 37 | # score limit is 100 in mediapipe and leads to overflows with IEEE 754 floats 38 | # this lower limit is safe for use with the sigmoid functions and float32 39 | RAW_SCORE_LIMIT = 80 40 | # threshold for confidence scores 41 | MIN_SCORE = 0.5 42 | # NMS similarity threshold 43 | MIN_SUPPRESSION_THRESHOLD = 0.3 44 | 45 | # from mediapipe module; irrelevant parts removed 46 | # (reference: mediapipe/modules/face_detection/face_detection_front_cpu.pbtxt) 47 | SSD_OPTIONS_FRONT = { 48 | 'num_layers': 4, 49 | 'input_size_height': 128, 50 | 'input_size_width': 128, 51 | 'anchor_offset_x': 0.5, 52 | 'anchor_offset_y': 0.5, 53 | 'strides': [8, 16, 16, 16], 54 | 'interpolated_scale_aspect_ratio': 1.0 55 | } 56 | 57 | # (reference: modules/face_detection/face_detection_back_desktop_live.pbtxt) 58 | SSD_OPTIONS_BACK = { 59 | 'num_layers': 4, 60 | 'input_size_height': 256, 61 | 'input_size_width': 256, 62 | 'anchor_offset_x': 0.5, 63 | 'anchor_offset_y': 0.5, 64 | 'strides': [16, 32, 32, 32], 65 | 'interpolated_scale_aspect_ratio': 1.0 66 | } 67 | 68 | # (reference: modules/face_detection/face_detection_short_range_common.pbtxt) 69 | SSD_OPTIONS_SHORT = { 70 | 'num_layers': 4, 71 | 'input_size_height': 128, 72 | 'input_size_width': 128, 73 | 'anchor_offset_x': 0.5, 74 | 'anchor_offset_y': 0.5, 75 | 'strides': [8, 16, 16, 16], 76 | 'interpolated_scale_aspect_ratio': 1.0 77 | } 78 | 79 | # (reference: modules/face_detection/face_detection_full_range_common.pbtxt) 80 | SSD_OPTIONS_FULL = { 81 | 'num_layers': 1, 82 | 'input_size_height': 192, 83 | 'input_size_width': 192, 84 | 'anchor_offset_x': 0.5, 85 | 'anchor_offset_y': 0.5, 86 | 'strides': [4], 87 | 'interpolated_scale_aspect_ratio': 0.0 88 | } 89 | 90 | 91 | class FaceIndex(IntEnum): 92 | """Indexes of keypoints returned by the face detection model. 93 | 94 | Use these with detection results (by indexing the result): 95 | ``` 96 | def get_left_eye_position(detection): 97 | x, y = detection[FaceIndex.LEFT_EYE] 98 | return x, y 99 | ``` 100 | """ 101 | LEFT_EYE = 0 102 | RIGHT_EYE = 1 103 | NOSE_TIP = 2 104 | MOUTH = 3 105 | LEFT_EYE_TRAGION = 4 106 | RIGHT_EYE_TRAGION = 5 107 | 108 | 109 | class FaceDetectionModel(IntEnum): 110 | """Face detection model option: 111 | 112 | FRONT_CAMERA - 128x128 image, assumed to be mirrored 113 | 114 | BACK_CAMERA - 256x256 image, not mirrored 115 | 116 | SHORT - 128x128 image, assumed to be mirrored; best for short range images 117 | (i.e. faces within 2 metres from the camera) 118 | 119 | FULL - 192x192 image, assumed to be mirrored; dense; best for mid-ranges 120 | (i.e. faces within 5 metres from the camera) 121 | 122 | FULL_SPARSE - 192x192 image, assumed to be mirrored; sparse; best for 123 | mid-ranges (i.e. faces within 5 metres from the camera) 124 | this model is up ~30% faster than `FULL` when run on the CPU 125 | """ 126 | FRONT_CAMERA = 0 127 | BACK_CAMERA = 1 128 | SHORT = 2 129 | FULL = 3 130 | FULL_SPARSE = 4 131 | 132 | 133 | class FaceDetection: 134 | """BlazeFace face detection model as used by Google MediaPipe. 135 | 136 | This model can detect multiple faces and returns a list of detections. 137 | Each detection contains the normalised [0,1] position and size of the 138 | detected face, as well as a number of keypoints (also normalised to 139 | [0,1]). 140 | 141 | The model is callable and accepts a PIL image instance, image file name, 142 | and Numpy array of shape (height, width, channels) as input. There is no 143 | size restriction, but smaller images are processed faster. 144 | 145 | Example: 146 | 147 | ``` 148 | detect_faces = FaceDetection(model_path='/var/mediapipe/models') 149 | detections = detect_faces('/home/user/pictures/group_photo.jpg') 150 | print(f'num. faces found: {len(detections)}') 151 | # convert normalised coordinates to pixels (assuming 3kx2k image): 152 | if len(detections): 153 | rect = detections[0].bbox.scale(3000, 2000) 154 | print(f'first face rect.: {rect}') 155 | else: 156 | print('no faces found') 157 | ``` 158 | 159 | Raises: 160 | InvalidEnumError: `model_type` contains an unsupported value 161 | """ 162 | def __init__( 163 | self, 164 | model_type: FaceDetectionModel = FaceDetectionModel.FRONT_CAMERA, 165 | model_path: Optional[str] = None 166 | ) -> None: 167 | ssd_opts = {} 168 | if model_path is None: 169 | my_path = os.path.abspath(__file__) 170 | model_path = os.path.join(os.path.dirname(my_path), 'data') 171 | if model_type == FaceDetectionModel.FRONT_CAMERA: 172 | self.model_path = os.path.join(model_path, MODEL_NAME_FRONT) 173 | ssd_opts = SSD_OPTIONS_FRONT 174 | elif model_type == FaceDetectionModel.BACK_CAMERA: 175 | self.model_path = os.path.join(model_path, MODEL_NAME_BACK) 176 | ssd_opts = SSD_OPTIONS_BACK 177 | elif model_type == FaceDetectionModel.SHORT: 178 | self.model_path = os.path.join(model_path, MODEL_NAME_SHORT) 179 | ssd_opts = SSD_OPTIONS_SHORT 180 | elif model_type == FaceDetectionModel.FULL: 181 | self.model_path = os.path.join(model_path, MODEL_NAME_FULL) 182 | ssd_opts = SSD_OPTIONS_FULL 183 | elif model_type == FaceDetectionModel.FULL_SPARSE: 184 | self.model_path = os.path.join(model_path, MODEL_NAME_FULL_SPARSE) 185 | ssd_opts = SSD_OPTIONS_FULL 186 | else: 187 | raise InvalidEnumError(f'unsupported model_type "{model_type}"') 188 | self.interpreter = tf.lite.Interpreter(model_path=self.model_path) 189 | self.interpreter.allocate_tensors() 190 | self.input_index = self.interpreter.get_input_details()[0]['index'] 191 | self.input_shape = self.interpreter.get_input_details()[0]['shape'] 192 | self.bbox_index = self.interpreter.get_output_details()[0]['index'] 193 | self.score_index = self.interpreter.get_output_details()[1]['index'] 194 | self.anchors = _ssd_generate_anchors(ssd_opts) 195 | 196 | def __call__( 197 | self, 198 | image: Union[Image, np.ndarray, str], 199 | roi: Optional[Rect] = None 200 | ) -> List[Detection]: 201 | """Run inference and return detections from a given image 202 | 203 | Args: 204 | image (Image|ndarray|str): Numpy array of shape 205 | `(height, width, 3)`, PIL Image instance or file name. 206 | 207 | roi (Rect|None): Optional region within the image that may 208 | contain faces. 209 | 210 | Returns: 211 | (list) List of detection results with relative coordinates. 212 | """ 213 | height, width = self.input_shape[1:3] 214 | image_data = image_to_tensor( 215 | image, 216 | roi, 217 | output_size=(width, height), 218 | keep_aspect_ratio=True, 219 | output_range=(-1, 1)) 220 | input_data = image_data.tensor_data[np.newaxis] 221 | self.interpreter.set_tensor(self.input_index, input_data) 222 | self.interpreter.invoke() 223 | raw_boxes = self.interpreter.get_tensor(self.bbox_index) 224 | raw_scores = self.interpreter.get_tensor(self.score_index) 225 | boxes = self._decode_boxes(raw_boxes) 226 | scores = self._get_sigmoid_scores(raw_scores) 227 | detections = FaceDetection._convert_to_detections(boxes, scores) 228 | pruned_detections = non_maximum_suppression( 229 | detections, 230 | MIN_SUPPRESSION_THRESHOLD, MIN_SCORE, 231 | weighted=True) 232 | detections = detection_letterbox_removal( 233 | pruned_detections, image_data.padding) 234 | return detections 235 | 236 | def _decode_boxes(self, raw_boxes: np.ndarray) -> np.ndarray: 237 | """Simplified version of 238 | mediapipe/calculators/tflite/tflite_tensors_to_detections_calculator.cc 239 | """ 240 | # width == height so scale is the same across the board 241 | scale = self.input_shape[1] 242 | num_points = raw_boxes.shape[-1] // 2 243 | # scale all values (applies to positions, width, and height alike) 244 | boxes = raw_boxes.reshape(-1, num_points, 2) / scale 245 | # adjust center coordinates and key points to anchor positions 246 | boxes[:, 0] += self.anchors 247 | for i in range(2, num_points): 248 | boxes[:, i] += self.anchors 249 | # convert x_center, y_center, w, h to xmin, ymin, xmax, ymax 250 | center = np.array(boxes[:, 0]) 251 | half_size = boxes[:, 1] / 2 252 | boxes[:, 0] = center - half_size 253 | boxes[:, 1] = center + half_size 254 | return boxes 255 | 256 | def _get_sigmoid_scores(self, raw_scores: np.ndarray) -> np.ndarray: 257 | """Extracted loop from ProcessCPU (line 327) in 258 | mediapipe/calculators/tflite/tflite_tensors_to_detections_calculator.cc 259 | """ 260 | # just a single class ("face"), which simplifies this a lot 261 | # 1) thresholding; adjusted from 100 to 80, since sigmoid of [-]100 262 | # causes overflow with IEEE single precision floats (max ~10e38) 263 | raw_scores[raw_scores < -RAW_SCORE_LIMIT] = -RAW_SCORE_LIMIT 264 | raw_scores[raw_scores > RAW_SCORE_LIMIT] = RAW_SCORE_LIMIT 265 | # 2) apply sigmoid function on clipped confidence scores 266 | return sigmoid(raw_scores) 267 | 268 | @staticmethod 269 | def _convert_to_detections( 270 | boxes: np.ndarray, 271 | scores: np.ndarray 272 | ) -> List[Detection]: 273 | """Apply detection threshold, filter invalid boxes and return 274 | detection instance. 275 | """ 276 | # return whether width and height are positive 277 | def is_valid(box: np.ndarray) -> bool: 278 | return np.all(box[1] > box[0]) 279 | 280 | score_above_threshold = scores > MIN_SCORE 281 | filtered_boxes = boxes[np.argwhere(score_above_threshold)[:, 1], :] 282 | filtered_scores = scores[score_above_threshold] 283 | return [Detection(box, score) 284 | for box, score in zip(filtered_boxes, filtered_scores) 285 | if is_valid(box)] 286 | 287 | 288 | def _ssd_generate_anchors(opts: dict) -> np.ndarray: 289 | """This is a trimmed down version of the C++ code; all irrelevant parts 290 | have been removed. 291 | (reference: mediapipe/calculators/tflite/ssd_anchors_calculator.cc) 292 | """ 293 | layer_id = 0 294 | num_layers = opts['num_layers'] 295 | strides = opts['strides'] 296 | assert len(strides) == num_layers 297 | input_height = opts['input_size_height'] 298 | input_width = opts['input_size_width'] 299 | anchor_offset_x = opts['anchor_offset_x'] 300 | anchor_offset_y = opts['anchor_offset_y'] 301 | interpolated_scale_aspect_ratio = opts['interpolated_scale_aspect_ratio'] 302 | anchors = [] 303 | while layer_id < num_layers: 304 | last_same_stride_layer = layer_id 305 | repeats = 0 306 | while (last_same_stride_layer < num_layers and 307 | strides[last_same_stride_layer] == strides[layer_id]): 308 | last_same_stride_layer += 1 309 | # aspect_ratios are added twice per iteration 310 | repeats += 2 if interpolated_scale_aspect_ratio == 1.0 else 1 311 | stride = strides[layer_id] 312 | feature_map_height = input_height // stride 313 | feature_map_width = input_width // stride 314 | for y in range(feature_map_height): 315 | y_center = (y + anchor_offset_y) / feature_map_height 316 | for x in range(feature_map_width): 317 | x_center = (x + anchor_offset_x) / feature_map_width 318 | for _ in range(repeats): 319 | anchors.append((x_center, y_center)) 320 | layer_id = last_same_stride_layer 321 | return np.array(anchors, dtype=np.float32) 322 | -------------------------------------------------------------------------------- /fdlite/face_landmark.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright © 2021 Patrick Levin 3 | # SPDX-Identifier: MIT 4 | import os 5 | import numpy as np 6 | import tensorflow as tf 7 | from PIL.Image import Image 8 | from typing import List, Optional, Sequence, Tuple, Union 9 | from fdlite import ModelDataError 10 | from fdlite.render import Annotation 11 | from fdlite.render import Color, landmarks_to_render_data 12 | from fdlite.transform import SizeMode, bbox_to_roi, image_to_tensor, sigmoid 13 | from fdlite.transform import project_landmarks 14 | from fdlite.types import Detection, Landmark, Rect 15 | from fdlite.face_detection import FaceIndex 16 | """Model for face landmark detection. 17 | 18 | Ported from Google® MediaPipe (https://google.github.io/mediapipe/). 19 | 20 | Model card: 21 | 22 | https://mediapipe.page.link/facemesh-mc 23 | 24 | Reference: 25 | 26 | Real-time Facial Surface Geometry from Monocular 27 | Video on Mobile GPUs, CVPR Workshop on Computer 28 | Vision for Augmented and Virtual Reality, Long Beach, 29 | CA, USA, 2019 30 | """ 31 | 32 | MODEL_NAME = 'face_landmark.tflite' 33 | NUM_DIMS = 3 # x, y, z 34 | NUM_LANDMARKS = 468 # number of points in the face mesh 35 | ROI_SCALE = (1.5, 1.5) # Scaling of the face detection ROI 36 | DETECTION_THRESHOLD = 0.5 # minimum score for detected faces 37 | 38 | # face landmark connections 39 | # (from face_landmarks_to_render_data_calculator.cc) 40 | FACE_LANDMARK_CONNECTIONS = [ 41 | # Lips. 42 | (61, 146), (146, 91), (91, 181), (181, 84), (84, 17), (17, 314), 43 | (314, 405), (405, 321), (321, 375), (375, 291), (61, 185), (185, 40), 44 | (40, 39), (39, 37), (37, 0), (0, 267), (267, 269), 45 | (269, 270), (270, 409), (409, 291), (78, 95), (95, 88), (88, 178), 46 | (178, 87), (87, 14), (14, 317), (317, 402), (402, 318), (318, 324), 47 | (324, 308), (78, 191), (191, 80), (80, 81), (81, 82), (82, 13), (13, 312), 48 | (312, 311), (311, 310), (310, 415), (415, 308), 49 | # Left eye. 50 | (33, 7), (7, 163), (163, 144), (144, 145), (145, 153), (153, 154), 51 | (154, 155), (155, 133), (33, 246), (246, 161), (161, 160), (160, 159), 52 | (159, 158), (158, 157), (157, 173), (173, 133), 53 | # Left eyebrow. 54 | (46, 53), (53, 52), (52, 65), (65, 55), (70, 63), (63, 105), (105, 66), 55 | (66, 107), 56 | # Right eye. 57 | (263, 249), (249, 390), (390, 373), (373, 374), (374, 380), (380, 381), 58 | (381, 382), (382, 362), (263, 466), (466, 388), (388, 387), (387, 386), 59 | (386, 385), (385, 384), (384, 398), (398, 362), 60 | # Right eyebrow. 61 | (276, 283), (283, 282), (282, 295), (295, 285), (300, 293), (293, 334), 62 | (334, 296), (296, 336), 63 | # Face oval. 64 | (10, 338), (338, 297), (297, 332), (332, 284), (284, 251), (251, 389), 65 | (389, 356), (356, 454), (454, 323), (323, 361), (361, 288), (288, 397), 66 | (397, 365), (365, 379), (379, 378), (378, 400), (400, 377), (377, 152), 67 | (152, 148), (148, 176), (176, 149), (149, 150), (150, 136), (136, 172), 68 | (172, 58), (58, 132), (132, 93), (93, 234), (234, 127), (127, 162), 69 | (162, 21), (21, 54), (54, 103), (103, 67), (67, 109), (109, 10), 70 | ] 71 | MAX_FACE_LANDMARK = len(FACE_LANDMARK_CONNECTIONS) 72 | 73 | 74 | def face_detection_to_roi( 75 | face_detection: Detection, 76 | image_size: Tuple[int, int] 77 | ) -> Rect: 78 | """Return a normalized ROI from a list of face detection results. 79 | 80 | The result of this function is intended to serve as the input of 81 | calls to `FaceLandmark`: 82 | 83 | ``` 84 | MODEL_PATH = '/var/mediapipe/models/' 85 | ... 86 | face_detect = FaceDetection(model_path=MODEL_PATH) 87 | face_landmarks = FaceLandmark(model_path=MODEL_PATH) 88 | image = Image.open('/home/user/pictures/photo.jpg') 89 | # detect faces 90 | detections = face_detect(image) 91 | for detection in detections: 92 | # find ROI from detection 93 | roi = face_detection_to_roi(detection) 94 | # extract face landmarks using ROI 95 | landmarks = face_landmarks(image, roi) 96 | ... 97 | ``` 98 | 99 | Args: 100 | face_detection (Detection): Normalized face detection result from a 101 | call to `FaceDetection`. 102 | 103 | image_size (tuple): A tuple of `(image_width, image_height)` denoting 104 | the size of the input image the face detection results came from. 105 | 106 | Returns: 107 | (Rect) Normalized ROI for passing to `FaceLandmark`. 108 | """ 109 | absolute_detection = face_detection.scaled(image_size) 110 | left_eye = absolute_detection[FaceIndex.LEFT_EYE] 111 | right_eye = absolute_detection[FaceIndex.RIGHT_EYE] 112 | return bbox_to_roi( 113 | face_detection.bbox, 114 | image_size, 115 | rotation_keypoints=[left_eye, right_eye], 116 | scale=ROI_SCALE, 117 | size_mode=SizeMode.SQUARE_LONG 118 | ) 119 | 120 | 121 | class FaceLandmark: 122 | """Face Landmark detection model as used by Google MediaPipe. 123 | 124 | This model detects facial landmarks from a face image. 125 | 126 | The model is callable and accepts a PIL image instance, image file name, 127 | and Numpy array of shape (height, width, channels) as input. There is no 128 | size restriction, but smaller images are processed faster. 129 | 130 | The output of the model is a list of 468 face landmarks in normalized 131 | coordinates (e.g. in the range [0, 1]). 132 | 133 | The preferred usage is to pass an ROI returned by a call to the 134 | `FaceDetection` model along with the image. 135 | 136 | Raises: 137 | ModelDataError: `model_path` points to an incompatible model 138 | """ 139 | def __init__( 140 | self, 141 | model_path: Optional[str] = None 142 | ) -> None: 143 | if model_path is None: 144 | my_path = os.path.abspath(__file__) 145 | model_path = os.path.join(os.path.dirname(my_path), 'data') 146 | self.model_path = os.path.join(model_path, MODEL_NAME) 147 | self.interpreter = tf.lite.Interpreter(model_path=self.model_path) 148 | self.input_index = self.interpreter.get_input_details()[0]['index'] 149 | self.input_shape = self.interpreter.get_input_details()[0]['shape'] 150 | self.data_index = self.interpreter.get_output_details()[0]['index'] 151 | self.face_index = self.interpreter.get_output_details()[1]['index'] 152 | data_shape = self.interpreter.get_output_details()[0]['shape'] 153 | num_exected_elements = NUM_DIMS * NUM_LANDMARKS 154 | if data_shape[-1] < num_exected_elements: 155 | raise ModelDataError(f'incompatible model: {data_shape} < ' 156 | f'{num_exected_elements}') 157 | self.interpreter.allocate_tensors() 158 | 159 | def __call__( 160 | self, 161 | image: Union[Image, np.ndarray, str], 162 | roi: Optional[Rect] = None 163 | ) -> List[Landmark]: 164 | """Run inference and return detections from a given image 165 | 166 | Args: 167 | image (Image|ndarray|str): Numpy array of shape 168 | `(height, width, 3)` or PIL Image instance or path to image. 169 | 170 | roi (Rect|None): Region within the image that contains a face. 171 | 172 | Returns: 173 | (list) List of face landmarks in nromalised coordinates relative to 174 | the input image, i.e. values ranging from [0, 1]. 175 | """ 176 | height, width = self.input_shape[1:3] 177 | image_data = image_to_tensor( 178 | image, 179 | roi, 180 | output_size=(width, height), 181 | keep_aspect_ratio=False, 182 | output_range=(0., 1.)) 183 | input_data = image_data.tensor_data[np.newaxis] 184 | self.interpreter.set_tensor(self.input_index, input_data) 185 | self.interpreter.invoke() 186 | raw_data = self.interpreter.get_tensor(self.data_index) 187 | raw_face = self.interpreter.get_tensor(self.face_index) 188 | # second tensor contains confidence score for a face detection 189 | face_flag = sigmoid(raw_face).flatten()[-1] 190 | # no data if no face was detected 191 | if face_flag <= DETECTION_THRESHOLD: 192 | return [] 193 | # extract and normalise landmark data 194 | height, width = self.input_shape[1:3] 195 | return project_landmarks(raw_data, 196 | tensor_size=(width, height), 197 | image_size=image_data.original_size, 198 | padding=image_data.padding, 199 | roi=roi) 200 | 201 | 202 | def face_landmarks_to_render_data( 203 | face_landmarks: Sequence[Landmark], 204 | landmark_color: Color, 205 | connection_color: Color, 206 | thickness: float = 2.0, 207 | output: Optional[List[Annotation]] = None 208 | ) -> List[Annotation]: 209 | """Convert face landmarks to render data. 210 | 211 | This post-processing function can be used to generate a list of rendering 212 | instructions from face landmark detection results. 213 | 214 | Args: 215 | face_landmarks (list): List of `Landmark` detection results returned 216 | by `FaceLandmark`. 217 | 218 | landmark_color (Color): Color of the individual landmark points. 219 | 220 | connection_color (Color): Color of the landmark connections that 221 | will be rendered as lines. 222 | 223 | thickness (float): Width of the lines and landmark point size in 224 | viewport units (e.g. pixels). 225 | 226 | output (list): Optional list of render annotations to add the items 227 | to. If not provided, a new list will be created. 228 | Use this to add multiple landmark detections into a single render 229 | annotation list. 230 | 231 | Returns: 232 | (list) List of render annotations that should be rendered. 233 | All positions are normalized, e.g. with a value range of [0, 1]. 234 | """ 235 | render_data = landmarks_to_render_data( 236 | face_landmarks, FACE_LANDMARK_CONNECTIONS, 237 | landmark_color, connection_color, thickness, 238 | normalized_positions=True, output=output) 239 | return render_data 240 | -------------------------------------------------------------------------------- /fdlite/iris_landmark.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright © 2021 Patrick Levin 3 | # SPDX-Identifier: MIT 4 | from dataclasses import dataclass 5 | from enum import IntEnum 6 | import numpy as np 7 | import os 8 | import tensorflow as tf 9 | from PIL.Image import Image 10 | from typing import List, Optional, Sequence, Tuple, Union 11 | from fdlite import ArgumentError, MissingExifDataError, ModelDataError, exif 12 | from fdlite.render import Annotation, Color, Point, RectOrOval 13 | from fdlite.render import landmarks_to_render_data 14 | from fdlite.transform import bbox_from_landmarks, bbox_to_roi, image_to_tensor 15 | from fdlite.transform import project_landmarks, SizeMode 16 | from fdlite.types import Landmark, Rect 17 | """Iris landmark detection model. 18 | 19 | Ported from Google® MediaPipe (https://google.github.io/mediapipe/). 20 | 21 | Model card: 22 | 23 | https://mediapipe.page.link/iris-mc 24 | 25 | Reference: 26 | N/A 27 | """ 28 | 29 | MODEL_NAME = 'iris_landmark.tflite' 30 | # ROI scale factor for 25% margin around eye 31 | ROI_SCALE = (2.3, 2.3) 32 | # Landmark index of the left eye start point 33 | LEFT_EYE_START = 33 34 | # Landmark index of the left eye end point 35 | LEFT_EYE_END = 133 36 | # Landmark index of the right eye start point 37 | RIGHT_EYE_START = 362 38 | # Landmark index of the right eye end point 39 | RIGHT_EYE_END = 263 40 | # Number of face landmarks (from face landmark results) 41 | NUM_FACE_LANDMARKS = 468 42 | 43 | # Landmark element count (x, y, z) 44 | NUM_DIMS = 3 45 | NUM_EYE_LANDMARKS = 71 46 | NUM_IRIS_LANDMARKS = 5 47 | 48 | # eye contour default visualisation settings 49 | # (from iris_and_depth_renderer_cpu.pbtxt) 50 | EYE_LANDMARK_CONNECTIONS = [ 51 | (0, 1), (1, 2), (2, 3), (3, 4), (4, 5), 52 | (5, 6), (6, 7), (7, 8), (9, 10), (10, 11), 53 | (11, 12), (12, 13), (13, 14), (0, 9), (8, 14) 54 | ] 55 | MAX_EYE_LANDMARK = len(EYE_LANDMARK_CONNECTIONS) 56 | 57 | # mapping from left eye contour index to face landmark index 58 | LEFT_EYE_TO_FACE_LANDMARK_INDEX = [ 59 | # eye lower contour 60 | 33, 7, 163, 144, 145, 153, 154, 155, 133, 61 | # eye upper contour excluding corners 62 | 246, 161, 160, 159, 158, 157, 173, 63 | # halo x2 lower contour 64 | 130, 25, 110, 24, 23, 22, 26, 112, 243, 65 | # halo x2 upper contour excluding corners 66 | 247, 30, 29, 27, 28, 56, 190, 67 | # halo x3 lower contour 68 | 226, 31, 228, 229, 230, 231, 232, 233, 244, 69 | # halo x3 upper contour excluding corners 70 | 113, 225, 224, 223, 222, 221, 189, 71 | # halo x4 upper contour (no upper due to mesh structure) 72 | # or eyebrow inner contour 73 | 35, 124, 46, 53, 52, 65, 74 | # halo x5 lower contour 75 | 143, 111, 117, 118, 119, 120, 121, 128, 245, 76 | # halo x5 upper contour excluding corners or eyebrow outer contour 77 | 156, 70, 63, 105, 66, 107, 55, 193, 78 | ] 79 | 80 | # mapping from right eye contour index to face landmark index 81 | RIGHT_EYE_TO_FACE_LANDMARK_INDEX = [ 82 | # eye lower contour 83 | 263, 249, 390, 373, 374, 380, 381, 382, 362, 84 | # eye upper contour excluding corners 85 | 466, 388, 387, 386, 385, 384, 398, 86 | # halo x2 lower contour 87 | 359, 255, 339, 254, 253, 252, 256, 341, 463, 88 | # halo x2 upper contour excluding corners 89 | 467, 260, 259, 257, 258, 286, 414, 90 | # halo x3 lower contour 91 | 446, 261, 448, 449, 450, 451, 452, 453, 464, 92 | # halo x3 upper contour excluding corners 93 | 342, 445, 444, 443, 442, 441, 413, 94 | # halo x4 upper contour (no upper due to mesh structure) 95 | # or eyebrow inner contour 96 | 265, 353, 276, 283, 282, 295, 97 | # halo x5 lower contour 98 | 372, 340, 346, 347, 348, 349, 350, 357, 465, 99 | # halo x5 upper contour excluding corners or eyebrow outer contour 100 | 383, 300, 293, 334, 296, 336, 285, 417, 101 | ] 102 | 103 | # 35mm camera sensor diagonal (36mm * 24mm) 104 | SENSOR_DIAGONAL_35MM = np.math.sqrt(36 ** 2 + 24 ** 2) 105 | # average human iris size 106 | IRIS_SIZE_IN_MM = 11.8 107 | 108 | 109 | class IrisIndex(IntEnum): 110 | """Index into iris landmarks as returned by `IrisLandmark` 111 | """ 112 | CENTER = 0 113 | LEFT = 1 114 | TOP = 2 115 | RIGHT = 3 116 | BOTTOM = 4 117 | 118 | 119 | @dataclass 120 | class IrisResults: 121 | """Iris detection results. 122 | 123 | contour data is 71 points defining the eye region 124 | 125 | iris data is 5 keypoints 126 | """ 127 | contour: List[Landmark] 128 | iris: List[Landmark] 129 | 130 | @property 131 | def eyeball_contour(self) -> List[Landmark]: 132 | """Visible eyeball contour landmarks""" 133 | return self.contour[:MAX_EYE_LANDMARK] 134 | 135 | 136 | def iris_roi_from_face_landmarks( 137 | face_landmarks: Sequence[Landmark], 138 | image_size: Tuple[int, int] 139 | ) -> Tuple[Rect, Rect]: 140 | """Extract iris landmark detection ROIs from face landmarks. 141 | 142 | Use this function to get the ROI for the left and right eye. The resulting 143 | bounding boxes are suitable for passing to `IrisDetection` as the ROI 144 | parameter. This is a pre-processing step originally found in the 145 | MediaPipe sub-graph "iris_landmark_landmarks_to_roi": 146 | 147 | ``` 148 | def iris_detection(image, face_landmarks, iris_landmark_model): 149 | # extract left- and right eye ROI from face landmarks 150 | roi_left_eye, roi_right_eye = iris_roi_from_face_landmarks( 151 | face_landmarks) 152 | # use ROIs with iris detection 153 | iris_results_left = iris_landmark_model(image, roi_left_eye) 154 | iris_results_right = iris_landmark_model(image, roi_right_eye, 155 | is_right_eye=True) 156 | return iris_result_left, iris_results_right 157 | ``` 158 | 159 | Args: 160 | landmarks (list): Result of a `FaceLandmark` call containing face 161 | landmark detection results. 162 | 163 | image_size (tuple): Tuple of `(width, height)` representing the size 164 | of the image in pixels. The image must be the same as the one used 165 | for face lanmark detection. 166 | 167 | Return: 168 | (Rect, Rect) Tuple of ROIs containing the absolute pixel coordinates 169 | of the left and right eye regions. The ROIs can be passed to 170 | `IrisDetetion` together with the original image to detect iris 171 | landmarks. 172 | """ 173 | left_eye_landmarks = ( 174 | face_landmarks[LEFT_EYE_START], 175 | face_landmarks[LEFT_EYE_END]) 176 | bbox = bbox_from_landmarks(left_eye_landmarks) 177 | rotation_keypoints = [(point.x, point.y) for point in left_eye_landmarks] 178 | w, h = image_size 179 | left_eye_roi = bbox_to_roi( 180 | bbox, (w, h), rotation_keypoints, ROI_SCALE, SizeMode.SQUARE_LONG) 181 | 182 | right_eye_landmarks = ( 183 | face_landmarks[RIGHT_EYE_START], 184 | face_landmarks[RIGHT_EYE_END]) 185 | bbox = bbox_from_landmarks(right_eye_landmarks) 186 | rotation_keypoints = [(point.x, point.y) for point in right_eye_landmarks] 187 | right_eye_roi = bbox_to_roi( 188 | bbox, (w, h), rotation_keypoints, ROI_SCALE, SizeMode.SQUARE_LONG) 189 | 190 | return left_eye_roi, right_eye_roi 191 | 192 | 193 | def eye_landmarks_to_render_data( 194 | eye_contour: Sequence[Landmark], 195 | landmark_color: Color, 196 | connection_color: Color, 197 | thickness: float = 2.0, 198 | output: Optional[List[Annotation]] = None 199 | ) -> List[Annotation]: 200 | """Convert eye contour to render data. 201 | 202 | This post-processing function can be used to generate a list of rendering 203 | instructions from iris landmark detection results. 204 | 205 | Args: 206 | eye_contour (list): List of `Landmark` detection results returned 207 | by `IrisLandmark`. 208 | 209 | landmark_color (Color): Color of the individual landmark points. 210 | 211 | connection_color (Color): Color of the landmark connections that 212 | will be rendered as lines. 213 | 214 | thickness (float): Width of the lines and landmark point size in 215 | viewport units (e.g. pixels). 216 | 217 | output (list): Optional list of render annotations to add the items 218 | to. If not provided, a new list will be created. 219 | Use this to add multiple landmark detections into a single render 220 | annotation list. 221 | 222 | Returns: 223 | (list) List of render annotations that should be rendered. 224 | All positions are normalized, e.g. with a value range of [0, 1]. 225 | """ 226 | render_data = landmarks_to_render_data( 227 | eye_contour[0:MAX_EYE_LANDMARK], EYE_LANDMARK_CONNECTIONS, 228 | landmark_color, connection_color, thickness, 229 | normalized_positions=True, output=output) 230 | return render_data 231 | 232 | 233 | def iris_landmarks_to_render_data( 234 | iris_landmarks: Sequence[Landmark], 235 | landmark_color: Optional[Color] = None, 236 | oval_color: Optional[Color] = None, 237 | thickness: float = 1.0, 238 | image_size: Tuple[int, int] = (-1, -1), 239 | output: Optional[List[Annotation]] = None 240 | ) -> List[Annotation]: 241 | """Convert iris landmarks to render data. 242 | 243 | This post-processing function can be used to generate a list of rendering 244 | instructions from iris landmark detection results. 245 | 246 | Args: 247 | iris_landmarks (list): List of `Landmark` detection results returned 248 | by `IrisLandmark`. 249 | 250 | landmark_color (Color|None): Color of the individual landmark points. 251 | 252 | oval_color (Color|None): Color of the iris oval. 253 | 254 | thickness (float): Width of the iris oval and landmark point size 255 | in viewport units (e.g. pixels). 256 | 257 | image_size (tuple): Image size as a tuple of `(width, height)`. 258 | 259 | output (list|None): Optional list of render annotations to add the 260 | items to. If not provided, a new list will be created. 261 | Use this to add multiple landmark detections into a single render 262 | annotation list. 263 | 264 | Raises: 265 | ArgumentError: `image_size` is too small or not provided and 266 | `oval_color` is not `None` 267 | 268 | Returns: 269 | (list) Render data bundle containing points for landmarks and 270 | lines for landmark connections. All positions are normalised with 271 | respect to the output viewport (e.g. value range is [0, 1]) 272 | """ 273 | annotations = [] 274 | if oval_color is not None: 275 | iris_radius = _get_iris_diameter(iris_landmarks, image_size) / 2 276 | width, height = image_size 277 | if width < 2 or height < 2: 278 | raise ArgumentError('oval_color requires a valid image_size arg') 279 | radius_h = iris_radius / width 280 | radius_v = iris_radius / height 281 | iris_center = iris_landmarks[IrisIndex.CENTER] 282 | oval = RectOrOval(iris_center.x - radius_h, iris_center.y - radius_v, 283 | iris_center.x + radius_h, iris_center.y + radius_v, 284 | oval=True) 285 | annotations.append(Annotation([oval], normalized_positions=True, 286 | thickness=thickness, color=oval_color)) 287 | if landmark_color is not None: 288 | points = [Point(pt.x, pt.y) for pt in iris_landmarks] 289 | annotations.append(Annotation(points, normalized_positions=True, 290 | thickness=thickness, 291 | color=landmark_color)) 292 | if output is None: 293 | output = annotations 294 | else: 295 | output += annotations 296 | return output 297 | 298 | 299 | def update_face_landmarks_with_iris_results( 300 | face_landmarks: Sequence[Landmark], 301 | iris_data_left: IrisResults, 302 | iris_data_right: IrisResults 303 | ) -> List[Landmark]: 304 | """Update face landmarks with iris detection results. 305 | 306 | Landmarks will be updated with refined results from iris tracking. 307 | 308 | Args: 309 | face_landmarks (list): Face landmark detection results with 310 | coarse eye landmarks 311 | 312 | iris_data_left (IrisResults): Iris landmark results for the left eye 313 | 314 | iris_data_right (IrisResults): Iris landmark results for the right eye 315 | 316 | Returns: 317 | (list) List of face landmarks with refined eye contours and iris 318 | landmarks 319 | 320 | Raises: 321 | ModelDataError: the number of results in `face_landmarks` doesn't match 322 | the supported model output 323 | """ 324 | if len(face_landmarks) != NUM_FACE_LANDMARKS: 325 | raise ModelDataError('unexpected number of items in face_landmarks') 326 | # copy landmarks 327 | refined_landmarks = [Landmark(pt.x, pt.y, pt.z) for pt in face_landmarks] 328 | # merge left eye contours 329 | for n, point in enumerate(iris_data_left.contour): 330 | index = LEFT_EYE_TO_FACE_LANDMARK_INDEX[n] 331 | refined_landmarks[index] = Landmark(point.x, point.y, point.z) 332 | # merge right eye contours 333 | for n, point in enumerate(iris_data_right.contour): 334 | index = RIGHT_EYE_TO_FACE_LANDMARK_INDEX[n] 335 | refined_landmarks[index] = Landmark(point.x, point.y, point.z) 336 | return refined_landmarks 337 | 338 | 339 | def iris_depth_in_mm_from_landmarks( 340 | image_or_focal_length: Union[Image, Tuple[int, int, int, int]], 341 | iris_data_left: IrisResults, 342 | iris_data_right: IrisResults 343 | ) -> Tuple[float, float]: 344 | """Calculate the distances to the left- and right eye from image meta data 345 | and iris landmarks. 346 | 347 | The calculation requires EXIF meta data to be present if an image is 348 | provided. Alternatively, focal length and image size in pixels can be 349 | provided directly. 350 | 351 | Args: 352 | image_or_focal_length (Image|tuple): Either a PIL image instance with 353 | EXIF meta data or a tuple of 354 | `(focal_length_35mm, focal_length_mm, image_width, image_height)` 355 | 356 | iris_data_left (IrisResults): Detection results from `IrisLandmark` 357 | for the left eye. 358 | 359 | iris_data_right (IrisResults): Detection results from `IrisLandmark` 360 | for the right eye. 361 | 362 | Raises: 363 | ArgumentError: `image_or_focal_length` does not have four (4) elements 364 | MissingExifDataError: `image_or_sensor_data` is a PIL image without the 365 | required EXIF meta data. 366 | 367 | Returns: 368 | (tuple) Tuple of `(left_eye_distance_in_mm, right_eye_distance_in_mm)` 369 | """ 370 | if isinstance(image_or_focal_length, Image): 371 | from_exif = exif.get_focal_length(image_or_focal_length) 372 | if from_exif is None: 373 | raise MissingExifDataError('missing EXIF data or unknown camera') 374 | focal_length = from_exif 375 | else: 376 | focal_length = image_or_focal_length 377 | if len(focal_length) != 4: 378 | raise ArgumentError('focal length must contain 4 elements') 379 | # calculate focal length in pixels 380 | focal_len_35mm, focal_len_mm, width_px, height_px = focal_length 381 | sensor_diagonal_mm = SENSOR_DIAGONAL_35MM / focal_len_35mm * focal_len_mm 382 | w, h = width_px, height_px 383 | pixel_size = (width_px, height_px) 384 | # treat the shorter dimension as width 385 | if height_px > width_px: 386 | w, h = h, w 387 | sqr_inv_aspect = (h / w) ** 2 388 | sensor_width = np.math.sqrt((sensor_diagonal_mm**2) / (1 + sqr_inv_aspect)) 389 | focal_len_px = w * focal_len_mm / sensor_width 390 | left_landmarks, right_landmarks = iris_data_left.iris, iris_data_right.iris 391 | left_iris_size = _get_iris_diameter(left_landmarks, pixel_size) 392 | right_iris_size = _get_iris_diameter(right_landmarks, pixel_size) 393 | left_depth_mm = _get_iris_depth(left_landmarks, focal_len_px, 394 | left_iris_size, pixel_size) 395 | right_depth_mm = _get_iris_depth(right_landmarks, focal_len_px, 396 | right_iris_size, pixel_size) 397 | return left_depth_mm, right_depth_mm 398 | 399 | 400 | class IrisLandmark: 401 | """Model for iris landmark detection from the image of an eye. 402 | 403 | The model expects the image of an eye as input, complete with brows and 404 | a 25% margin around the eye. 405 | 406 | The outputs of the model are 71 normalized eye contour landmarks and a 407 | separate list of 5 normalized iris landmarks. 408 | 409 | The model is callable and accepts a PIL image instance, image file name, 410 | and Numpy array of shape (height, width, channels) as input. There is no 411 | size restriction, but smaller images are processed faster. 412 | 413 | The provided image either matches the model's input spec or an ROI is 414 | provided, which denotes the eye location within the image. 415 | The ROI can be obtained from calling the `FaceDetection` model with the 416 | image and converting the iris ROI from the result: 417 | 418 | ``` 419 | MODEL_PATH = '/var/mediapipe/models' 420 | 421 | img = PIL.Image.open('/home/usr/pictures/group.jpg', mode='RGB') 422 | # 1) load models 423 | detect_face = FaceDetection(model_path=MODEL_PATH) 424 | detect_face_landmarks = FaceLandmark(model_path=MODEL_PATH) 425 | detect_iris_landmarks = IrisLandmark(model_path=MODEL_PATH) 426 | # 2) run face detection 427 | face_detections = detect_face(img) 428 | # 3) detect face landmarks 429 | for detection in face_detections: 430 | face_roi = face_detection_to_roi(detection, img.size) 431 | face_landmarks = detect_face_landmarks(img, face_roi) 432 | # 4) run iris detection 433 | iris_roi = iris_roi_from_landmarks(face_landmarks, img.size) 434 | left_eye_detection = detect_iris_landmarks(img, iris_roi[0]) 435 | right_eye_detection = detect_iris_landmarks( 436 | img, iris_roi[1], is_right_eye=True) 437 | ... 438 | ``` 439 | 440 | Raises: 441 | ModelDataError: `model_path` refers to an incompatible detection model 442 | """ 443 | def __init__(self, model_path: Optional[str] = None) -> None: 444 | if model_path is None: 445 | my_path = os.path.abspath(__file__) 446 | model_path = os.path.join(os.path.dirname(my_path), 'data') 447 | self.model_path = os.path.join(model_path, MODEL_NAME) 448 | self.interpreter = tf.lite.Interpreter(model_path=self.model_path) 449 | self.input_index = self.interpreter.get_input_details()[0]['index'] 450 | self.input_shape = self.interpreter.get_input_details()[0]['shape'] 451 | self.eye_index = self.interpreter.get_output_details()[0]['index'] 452 | self.iris_index = self.interpreter.get_output_details()[1]['index'] 453 | 454 | eye_shape = self.interpreter.get_output_details()[0]['shape'] 455 | if eye_shape[-1] != NUM_DIMS * NUM_EYE_LANDMARKS: 456 | raise ModelDataError('unexpected number of eye landmarks: ' 457 | f'{eye_shape[-1]}') 458 | iris_shape = self.interpreter.get_output_details()[1]['shape'] 459 | if iris_shape[-1] != NUM_DIMS * NUM_IRIS_LANDMARKS: 460 | raise ModelDataError('unexpected number of iris landmarks: ' 461 | f'{eye_shape[-1]}') 462 | self.interpreter.allocate_tensors() 463 | 464 | def __call__( 465 | self, 466 | image: Union[Image, np.ndarray, str], 467 | roi: Optional[Rect] = None, 468 | is_right_eye: bool = False 469 | ) -> IrisResults: 470 | height, width = self.input_shape[1:3] 471 | image_data = image_to_tensor( 472 | image, 473 | roi, 474 | output_size=(width, height), 475 | keep_aspect_ratio=True, # equivalent to scale_mode=FIT 476 | output_range=(0, 1), # see iris_landmark_cpu.pbtxt 477 | flip_horizontal=is_right_eye 478 | ) 479 | input_data = image_data.tensor_data[np.newaxis] 480 | self.interpreter.set_tensor(self.input_index, input_data) 481 | self.interpreter.invoke() 482 | raw_eye_landmarks = self.interpreter.get_tensor(self.eye_index) 483 | raw_iris_landmarks = self.interpreter.get_tensor(self.iris_index) 484 | height, width = self.input_shape[1:3] 485 | eye_contour = project_landmarks( 486 | raw_eye_landmarks, 487 | tensor_size=(width, height), 488 | image_size=image_data.original_size, 489 | padding=image_data.padding, 490 | roi=roi, 491 | flip_horizontal=is_right_eye) 492 | iris_landmarks = project_landmarks( 493 | raw_iris_landmarks, 494 | tensor_size=(width, height), 495 | image_size=image_data.original_size, 496 | padding=image_data.padding, 497 | roi=roi, 498 | flip_horizontal=is_right_eye) 499 | return IrisResults(eye_contour, iris_landmarks) 500 | 501 | 502 | def _get_iris_diameter( 503 | iris_landmarks: Sequence[Landmark], image_size: Tuple[int, int] 504 | ) -> float: 505 | """Calculate the iris diameter in pixels""" 506 | width, height = image_size 507 | 508 | def get_landmark_depth(a: Landmark, b: Landmark) -> float: 509 | x0, y0, x1, y1 = a.x * width, a.y * height, b.x * width, b.y * height 510 | return np.math.sqrt((x0 - x1) ** 2 + (y0 - y1) ** 2) 511 | 512 | iris_size_horiz = get_landmark_depth(iris_landmarks[IrisIndex.LEFT], 513 | iris_landmarks[IrisIndex.RIGHT]) 514 | iris_size_vert = get_landmark_depth(iris_landmarks[IrisIndex.TOP], 515 | iris_landmarks[IrisIndex.BOTTOM]) 516 | return (iris_size_vert + iris_size_horiz) / 2 517 | 518 | 519 | def _get_iris_depth( 520 | iris_landmarks: Sequence[Landmark], 521 | focal_length_mm: float, 522 | iris_size_px: float, 523 | image_size: Tuple[int, int] 524 | ) -> float: 525 | """Calculate iris depth in mm from landmarks and lens focal length in mm""" 526 | width, height = image_size 527 | center = iris_landmarks[IrisIndex.CENTER] 528 | x0, y0 = width / 2, height / 2 529 | x1, y1 = center.x * width, center.y * height 530 | y = np.math.sqrt((x0 - x1) ** 2 + (y0 - y1) ** 2) 531 | x = np.math.sqrt(focal_length_mm ** 2 + y ** 2) 532 | return IRIS_SIZE_IN_MM * x / iris_size_px 533 | -------------------------------------------------------------------------------- /fdlite/nms.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright © 2021 Patrick Levin 3 | # SPDX-Identifier: MIT 4 | import numpy as np 5 | from typing import List, Optional, Tuple 6 | from fdlite.types import BBox, Detection 7 | """Implementation of non-maximum-suppression (NMS) for detections.""" 8 | 9 | 10 | def non_maximum_suppression( 11 | detections: List[Detection], 12 | min_suppression_threshold: float, 13 | min_score: Optional[float], 14 | weighted: bool = False 15 | ) -> List[Detection]: 16 | """Return only significant detections. 17 | 18 | Args: 19 | detections (list): List of detections. 20 | 21 | min_suppression_threshold (float): Discard detections whose similarity 22 | is above this threshold value 23 | 24 | min_score (float): Minimum score of valid detections. 25 | 26 | Returns: 27 | (list) List of sufficiently relevant detections 28 | """ 29 | scores = [detection.score for detection in detections] 30 | indexed_scores = [(n, score) for n, score in enumerate(scores)] 31 | indexed_scores = sorted(indexed_scores, key=lambda p: p[1], reverse=True) 32 | if weighted: 33 | return _weighted_non_maximum_suppression( 34 | indexed_scores, detections, min_suppression_threshold, min_score) 35 | else: 36 | return _non_maximum_suppression( 37 | indexed_scores, detections, min_suppression_threshold, min_score) 38 | 39 | 40 | def _overlap_similarity(box1: BBox, box2: BBox) -> float: 41 | """Return intersection-over-union similarity of two bounding boxes""" 42 | intersection = box1.intersect(box2) 43 | if intersection is None: 44 | return 0. 45 | intersect_area = intersection.area 46 | denominator = box1.area + box2.area - intersect_area 47 | return intersect_area / denominator if denominator > 0. else 0. 48 | 49 | 50 | def _non_maximum_suppression( 51 | indexed_scores: List[Tuple[int, float]], 52 | detections: List[Detection], 53 | min_suppression_threshold: float, 54 | min_score: Optional[float] 55 | ) -> List[Detection]: 56 | """Return only most significant detections""" 57 | kept_boxes: List[BBox] = [] 58 | outputs = [] 59 | for index, score in indexed_scores: 60 | # exit loop if remaining scores are below threshold 61 | if min_score is not None and score < min_score: 62 | break 63 | detection = detections[index] 64 | bbox = detection.bbox 65 | suppressed = False 66 | for kept in kept_boxes: 67 | similarity = _overlap_similarity(kept, bbox) 68 | if similarity > min_suppression_threshold: 69 | suppressed = True 70 | break 71 | if not suppressed: 72 | outputs.append(detection) 73 | kept_boxes.append(bbox) 74 | return outputs 75 | 76 | 77 | def _weighted_non_maximum_suppression( 78 | indexed_scores: List[Tuple[int, float]], 79 | detections: List[Detection], 80 | min_suppression_threshold: float, 81 | min_score: Optional[float] 82 | ) -> List[Detection]: 83 | """Return only most significant detections; merge similar detections""" 84 | remaining_indexed_scores = list(indexed_scores) 85 | remaining: List[Tuple[int, float]] = [] 86 | candidates: List[Tuple[int, float]] = [] 87 | outputs: List[Detection] = [] 88 | 89 | while len(remaining_indexed_scores): 90 | detection = detections[remaining_indexed_scores[0][0]] 91 | # exit loop if remaining scores are below threshold 92 | if min_score is not None and detection.score < min_score: 93 | break 94 | num_prev_indexed_scores = len(remaining_indexed_scores) 95 | detection_bbox = detection.bbox 96 | remaining.clear() 97 | candidates.clear() 98 | weighted_detection = detection 99 | for (index, score) in remaining_indexed_scores: 100 | remaining_bbox = detections[index].bbox 101 | similarity = _overlap_similarity(remaining_bbox, detection_bbox) 102 | if similarity > min_suppression_threshold: 103 | candidates.append((index, score)) 104 | else: 105 | remaining.append((index, score)) 106 | # weighted merging of similar (close) boxes 107 | if len(candidates): 108 | weighted = np.zeros((2 + len(detection), 2), dtype=np.float32) 109 | total_score = 0. 110 | for index, score in candidates: 111 | total_score += score 112 | weighted += detections[index].data * score 113 | weighted /= total_score 114 | weighted_detection = Detection(weighted, detection.score) 115 | outputs.append(weighted_detection) 116 | # exit the loop if the number of indexed scores didn't change 117 | if num_prev_indexed_scores == len(remaining): 118 | break 119 | remaining_indexed_scores = list(remaining) 120 | return outputs 121 | -------------------------------------------------------------------------------- /fdlite/render.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright © 2021 Patrick Levin 3 | # SPDX-Identifier: MIT 4 | from dataclasses import dataclass 5 | from typing import List, Optional, Sequence, Tuple, Union 6 | from PIL import ImageDraw 7 | from PIL.Image import Image as PILImage 8 | from fdlite import CoordinateRangeError 9 | from fdlite.types import Detection, Landmark 10 | """Types and functions related to rendering detection results""" 11 | 12 | 13 | @dataclass 14 | class Color: 15 | """Color for rendering annotations""" 16 | r: int = 0 17 | g: int = 0 18 | b: int = 0 19 | a: Optional[int] = None 20 | 21 | @property 22 | def as_tuple( 23 | self 24 | ) -> Union[Tuple[int, int, int], Tuple[int, int, int, int]]: 25 | """Return color components as tuple""" 26 | r, g, b, a = self.r, self.g, self.b, self.a 27 | return (r, g, b) if a is None else (r, g, b, a) 28 | 29 | 30 | class Colors: 31 | """Predefined common color values for use with annotations rendering""" 32 | BLACK = Color() 33 | RED = Color(r=255) 34 | GREEN = Color(g=255) 35 | BLUE = Color(b=255) 36 | PINK = Color(r=255, b=255) 37 | WHITE = Color(r=255, g=255, b=255) 38 | 39 | 40 | @dataclass 41 | class Point: 42 | """A point to be rendered""" 43 | x: float 44 | y: float 45 | 46 | @property 47 | def as_tuple(self) -> Tuple[float, float]: 48 | """Values as a tuple of (x, y)""" 49 | return self.x, self.y 50 | 51 | def scaled(self, factor: Tuple[float, float]) -> 'Point': 52 | """Return a point with an absolute position""" 53 | sx, sy = factor 54 | return Point(self.x * sx, self.y * sy) 55 | 56 | 57 | @dataclass 58 | class RectOrOval: 59 | """A rectangle or oval to be rendered""" 60 | left: float 61 | top: float 62 | right: float 63 | bottom: float 64 | oval: bool = False 65 | 66 | @property 67 | def as_tuple(self) -> Tuple[float, float, float, float]: 68 | """Values as a tuple of (left, top, right, bottom)""" 69 | return self.left, self.top, self.right, self.bottom 70 | 71 | def scaled(self, factor: Tuple[float, float]) -> 'RectOrOval': 72 | """Return a rect or oval with absolute positions""" 73 | sx, sy = factor 74 | return RectOrOval(self.left * sx, self.top * sy, 75 | self.right * sx, self.bottom * sy, self.oval) 76 | 77 | 78 | @dataclass 79 | class FilledRectOrOval: 80 | """A filled rectangle or oval to be rendered""" 81 | rect: RectOrOval 82 | fill: Color 83 | 84 | def scaled(self, factor: Tuple[float, float]) -> 'FilledRectOrOval': 85 | """Return a filled rect or oval with absolute positions""" 86 | return FilledRectOrOval(self.rect.scaled(factor), self.fill) 87 | 88 | 89 | @dataclass 90 | class Line: 91 | """A solid or dashed line to be rendered""" 92 | x_start: float 93 | y_start: float 94 | x_end: float 95 | y_end: float 96 | dashed: bool = False 97 | 98 | @property 99 | def as_tuple(self) -> Tuple[float, float, float, float]: 100 | """Values as a tuple of (x_start, y_start, x_end, y_end)""" 101 | return self.x_start, self.y_start, self.x_end, self.y_end 102 | 103 | def scaled(self, factor: Tuple[float, float]) -> 'Line': 104 | """Return line with absolute positions""" 105 | sx, sy = factor 106 | return Line(self.x_start * sx, self.y_start * sy, 107 | self.x_end * sx, self.y_end * sy, self.dashed) 108 | 109 | 110 | @dataclass 111 | class Annotation: 112 | """Graphical annotation to be rendered 113 | 114 | Massively cut-down version of the MediaPipe type. 115 | Annotation data is bundled for higher data efficiency. 116 | Normalisation flag, thickness, and color apply to all 117 | items in the data list to reduce redundancy. 118 | 119 | The corresponding converter functions will automatically 120 | bundle data by type and format. 121 | """ 122 | data: Sequence[Union[Point, RectOrOval, FilledRectOrOval, Line]] 123 | normalized_positions: bool 124 | thickness: float 125 | color: Color 126 | 127 | def scaled(self, factor: Tuple[float, float]) -> 'Annotation': 128 | """Return an annotation with all positions scaled yb a given factor 129 | 130 | Args: 131 | factor (tuple): Scaling factor as a tuple of `(width, height)`. 132 | 133 | Raises: 134 | CoordinateRangeError: position data is not normalized 135 | 136 | Returns: 137 | (Annotation) Annotation with all elements scaled to the given 138 | size. 139 | """ 140 | if not self.normalized_positions: 141 | raise CoordinateRangeError('position data must be normalized') 142 | scaled_data = [item.scaled(factor) for item in self.data] 143 | return Annotation(scaled_data, normalized_positions=False, 144 | thickness=self.thickness, color=self.color) 145 | 146 | 147 | def detections_to_render_data( 148 | detections: Sequence[Detection], 149 | bounds_color: Optional[Color] = None, 150 | keypoint_color: Optional[Color] = None, 151 | line_width: int = 1, 152 | point_width: int = 3, 153 | normalized_positions: bool = True, 154 | output: Optional[List[Annotation]] = None 155 | ) -> List[Annotation]: 156 | """Convert detections to render data. 157 | 158 | This is an implementation of the MediaPipe DetectionToRenderDataCalculator 159 | node with keypoints added. 160 | 161 | Args: 162 | detections (list): List of detects, which will be converted to 163 | individual points and a bounding box rectangle for rendering. 164 | 165 | bounds_color (RenderColor|None): Color of the bounding box; if `None` 166 | the bounds won't be rendered. 167 | 168 | keypoint_color (RenderColor|None): Color of the keypoints that will 169 | will be rendered as points; set to `None` to disable keypoint 170 | rendering. 171 | 172 | line_width (int): Thickness of the lines in viewport units 173 | (e.g. pixels). 174 | 175 | point_width (int): Size of the keypoints in viewport units 176 | (e.g. pixels). 177 | 178 | normalized_positions (bool): Flag indicating whether the detections 179 | contain normalised data (e.g. range [0,1]). 180 | 181 | output (RenderData): Optional render data instance to add the items 182 | to. If not provided, a new instance of `RenderData` will be 183 | created. 184 | Use this to add multiple landmark detections into a single render 185 | data bundle. 186 | 187 | Returns: 188 | (list) List of annotations for rendering landmarks. 189 | """ 190 | 191 | def to_rect(detection: Detection) -> RectOrOval: 192 | bbox = detection.bbox 193 | return RectOrOval(bbox.xmin, bbox.ymin, bbox.xmax, bbox.ymax) 194 | 195 | annotations = [] 196 | if bounds_color is not None and line_width > 0: 197 | bounds = Annotation([to_rect(detection) for detection in detections], 198 | normalized_positions, thickness=line_width, 199 | color=bounds_color) 200 | annotations.append(bounds) 201 | if keypoint_color is not None and point_width > 0: 202 | points = Annotation([Point(x, y) 203 | for detection in detections 204 | for (x, y) in detection], 205 | normalized_positions, thickness=point_width, 206 | color=keypoint_color) 207 | annotations.append(points) 208 | if output is not None: 209 | output += annotations 210 | else: 211 | output = annotations 212 | return output 213 | 214 | 215 | def landmarks_to_render_data( 216 | landmarks: Sequence[Landmark], 217 | landmark_connections: Sequence[Tuple[int, int]], 218 | landmark_color: Color = Colors.RED, 219 | connection_color: Color = Colors.RED, 220 | thickness: float = 1., 221 | normalized_positions: bool = True, 222 | output: Optional[List[Annotation]] = None 223 | ) -> List[Annotation]: 224 | """Convert detected landmarks to render data. 225 | 226 | This is an implementation of the MediaPipe LandmarksToRenderDataCalculator 227 | node. 228 | 229 | Args: 230 | landmarks (list): List of detected landmarks, which will be converted 231 | to individual points for rendering. 232 | 233 | landmark_connections (list): List of tuples in the form of 234 | `(offset_from, offset_to)` that represent connections between 235 | landmarks. The offsets are zero-based indexes into `landmarks` 236 | and each tuple will be converted to a line for rendering. 237 | 238 | landmark_color (RenderColor): Color of the individual landmark points. 239 | 240 | connection_color (RenderColor): Color of the landmark connections that 241 | will be rendered as lines. 242 | 243 | thickness (float): Thickness of the lines and landmark point size in 244 | viewport units (e.g. pixels). 245 | 246 | normalized_positions (bool): Flag indicating whether the landmarks 247 | contain normalised data (e.g. range [0,1]). 248 | 249 | output (RenderData): Optional render data instance to add the items 250 | to. If not provided, a new instance of `RenderData` will be 251 | created. 252 | Use this to add multiple landmark detections into a single render 253 | data bundle. 254 | 255 | Returns: 256 | (list) List of annotations for rendering landmarks. 257 | """ 258 | lm = landmarks 259 | lines = [Line(lm[start].x, lm[start].y, lm[end].x, lm[end].y) 260 | for start, end in landmark_connections] 261 | points = [Point(landmark.x, landmark.y) for landmark in landmarks] 262 | la = Annotation(lines, normalized_positions, thickness, connection_color) 263 | pa = Annotation(points, normalized_positions, thickness, landmark_color) 264 | if output is not None: 265 | output += [la, pa] 266 | else: 267 | output = [la, pa] 268 | return output 269 | 270 | 271 | def render_to_image( 272 | annotations: Sequence[Annotation], 273 | image: PILImage, 274 | blend: bool = False 275 | ) -> PILImage: 276 | """Render annotations to an image. 277 | 278 | Args: 279 | annotations (list): List of annotations. 280 | 281 | image (Image): PIL Image instance to render to. 282 | 283 | blend (bool): If `True`, allows for alpha-blending annotations on 284 | top of the image. 285 | 286 | Returns: 287 | (Image) Returns the modified image. 288 | """ 289 | draw = ImageDraw.Draw(image, mode='RGBA' if blend else 'RGB') 290 | for annotation in annotations: 291 | if annotation.normalized_positions: 292 | scaled = annotation.scaled(image.size) 293 | else: 294 | scaled = annotation 295 | if not len(scaled.data): 296 | continue 297 | thickness = int(scaled.thickness) 298 | color = scaled.color 299 | for item in scaled.data: 300 | if isinstance(item, Point): 301 | w = max(thickness // 2, 1) 302 | rc = [item.x - w, item.y - w, item.x + w, item.y + w] 303 | draw.rectangle(rc, fill=color.as_tuple, outline=color.as_tuple) 304 | elif isinstance(item, Line): 305 | coords = [item.x_start, item.y_start, item.x_end, item.y_end] 306 | draw.line(coords, fill=color.as_tuple, width=thickness) 307 | elif isinstance(item, RectOrOval): 308 | rc = [item.left, item.top, item.right, item.bottom] 309 | if item.oval: 310 | draw.ellipse(rc, outline=color.as_tuple, width=thickness) 311 | else: 312 | draw.rectangle(rc, outline=color.as_tuple, width=thickness) 313 | elif isinstance(item, FilledRectOrOval): 314 | rgb = color.as_tuple 315 | rect, fill = item.rect, item.fill.as_tuple 316 | rc = [rect.left, rect.top, rect.right, rect.bottom] 317 | if rect.oval: 318 | draw.ellipse(rc, fill=fill, outline=rgb, width=thickness) 319 | else: 320 | draw.rectangle(rc, fill=fill, outline=rgb, width=thickness) 321 | else: 322 | # don't know how to render this 323 | pass 324 | return image 325 | -------------------------------------------------------------------------------- /fdlite/transform.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright © 2021 Patrick Levin 3 | # SPDX-Identifier: MIT 4 | from enum import IntEnum 5 | from typing import List, Optional, Sequence, Tuple, Union 6 | import numpy as np 7 | from PIL import Image 8 | from PIL.Image import Image as PILImage, Resampling, Transform, Transpose 9 | from fdlite import ArgumentError, CoordinateRangeError, InvalidEnumError 10 | from fdlite.types import BBox, Detection, ImageTensor, Landmark, Rect 11 | """Functions for data transformations that are used by the detection models""" 12 | 13 | 14 | def image_to_tensor( 15 | image: Union[PILImage, np.ndarray, str], 16 | roi: Optional[Rect] = None, 17 | output_size: Optional[Tuple[int, int]] = None, 18 | keep_aspect_ratio: bool = False, 19 | output_range: Tuple[float, float] = (0., 1.), 20 | flip_horizontal: bool = False 21 | ) -> ImageTensor: 22 | """Load an image into an array and return data, image size, and padding. 23 | 24 | This function combines the mediapipe calculator-nodes ImageToTensor, 25 | ImageCropping, and ImageTransformation into one function. 26 | 27 | Args: 28 | image (Image|ndarray|str): Input image; preferably RGB, but will be 29 | converted if necessary; loaded from file if a string is given 30 | 31 | roi (Rect|None): Location within the image where to convert; can be 32 | `None`, in which case the entire image is converted. Rotation is 33 | supported. 34 | 35 | output_size (tuple|None): Tuple of `(width, height)` describing the 36 | output tensor size; defaults to ROI if `None`. 37 | 38 | keep_aspect_ratio (bool): `False` (default) will scale the image to 39 | the output size; `True` will keep the ROI aspect ratio and apply 40 | letterboxing. 41 | 42 | output_range (tuple): Tuple of `(min_val, max_val)` containing the 43 | minimum and maximum value of the output tensor. 44 | Defaults to (0, 1). 45 | 46 | flip_horizontal (bool): Flip the resulting image horizontally if set 47 | to `True`. Default: `False` 48 | 49 | Returns: 50 | (ImageTensor) Tensor data, padding for reversing letterboxing and 51 | original image dimensions. 52 | """ 53 | img = _normalize_image(image) 54 | image_size = img.size 55 | if roi is None: 56 | roi = Rect(0.5, 0.5, 1.0, 1.0, rotation=0.0, normalized=True) 57 | roi = roi.scaled(image_size) 58 | if output_size is None: 59 | output_size = (int(roi.size[0]), int(roi.size[1])) 60 | width, height = (roi.size if keep_aspect_ratio # type: ignore[misc] 61 | else output_size) 62 | src_points = roi.points() 63 | dst_points = [(0., 0.), (width, 0.), (width, height), (0., height)] 64 | coeffs = _perspective_transform_coeff(src_points, dst_points) 65 | roi_image = img.transform(size=(width, height), method=Transform.PERSPECTIVE, 66 | data=coeffs, resample=Resampling.BILINEAR) 67 | # free some memory - we don't need the temporary image anymore 68 | if img != image: 69 | img.close() 70 | pad_x, pad_y = 0., 0. 71 | if keep_aspect_ratio: 72 | # perform letterboxing if required 73 | out_aspect = output_size[1] / output_size[0] # type: ignore[index] 74 | roi_aspect = roi.height / roi.width 75 | new_width, new_height = int(roi.width), int(roi.height) 76 | if out_aspect > roi_aspect: 77 | new_height = int(roi.width * out_aspect) 78 | pad_y = (1 - roi_aspect / out_aspect) / 2 79 | else: 80 | new_width = int(roi.height / out_aspect) 81 | pad_x = (1 - out_aspect / roi_aspect) / 2 82 | if new_width != int(roi.width) or new_height != int(roi.height): 83 | pad_h, pad_v = int(pad_x * new_width), int(pad_y * new_height) 84 | roi_image = roi_image.transform( 85 | size=(new_width, new_height), method=Transform.EXTENT, 86 | data=(-pad_h, -pad_v, new_width - pad_h, new_height - pad_v)) 87 | roi_image = roi_image.resize(output_size, resample=Resampling.BILINEAR) 88 | if flip_horizontal: 89 | roi_image = roi_image.transpose(method=Transpose.FLIP_LEFT_RIGHT) 90 | # finally, apply value range transform 91 | min_val, max_val = output_range 92 | tensor_data = np.asarray(roi_image, dtype=np.float32) 93 | tensor_data *= (max_val - min_val) / 255 94 | tensor_data += min_val 95 | return ImageTensor(tensor_data, 96 | padding=(pad_x, pad_y, pad_x, pad_y), 97 | original_size=image_size) 98 | 99 | 100 | def sigmoid(data: np.ndarray) -> np.ndarray: 101 | """Return sigmoid activation of the given data 102 | 103 | Args: 104 | data (ndarray): Numpy array containing data 105 | 106 | Returns: 107 | (ndarray) Sigmoid activation of the data with element range (0,1] 108 | """ 109 | return 1 / (1 + np.exp(-data)) 110 | 111 | 112 | def detection_letterbox_removal( 113 | detections: Sequence[Detection], 114 | padding: Tuple[float, float, float, float] 115 | ) -> List[Detection]: 116 | """Return detections with bounding box and keypoints adjusted for padding 117 | 118 | Args: 119 | detections (list): List of detection results with relative coordinates 120 | 121 | padding (tuple): Tuple of (`float`,`float`,`float`,`float`) containing 122 | the padding value [0,1) for left, top, right and bottom sides. 123 | 124 | Returns: 125 | (list) List of detections with relative coordinates adjusted to remove 126 | letterboxing. 127 | """ 128 | left, top, right, bottom = padding 129 | h_scale = 1 - (left + right) 130 | v_scale = 1 - (top + bottom) 131 | 132 | def adjust_data(detection: Detection) -> Detection: 133 | adjusted = (detection.data - (left, top)) / (h_scale, v_scale) 134 | return Detection(adjusted, detection.score) 135 | 136 | return [adjust_data(detection) for detection in detections] 137 | 138 | 139 | class SizeMode(IntEnum): 140 | """Size mode for `bbox_to_roi` 141 | 142 | DEFAULT - keep width and height as calculated 143 | 144 | SQUARE_LONG - make square using `max(width, height)` 145 | 146 | SQUARE_SHORT - make square using `min(width, height)` 147 | """ 148 | DEFAULT = 0 149 | SQUARE_LONG = 1 150 | SQUARE_SHORT = 2 151 | 152 | 153 | def bbox_to_roi( 154 | bbox: BBox, 155 | image_size: Tuple[int, int], 156 | rotation_keypoints: Optional[Sequence[Tuple[float, float]]] = None, 157 | scale: Tuple[float, float] = (1., 1.), 158 | size_mode: SizeMode = SizeMode.DEFAULT 159 | ) -> Rect: 160 | """Convert a normalized bounding box into a ROI with optional scaling and 161 | and rotation. 162 | 163 | This function combines parts of DetectionsToRect and RectTransformation 164 | MediaPipe nodes. 165 | 166 | Args: 167 | bbox (bbox): Normalized bounding box to convert. 168 | 169 | image_size (tuple): Image size for the bounding box as a tuple 170 | of `(image_width, image_height)`. 171 | 172 | rotation_keypoints (list|None): Optional list of keypoints to get the 173 | target rotation from; expected format: `[(x1, y1), (x2, y2)]` 174 | 175 | scale (tuple): Tuple of `(scale_x, scale_y)` that determines the 176 | scaling of the requested ROI. 177 | 178 | size_mode (SizeMode): Determines the way the ROI dimensions should be 179 | determined. Default keeps the bounding box proportions as-is, 180 | while the other modes result in a square with a length matching 181 | either the shorter or longer side. 182 | 183 | Returns: 184 | (Rect) Normalized and possibly rotated ROI rectangle. 185 | 186 | Raises: 187 | CoordinateRangeError: bbox is not in normalised coordinates (0 to 1) 188 | InvalidEnumError: `size_mode` contains an unsupported value 189 | """ 190 | if not bbox.normalized: 191 | raise CoordinateRangeError('bbox must be normalized') 192 | PI = np.math.pi 193 | TWO_PI = 2 * PI 194 | # select ROI dimensions 195 | width, height = _select_roi_size(bbox, image_size, size_mode) 196 | scale_x, scale_y = scale 197 | # calculate ROI size and -centre 198 | width, height = width * scale_x, height * scale_y 199 | cx, cy = bbox.xmin + bbox.width / 2, bbox.ymin + bbox.height / 2 200 | # calculate rotation of required 201 | if rotation_keypoints is None or len(rotation_keypoints) < 2: 202 | return Rect(cx, cy, width, height, rotation=0., normalized=True) 203 | x0, y0 = rotation_keypoints[0] 204 | x1, y1 = rotation_keypoints[1] 205 | angle = -np.math.atan2(y0 - y1, x1 - x0) 206 | # normalise to [0, 2*PI] 207 | rotation = angle - TWO_PI * np.math.floor((angle + PI) / TWO_PI) 208 | return Rect(cx, cy, width, height, rotation, normalized=True) 209 | 210 | 211 | def bbox_from_landmarks(landmarks: Sequence[Landmark]) -> BBox: 212 | """Return the bounding box that encloses all landmarks in a given list. 213 | 214 | This function combines the MediaPipe nodes LandmarksToDetectionCalculator 215 | and DetectionToRectCalculator. 216 | 217 | Args: 218 | landmarks (list): List of landmark detection results. Must contain ar 219 | least two items. 220 | 221 | Returns: 222 | (BBox) Bounding box that contains all points defined by the landmarks. 223 | 224 | Raises: 225 | ArgumentError: `landmarks` contains less than two (2) items 226 | """ 227 | if len(landmarks) < 2: 228 | raise ArgumentError('landmarks must contain at least 2 items') 229 | xmin, ymin = 999999., 999999. 230 | xmax, ymax = -999999., -999999. 231 | for landmark in landmarks: 232 | x, y = landmark.x, landmark.y 233 | xmin, ymin = min(xmin, x), min(ymin, y) 234 | xmax, ymax = max(xmax, x), max(ymax, y) 235 | return BBox(xmin, ymin, xmax, ymax) 236 | 237 | 238 | def project_landmarks( 239 | data: Union[Sequence[Landmark], np.ndarray], 240 | *, 241 | tensor_size: Tuple[int, int], 242 | image_size: Tuple[int, int], 243 | padding: Tuple[float, float, float, float], 244 | roi: Optional[Rect], 245 | flip_horizontal: bool = False 246 | ) -> List[Landmark]: 247 | """Transform landmarks or raw detection results from tensor coordinates 248 | into normalized image coordinates, removing letterboxing if required. 249 | 250 | Args: 251 | data (list|ndarray): List of landmarks or numpy array with number of 252 | elements divisible by 3. 253 | 254 | tensor_size (tuple): Tuple of `(width, height)` denoting the input 255 | tensor size. 256 | 257 | image_size (tuple): Tuple of `(width, height)` denoting the image 258 | size. 259 | 260 | padding (tuple): Tuple of `(pad_left, pad_top, pad_right, pad_bottom)` 261 | denoting padding from letterboxing. 262 | 263 | roi (Rect|None): Optional ROI from which the input data was taken. 264 | 265 | flip_horizontal (bool): Flip the image from left to right if `True` 266 | 267 | Returns: 268 | (list) List of normalized landmarks projected into image space. 269 | """ 270 | # normalize input type 271 | if not isinstance(data, np.ndarray): 272 | points = np.array([(pt.x, pt.y, pt.z) for pt in data], dtype='float32') 273 | else: 274 | points = data.reshape(-1, 3) 275 | # normalize to tensor coordinates 276 | width, height = tensor_size 277 | points /= (width, height, width) 278 | # flip left to right if requested 279 | if flip_horizontal: 280 | points[:, 0] *= -1 281 | points[:, 0] += 1 282 | # letterbox removal if required 283 | if any(padding): 284 | left, top, right, bottom = padding 285 | h_scale = 1 - (left + right) 286 | v_scale = 1 - (top + bottom) 287 | points -= (left, top, 0.) 288 | points /= (h_scale, v_scale, h_scale) 289 | # convert to landmarks if coordinate system doesn't change 290 | if roi is None: 291 | return [Landmark(x, y, z) for (x, y, z) in points] 292 | # coordinate system transformation from ROI- to image space 293 | norm_roi = roi.scaled(image_size, normalize=True) 294 | sin, cos = np.math.sin(roi.rotation), np.math.cos(roi.rotation) 295 | matrix = np.array([[cos, sin, 0.], [-sin, cos, 0.], [1., 1., 1.]]) 296 | points -= (0.5, 0.5, 0.0) 297 | rotated = np.matmul(points * (1, 1, 0), matrix) 298 | points *= (0, 0, 1) 299 | points += rotated 300 | points *= (norm_roi.width, norm_roi.height, norm_roi.width) 301 | points += (norm_roi.x_center, norm_roi.y_center, 0.0) 302 | return [Landmark(x, y, z) for (x, y, z) in points] 303 | 304 | 305 | def _perspective_transform_coeff( 306 | src_points: np.ndarray, 307 | dst_points: np.ndarray 308 | ) -> np.ndarray: 309 | """Calculate coefficients for a perspective transform given source- and 310 | target points. Note: argument order is reversed for more intuitive 311 | usage. 312 | 313 | Reference: 314 | https://web.archive.org/web/20150222120106/xenia.media.mit.edu/~cwren/interpolator/ 315 | """ 316 | matrix = [] 317 | for (x, y), (X, Y) in zip(dst_points, src_points): 318 | matrix.extend([ 319 | [x, y, 1., 0., 0., 0., -X*x, -X*y], 320 | [0., 0., 0., x, y, 1., -Y*x, -Y*y] 321 | ]) 322 | A = np.array(matrix, dtype=np.float32) 323 | B = np.array(src_points, dtype=np.float32).reshape(8) 324 | return np.linalg.solve(A, B) 325 | 326 | 327 | def _normalize_image(image: Union[PILImage, np.ndarray, str]) -> PILImage: 328 | """Return PIL Image instance in RGB-mode from input""" 329 | if isinstance(image, PILImage) and image.mode != 'RGB': 330 | return image.convert(mode='RGB') 331 | if isinstance(image, np.ndarray): 332 | return Image.fromarray(image, mode='RGB') 333 | if not isinstance(image, PILImage): 334 | return Image.open(image) 335 | return image 336 | 337 | 338 | def _select_roi_size( 339 | bbox: BBox, 340 | image_size: Tuple[int, int], 341 | size_mode: SizeMode 342 | ) -> Tuple[float, float]: 343 | """Return the size of an ROI based on bounding box, image size and mode""" 344 | abs_box = bbox.absolute(image_size) 345 | width, height = abs_box.width, abs_box.height 346 | image_width, image_height = image_size 347 | if size_mode == SizeMode.SQUARE_LONG: 348 | long_size = max(width, height) 349 | width, height = long_size / image_width, long_size / image_height 350 | elif size_mode == SizeMode.SQUARE_SHORT: 351 | short_side = min(width, height) 352 | width, height = short_side / image_width, short_side / image_height 353 | elif size_mode != SizeMode.DEFAULT: 354 | raise InvalidEnumError(f'unsupported size_mode: {size_mode}') 355 | return width, height 356 | -------------------------------------------------------------------------------- /fdlite/types.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright © 2021 Patrick Levin 3 | # SPDX-Identifier: MIT 4 | from dataclasses import dataclass 5 | import numpy as np 6 | from typing import Optional, Tuple, Union 7 | """Types used throughout the library""" 8 | 9 | 10 | @dataclass 11 | class ImageTensor: 12 | """Tensor data obtained from an image with optional letterboxing. 13 | The original image size is kept track of. 14 | 15 | The data may contain an extra dimension for batching (the default). 16 | """ 17 | tensor_data: np.ndarray 18 | padding: Tuple[float, float, float, float] 19 | original_size: Tuple[int, int] 20 | 21 | 22 | @dataclass 23 | class Rect: 24 | """A rotated rectangle 25 | 26 | Rotation is given in radians (clockwise). 27 | Normalized indicates whether properties are relative to image size 28 | (i.e. value range is [0,1]). 29 | """ 30 | x_center: float 31 | y_center: float 32 | width: float 33 | height: float 34 | rotation: float 35 | normalized: bool 36 | 37 | @property 38 | def size(self) -> Union[Tuple[float, float], Tuple[int, int]]: 39 | """Tuple of `(width, height)`""" 40 | w, h = self.width, self.height 41 | return (w, h) if self.normalized else (int(w), int(h)) 42 | 43 | def scaled( 44 | self, size: Tuple[float, float], normalize: bool = False 45 | ) -> 'Rect': 46 | """Return a scaled version or self, if not normalized.""" 47 | if self.normalized == normalize: 48 | return self 49 | sx, sy = size 50 | if normalize: 51 | sx, sy = 1 / sx, 1 / sy 52 | return Rect(self.x_center * sx, self.y_center * sy, 53 | self.width * sx, self.height * sy, 54 | self.rotation, normalized=False) 55 | 56 | def points(self) -> np.ndarray: 57 | """Return the corners of the box as a list of tuples `[(x, y), ...]` 58 | """ 59 | x, y = self.x_center, self.y_center 60 | w, h = self.width / 2, self.height / 2 61 | pts = [(x - w, y - h), (x + w, y - h), (x + w, y + h), (x - w, y + h)] 62 | if self.rotation == 0: 63 | return pts 64 | s, c = np.math.sin(self.rotation), np.math.cos(self.rotation) 65 | t = np.array(pts) - (x, y) 66 | r = np.array([[c, s], [-s, c]]) 67 | return np.matmul(t, r) + (x, y) 68 | 69 | 70 | @dataclass 71 | class BBox: 72 | """A non-rotated bounding box. 73 | 74 | The bounds can be relative to image size (normalized, range [0,1]) or 75 | absolute (i.e. pixel-) coordinates. 76 | """ 77 | xmin: float 78 | ymin: float 79 | xmax: float 80 | ymax: float 81 | 82 | @property 83 | def as_tuple(self) -> Tuple[float, float, float, float]: 84 | """Upper-left and bottom-right as a tuple""" 85 | return self.xmin, self.ymin, self.xmax, self.ymax 86 | 87 | @property 88 | def width(self) -> float: 89 | """Width of the box""" 90 | return self.xmax - self.xmin 91 | 92 | @property 93 | def height(self) -> float: 94 | """Height of the box""" 95 | return self.ymax - self.ymin 96 | 97 | @property 98 | def empty(self) -> bool: 99 | """True if the box is empty""" 100 | return self.width <= 0 or self.height <= 0 101 | 102 | @property 103 | def normalized(self) -> bool: 104 | """True if the box contains normalized coordinates""" 105 | return self.xmin >= -1 and self.xmax < 2 and self.ymin >= -1 106 | 107 | @property 108 | def area(self) -> float: 109 | """Area of the bounding box, 0 if empty""" 110 | return self.width * self.height if not self.empty else 0 111 | 112 | def intersect(self, other: 'BBox') -> Optional['BBox']: 113 | """Return the intersection with another (non-rotated) bounding box 114 | 115 | Args: 116 | other (BBox): Bounding box to intersect with 117 | 118 | Returns: 119 | (BBox) Intersection between the two bounding boxes; `None` if the 120 | boxes are disjoint. 121 | """ 122 | xmin, ymin = max(self.xmin, other.xmin), max(self.ymin, other.ymin) 123 | xmax, ymax = min(self.xmax, other.xmax), min(self.ymax, other.ymax) 124 | if xmin < xmax and ymin < ymax: 125 | return BBox(xmin, ymin, xmax, ymax) 126 | else: 127 | return None 128 | 129 | def scale(self, size: Tuple[float, float]) -> 'BBox': 130 | """Scale the bounding box""" 131 | sx, sy = size 132 | xmin, ymin = self.xmin * sx, self.ymin * sy 133 | xmax, ymax = self.xmax * sx, self.ymax * sy 134 | return BBox(xmin, ymin, xmax, ymax) 135 | 136 | def absolute(self, size: Tuple[int, int]) -> 'BBox': 137 | """Return the box in absolute coordinates (if normalized) 138 | 139 | Args: 140 | size (tuple): Tuple of `(image_width, image_height)` that denotes 141 | the image dimensions. Ignored if the box is not normalized. 142 | 143 | Returns: 144 | (BBox) Bounding box in absolute pixel coordinates. 145 | """ 146 | if not self.normalized: 147 | return self 148 | return self.scale(size) 149 | 150 | 151 | @dataclass 152 | class Landmark: 153 | """An object landmark (3d point) detected by a model""" 154 | x: float 155 | y: float 156 | z: float 157 | 158 | 159 | class Detection(object): 160 | """An object detection made by a model. 161 | 162 | A detection consists of a bounding box and zero or more 2d keypoints. 163 | Keypoints can be accessed directly via indexing or iterating a detection: 164 | 165 | ``` 166 | detection = ... 167 | # loop through keypoints 168 | for keypoint in detection: 169 | print(keypoint) 170 | # access a keypoint by index 171 | nosetip = detection[3] 172 | ``` 173 | """ 174 | def __init__(self, data: np.ndarray, score: float) -> None: 175 | """Initialize a detection from data points. 176 | 177 | Args: 178 | data (ndarray): Array of `[xmin, ymin, xmax, ymax, ...]` followed 179 | by zero or more x and y coordinates. 180 | 181 | score (float): Confidence score of the detection; range [0, 1]. 182 | """ 183 | self.data = data.reshape(-1, 2) 184 | self.score = score 185 | 186 | def __len__(self) -> int: 187 | """Number of keypoints""" 188 | return len(self.data) - 2 189 | 190 | def __getitem__(self, key: int) -> Tuple[float, float]: 191 | """Keypoint by index""" 192 | x, y = self.data[key + 2] 193 | return x, y 194 | 195 | def __iter__(self): 196 | """Keypoints iterator""" 197 | return iter(self.data[2:]) 198 | 199 | @property 200 | def bbox(self) -> BBox: 201 | """The bounding box of this detection.""" 202 | xmin, ymin = self.data[0] 203 | xmax, ymax = self.data[1] 204 | return BBox(xmin, ymin, xmax, ymax) 205 | 206 | def scaled(self, factor: Union[Tuple[float, float], float]) -> 'Detection': 207 | """Return a scaled version of the bounding box and keypoints""" 208 | return Detection(self.data * factor, self.score) 209 | -------------------------------------------------------------------------------- /mypy.ini: -------------------------------------------------------------------------------- 1 | [mypy] 2 | ignore_missing_imports = True 3 | 4 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = [ 3 | "setuptools>=42", 4 | "wheel" 5 | ] 6 | build-backend = "setuptools.build_meta" 7 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | [metadata] 3 | name = face-detection-tflite 4 | version = 0.6.0 5 | author = Patrick Levin 6 | author_email = vertical-pink@protonmail.com 7 | description = A Python port of Google MediaPipe Face Detection modules 8 | long_description = file: README.md 9 | long_description_content_type = text/markdown 10 | keywords = AI, face-detection, tensorflow, tflite, face-landmarks, iris-detection, face-mesh 11 | url = https://github.com/patlevin/face-detection-tflite 12 | project_urls = 13 | Bug Tracker = https://github.com/patlevin/face-detection-tflite/issues 14 | license = MIT 15 | license_file = LICENSE 16 | classifiers = 17 | Programming Language :: Python :: 3 18 | License :: OSI Approved :: MIT License 19 | Operating System :: OS Independent 20 | Development Status :: 4 - Beta 21 | Intended Audience :: Developers 22 | Topic :: Multimedia :: Graphics 23 | Topic :: Scientific/Engineering :: Artificial Intelligence 24 | Topic :: Scientific/Engineering :: Image Recognition 25 | Topic :: Software Development :: Libraries :: Python Modules 26 | 27 | [options] 28 | packages = find: 29 | python_requires = >= 3.8 30 | include_package_data = true 31 | install_requires = 32 | tensorflow>=2.3 33 | Pillow>=10.3.0 34 | 35 | [flake8] 36 | exclude = 37 | .eggs, 38 | .git, 39 | __pycache__, 40 | build, 41 | dist 42 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Copyright © 2021,2024 Patrick Levin 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy of 7 | this software and associated documentation files (the "Software"), to deal in 8 | the Software without restriction, including without limitation the rights to 9 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 10 | of the Software, and to permit persons to whom the Software is furnished to do 11 | so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | """ 24 | from setuptools import setup, __version__ 25 | from pkg_resources import parse_version 26 | 27 | minimum_version = parse_version('42.0.0') 28 | 29 | if parse_version(__version__) < minimum_version: 30 | raise RuntimeError( 31 | f'Package setuptools must be at least version {minimum_version}') 32 | 33 | setup() 34 | --------------------------------------------------------------------------------