├── .gitignore ├── LICENSE.md ├── README.md ├── face_mesh.blend ├── face_mesh.obj ├── face_mesh_perspective.blend ├── face_track.py └── shell.nix /.gitignore: -------------------------------------------------------------------------------- 1 | *.mdd 2 | *.blend 3 | *.blend? 4 | videos 5 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | # Creative Commons CC0 1.0 Universal 2 | 3 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED HEREUNDER. 4 | 5 | ## Statement of Purpose 6 | 7 | The laws of most jurisdictions throughout the world automatically confer exclusive Copyright and Related Rights (defined below) upon the creator and subsequent owner(s) (each and all, an "owner") of an original work of authorship and/or a database (each, a "Work"). 8 | 9 | Certain owners wish to permanently relinquish those rights to a Work for the purpose of contributing to a commons of creative, cultural and scientific works ("Commons") that the public can reliably and without fear of later claims of infringement build upon, modify, incorporate in other works, reuse and redistribute as freely as possible in any form whatsoever and for any purposes, including without limitation commercial purposes. These owners may contribute to the Commons to promote the ideal of a free culture and the further production of creative, cultural and scientific works, or to gain reputation or greater distribution for their Work in part through the use and efforts of others. 10 | 11 | For these and/or other purposes and motivations, and without any expectation of additional consideration or compensation, the person associating CC0 with a Work (the "Affirmer"), to the extent that he or she is an owner of Copyright and Related Rights in the Work, voluntarily elects to apply CC0 to the Work and publicly distribute the Work under its terms, with knowledge of his or her Copyright and Related Rights in the Work and the meaning and intended legal effect of CC0 on those rights. 12 | 13 | 1. __Copyright and Related Rights.__ A Work made available under CC0 may be protected by copyright and related or neighboring rights ("Copyright and Related Rights"). Copyright and Related Rights include, but are not limited to, the following: 14 | 15 | i. the right to reproduce, adapt, distribute, perform, display, communicate, and translate a Work; 16 | 17 | ii. moral rights retained by the original author(s) and/or performer(s); 18 | 19 | iii. publicity and privacy rights pertaining to a person's image or likeness depicted in a Work; 20 | 21 | iv. rights protecting against unfair competition in regards to a Work, subject to the limitations in paragraph 4(a), below; 22 | 23 | v. rights protecting the extraction, dissemination, use and reuse of data in a Work; 24 | 25 | vi. database rights (such as those arising under Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, and under any national implementation thereof, including any amended or successor version of such directive); and 26 | 27 | vii. other similar, equivalent or corresponding rights throughout the world based on applicable law or treaty, and any national implementations thereof. 28 | 29 | 2. __Waiver.__ To the greatest extent permitted by, but not in contravention of, applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and unconditionally waives, abandons, and surrenders all of Affirmer's Copyright and Related Rights and associated claims and causes of action, whether now known or unknown (including existing as well as future claims and causes of action), in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each member of the public at large and to the detriment of Affirmer's heirs and successors, fully intending that such Waiver shall not be subject to revocation, rescission, cancellation, termination, or any other legal or equitable action to disrupt the quiet enjoyment of the Work by the public as contemplated by Affirmer's express Statement of Purpose. 30 | 31 | 3. __Public License Fallback.__ Should any part of the Waiver for any reason be judged legally invalid or ineffective under applicable law, then the Waiver shall be preserved to the maximum extent permitted taking into account Affirmer's express Statement of Purpose. In addition, to the extent the Waiver is so judged Affirmer hereby grants to each affected person a royalty-free, non transferable, non sublicensable, non exclusive, irrevocable and unconditional license to exercise Affirmer's Copyright and Related Rights in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the "License"). The License shall be deemed effective as of the date CC0 was applied by Affirmer to the Work. Should any part of the License for any reason be judged legally invalid or ineffective under applicable law, such partial invalidity or ineffectiveness shall not invalidate the remainder of the License, and in such case Affirmer hereby affirms that he or she will not (i) exercise any of his or her remaining Copyright and Related Rights in the Work or (ii) assert any associated claims and causes of action with respect to the Work, in either case contrary to Affirmer's express Statement of Purpose. 32 | 33 | 4. __Limitations and Disclaimers.__ 34 | 35 | a. No trademark or patent rights held by Affirmer are waived, abandoned, surrendered, licensed or otherwise affected by this document. 36 | 37 | b. Affirmer offers the Work as-is and makes no representations or warranties of any kind concerning the Work, express, implied, statutory or otherwise, including without limitation warranties of title, merchantability, fitness for a particular purpose, non infringement, or the absence of latent or other defects, accuracy, or the present or absence of errors, whether or not discoverable, all to the greatest extent permissible under applicable law. 38 | 39 | c. Affirmer disclaims responsibility for clearing rights of other persons that may apply to the Work or any use thereof, including without limitation any person's Copyright and Related Rights in the Work. Further, Affirmer disclaims responsibility for obtaining any necessary consents, permissions or other rights required for any use of the Work. 40 | 41 | d. Affirmer understands and acknowledges that Creative Commons is not a party to this document and has no duty or obligation with respect to this CC0 or use of the Work. 42 | 43 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Face Track 2 | 3 | A very simple Python script that uses a not-at-all simple Python library to generate animated face meshes from video files. 4 | 5 | Basic usage: 6 | 7 | ```bash 8 | $ face_track.py path/to/video_file.mp4 path/to/output.mdd 9 | ``` 10 | 11 | This will create `output.mdd`, a NewTek MDD file. This file contains animated vertex data, which can be used to animate the face mesh in `face_mesh.obj` or `face_mesh.blend`. For example, by using the Mesh Cache modifier in Blender. 12 | 13 | 14 | ## Accounting for Perspective 15 | 16 | By default, this script just spits out the direct results of Google's Mediapipe library. Those results assume an orthographic camera, meaning that as the face gets closer/further away from the camera, the resulting face mesh actually gets bigger/smaller rather than adjusting its z-depth. 17 | 18 | To get a face mesh that (more or less) accounts for perspective, you can specify the camera's field of view. The script will then process the face mesh data to be consistent with that. (Note that this does not produce perfect results by any stretch, but it may be workable for certain use cases at least.) 19 | 20 | You can specify the field of view in one of two ways. Either in (horizontal) degrees: 21 | 22 | ```bash 23 | $ face_track.py --fov 85 path/to/video_file.mp4 path/to/output.mdd 24 | ``` 25 | 26 | Or with a sensor-size/lens-length pair: 27 | 28 | ```bash 29 | $ face_track.py --lens_len 35/50 path/to/video_file.mp4 path/to/output.mdd 30 | ``` 31 | 32 | Both options are designed to match the perspective camera model in Blender, and using the exact same camera settings should result in perfect alignment with that camera. 33 | 34 | A separate blend file, `face_mesh_perspective.blend`, is provided with the basic setup for a perspective-corrected face mesh generated this way. 35 | 36 | 37 | ## Requirements 38 | 39 | face_track.py requires the following: 40 | 41 | - [Python 3](https://www.python.org) 42 | - [Numpy](https://numpy.org) 43 | - [OpenCV](https://opencv.org) and its Python bindings 44 | - [Mediapipe](https://mediapipe.dev) and its Python bindings 45 | 46 | 47 | ## Support 48 | 49 | This project is a quick hack held together by ducttape and glue, and is 100% unsupported. Issue reports and contributions are welcome, but we make no promises they will be acted upon. 50 | 51 | If you would like to further develop this, we recommend forking it into a new project. 52 | 53 | 54 | ## License 55 | 56 | All files in this project are licensed under [CC0](https://creativecommons.org/publicdomain/zero/1.0/). Please see LICENSE.md for details. 57 | -------------------------------------------------------------------------------- /face_mesh.blend: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EatTheFuture/face_track/133b504ef590f1870788ce7f86952197f98979c8/face_mesh.blend -------------------------------------------------------------------------------- /face_mesh.obj: -------------------------------------------------------------------------------- 1 | # Blender 3.1.0 2 | # www.blender.org 3 | o FaceMesh 4 | v 0.000000 -0.263889 0.283205 5 | v 0.000000 -0.122932 0.405047 6 | v 0.000000 -0.169293 0.289391 7 | v -0.034465 0.031768 0.352123 8 | v 0.000000 -0.075300 0.420478 9 | v 0.000000 -0.011632 0.404370 10 | v 0.000000 0.140255 0.297376 11 | v -0.297307 0.143263 0.127989 12 | v 0.000000 0.250964 0.275512 13 | v 0.000000 0.315281 0.289007 14 | v 0.000000 0.565825 0.253535 15 | v 0.000000 -0.275664 0.275677 16 | v -0.000000 -0.286984 0.260445 17 | v -0.000000 -0.295881 0.237021 18 | v -0.000000 -0.323612 0.237978 19 | v -0.000000 -0.344161 0.241477 20 | v -0.000000 -0.368054 0.249885 21 | v -0.000000 -0.394041 0.242819 22 | v -0.000000 -0.429621 0.198844 23 | v 0.000000 -0.145517 0.382276 24 | v -0.040648 -0.146375 0.321534 25 | v -0.487018 0.326126 -0.064496 26 | v -0.175510 0.113089 0.170875 27 | v -0.220667 0.106584 0.168276 28 | v -0.263155 0.107246 0.156493 29 | v -0.314637 0.127688 0.119454 30 | v -0.136815 0.126104 0.166133 31 | v -0.251274 0.236447 0.188405 32 | v -0.198274 0.235071 0.189018 33 | v -0.297237 0.224312 0.173625 34 | v -0.324375 0.204819 0.156055 35 | v -0.353247 0.099998 0.095948 36 | v -0.150927 -0.500528 0.143122 37 | v -0.311847 0.153508 0.113965 38 | v -0.490938 0.117109 -0.096254 39 | v -0.403715 0.129361 0.054463 40 | v -0.212945 -0.078765 0.199952 41 | v -0.052806 -0.253882 0.277921 42 | v -0.049170 -0.282669 0.253838 43 | v -0.101907 -0.263607 0.251173 44 | v -0.136307 -0.274110 0.217061 45 | v -0.089912 -0.282736 0.236090 46 | v -0.121775 -0.286704 0.204762 47 | v -0.196187 -0.345424 0.142966 48 | v -0.033532 -0.121547 0.401506 49 | v -0.038638 -0.075345 0.415629 50 | v -0.392703 0.239141 0.149856 51 | v -0.125065 0.036564 0.211973 52 | v -0.132245 -0.105122 0.293642 53 | v -0.134116 -0.079157 0.280372 54 | v -0.341150 -0.072273 0.145121 55 | v -0.036803 -0.017211 0.391523 56 | v -0.295414 0.291856 0.227264 57 | v -0.354734 0.272993 0.193410 58 | v -0.444801 0.414976 0.026229 59 | v -0.096529 0.262349 0.262807 60 | v -0.151127 0.217788 0.175688 61 | v -0.229745 -0.303619 0.135687 62 | v -0.412327 -0.312806 -0.245365 63 | v -0.101953 -0.133921 0.276470 64 | v -0.069045 -0.147882 0.278406 65 | v -0.184733 -0.292566 0.138201 66 | v -0.173839 -0.293253 0.150372 67 | v -0.380187 0.300716 0.174633 68 | v -0.134905 -0.122756 0.268018 69 | v -0.210804 0.292076 0.251089 70 | v -0.221546 0.329749 0.262832 71 | v -0.259704 0.537125 0.197420 72 | v -0.407678 0.365050 0.111150 73 | v -0.229604 0.438800 0.234339 74 | v -0.418869 0.258460 0.114427 75 | v -0.452483 0.294738 0.031800 76 | v -0.051733 -0.269656 0.270674 77 | v -0.096057 -0.273817 0.245046 78 | v -0.129444 -0.280202 0.214102 79 | v -0.092930 -0.141808 0.267045 80 | v -0.179117 -0.292889 0.144762 81 | v -0.158346 -0.308041 0.170302 82 | v -0.168246 -0.294183 0.152010 83 | v -0.079837 -0.119896 0.339108 84 | v -0.115269 -0.291756 0.195425 85 | v -0.083907 -0.290473 0.215602 86 | v -0.045791 -0.292015 0.231732 87 | v -0.058825 -0.427223 0.196704 88 | v -0.060698 -0.387694 0.239772 89 | v -0.057209 -0.361855 0.245652 90 | v -0.052446 -0.337935 0.236394 91 | v -0.047273 -0.319170 0.232587 92 | v -0.116041 -0.306412 0.194949 93 | v -0.124789 -0.313596 0.199031 94 | v -0.135024 -0.324392 0.203629 95 | v -0.143375 -0.338393 0.195496 96 | v -0.181623 -0.235402 0.205351 97 | v -0.476112 -0.096530 -0.293111 98 | v 0.000000 -0.159433 0.322798 99 | v -0.139065 -0.301747 0.169998 100 | v -0.148590 -0.304036 0.171751 101 | v -0.055505 -0.165300 0.277229 102 | v -0.121208 -0.149921 0.233632 103 | v -0.061852 -0.156634 0.279580 104 | v -0.169964 0.010398 0.195046 105 | v -0.241249 -0.020161 0.181721 106 | v -0.142094 -0.098649 0.254480 107 | v -0.372121 0.487772 0.118131 108 | v -0.336402 0.415356 0.176184 109 | v -0.313014 0.327796 0.222598 110 | v -0.159668 -0.377715 0.164436 111 | v -0.114838 0.322812 0.283263 112 | v -0.121510 0.445106 0.266960 113 | v -0.139617 0.561253 0.242629 114 | v -0.296390 0.113178 0.137435 115 | v -0.399777 0.068140 0.070085 116 | v -0.111763 0.137747 0.161092 117 | v -0.365299 0.184027 0.120040 118 | v -0.095254 0.065954 0.228925 119 | v -0.105568 -0.090170 0.337875 120 | v -0.446520 0.035298 0.018690 121 | v -0.371849 0.037444 0.107105 122 | v -0.317473 0.023217 0.146111 123 | v -0.233542 0.032099 0.165600 124 | v -0.171207 0.052876 0.175547 125 | v -0.125601 0.076088 0.188733 126 | v -0.040172 0.127780 0.280573 127 | v -0.433958 -0.059646 0.024988 128 | v -0.401552 0.189996 0.101505 129 | v -0.020418 -0.144151 0.379778 130 | v -0.121202 -0.009632 0.227633 131 | v -0.507279 0.115622 -0.242717 132 | v -0.091897 0.100409 0.197476 133 | v -0.146094 -0.102362 0.209500 134 | v -0.332817 0.152099 0.104972 135 | v -0.106691 -0.058554 0.321661 136 | v -0.449489 -0.205137 -0.280850 137 | v -0.110893 0.152969 0.152538 138 | v -0.072542 -0.032834 0.364855 139 | v -0.320615 -0.403057 -0.017087 140 | v -0.311413 -0.455047 -0.099471 141 | v -0.469351 -0.080943 -0.132607 142 | v -0.373764 -0.346940 -0.068802 143 | v -0.474851 0.212187 -0.043923 144 | v -0.154917 -0.544387 0.113825 145 | v -0.017165 -0.157696 0.319807 146 | v -0.160231 -0.042220 0.209580 147 | v -0.438091 0.121180 0.010679 148 | v -0.257949 0.131088 0.154056 149 | v -0.221116 0.129978 0.164737 150 | v -0.166891 -0.313903 0.162448 151 | v -0.420730 -0.152186 0.004273 152 | v -0.080699 -0.602686 0.101272 153 | v -0.196938 -0.548435 0.016799 154 | v -0.248476 -0.510189 -0.033728 155 | v 0.000000 0.444401 0.276122 156 | v -0.000000 -0.608949 0.112730 157 | v -0.183027 0.135802 0.166542 158 | v -0.146278 0.144947 0.160584 159 | v -0.122403 0.149110 0.152849 160 | v -0.431216 0.198522 0.058093 161 | v -0.153451 0.184971 0.168927 162 | v -0.194589 0.194532 0.176885 163 | v -0.234053 0.192799 0.175182 164 | v -0.270414 0.182663 0.164159 165 | v -0.291387 0.170825 0.149335 166 | v -0.506066 0.230429 -0.162813 167 | v -0.281560 0.136520 0.140106 168 | v 0.000000 -0.203097 0.276987 169 | v -0.137880 -0.212911 0.231581 170 | v -0.101198 -0.126939 0.294885 171 | v -0.060108 -0.203204 0.272761 172 | v 0.000000 0.197207 0.269755 173 | v -0.263925 -0.459437 0.022718 174 | v -0.210134 -0.502168 0.064913 175 | v -0.085384 -0.572383 0.155056 176 | v -0.364113 -0.394988 -0.176891 177 | v -0.124331 0.166759 0.159667 178 | v -0.066597 0.052672 0.289186 179 | v -0.000000 -0.578587 0.164300 180 | v -0.143242 -0.580928 0.068465 181 | v -0.446695 -0.179831 -0.133052 182 | v -0.084771 -0.312774 0.216022 183 | v -0.093267 -0.326374 0.220745 184 | v -0.101393 -0.344570 0.226991 185 | v -0.107611 -0.367615 0.220627 186 | v -0.114770 -0.406154 0.186358 187 | v -0.148611 -0.290543 0.176639 188 | v -0.156503 -0.287082 0.176876 189 | v -0.163061 -0.283396 0.177957 190 | v -0.215184 -0.265914 0.172453 191 | v -0.369709 -0.171735 0.092983 192 | v -0.066904 0.099474 0.251626 193 | v -0.086026 0.188879 0.183934 194 | v -0.109377 0.183111 0.167949 195 | v -0.140630 -0.294412 0.170903 196 | v -0.366554 -0.281203 0.019533 197 | v -0.054269 0.194675 0.239542 198 | v -0.134122 -0.452785 0.162010 199 | v 0.000000 0.039672 0.366679 200 | v -0.037609 0.077788 0.321774 201 | v 0.000000 0.087862 0.331513 202 | v -0.095189 -0.019665 0.285230 203 | v -0.000000 -0.535124 0.192072 204 | v -0.000000 -0.480198 0.194835 205 | v -0.070381 -0.477169 0.186492 206 | v -0.229444 -0.369637 0.118042 207 | v -0.182484 -0.137866 0.197485 208 | v -0.184704 -0.417072 0.137522 209 | v -0.282317 -0.131560 0.179666 210 | v -0.224195 -0.182393 0.186130 211 | v -0.316407 -0.203505 0.140319 212 | v -0.082427 -0.530425 0.181462 213 | v -0.120326 -0.044438 0.251752 214 | v -0.258193 -0.408060 0.075346 215 | v -0.203416 -0.457068 0.102043 216 | v -0.266313 -0.314552 0.117083 217 | v -0.404977 -0.229095 -0.016389 218 | v -0.313195 -0.325102 0.074647 219 | v -0.418873 -0.269042 -0.121863 220 | v -0.258742 -0.236054 0.162248 221 | v -0.092953 0.021770 0.255819 222 | v -0.092741 -0.111149 0.348325 223 | v -0.119523 -0.121442 0.298281 224 | v -0.074165 -0.081430 0.379026 225 | v -0.124685 0.228443 0.199423 226 | v -0.202782 0.253817 0.208013 227 | v -0.269202 0.255239 0.204294 228 | v -0.321145 0.242181 0.185399 229 | v -0.353474 0.219218 0.158110 230 | v -0.364049 0.142133 0.087004 231 | v -0.486185 0.018054 -0.117006 232 | v -0.329495 0.081523 0.117221 233 | v -0.286957 0.070982 0.143199 234 | v -0.228154 0.072828 0.160353 235 | v -0.171910 0.085646 0.168130 236 | v -0.128295 0.102443 0.171098 237 | v -0.098888 0.117475 0.174487 238 | v -0.494131 0.009045 -0.282189 239 | v -0.119518 -0.132083 0.276039 240 | v -0.066502 0.009986 0.320764 241 | v -0.063178 -0.114888 0.381381 242 | v -0.042368 -0.136711 0.362141 243 | v -0.063548 -0.122285 0.360056 244 | v -0.108751 -0.144871 0.259835 245 | v -0.033866 -0.141075 0.373545 246 | v -0.030724 -0.153831 0.320803 247 | v -0.096747 0.151580 0.158201 248 | v -0.077255 0.142857 0.182919 249 | v -0.067639 0.136676 0.212006 250 | v -0.302915 0.161928 0.133935 251 | v -0.339065 0.182307 0.130742 252 | v 0.034465 0.031768 0.352123 253 | v 0.297307 0.143263 0.127989 254 | v 0.040648 -0.146375 0.321534 255 | v 0.487018 0.326126 -0.064496 256 | v 0.175510 0.113089 0.170875 257 | v 0.220667 0.106584 0.168276 258 | v 0.263155 0.107246 0.156493 259 | v 0.314637 0.127688 0.119454 260 | v 0.136815 0.126104 0.166133 261 | v 0.251274 0.236447 0.188405 262 | v 0.198274 0.235071 0.189018 263 | v 0.297237 0.224312 0.173625 264 | v 0.324375 0.204819 0.156055 265 | v 0.353247 0.099998 0.095948 266 | v 0.150927 -0.500528 0.143122 267 | v 0.311847 0.153508 0.113965 268 | v 0.490938 0.117109 -0.096254 269 | v 0.403715 0.129361 0.054463 270 | v 0.212945 -0.078765 0.199952 271 | v 0.052806 -0.253882 0.277921 272 | v 0.049170 -0.282669 0.253838 273 | v 0.101907 -0.263607 0.251173 274 | v 0.136307 -0.274110 0.217061 275 | v 0.089912 -0.282736 0.236090 276 | v 0.121775 -0.286704 0.204762 277 | v 0.196187 -0.345424 0.142966 278 | v 0.033532 -0.121547 0.401506 279 | v 0.038638 -0.075345 0.415629 280 | v 0.392703 0.239141 0.149856 281 | v 0.125065 0.036564 0.211973 282 | v 0.132245 -0.105122 0.293642 283 | v 0.134116 -0.079157 0.280372 284 | v 0.341150 -0.072273 0.145121 285 | v 0.036803 -0.017211 0.391523 286 | v 0.295414 0.291856 0.227264 287 | v 0.354734 0.272993 0.193410 288 | v 0.444801 0.414976 0.026229 289 | v 0.096529 0.262349 0.262807 290 | v 0.151127 0.217788 0.175688 291 | v 0.229745 -0.303619 0.135687 292 | v 0.412327 -0.312806 -0.245365 293 | v 0.101953 -0.133921 0.276470 294 | v 0.069045 -0.147882 0.278406 295 | v 0.184733 -0.292566 0.138201 296 | v 0.173839 -0.293253 0.150372 297 | v 0.380187 0.300716 0.174633 298 | v 0.134905 -0.122756 0.268018 299 | v 0.210804 0.292076 0.251089 300 | v 0.221546 0.329749 0.262832 301 | v 0.259704 0.537125 0.197420 302 | v 0.407678 0.365050 0.111150 303 | v 0.229604 0.438800 0.234339 304 | v 0.418869 0.258460 0.114427 305 | v 0.452483 0.294738 0.031800 306 | v 0.051733 -0.269656 0.270674 307 | v 0.096057 -0.273817 0.245046 308 | v 0.129444 -0.280202 0.214102 309 | v 0.092930 -0.141808 0.267045 310 | v 0.179117 -0.292889 0.144762 311 | v 0.158346 -0.308041 0.170302 312 | v 0.168246 -0.294183 0.152010 313 | v 0.079837 -0.119896 0.339108 314 | v 0.115269 -0.291756 0.195425 315 | v 0.083907 -0.290473 0.215602 316 | v 0.045791 -0.292015 0.231732 317 | v 0.058825 -0.427223 0.196704 318 | v 0.050955 -0.388113 0.238617 319 | v 0.057209 -0.361855 0.245652 320 | v 0.052446 -0.337935 0.236394 321 | v 0.047273 -0.319170 0.232587 322 | v 0.116041 -0.306412 0.194949 323 | v 0.124789 -0.313596 0.199031 324 | v 0.135024 -0.324392 0.203629 325 | v 0.143375 -0.338393 0.195496 326 | v 0.181623 -0.235402 0.205351 327 | v 0.476112 -0.096530 -0.293111 328 | v 0.139065 -0.301747 0.169998 329 | v 0.148590 -0.304036 0.171751 330 | v 0.055505 -0.165300 0.277229 331 | v 0.121208 -0.149921 0.233632 332 | v 0.061852 -0.156634 0.279580 333 | v 0.169964 0.010398 0.195046 334 | v 0.241249 -0.020161 0.181721 335 | v 0.142094 -0.098649 0.254480 336 | v 0.372121 0.487772 0.118131 337 | v 0.336402 0.415356 0.176184 338 | v 0.313014 0.327796 0.222598 339 | v 0.159668 -0.377715 0.164436 340 | v 0.114838 0.322812 0.283263 341 | v 0.121510 0.445106 0.266960 342 | v 0.139617 0.561253 0.242629 343 | v 0.296390 0.113178 0.137435 344 | v 0.399777 0.068140 0.070085 345 | v 0.111763 0.137747 0.161092 346 | v 0.365299 0.184027 0.120040 347 | v 0.095254 0.065954 0.228925 348 | v 0.105568 -0.090170 0.337875 349 | v 0.446520 0.035298 0.018690 350 | v 0.371849 0.037444 0.107105 351 | v 0.317473 0.023217 0.146111 352 | v 0.233542 0.032099 0.165600 353 | v 0.171207 0.052876 0.175547 354 | v 0.125601 0.076088 0.188733 355 | v 0.040172 0.127780 0.280573 356 | v 0.433958 -0.059646 0.024988 357 | v 0.401552 0.189996 0.101505 358 | v 0.020418 -0.144151 0.379778 359 | v 0.121202 -0.009632 0.227633 360 | v 0.507279 0.115622 -0.242717 361 | v 0.091897 0.100409 0.197476 362 | v 0.146094 -0.102362 0.209500 363 | v 0.332817 0.152099 0.104972 364 | v 0.106691 -0.058554 0.321661 365 | v 0.449489 -0.205137 -0.280850 366 | v 0.110893 0.152969 0.152538 367 | v 0.072542 -0.032834 0.364855 368 | v 0.320615 -0.403057 -0.017087 369 | v 0.311413 -0.455047 -0.099471 370 | v 0.469351 -0.080943 -0.132607 371 | v 0.373764 -0.346940 -0.068802 372 | v 0.474851 0.212187 -0.043923 373 | v 0.154917 -0.544387 0.113825 374 | v 0.017165 -0.157696 0.319807 375 | v 0.160231 -0.042220 0.209580 376 | v 0.438091 0.121180 0.010679 377 | v 0.257949 0.131088 0.154056 378 | v 0.221116 0.129978 0.164737 379 | v 0.166891 -0.313903 0.162448 380 | v 0.420730 -0.152186 0.004273 381 | v 0.080699 -0.602686 0.101272 382 | v 0.196938 -0.548435 0.016799 383 | v 0.248476 -0.510189 -0.033728 384 | v 0.183027 0.135802 0.166542 385 | v 0.146278 0.144947 0.160584 386 | v 0.122403 0.149110 0.152849 387 | v 0.431216 0.198522 0.058093 388 | v 0.153451 0.184971 0.168927 389 | v 0.194589 0.194532 0.176885 390 | v 0.234053 0.192799 0.175182 391 | v 0.270414 0.182663 0.164159 392 | v 0.291387 0.170825 0.149335 393 | v 0.506066 0.230429 -0.162813 394 | v 0.281560 0.136520 0.140106 395 | v 0.137880 -0.212911 0.231581 396 | v 0.101198 -0.126939 0.294885 397 | v 0.060108 -0.203204 0.272761 398 | v 0.263925 -0.459437 0.022718 399 | v 0.210134 -0.502168 0.064913 400 | v 0.085384 -0.572383 0.155056 401 | v 0.364113 -0.394988 -0.176891 402 | v 0.124331 0.166759 0.159667 403 | v 0.066597 0.052672 0.289186 404 | v 0.143242 -0.580928 0.068465 405 | v 0.446695 -0.179831 -0.133052 406 | v 0.084771 -0.312774 0.216022 407 | v 0.093267 -0.326374 0.220745 408 | v 0.101393 -0.344570 0.226991 409 | v 0.107611 -0.367615 0.220627 410 | v 0.114770 -0.406154 0.186358 411 | v 0.148611 -0.290543 0.176639 412 | v 0.156503 -0.287082 0.176876 413 | v 0.163061 -0.283396 0.177957 414 | v 0.215184 -0.265914 0.172453 415 | v 0.369709 -0.171735 0.092983 416 | v 0.066904 0.099474 0.251626 417 | v 0.086026 0.188879 0.183934 418 | v 0.109377 0.183111 0.167949 419 | v 0.140630 -0.294412 0.170903 420 | v 0.366554 -0.281203 0.019533 421 | v 0.054269 0.194675 0.239542 422 | v 0.134122 -0.452785 0.162010 423 | v 0.037609 0.077788 0.321774 424 | v 0.095189 -0.019665 0.285230 425 | v 0.070381 -0.477169 0.186492 426 | v 0.229444 -0.369637 0.118042 427 | v 0.182484 -0.137866 0.197485 428 | v 0.184704 -0.417072 0.137522 429 | v 0.282317 -0.131560 0.179666 430 | v 0.224195 -0.182393 0.186130 431 | v 0.316407 -0.203505 0.140319 432 | v 0.082427 -0.530425 0.181462 433 | v 0.120326 -0.044438 0.251752 434 | v 0.258193 -0.408060 0.075346 435 | v 0.203416 -0.457068 0.102043 436 | v 0.266313 -0.314552 0.117083 437 | v 0.404977 -0.229095 -0.016389 438 | v 0.313195 -0.325102 0.074647 439 | v 0.418873 -0.269042 -0.121863 440 | v 0.258742 -0.236054 0.162248 441 | v 0.092953 0.021770 0.255819 442 | v 0.092741 -0.111149 0.348325 443 | v 0.119523 -0.121442 0.298281 444 | v 0.074165 -0.081430 0.379026 445 | v 0.124685 0.228443 0.199423 446 | v 0.202782 0.253817 0.208013 447 | v 0.269202 0.255239 0.204294 448 | v 0.321145 0.242181 0.185399 449 | v 0.353474 0.219218 0.158110 450 | v 0.364049 0.142133 0.087004 451 | v 0.486185 0.018054 -0.117006 452 | v 0.329495 0.081523 0.117221 453 | v 0.286957 0.070982 0.143199 454 | v 0.228154 0.072828 0.160353 455 | v 0.171910 0.085646 0.168130 456 | v 0.128295 0.102443 0.171098 457 | v 0.098888 0.117475 0.174487 458 | v 0.494131 0.009045 -0.282189 459 | v 0.119518 -0.132083 0.276039 460 | v 0.066502 0.009986 0.320764 461 | v 0.063178 -0.114888 0.381381 462 | v 0.042368 -0.136711 0.362141 463 | v 0.063548 -0.122285 0.360056 464 | v 0.108751 -0.144871 0.259835 465 | v 0.033866 -0.141075 0.373545 466 | v 0.030724 -0.153831 0.320803 467 | v 0.096747 0.151580 0.158201 468 | v 0.077255 0.142857 0.182919 469 | v 0.067639 0.136676 0.212006 470 | v 0.302915 0.161928 0.133935 471 | v 0.339065 0.182307 0.130742 472 | v -0.210967 0.164795 0.141925 473 | v -0.163637 0.164795 0.141925 474 | v -0.212157 0.206706 0.141925 475 | v -0.259397 0.164795 0.141925 476 | v -0.210472 0.126854 0.141925 477 | v 0.210967 0.164795 0.141925 478 | v 0.259401 0.164795 0.141925 479 | v 0.212157 0.206706 0.141925 480 | v 0.163640 0.164795 0.141925 481 | v 0.210473 0.126854 0.141925 482 | s 0 483 | f 176 153 378 397 484 | f 397 378 401 370 485 | f 396 370 401 379 486 | f 395 396 379 380 487 | f 366 365 395 380 488 | f 398 368 365 366 489 | f 397 429 200 176 490 | f 263 429 397 370 491 | f 263 370 396 432 492 | f 432 396 395 431 493 | f 435 431 395 365 494 | f 417 435 365 368 495 | f 289 436 368 398 496 | f 436 434 417 368 497 | f 362 402 436 289 498 | f 324 367 402 362 499 | f 455 448 367 324 500 | f 455 357 265 448 501 | f 390 369 265 357 502 | f 252 302 369 390 503 | f 285 299 302 252 504 | f 333 334 299 285 505 | f 298 300 334 333 506 | f 339 338 300 298 507 | f 11 152 338 339 508 | f 110 109 152 11 509 | f 68 70 109 110 510 | f 104 105 70 68 511 | f 55 69 105 104 512 | f 22 72 69 55 513 | f 163 140 72 22 514 | f 128 35 140 163 515 | f 228 35 128 235 516 | f 94 138 228 235 517 | f 133 178 138 94 518 | f 59 216 178 133 519 | f 173 139 216 59 520 | f 137 136 139 173 521 | f 151 170 136 137 522 | f 151 150 171 170 523 | f 150 177 141 171 524 | f 141 177 149 172 525 | f 176 172 149 153 526 | f 10 337 338 152 527 | f 297 300 338 337 528 | f 335 334 300 297 529 | f 294 299 334 335 530 | f 301 302 299 294 531 | f 301 384 369 302 532 | f 384 373 265 369 533 | f 346 448 265 373 534 | f 346 353 367 448 535 | f 353 377 402 367 536 | f 436 402 377 434 537 | f 209 172 176 200 538 | f 33 141 172 209 539 | f 33 212 171 141 540 | f 212 211 170 171 541 | f 215 136 170 211 542 | f 193 139 136 215 543 | f 214 216 139 193 544 | f 148 178 216 214 545 | f 124 138 178 148 546 | f 117 228 138 124 547 | f 228 117 144 35 548 | f 157 140 35 144 549 | f 71 72 140 157 550 | f 71 64 69 72 551 | f 64 106 105 69 552 | f 106 67 70 105 553 | f 67 108 109 70 554 | f 109 108 10 152 555 | f 435 433 423 431 556 | f 428 437 433 435 557 | f 428 435 417 412 558 | f 426 427 437 428 559 | f 267 424 427 426 560 | f 425 432 431 423 561 | f 419 263 432 425 562 | f 422 429 263 419 563 | f 422 201 200 429 564 | f 201 202 209 200 565 | f 202 195 33 209 566 | f 195 205 212 33 567 | f 205 203 211 212 568 | f 203 213 215 211 569 | f 215 213 217 208 570 | f 208 217 207 206 571 | f 206 207 204 37 572 | f 203 44 58 213 573 | f 107 44 203 205 574 | f 183 107 205 195 575 | f 84 183 195 202 576 | f 19 84 202 201 577 | f 314 19 201 422 578 | f 407 314 422 419 579 | f 336 407 419 425 580 | f 16 15 88 87 581 | f 87 88 179 180 582 | f 180 179 89 90 583 | f 90 89 96 97 584 | f 97 96 79 63 585 | f 63 79 192 184 586 | f 184 192 81 43 587 | f 43 81 82 42 588 | f 42 82 83 39 589 | f 39 83 14 13 590 | f 13 14 313 269 591 | f 269 313 312 272 592 | f 272 312 311 273 593 | f 273 311 416 408 594 | f 408 416 309 293 595 | f 293 309 325 326 596 | f 326 325 319 320 597 | f 320 319 403 404 598 | f 404 403 318 317 599 | f 317 318 15 16 600 | f 13 12 73 39 601 | f 39 73 74 42 602 | f 42 74 75 43 603 | f 43 75 185 184 604 | f 184 185 77 63 605 | f 63 77 78 97 606 | f 97 78 91 90 607 | f 90 91 181 180 608 | f 180 181 86 87 609 | f 87 86 17 16 610 | f 16 17 316 317 611 | f 317 316 405 404 612 | f 404 405 321 320 613 | f 320 321 308 326 614 | f 326 308 307 293 615 | f 293 307 409 408 616 | f 408 409 305 273 617 | f 273 305 304 272 618 | f 272 304 303 269 619 | f 269 303 12 13 620 | f 316 315 406 405 621 | f 405 406 322 321 622 | f 321 322 376 308 623 | f 308 376 292 307 624 | f 307 292 410 409 625 | f 409 410 271 305 626 | f 305 271 270 304 627 | f 304 270 268 303 628 | f 303 268 1 12 629 | f 12 1 38 73 630 | f 73 38 40 74 631 | f 74 40 41 75 632 | f 75 41 186 185 633 | f 185 186 62 77 634 | f 77 62 147 78 635 | f 78 147 92 91 636 | f 91 92 182 181 637 | f 181 182 85 86 638 | f 86 85 18 17 639 | f 17 18 315 316 640 | f 147 44 107 92 641 | f 92 107 183 182 642 | f 182 183 84 85 643 | f 85 84 19 18 644 | f 18 19 314 315 645 | f 315 314 407 406 646 | f 406 407 336 322 647 | f 322 336 274 376 648 | f 376 274 288 292 649 | f 292 288 411 410 650 | f 410 411 323 271 651 | f 271 323 392 270 652 | f 270 392 394 268 653 | f 268 394 165 1 654 | f 1 165 168 38 655 | f 38 168 166 40 656 | f 40 166 93 41 657 | f 41 93 187 186 658 | f 186 187 58 62 659 | f 62 58 44 147 660 | f 425 423 274 336 661 | f 433 288 274 423 662 | f 341 346 373 266 663 | f 266 373 384 354 664 | f 354 384 301 277 665 | f 277 301 294 284 666 | f 284 294 335 283 667 | f 283 335 297 296 668 | f 296 297 337 286 669 | f 286 337 10 9 670 | f 9 10 108 56 671 | f 56 108 67 66 672 | f 66 67 106 53 673 | f 53 106 64 54 674 | f 54 64 71 47 675 | f 47 71 157 125 676 | f 125 157 144 36 677 | f 36 144 117 112 678 | f 45 2 5 46 679 | f 46 5 6 52 680 | f 52 6 196 4 681 | f 4 196 198 197 682 | f 197 198 7 123 683 | f 123 7 169 194 684 | f 194 169 9 56 685 | f 286 9 169 418 686 | f 418 169 7 352 687 | f 352 7 198 420 688 | f 420 198 196 249 689 | f 249 196 6 282 690 | f 282 6 5 276 691 | f 276 5 2 275 692 | f 166 204 207 93 693 | f 93 207 217 187 694 | f 187 217 213 58 695 | f 288 433 437 411 696 | f 411 437 427 323 697 | f 323 427 424 392 698 | f 417 434 377 412 699 | f 208 188 193 215 700 | f 193 188 148 214 701 | f 95 142 98 3 702 | f 243 100 98 142 703 | f 242 238 240 239 704 | f 238 219 80 240 705 | f 219 220 167 80 706 | f 220 236 60 167 707 | f 236 241 76 60 708 | f 241 100 61 76 709 | f 100 243 21 61 710 | f 243 242 239 21 711 | f 462 459 460 458 712 | f 458 460 310 439 713 | f 439 310 393 440 714 | f 440 393 290 456 715 | f 456 290 306 461 716 | f 461 306 291 329 717 | f 329 291 251 463 718 | f 463 251 459 462 719 | f 95 3 327 371 720 | f 463 371 327 329 721 | f 463 462 355 371 722 | f 95 371 355 20 723 | f 95 20 126 142 724 | f 243 142 126 242 725 | f 242 126 45 238 726 | f 20 2 45 126 727 | f 20 355 275 2 728 | f 462 458 275 355 729 | f 168 165 3 98 730 | f 394 327 3 165 731 | f 328 327 394 392 732 | f 99 166 168 98 733 | f 241 99 98 100 734 | f 461 329 327 328 735 | f 439 345 441 458 736 | f 276 275 458 441 737 | f 440 279 345 439 738 | f 456 295 279 440 739 | f 461 328 295 456 740 | f 46 221 238 45 741 | f 219 238 221 116 742 | f 220 219 116 49 743 | f 236 220 49 65 744 | f 241 236 65 99 745 | f 424 359 328 392 746 | f 328 359 332 295 747 | f 295 332 280 279 748 | f 345 279 280 361 749 | f 441 345 361 364 750 | f 282 276 441 364 751 | f 204 166 99 130 752 | f 99 65 103 130 753 | f 65 49 50 103 754 | f 116 132 50 49 755 | f 221 135 132 116 756 | f 52 135 221 46 757 | f 249 282 364 457 758 | f 364 361 421 457 759 | f 280 430 421 361 760 | f 280 332 359 430 761 | f 267 372 359 424 762 | f 356 430 359 372 763 | f 130 103 50 210 764 | f 199 132 135 237 765 | f 52 4 237 135 766 | f 132 199 210 50 767 | f 421 430 356 438 768 | f 421 438 400 457 769 | f 420 249 457 400 770 | f 352 420 400 413 771 | f 400 438 344 413 772 | f 438 356 278 344 773 | f 372 330 278 356 774 | f 267 331 330 372 775 | f 267 426 281 331 776 | f 412 281 426 428 777 | f 37 204 130 143 778 | f 130 210 127 143 779 | f 199 218 127 210 780 | f 199 237 175 218 781 | f 197 175 237 4 782 | f 465 464 415 414 783 | f 414 415 287 442 784 | f 442 287 259 443 785 | f 443 259 258 444 786 | f 444 258 260 445 787 | f 445 260 261 446 788 | f 446 261 468 343 789 | f 343 468 360 447 790 | f 447 360 256 262 791 | f 262 256 340 449 792 | f 449 340 255 450 793 | f 450 255 254 451 794 | f 451 254 253 452 795 | f 452 253 257 453 796 | f 453 257 342 454 797 | f 454 342 464 465 798 | f 32 112 118 229 799 | f 229 118 119 230 800 | f 230 119 120 231 801 | f 231 120 121 232 802 | f 232 121 122 233 803 | f 233 122 129 234 804 | f 234 129 246 245 805 | f 245 246 194 190 806 | f 190 194 56 222 807 | f 222 56 66 223 808 | f 223 66 53 224 809 | f 224 53 54 225 810 | f 225 54 47 226 811 | f 226 47 125 114 812 | f 114 125 36 227 813 | f 227 36 112 32 814 | f 442 286 418 414 815 | f 414 418 466 465 816 | f 465 466 358 454 817 | f 454 358 351 453 818 | f 453 351 350 452 819 | f 452 350 349 451 820 | f 451 349 348 450 821 | f 450 348 347 449 822 | f 449 347 341 262 823 | f 262 341 266 447 824 | f 447 266 354 343 825 | f 343 354 277 446 826 | f 446 277 284 445 827 | f 445 284 283 444 828 | f 444 283 296 443 829 | f 443 296 286 442 830 | f 418 352 413 466 831 | f 466 413 344 358 832 | f 358 344 278 351 833 | f 351 278 330 350 834 | f 350 330 331 349 835 | f 349 331 281 348 836 | f 119 51 102 120 837 | f 120 102 101 121 838 | f 121 101 48 122 839 | f 122 48 115 129 840 | f 129 115 189 246 841 | f 246 189 123 194 842 | f 188 208 206 51 843 | f 51 206 37 102 844 | f 102 37 143 101 845 | f 101 143 127 48 846 | f 48 127 218 115 847 | f 115 218 175 189 848 | f 189 175 197 123 849 | f 353 346 341 347 850 | f 347 348 281 353 851 | f 377 353 281 412 852 | f 117 124 118 112 853 | f 118 124 51 119 854 | f 148 188 51 124 855 | f 248 114 227 131 856 | f 131 227 32 26 857 | f 26 32 229 111 858 | f 111 229 230 25 859 | f 25 230 231 24 860 | f 24 231 232 23 861 | f 23 232 233 27 862 | f 27 233 234 113 863 | f 113 234 245 244 864 | f 244 245 190 191 865 | f 191 190 222 57 866 | f 57 222 223 29 867 | f 29 223 224 28 868 | f 28 224 225 30 869 | f 30 225 226 31 870 | f 31 226 114 248 871 | f 383 342 257 382 872 | f 382 257 253 381 873 | f 381 253 254 375 874 | f 375 254 255 374 875 | f 374 255 340 391 876 | f 391 340 256 250 877 | f 250 256 360 264 878 | f 264 360 468 467 879 | f 467 468 261 389 880 | f 389 261 260 388 881 | f 388 260 258 387 882 | f 387 258 259 386 883 | f 386 259 287 385 884 | f 385 287 415 399 885 | f 399 415 464 363 886 | f 363 464 342 383 887 | f 174 191 57 158 888 | f 158 57 29 159 889 | f 159 29 28 160 890 | f 160 28 30 161 891 | f 161 30 31 162 892 | f 162 31 248 247 893 | f 247 248 131 34 894 | f 34 131 26 8 895 | f 8 26 111 164 896 | f 164 111 25 145 897 | f 145 25 24 146 898 | f 146 24 23 154 899 | f 154 23 27 155 900 | f 155 27 113 156 901 | f 156 113 244 134 902 | f 134 244 191 174 903 | f 476 474 477 904 | f 474 476 475 905 | f 474 475 478 906 | f 474 478 477 907 | f 469 472 471 908 | f 469 471 470 909 | f 473 472 469 470 910 | f 472 473 469 911 | -------------------------------------------------------------------------------- /face_mesh_perspective.blend: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EatTheFuture/face_track/133b504ef590f1870788ce7f86952197f98979c8/face_mesh_perspective.blend -------------------------------------------------------------------------------- /face_track.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import math 5 | from struct import pack 6 | import numpy as np 7 | import cv2 as cv 8 | import mediapipe as mp 9 | 10 | 11 | eye_l_idxs = [474, 475, 476, 477] 12 | eye_r_idxs = [469, 470, 471, 472] 13 | head_idxs = [34, 264] 14 | 15 | # For averaging depth estimate out over multiple frames. 16 | WINDOW_WIDTH = 3 17 | 18 | 19 | def distance(a, b): 20 | x = a.x - b.x 21 | y = a.y - b.y 22 | z = a.z - b.z 23 | 24 | return ((x * x) + (y * y) + (z * z))**0.5 25 | 26 | 27 | if __name__ == "__main__": 28 | arg_parser = argparse.ArgumentParser(description= 29 | """ 30 | Processes video of a human face into 3D face mesh animation 31 | data. Currently only supports video files with a single face. 32 | """ 33 | ) 34 | arg_parser.add_argument("--fov", help="Attempt to account for perspective projection based on the given horizontal fov (in degrees).") 35 | arg_parser.add_argument("--focal_len", help="Same as --fov except it takes sensor size / focal length (e.g. \"35/60\" for a 35mm sensor and 60mm lens).") 36 | arg_parser.add_argument("input_video", help="The input video file with a person's face in it.") 37 | arg_parser.add_argument("output_mdd", help="The .mdd file to write the mesh animation data to.") 38 | args = arg_parser.parse_args() 39 | 40 | video_path = args.input_video 41 | mdd_path = args.output_mdd 42 | camera_scale = None 43 | if args.fov is not None: 44 | try: 45 | fov = float(args.fov) 46 | camera_scale = 2.0 * math.tan(math.radians(fov / 2)) 47 | except: 48 | print("Error: the specified fov, '{}', is not a number.".format(args.fov)) 49 | exit() 50 | elif args.focal_len is not None: 51 | error_msg = "Error: focal_len must be specified as two numbers separated by a slash (no spaces). E.g. \"35/60\" for a 35mm sensor and 60mm lens." 52 | try: 53 | sensor_lens = args.focal_len.split("/") 54 | if len(sensor_lens) != 2: 55 | raise None 56 | sensor = float(sensor_lens[0]) 57 | lens = float(sensor_lens[1]) 58 | camera_scale = sensor / lens 59 | except: 60 | print(error_msg) 61 | exit() 62 | 63 | 64 | 65 | #---------------------------------------------------------- 66 | # Compute the mesh points from the input video. 67 | meshes = [] # One mesh per frame. 68 | point_count = 0 69 | width = 0 70 | height = 0 71 | fps = 0 72 | aspect_ratio = 1.0 73 | with mp.solutions.face_mesh.FaceMesh( 74 | static_image_mode=False, # Set false for video. 75 | max_num_faces=1, 76 | refine_landmarks=True, 77 | min_detection_confidence=0.5) as face_mesh: 78 | 79 | video = cv.VideoCapture(video_path) 80 | 81 | width = video.get(cv.CAP_PROP_FRAME_WIDTH) 82 | height = video.get(cv.CAP_PROP_FRAME_HEIGHT) 83 | fps = video.get(cv.CAP_PROP_FPS) 84 | aspect_ratio = width / height 85 | print("Input video fps: {}".format(fps)) 86 | print("Input video resolution: {}x{}".format(int(width), int(height))) 87 | 88 | i = 1 89 | while video.isOpened(): 90 | ret, image = video.read() 91 | print("\rReading frame", i, end = "") 92 | if not ret: 93 | break 94 | 95 | results = face_mesh.process(cv.cvtColor(image, cv.COLOR_BGR2RGB)) 96 | if not results.multi_face_landmarks: 97 | meshes += [None] 98 | else: 99 | point_count = len(results.multi_face_landmarks[0].landmark) 100 | meshes += [results.multi_face_landmarks[0].landmark] 101 | 102 | i += 1 103 | video.release() 104 | print("\rRead {} frames. ".format(len(meshes))) 105 | print("Generated vert count:", point_count) 106 | 107 | 108 | #---------------------------------------------------------- 109 | # Compute distance and average Z coordinate of our metric head width 110 | # vertices for each frame. 111 | width_2d_and_z = [] 112 | for mesh in meshes: 113 | if mesh is None: 114 | width_2d_and_z += [None] 115 | else: 116 | d = distance(mesh[head_idxs[0]], mesh[head_idxs[1]]) 117 | z = (mesh[head_idxs[0]].z + mesh[head_idxs[1]].z) * 0.5 118 | width_2d_and_z += [(d, z)] 119 | 120 | 121 | #---------------------------------------------------------- 122 | # Write the mdd file. 123 | frame_count = len(meshes) 124 | if frame_count > 0: 125 | with open(mdd_path, 'wb') as mdd: 126 | 127 | mdd.write(pack(">2i", frame_count, point_count)) 128 | mdd.write(pack(">%df" % (frame_count), *[frame / fps for frame in range(frame_count)])) 129 | 130 | for mesh, i in zip(meshes, range(len(meshes))): 131 | print("\rWriting frame", i, end = "") 132 | if mesh is None: 133 | # Put all vertices at the origin for bogus frames. 134 | for n in range(point_count): 135 | mdd.write(pack(">3f", 0.0, 0.0, 0.0)) 136 | elif camera_scale is None: 137 | # No camera fov, so just do simple orthographic. 138 | for point in mesh: 139 | x = point.x - 0.5 140 | y = point.y - 0.5 141 | mdd.write(pack(">3f", x, -y / aspect_ratio, -point.z)) 142 | else: 143 | # Compute a rolling average of width_2d_and_z. 144 | w2d = width_2d_and_z[i][0] 145 | wz = width_2d_and_z[i][1] 146 | k = 1 147 | for j in range(1, WINDOW_WIDTH + 1): 148 | if (i + j) < len(width_2d_and_z): 149 | w2d_z = width_2d_and_z[i + j] 150 | if w2d_z is None: 151 | break 152 | w2d += w2d_z[0] 153 | wz += w2d_z[1] 154 | k += 1 155 | for j in range(1, WINDOW_WIDTH + 1): 156 | if (i - j) >= 0: 157 | w2d_z = width_2d_and_z[i - j] 158 | if w2d_z is None: 159 | break 160 | w2d += w2d_z[0] 161 | wz += w2d_z[1] 162 | k += 1 163 | w2d /= k 164 | wz /= k 165 | 166 | # Compute and write out mesh coordinates. 167 | scale = 1.0 / w2d 168 | for point in mesh: 169 | z = ((point.z - wz) * camera_scale * scale) + scale 170 | x = (point.x - 0.5) * camera_scale * z 171 | y = ((point.y - 0.5) * camera_scale / aspect_ratio) * z 172 | 173 | mdd.write(pack( 174 | ">3f", 175 | x / camera_scale, 176 | -y / camera_scale, 177 | -z / camera_scale, 178 | )) 179 | 180 | print("\rWrote {} frames. ".format(len(meshes))) 181 | 182 | 183 | -------------------------------------------------------------------------------- /shell.nix: -------------------------------------------------------------------------------- 1 | let 2 | # You can replace "unstable" with any channel name to use that channel. 3 | pkgs = import {}; 4 | pypkgs = pkgs.python310Packages; 5 | name = "face_track_dev_env"; 6 | in pkgs.stdenv.mkDerivation { 7 | inherit name; 8 | nativeBuildInputs = with pkgs; [ 9 | pypkgs.python 10 | pypkgs.pip 11 | pypkgs.numpy 12 | pypkgs.opencv4 13 | pypkgs.dlib 14 | ]; 15 | 16 | shellHook = with pkgs; '' 17 | export PYTHONPATH="$PYTHONPATH:$HOME/.local/lib/python3.10/site-packages" 18 | export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python 19 | ''; 20 | } 21 | --------------------------------------------------------------------------------