├── .github └── ISSUE_TEMPLATE │ ├── bug-report.md │ └── discussion.md ├── .gitignore ├── .gitmodules ├── Dockerfile ├── LICENSE.md ├── README.md ├── afy ├── arguments.py ├── cam_fomm.py ├── camera_selector.py ├── networking.py ├── predictor_local.py ├── predictor_remote.py ├── predictor_worker.py ├── utils.py └── videocaptureasync.py ├── avatarify.ipynb ├── avatars ├── .geetkeep ├── einstein.jpg ├── eminem.jpg ├── jobs.jpg ├── mona.jpg ├── obama.jpg ├── potter.jpg ├── ronaldo.png └── schwarzenegger.png ├── config.yaml ├── docs ├── README.md ├── appstore-badge.png ├── google-play-badge.png ├── mona.gif ├── skype.jpg ├── slack.jpg ├── teams.jpg └── zoom.jpg ├── requirements.txt ├── requirements_client.txt ├── run.sh ├── run_mac.sh ├── run_windows.bat ├── scripts ├── create_virtual_camera.sh ├── download_data.sh ├── get_ngrok.sh ├── install.sh ├── install_docker.sh ├── install_mac.sh ├── install_windows.bat ├── open_tunnel_ngrok.sh ├── open_tunnel_ssh.sh ├── settings.sh └── settings_windows.bat └── var └── log └── .geetkeep /.github/ISSUE_TEMPLATE/bug-report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | 12 | 13 | **To Reproduce** 14 | 15 | 16 | **Info (please complete the following information):** 17 | - OS (e.g., Linux): 18 | - GPU model: 19 | 23 | - Any other relevant information: 24 | 41 | 42 | **Screenshots** 43 | 44 | 45 | **Logs** 46 | 47 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/discussion.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Discussion 3 | about: Discuss a new feature or improvement. 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | 11 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | vox-adv-cpk.pth.tar 2 | 3 | __pycache__ 4 | 5 | *.log 6 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/.gitmodules -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM nvcr.io/nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04 2 | 3 | RUN DEBIAN_FRONTEND=noninteractive apt-get -qq update \ 4 | && DEBIAN_FRONTEND=noninteractive apt-get -qqy install curl python3-pip python3-tk python3.7-dev ffmpeg git less nano libsm6 libxext6 libxrender-dev \ 5 | && rm -rf /var/lib/apt/lists/* 6 | 7 | # use python3.7 by default 8 | RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.6 1 \ 9 | && update-alternatives --install /usr/bin/python python /usr/bin/python3.7 2 \ 10 | && update-alternatives --set python /usr/bin/python3.7 \ 11 | && update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1 \ 12 | && update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 2 \ 13 | && update-alternatives --set python3 /usr/bin/python3.7 \ 14 | && python -m pip install --upgrade setuptools pip wheel 15 | 16 | ARG PYTORCH_WHEEL="https://download.pytorch.org/whl/cu100/torch-1.0.0-cp37-cp37m-linux_x86_64.whl" 17 | ARG FACE_ALIGNMENT_GIT="git+https://github.com/1adrianb/face-alignment" 18 | ARG AVATARIFY_COMMIT="a300fcaadb6a6964e69d4a90db9e7d72bb87e791" 19 | ARG FOMM_COMMIT="efbe0a6f17b38360ff9a446fddfbb3ce5493534c" 20 | 21 | RUN git clone https://github.com/alievk/avatarify-python.git /app/avatarify && cd /app/avatarify && git checkout ${AVATARIFY_COMMIT} \ 22 | && git clone https://github.com/alievk/first-order-model.git /app/avatarify/fomm && cd /app/avatarify/fomm && git checkout ${FOMM_COMMIT} 23 | 24 | WORKDIR /app/avatarify 25 | 26 | RUN bash scripts/download_data.sh 27 | 28 | RUN pip3 install ${PYTORCH_WHEEL} -r requirements.txt \ 29 | && pip3 install ${PYTORCH_WHEEL} -r fomm/requirements.txt \ 30 | && rm -rf /root/.cache/pip 31 | 32 | ENV PYTHONPATH="/app/avatarify:/app/avatarify/fomm" 33 | 34 | EXPOSE 5557 35 | EXPOSE 5558 36 | 37 | CMD ["python3", "afy/cam_fomm.py", "--config", "fomm/config/vox-adv-256.yaml", "--checkpoint", "vox-adv-cpk.pth.tar", "--virt-cam", "9", "--relative", "--adapt_scale", "--is-worker"] 38 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | # Attribution-NonCommercial 4.0 International 2 | 3 | Creative Commons Corporation (“Creative Commons”) is not a law firm and does not provide legal services or legal advice. Distribution of Creative Commons public licenses does not create a lawyer-client or other relationship. Creative Commons makes its licenses and related information available on an “as-is” basis. Creative Commons gives no warranties regarding its licenses, any material licensed under their terms and conditions, or any related information. Creative Commons disclaims all liability for damages resulting from their use to the fullest extent possible. 4 | 5 | ### Using Creative Commons Public Licenses 6 | 7 | Creative Commons public licenses provide a standard set of terms and conditions that creators and other rights holders may use to share original works of authorship and other material subject to copyright and certain other rights specified in the public license below. The following considerations are for informational purposes only, are not exhaustive, and do not form part of our licenses. 8 | 9 | * __Considerations for licensors:__ Our public licenses are intended for use by those authorized to give the public permission to use material in ways otherwise restricted by copyright and certain other rights. Our licenses are irrevocable. Licensors should read and understand the terms and conditions of the license they choose before applying it. Licensors should also secure all rights necessary before applying our licenses so that the public can reuse the material as expected. Licensors should clearly mark any material not subject to the license. This includes other CC-licensed material, or material used under an exception or limitation to copyright. [More considerations for licensors](http://wiki.creativecommons.org/Considerations_for_licensors_and_licensees#Considerations_for_licensors). 10 | 11 | * __Considerations for the public:__ By using one of our public licenses, a licensor grants the public permission to use the licensed material under specified terms and conditions. If the licensor’s permission is not necessary for any reason–for example, because of any applicable exception or limitation to copyright–then that use is not regulated by the license. Our licenses grant only permissions under copyright and certain other rights that a licensor has authority to grant. Use of the licensed material may still be restricted for other reasons, including because others have copyright or other rights in the material. A licensor may make special requests, such as asking that all changes be marked or described. Although not required by our licenses, you are encouraged to respect those requests where reasonable. [More considerations for the public](http://wiki.creativecommons.org/Considerations_for_licensors_and_licensees#Considerations_for_licensees). 12 | 13 | ## Creative Commons Attribution-NonCommercial 4.0 International Public License 14 | 15 | By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution-NonCommercial 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions. 16 | 17 | ### Section 1 – Definitions. 18 | 19 | a. __Adapted Material__ means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image. 20 | 21 | b. __Adapter's License__ means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License. 22 | 23 | c. __Copyright and Similar Rights__ means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights. 24 | 25 | d. __Effective Technological Measures__ means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements. 26 | 27 | e. __Exceptions and Limitations__ means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material. 28 | 29 | f. __Licensed Material__ means the artistic or literary work, database, or other material to which the Licensor applied this Public License. 30 | 31 | g. __Licensed Rights__ means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license. 32 | 33 | h. __Licensor__ means the individual(s) or entity(ies) granting rights under this Public License. 34 | 35 | i. __NonCommercial__ means not primarily intended for or directed towards commercial advantage or monetary compensation. For purposes of this Public License, the exchange of the Licensed Material for other material subject to Copyright and Similar Rights by digital file-sharing or similar means is NonCommercial provided there is no payment of monetary compensation in connection with the exchange. 36 | 37 | j. __Share__ means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them. 38 | 39 | k. __Sui Generis Database Rights__ means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world. 40 | 41 | l. __You__ means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning. 42 | 43 | ### Section 2 – Scope. 44 | 45 | a. ___License grant.___ 46 | 47 | 1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to: 48 | 49 | A. reproduce and Share the Licensed Material, in whole or in part, for NonCommercial purposes only; and 50 | 51 | B. produce, reproduce, and Share Adapted Material for NonCommercial purposes only. 52 | 53 | 2. __Exceptions and Limitations.__ For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions. 54 | 55 | 3. __Term.__ The term of this Public License is specified in Section 6(a). 56 | 57 | 4. __Media and formats; technical modifications allowed.__ The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material. 58 | 59 | 5. __Downstream recipients.__ 60 | 61 | A. __Offer from the Licensor – Licensed Material.__ Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License. 62 | 63 | B. __No downstream restrictions.__ You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material. 64 | 65 | 6. __No endorsement.__ Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i). 66 | 67 | b. ___Other rights.___ 68 | 69 | 1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise. 70 | 71 | 2. Patent and trademark rights are not licensed under this Public License. 72 | 73 | 3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties, including when the Licensed Material is used other than for NonCommercial purposes. 74 | 75 | ### Section 3 – License Conditions. 76 | 77 | Your exercise of the Licensed Rights is expressly made subject to the following conditions. 78 | 79 | a. ___Attribution.___ 80 | 81 | 1. If You Share the Licensed Material (including in modified form), You must: 82 | 83 | A. retain the following if it is supplied by the Licensor with the Licensed Material: 84 | 85 | i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated); 86 | 87 | ii. a copyright notice; 88 | 89 | iii. a notice that refers to this Public License; 90 | 91 | iv. a notice that refers to the disclaimer of warranties; 92 | 93 | v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable; 94 | 95 | B. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and 96 | 97 | C. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License. 98 | 99 | 2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information. 100 | 101 | 3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable. 102 | 103 | 4. If You Share Adapted Material You produce, the Adapter's License You apply must not prevent recipients of the Adapted Material from complying with this Public License. 104 | 105 | ### Section 4 – Sui Generis Database Rights. 106 | 107 | Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material: 108 | 109 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database for NonCommercial purposes only; 110 | 111 | b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material; and 112 | 113 | c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database. 114 | 115 | For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights. 116 | 117 | ### Section 5 – Disclaimer of Warranties and Limitation of Liability. 118 | 119 | a. __Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.__ 120 | 121 | b. __To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.__ 122 | 123 | c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability. 124 | 125 | ### Section 6 – Term and Termination. 126 | 127 | a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically. 128 | 129 | b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates: 130 | 131 | 1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or 132 | 133 | 2. upon express reinstatement by the Licensor. 134 | 135 | For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License. 136 | 137 | c. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License. 138 | 139 | d. Sections 1, 5, 6, 7, and 8 survive termination of this Public License. 140 | 141 | ### Section 7 – Other Terms and Conditions. 142 | 143 | a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed. 144 | 145 | b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License. 146 | 147 | ### Section 8 – Interpretation. 148 | 149 | a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License. 150 | 151 | b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions. 152 | 153 | c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor. 154 | 155 | d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority. 156 | 157 | > Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at [creativecommons.org/policies](http://creativecommons.org/policies), Creative Commons does not authorize the use of the trademark “Creative Commons” or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses. 158 | > 159 | > Creative Commons may be contacted at creativecommons.org 160 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![](docs/mona.gif) 2 | 3 | # Avatarify Python 4 | 5 | Photorealistic avatars for video-conferencing. 6 | 7 | Avatarify Python requires manually downloading and installing some dependencies, and is therefore best suited for users who have some experience with command line applications. [Avatarify Desktop](https://github.com/alievk/avatarify-desktop), which aims to be easier to install and use, is recommended for most users. If you still want to use Avatarify Python, proceed to the [install instructions](docs/). 8 | 9 | Based on [First Order Motion Model](https://github.com/AliaksandrSiarohin/first-order-model). 10 | 11 | ## News 12 | - **7 Mar 2021.** Renamed project to Avatarify Python to distinguish it from other versions of Avatarify 13 | - **14 December 2020.** Released Avatarify Desktop. Check it out [here](https://github.com/alievk/avatarify-desktop). 14 | - **11 July 2020.** Added Docker support. Now you can run Avatarify from Docker on [Linux](https://github.com/alievk/avatarify-python/blob/master/docs/README.md#docker). Thanks to [mikaelhg](https://github.com/mikaelhg) and [mintmaker](https://github.com/mintmaker) for contribution! 15 | - **22 May 2020.** Added [Google Colab](https://colab.research.google.com/github/alievk/avatarify/blob/master/avatarify.ipynb) mode. Now you can run Avatarify on any computer without GPU! 16 | - **7 May 2020.** Added remote GPU support for all platforms (based on [mynameisfiber's](https://github.com/mynameisfiber) solution). [Demo](https://youtu.be/3Dz_bUIPYFM). Deployment [instructions](https://github.com/alievk/avatarify-python/wiki/Remote-GPU). 17 | - **24 April 2020.** Added Windows installation [tutorial](https://www.youtube.com/watch?v=lym9ANVb120). 18 | - **17 April 2020.** Created Slack community. Please join via [invitation link](https://join.slack.com/t/avatarify/shared_invite/zt-dyoqy8tc-~4U2ObQ6WoxuwSaWKKVOgg). 19 | - **15 April 2020.** Added [StyleGAN-generated](https://www.thispersondoesnotexist.com) avatars. Just press `Q` and now you drive a person that never existed. Every time you push the button – new avatar is sampled. 20 | - **13 April 2020.** Added Windows support (kudos to [9of9](https://github.com/9of9)). 21 | 22 | ## Avatarify apps 23 | 24 | We have deployed Avatarify on iOS and Android devices using our proprietary inference engine. The iOS version features the Life mode for recording animations in real time. However, the Life mode is not available on Android devices due to the diversity of the devices we have to support. 25 | 26 | [drawing](https://apps.apple.com/app/apple-store/id1512669147?pt=121960189&ct=GitHub&mt=8) 27 | [drawing](https://play.google.com/store/apps/details?id=com.avatarify.android) 28 | -------------------------------------------------------------------------------- /afy/arguments.py: -------------------------------------------------------------------------------- 1 | from argparse import ArgumentParser 2 | 3 | parser = ArgumentParser() 4 | parser.add_argument("--config", help="path to config") 5 | parser.add_argument("--checkpoint", default='vox-cpk.pth.tar', help="path to checkpoint to restore") 6 | 7 | parser.add_argument("--relative", dest="relative", action="store_true", help="use relative or absolute keypoint coordinates") 8 | parser.add_argument("--adapt_scale", dest="adapt_scale", action="store_true", help="adapt movement scale based on convex hull of keypoints") 9 | parser.add_argument("--no-pad", dest="no_pad", action="store_true", help="don't pad output image") 10 | parser.add_argument("--enc_downscale", default=1, type=float, help="Downscale factor for encoder input. Improves performance with cost of quality.") 11 | 12 | parser.add_argument("--virt-cam", type=int, default=0, help="Virtualcam device ID") 13 | parser.add_argument("--no-stream", action="store_true", help="On Linux, force no streaming") 14 | 15 | parser.add_argument("--verbose", action="store_true", help="Print additional information") 16 | parser.add_argument("--hide-rect", action="store_true", default=False, help="Hide the helper rectangle in preview window") 17 | 18 | parser.add_argument("--avatars", default="./avatars", help="path to avatars directory") 19 | 20 | parser.add_argument("--is-worker", action="store_true", help="Whether to run this process as a remote GPU worker") 21 | parser.add_argument("--is-client", action="store_true", help="Whether to run this process as a client") 22 | parser.add_argument("--in-port", type=int, default=5557, help="Remote worker input port") 23 | parser.add_argument("--out-port", type=int, default=5558, help="Remote worker output port") 24 | parser.add_argument("--in-addr", type=str, default=None, help="Socket address for incoming messages, like example.com:5557") 25 | parser.add_argument("--out-addr", type=str, default=None, help="Socker address for outcoming messages, like example.com:5558") 26 | parser.add_argument("--jpg_quality", type=int, default=95, help="Jpeg copression quality for image transmission") 27 | 28 | parser.set_defaults(relative=False) 29 | parser.set_defaults(adapt_scale=False) 30 | parser.set_defaults(no_pad=False) 31 | 32 | opt = parser.parse_args() 33 | 34 | if opt.is_client and (opt.in_addr is None or opt.out_addr is None): 35 | raise ValueError("You have to set --in-addr and --out-addr") 36 | -------------------------------------------------------------------------------- /afy/cam_fomm.py: -------------------------------------------------------------------------------- 1 | import os, sys 2 | from sys import platform as _platform 3 | import glob 4 | import yaml 5 | import time 6 | import requests 7 | 8 | import numpy as np 9 | import cv2 10 | 11 | from afy.videocaptureasync import VideoCaptureAsync 12 | from afy.arguments import opt 13 | from afy.utils import info, Once, Tee, crop, pad_img, resize, TicToc 14 | import afy.camera_selector as cam_selector 15 | 16 | 17 | log = Tee('./var/log/cam_fomm.log') 18 | 19 | # Where to split an array from face_alignment to separate each landmark 20 | LANDMARK_SLICE_ARRAY = np.array([17, 22, 27, 31, 36, 42, 48, 60]) 21 | 22 | if _platform == 'darwin': 23 | if not opt.is_client: 24 | info('\nOnly remote GPU mode is supported for Mac (use --is-client and --connect options to connect to the server)') 25 | info('Standalone version will be available lately!\n') 26 | exit() 27 | 28 | 29 | def is_new_frame_better(source, driving, predictor): 30 | global avatar_kp 31 | global display_string 32 | 33 | if avatar_kp is None: 34 | display_string = "No face detected in avatar." 35 | return False 36 | 37 | if predictor.get_start_frame() is None: 38 | display_string = "No frame to compare to." 39 | return True 40 | 41 | driving_smaller = resize(driving, (128, 128))[..., :3] 42 | new_kp = predictor.get_frame_kp(driving) 43 | 44 | if new_kp is not None: 45 | new_norm = (np.abs(avatar_kp - new_kp) ** 2).sum() 46 | old_norm = (np.abs(avatar_kp - predictor.get_start_frame_kp()) ** 2).sum() 47 | 48 | out_string = "{0} : {1}".format(int(new_norm * 100), int(old_norm * 100)) 49 | display_string = out_string 50 | log(out_string) 51 | 52 | return new_norm < old_norm 53 | else: 54 | display_string = "No face found!" 55 | return False 56 | 57 | 58 | def load_stylegan_avatar(): 59 | url = "https://thispersondoesnotexist.com/image" 60 | r = requests.get(url, headers={'User-Agent': "My User Agent 1.0"}).content 61 | 62 | image = np.frombuffer(r, np.uint8) 63 | image = cv2.imdecode(image, cv2.IMREAD_COLOR) 64 | image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) 65 | 66 | image = resize(image, (IMG_SIZE, IMG_SIZE)) 67 | 68 | return image 69 | 70 | def load_images(IMG_SIZE = 256): 71 | avatars = [] 72 | filenames = [] 73 | images_list = sorted(glob.glob(f'{opt.avatars}/*')) 74 | for i, f in enumerate(images_list): 75 | if f.endswith('.jpg') or f.endswith('.jpeg') or f.endswith('.png'): 76 | img = cv2.imread(f) 77 | if img is None: 78 | log("Failed to open image: {}".format(f)) 79 | continue 80 | 81 | if img.ndim == 2: 82 | img = np.tile(img[..., None], [1, 1, 3]) 83 | img = img[..., :3][..., ::-1] 84 | img = resize(img, (IMG_SIZE, IMG_SIZE)) 85 | avatars.append(img) 86 | filenames.append(f) 87 | return avatars, filenames 88 | 89 | def change_avatar(predictor, new_avatar): 90 | global avatar, avatar_kp, kp_source 91 | avatar_kp = predictor.get_frame_kp(new_avatar) 92 | kp_source = None 93 | avatar = new_avatar 94 | predictor.set_source_image(avatar) 95 | 96 | 97 | def draw_rect(img, rw=0.6, rh=0.8, color=(255, 0, 0), thickness=2): 98 | h, w = img.shape[:2] 99 | l = w * (1 - rw) // 2 100 | r = w - l 101 | u = h * (1 - rh) // 2 102 | d = h - u 103 | img = cv2.rectangle(img, (int(l), int(u)), (int(r), int(d)), color, thickness) 104 | 105 | def kp_to_pixels(arr): 106 | '''Convert normalized landmark locations to screen pixels''' 107 | return ((arr + 1) * 127).astype(np.int32) 108 | 109 | def draw_face_landmarks(img, face_kp, color=(20, 80, 255)): 110 | if face_kp is not None: 111 | img = cv2.polylines(img, np.split(kp_to_pixels(face_kp), LANDMARK_SLICE_ARRAY), False, color) 112 | 113 | def print_help(): 114 | info('\n\n=== Control keys ===') 115 | info('1-9: Change avatar') 116 | for i, fname in enumerate(avatar_names): 117 | key = i + 1 118 | name = fname.split('/')[-1] 119 | info(f'{key}: {name}') 120 | info('W: Zoom camera in') 121 | info('S: Zoom camera out') 122 | info('A: Previous avatar in folder') 123 | info('D: Next avatar in folder') 124 | info('Q: Get random avatar') 125 | info('X: Calibrate face pose') 126 | info('I: Show FPS') 127 | info('ESC: Quit') 128 | info('\nFull key list: https://github.com/alievk/avatarify#controls') 129 | info('\n\n') 130 | 131 | 132 | def draw_fps(frame, fps, timing, x0=10, y0=20, ystep=30, fontsz=0.5, color=(255, 255, 255)): 133 | frame = frame.copy() 134 | cv2.putText(frame, f"FPS: {fps:.1f}", (x0, y0 + ystep * 0), 0, fontsz * IMG_SIZE / 256, color, 1) 135 | cv2.putText(frame, f"Model time (ms): {timing['predict']:.1f}", (x0, y0 + ystep * 1), 0, fontsz * IMG_SIZE / 256, color, 1) 136 | cv2.putText(frame, f"Preproc time (ms): {timing['preproc']:.1f}", (x0, y0 + ystep * 2), 0, fontsz * IMG_SIZE / 256, color, 1) 137 | cv2.putText(frame, f"Postproc time (ms): {timing['postproc']:.1f}", (x0, y0 + ystep * 3), 0, fontsz * IMG_SIZE / 256, color, 1) 138 | return frame 139 | 140 | 141 | def draw_landmark_text(frame, thk=2, fontsz=0.5, color=(0, 0, 255)): 142 | frame = frame.copy() 143 | cv2.putText(frame, "ALIGN FACES", (60, 20), 0, fontsz * IMG_SIZE / 255, color, thk) 144 | cv2.putText(frame, "THEN PRESS X", (60, 245), 0, fontsz * IMG_SIZE / 255, color, thk) 145 | return frame 146 | 147 | 148 | def draw_calib_text(frame, thk=2, fontsz=0.5, color=(0, 0, 255)): 149 | frame = frame.copy() 150 | cv2.putText(frame, "FIT FACE IN RECTANGLE", (40, 20), 0, fontsz * IMG_SIZE / 255, color, thk) 151 | cv2.putText(frame, "W - ZOOM IN", (60, 40), 0, fontsz * IMG_SIZE / 255, color, thk) 152 | cv2.putText(frame, "S - ZOOM OUT", (60, 60), 0, fontsz * IMG_SIZE / 255, color, thk) 153 | cv2.putText(frame, "THEN PRESS X", (60, 245), 0, fontsz * IMG_SIZE / 255, color, thk) 154 | return frame 155 | 156 | 157 | def select_camera(config): 158 | cam_config = config['cam_config'] 159 | cam_id = None 160 | 161 | if os.path.isfile(cam_config): 162 | with open(cam_config, 'r') as f: 163 | cam_config = yaml.load(f, Loader=yaml.FullLoader) 164 | cam_id = cam_config['cam_id'] 165 | else: 166 | cam_frames = cam_selector.query_cameras(config['query_n_cams']) 167 | 168 | if cam_frames: 169 | if len(cam_frames) == 1: 170 | cam_id = list(cam_frames)[0] 171 | else: 172 | cam_id = cam_selector.select_camera(cam_frames, window="CLICK ON YOUR CAMERA") 173 | log(f"Selected camera {cam_id}") 174 | 175 | with open(cam_config, 'w') as f: 176 | yaml.dump({'cam_id': cam_id}, f) 177 | else: 178 | log("No cameras are available") 179 | 180 | return cam_id 181 | 182 | 183 | if __name__ == "__main__": 184 | with open('config.yaml', 'r') as f: 185 | config = yaml.load(f, Loader=yaml.FullLoader) 186 | 187 | global display_string 188 | display_string = "" 189 | 190 | IMG_SIZE = 256 191 | 192 | log('Loading Predictor') 193 | predictor_args = { 194 | 'config_path': opt.config, 195 | 'checkpoint_path': opt.checkpoint, 196 | 'relative': opt.relative, 197 | 'adapt_movement_scale': opt.adapt_scale, 198 | 'enc_downscale': opt.enc_downscale 199 | } 200 | if opt.is_worker: 201 | from afy import predictor_worker 202 | predictor_worker.run_worker(opt.in_port, opt.out_port) 203 | sys.exit(0) 204 | elif opt.is_client: 205 | from afy import predictor_remote 206 | try: 207 | predictor = predictor_remote.PredictorRemote( 208 | in_addr=opt.in_addr, out_addr=opt.out_addr, 209 | **predictor_args 210 | ) 211 | except ConnectionError as err: 212 | log(err) 213 | sys.exit(1) 214 | predictor.start() 215 | else: 216 | from afy import predictor_local 217 | predictor = predictor_local.PredictorLocal( 218 | **predictor_args 219 | ) 220 | 221 | cam_id = select_camera(config) 222 | 223 | if cam_id is None: 224 | exit(1) 225 | 226 | cap = VideoCaptureAsync(cam_id) 227 | cap.start() 228 | 229 | avatars, avatar_names = load_images() 230 | 231 | enable_vcam = not opt.no_stream 232 | 233 | ret, frame = cap.read() 234 | stream_img_size = frame.shape[1], frame.shape[0] 235 | 236 | if enable_vcam: 237 | if _platform in ['linux', 'linux2']: 238 | try: 239 | import pyfakewebcam 240 | except ImportError: 241 | log("pyfakewebcam is not installed.") 242 | exit(1) 243 | 244 | stream = pyfakewebcam.FakeWebcam(f'/dev/video{opt.virt_cam}', *stream_img_size) 245 | else: 246 | enable_vcam = False 247 | # log("Virtual camera is supported only on Linux.") 248 | 249 | # if not enable_vcam: 250 | # log("Virtual camera streaming will be disabled.") 251 | 252 | cur_ava = 0 253 | avatar = None 254 | change_avatar(predictor, avatars[cur_ava]) 255 | passthrough = False 256 | 257 | cv2.namedWindow('cam', cv2.WINDOW_GUI_NORMAL) 258 | cv2.moveWindow('cam', 500, 250) 259 | 260 | frame_proportion = 0.9 261 | frame_offset_x = 0 262 | frame_offset_y = 0 263 | 264 | overlay_alpha = 0.0 265 | preview_flip = False 266 | output_flip = False 267 | find_keyframe = False 268 | is_calibrated = False 269 | 270 | show_landmarks = False 271 | 272 | fps_hist = [] 273 | fps = 0 274 | show_fps = False 275 | 276 | print_help() 277 | 278 | try: 279 | while True: 280 | tt = TicToc() 281 | 282 | timing = { 283 | 'preproc': 0, 284 | 'predict': 0, 285 | 'postproc': 0 286 | } 287 | 288 | green_overlay = False 289 | 290 | tt.tic() 291 | 292 | ret, frame = cap.read() 293 | if not ret: 294 | log("Can't receive frame (stream end?). Exiting ...") 295 | break 296 | 297 | frame = frame[..., ::-1] 298 | frame_orig = frame.copy() 299 | 300 | frame, (frame_offset_x, frame_offset_y) = crop(frame, p=frame_proportion, offset_x=frame_offset_x, offset_y=frame_offset_y) 301 | 302 | frame = resize(frame, (IMG_SIZE, IMG_SIZE))[..., :3] 303 | 304 | if find_keyframe: 305 | if is_new_frame_better(avatar, frame, predictor): 306 | log("Taking new frame!") 307 | green_overlay = True 308 | predictor.reset_frames() 309 | 310 | timing['preproc'] = tt.toc() 311 | 312 | if passthrough: 313 | out = frame 314 | elif is_calibrated: 315 | tt.tic() 316 | out = predictor.predict(frame) 317 | if out is None: 318 | log('predict returned None') 319 | timing['predict'] = tt.toc() 320 | else: 321 | out = None 322 | 323 | tt.tic() 324 | 325 | key = cv2.waitKey(1) 326 | 327 | if cv2.getWindowProperty('cam', cv2.WND_PROP_VISIBLE) < 1.0: 328 | break 329 | elif is_calibrated and cv2.getWindowProperty('avatarify', cv2.WND_PROP_VISIBLE) < 1.0: 330 | break 331 | 332 | if key == 27: # ESC 333 | break 334 | elif key == ord('d'): 335 | cur_ava += 1 336 | if cur_ava >= len(avatars): 337 | cur_ava = 0 338 | passthrough = False 339 | change_avatar(predictor, avatars[cur_ava]) 340 | elif key == ord('a'): 341 | cur_ava -= 1 342 | if cur_ava < 0: 343 | cur_ava = len(avatars) - 1 344 | passthrough = False 345 | change_avatar(predictor, avatars[cur_ava]) 346 | elif key == ord('w'): 347 | frame_proportion -= 0.05 348 | frame_proportion = max(frame_proportion, 0.1) 349 | elif key == ord('s'): 350 | frame_proportion += 0.05 351 | frame_proportion = min(frame_proportion, 1.0) 352 | elif key == ord('H'): 353 | frame_offset_x -= 1 354 | elif key == ord('h'): 355 | frame_offset_x -= 5 356 | elif key == ord('K'): 357 | frame_offset_x += 1 358 | elif key == ord('k'): 359 | frame_offset_x += 5 360 | elif key == ord('J'): 361 | frame_offset_y -= 1 362 | elif key == ord('j'): 363 | frame_offset_y -= 5 364 | elif key == ord('U'): 365 | frame_offset_y += 1 366 | elif key == ord('u'): 367 | frame_offset_y += 5 368 | elif key == ord('Z'): 369 | frame_offset_x = 0 370 | frame_offset_y = 0 371 | frame_proportion = 0.9 372 | elif key == ord('x'): 373 | predictor.reset_frames() 374 | 375 | if not is_calibrated: 376 | cv2.namedWindow('avatarify', cv2.WINDOW_GUI_NORMAL) 377 | cv2.moveWindow('avatarify', 600, 250) 378 | 379 | is_calibrated = True 380 | show_landmarks = False 381 | elif key == ord('z'): 382 | overlay_alpha = max(overlay_alpha - 0.1, 0.0) 383 | elif key == ord('c'): 384 | overlay_alpha = min(overlay_alpha + 0.1, 1.0) 385 | elif key == ord('r'): 386 | preview_flip = not preview_flip 387 | elif key == ord('t'): 388 | output_flip = not output_flip 389 | elif key == ord('f'): 390 | find_keyframe = not find_keyframe 391 | elif key == ord('o'): 392 | show_landmarks = not show_landmarks 393 | elif key == ord('q'): 394 | try: 395 | log('Loading StyleGAN avatar...') 396 | avatar = load_stylegan_avatar() 397 | passthrough = False 398 | change_avatar(predictor, avatar) 399 | except: 400 | log('Failed to load StyleGAN avatar') 401 | elif key == ord('l'): 402 | try: 403 | log('Reloading avatars...') 404 | avatars, avatar_names = load_images() 405 | passthrough = False 406 | log("Images reloaded") 407 | except: 408 | log('Image reload failed') 409 | elif key == ord('i'): 410 | show_fps = not show_fps 411 | elif 48 < key < 58: 412 | cur_ava = min(key - 49, len(avatars) - 1) 413 | passthrough = False 414 | change_avatar(predictor, avatars[cur_ava]) 415 | elif key == 48: 416 | passthrough = not passthrough 417 | elif key != -1: 418 | log(key) 419 | 420 | if overlay_alpha > 0: 421 | preview_frame = cv2.addWeighted( avatar, overlay_alpha, frame, 1.0 - overlay_alpha, 0.0) 422 | else: 423 | preview_frame = frame.copy() 424 | 425 | if show_landmarks: 426 | # Dim the background to make it easier to see the landmarks 427 | preview_frame = cv2.convertScaleAbs(preview_frame, alpha=0.5, beta=0.0) 428 | 429 | draw_face_landmarks(preview_frame, avatar_kp, (200, 20, 10)) 430 | frame_kp = predictor.get_frame_kp(frame) 431 | draw_face_landmarks(preview_frame, frame_kp) 432 | 433 | if preview_flip: 434 | preview_frame = cv2.flip(preview_frame, 1) 435 | 436 | if green_overlay: 437 | green_alpha = 0.8 438 | overlay = preview_frame.copy() 439 | overlay[:] = (0, 255, 0) 440 | preview_frame = cv2.addWeighted( preview_frame, green_alpha, overlay, 1.0 - green_alpha, 0.0) 441 | 442 | timing['postproc'] = tt.toc() 443 | 444 | if find_keyframe: 445 | preview_frame = cv2.putText(preview_frame, display_string, (10, 220), 0, 0.5 * IMG_SIZE / 256, (255, 255, 255), 1) 446 | 447 | if show_fps: 448 | preview_frame = draw_fps(preview_frame, fps, timing) 449 | 450 | if not is_calibrated: 451 | preview_frame = draw_calib_text(preview_frame) 452 | elif show_landmarks: 453 | preview_frame = draw_landmark_text(preview_frame) 454 | 455 | if not opt.hide_rect: 456 | draw_rect(preview_frame) 457 | 458 | cv2.imshow('cam', preview_frame[..., ::-1]) 459 | 460 | if out is not None: 461 | if not opt.no_pad: 462 | out = pad_img(out, stream_img_size) 463 | 464 | if output_flip: 465 | out = cv2.flip(out, 1) 466 | 467 | if enable_vcam: 468 | out = resize(out, stream_img_size) 469 | stream.schedule_frame(out) 470 | 471 | cv2.imshow('avatarify', out[..., ::-1]) 472 | 473 | fps_hist.append(tt.toc(total=True)) 474 | if len(fps_hist) == 10: 475 | fps = 10 / (sum(fps_hist) / 1000) 476 | fps_hist = [] 477 | except KeyboardInterrupt: 478 | log("main: user interrupt") 479 | 480 | log("stopping camera") 481 | cap.stop() 482 | 483 | cv2.destroyAllWindows() 484 | 485 | if opt.is_client: 486 | log("stopping remote predictor") 487 | predictor.stop() 488 | 489 | log("main: exit") 490 | -------------------------------------------------------------------------------- /afy/camera_selector.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import numpy as np 3 | import yaml 4 | 5 | from afy.utils import log 6 | 7 | 8 | g_selected_cam = None 9 | 10 | 11 | def query_cameras(n_cams): 12 | cam_frames = {} 13 | cap = None 14 | for camid in range(n_cams): 15 | log(f"Trying camera with id {camid}") 16 | cap = cv2.VideoCapture(camid) 17 | 18 | if not cap.isOpened(): 19 | log(f"Camera with id {camid} is not available") 20 | continue 21 | 22 | ret, frame = cap.read() 23 | 24 | if not ret or frame is None: 25 | log(f"Could not read from camera with id {camid}") 26 | cap.release() 27 | continue 28 | 29 | for i in range(10): 30 | ret, frame = cap.read() 31 | 32 | cam_frames[camid] = frame.copy() 33 | 34 | cap.release() 35 | 36 | return cam_frames 37 | 38 | 39 | def make_grid(images, cell_size=(320, 240), cols=2): 40 | w0, h0 = cell_size 41 | _rows = len(images) // cols + int(len(images) % cols) 42 | _cols = min(len(images), cols) 43 | grid = np.zeros((h0 * _rows, w0 * _cols, 3), dtype=np.uint8) 44 | for i, (camid, img) in enumerate(images.items()): 45 | img = cv2.resize(img, (w0, h0)) 46 | # add rect 47 | img = cv2.rectangle(img, (1, 1), (w0 - 1, h0 - 1), (0, 0, 255), 2) 48 | # add id 49 | img = cv2.putText(img, f'Camera {camid}', (10, 30), 0, 1, (0, 255, 0), 2) 50 | c = i % cols 51 | r = i // cols 52 | grid[r * h0:(r + 1) * h0, c * w0:(c + 1) * w0] = img[..., :3] 53 | return grid 54 | 55 | 56 | def mouse_callback(event, x, y, flags, userdata): 57 | global g_selected_cam 58 | if event == 1: 59 | cell_size, grid_cols, cam_frames = userdata 60 | c = x // cell_size[0] 61 | r = y // cell_size[1] 62 | camid = r * grid_cols + c 63 | if camid < len(cam_frames): 64 | g_selected_cam = camid 65 | 66 | 67 | def select_camera(cam_frames, window="Camera selector"): 68 | cell_size = 320, 240 69 | grid_cols = 2 70 | grid = make_grid(cam_frames, cols=grid_cols) 71 | 72 | # to fit the text if only one cam available 73 | if grid.shape[1] == 320: 74 | cell_size = 640, 480 75 | grid = cv2.resize(grid, cell_size) 76 | 77 | cv2.putText(grid, f'Click on the web camera to use', (10, grid.shape[0] - 30), 0, 0.7, (200, 200, 200), 2) 78 | 79 | cv2.namedWindow(window) 80 | cv2.setMouseCallback(window, mouse_callback, (cell_size, grid_cols, cam_frames)) 81 | cv2.imshow(window, grid) 82 | 83 | while True: 84 | key = cv2.waitKey(10) 85 | 86 | if g_selected_cam is not None: 87 | break 88 | 89 | if key == 27: 90 | break 91 | 92 | cv2.destroyAllWindows() 93 | 94 | if g_selected_cam is not None: 95 | return list(cam_frames)[g_selected_cam] 96 | else: 97 | return list(cam_frames)[0] 98 | 99 | 100 | if __name__ == '__main__': 101 | with open('config.yaml', 'r') as f: 102 | config = yaml.load(f, Loader=yaml.FullLoader) 103 | 104 | cam_frames = query_cameras(config['query_n_cams']) 105 | 106 | if cam_frames: 107 | selected_cam = select_camera(cam_frames) 108 | print(f"Selected camera {selected_cam}") 109 | else: 110 | log("No cameras are available") 111 | 112 | -------------------------------------------------------------------------------- /afy/networking.py: -------------------------------------------------------------------------------- 1 | import zmq 2 | import numpy as np 3 | import msgpack 4 | import msgpack_numpy as m 5 | m.patch() 6 | 7 | from afy.utils import log 8 | 9 | 10 | def check_connection(socket, timeout=1000): 11 | old_rcvtimeo = socket.RCVTIMEO 12 | socket.RCVTIMEO = timeout 13 | 14 | try: 15 | data = msgpack.packb(([], {})) 16 | socket.send_data('hello', data) 17 | attr_recv, data_recv = socket.recv_data() 18 | response = msgpack.unpackb(data_recv) 19 | except zmq.error.Again: 20 | return False 21 | finally: 22 | socket.RCVTIMEO = old_rcvtimeo 23 | 24 | log(f"Response to hello is {response}") 25 | 26 | return response == 'OK' 27 | 28 | 29 | class SerializingSocket(zmq.Socket): 30 | """Numpy array serialization methods. 31 | 32 | Based on https://github.com/jeffbass/imagezmq/blob/master/imagezmq/imagezmq.py#L291 33 | 34 | Used for sending / receiving OpenCV images, which are Numpy arrays. 35 | Also used for sending / receiving jpg compressed OpenCV images. 36 | """ 37 | 38 | def send_array(self, A, msg='NoName', flags=0, copy=True, track=False): 39 | """Sends a numpy array with metadata and text message. 40 | 41 | Sends a numpy array with the metadata necessary for reconstructing 42 | the array (dtype,shape). Also sends a text msg, often the array or 43 | image name. 44 | 45 | Arguments: 46 | A: numpy array or OpenCV image. 47 | msg: (optional) array name, image name or text message. 48 | flags: (optional) zmq flags. 49 | copy: (optional) zmq copy flag. 50 | track: (optional) zmq track flag. 51 | """ 52 | 53 | md = dict( 54 | msg=msg, 55 | dtype=str(A.dtype), 56 | shape=A.shape, 57 | ) 58 | self.send_json(md, flags | zmq.SNDMORE) 59 | return self.send(A, flags, copy=copy, track=track) 60 | 61 | def send_data(self, 62 | msg='NoName', 63 | data=b'00', 64 | flags=0, 65 | copy=True, 66 | track=False): 67 | """Send a jpg buffer with a text message. 68 | 69 | Sends a jpg bytestring of an OpenCV image. 70 | Also sends text msg, often the image name. 71 | 72 | Arguments: 73 | msg: image name or text message. 74 | data: binary data to be sent. 75 | flags: (optional) zmq flags. 76 | copy: (optional) zmq copy flag. 77 | track: (optional) zmq track flag. 78 | """ 79 | 80 | md = dict(msg=msg, ) 81 | self.send_json(md, flags | zmq.SNDMORE) 82 | return self.send(data, flags, copy=copy, track=track) 83 | 84 | def recv_array(self, flags=0, copy=True, track=False): 85 | """Receives a numpy array with metadata and text message. 86 | 87 | Receives a numpy array with the metadata necessary 88 | for reconstructing the array (dtype,shape). 89 | Returns the array and a text msg, often the array or image name. 90 | 91 | Arguments: 92 | flags: (optional) zmq flags. 93 | copy: (optional) zmq copy flag. 94 | track: (optional) zmq track flag. 95 | 96 | Returns: 97 | msg: image name or text message. 98 | A: numpy array or OpenCV image reconstructed with dtype and shape. 99 | """ 100 | 101 | md = self.recv_json(flags=flags) 102 | msg = self.recv(flags=flags, copy=copy, track=track) 103 | A = np.frombuffer(msg, dtype=md['dtype']) 104 | return (md['msg'], A.reshape(md['shape'])) 105 | 106 | def recv_data(self, flags=0, copy=True, track=False): 107 | """Receives a jpg buffer and a text msg. 108 | 109 | Receives a jpg bytestring of an OpenCV image. 110 | Also receives a text msg, often the image name. 111 | 112 | Arguments: 113 | flags: (optional) zmq flags. 114 | copy: (optional) zmq copy flag. 115 | track: (optional) zmq track flag. 116 | 117 | Returns: 118 | msg: image name or text message. 119 | data: bytestring, containing data. 120 | """ 121 | 122 | md = self.recv_json(flags=flags) # metadata text 123 | data = self.recv(flags=flags, copy=copy, track=track) 124 | return (md['msg'], data) 125 | 126 | 127 | class SerializingContext(zmq.Context): 128 | _socket_class = SerializingSocket 129 | -------------------------------------------------------------------------------- /afy/predictor_local.py: -------------------------------------------------------------------------------- 1 | from scipy.spatial import ConvexHull 2 | import torch 3 | import yaml 4 | from modules.keypoint_detector import KPDetector 5 | from modules.generator_optim import OcclusionAwareGenerator 6 | from sync_batchnorm import DataParallelWithCallback 7 | import numpy as np 8 | import face_alignment 9 | 10 | 11 | def normalize_kp(kp_source, kp_driving, kp_driving_initial, adapt_movement_scale=False, 12 | use_relative_movement=False, use_relative_jacobian=False): 13 | if adapt_movement_scale: 14 | source_area = ConvexHull(kp_source['value'][0].data.cpu().numpy()).volume 15 | driving_area = ConvexHull(kp_driving_initial['value'][0].data.cpu().numpy()).volume 16 | adapt_movement_scale = np.sqrt(source_area) / np.sqrt(driving_area) 17 | else: 18 | adapt_movement_scale = 1 19 | 20 | kp_new = {k: v for k, v in kp_driving.items()} 21 | 22 | if use_relative_movement: 23 | kp_value_diff = (kp_driving['value'] - kp_driving_initial['value']) 24 | kp_value_diff *= adapt_movement_scale 25 | kp_new['value'] = kp_value_diff + kp_source['value'] 26 | 27 | if use_relative_jacobian: 28 | jacobian_diff = torch.matmul(kp_driving['jacobian'], torch.inverse(kp_driving_initial['jacobian'])) 29 | kp_new['jacobian'] = torch.matmul(jacobian_diff, kp_source['jacobian']) 30 | 31 | return kp_new 32 | 33 | 34 | def to_tensor(a): 35 | return torch.tensor(a[np.newaxis].astype(np.float32)).permute(0, 3, 1, 2) / 255 36 | 37 | 38 | class PredictorLocal: 39 | def __init__(self, config_path, checkpoint_path, relative=False, adapt_movement_scale=False, device=None, enc_downscale=1): 40 | self.device = device or ('cuda' if torch.cuda.is_available() else 'cpu') 41 | self.relative = relative 42 | self.adapt_movement_scale = adapt_movement_scale 43 | self.start_frame = None 44 | self.start_frame_kp = None 45 | self.kp_driving_initial = None 46 | self.config_path = config_path 47 | self.checkpoint_path = checkpoint_path 48 | self.generator, self.kp_detector = self.load_checkpoints() 49 | self.fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D, flip_input=True, device=self.device) 50 | self.source = None 51 | self.kp_source = None 52 | self.enc_downscale = enc_downscale 53 | 54 | def load_checkpoints(self): 55 | with open(self.config_path) as f: 56 | config = yaml.load(f, Loader=yaml.FullLoader) 57 | 58 | generator = OcclusionAwareGenerator(**config['model_params']['generator_params'], 59 | **config['model_params']['common_params']) 60 | generator.to(self.device) 61 | 62 | kp_detector = KPDetector(**config['model_params']['kp_detector_params'], 63 | **config['model_params']['common_params']) 64 | kp_detector.to(self.device) 65 | 66 | checkpoint = torch.load(self.checkpoint_path, map_location=self.device) 67 | generator.load_state_dict(checkpoint['generator']) 68 | kp_detector.load_state_dict(checkpoint['kp_detector']) 69 | 70 | generator.eval() 71 | kp_detector.eval() 72 | 73 | return generator, kp_detector 74 | 75 | def reset_frames(self): 76 | self.kp_driving_initial = None 77 | 78 | def set_source_image(self, source_image): 79 | self.source = to_tensor(source_image).to(self.device) 80 | self.kp_source = self.kp_detector(self.source) 81 | 82 | if self.enc_downscale > 1: 83 | h, w = int(self.source.shape[2] / self.enc_downscale), int(self.source.shape[3] / self.enc_downscale) 84 | source_enc = torch.nn.functional.interpolate(self.source, size=(h, w), mode='bilinear') 85 | else: 86 | source_enc = self.source 87 | 88 | self.generator.encode_source(source_enc) 89 | 90 | def predict(self, driving_frame): 91 | assert self.kp_source is not None, "call set_source_image()" 92 | 93 | with torch.no_grad(): 94 | driving = to_tensor(driving_frame).to(self.device) 95 | 96 | if self.kp_driving_initial is None: 97 | self.kp_driving_initial = self.kp_detector(driving) 98 | self.start_frame = driving_frame.copy() 99 | self.start_frame_kp = self.get_frame_kp(driving_frame) 100 | 101 | kp_driving = self.kp_detector(driving) 102 | kp_norm = normalize_kp(kp_source=self.kp_source, kp_driving=kp_driving, 103 | kp_driving_initial=self.kp_driving_initial, use_relative_movement=self.relative, 104 | use_relative_jacobian=self.relative, adapt_movement_scale=self.adapt_movement_scale) 105 | 106 | out = self.generator(self.source, kp_source=self.kp_source, kp_driving=kp_norm) 107 | 108 | out = np.transpose(out['prediction'].data.cpu().numpy(), [0, 2, 3, 1])[0] 109 | out = (np.clip(out, 0, 1) * 255).astype(np.uint8) 110 | 111 | return out 112 | 113 | def get_frame_kp(self, image): 114 | kp_landmarks = self.fa.get_landmarks(image) 115 | if kp_landmarks: 116 | kp_image = kp_landmarks[0] 117 | kp_image = self.normalize_alignment_kp(kp_image) 118 | return kp_image 119 | else: 120 | return None 121 | 122 | @staticmethod 123 | def normalize_alignment_kp(kp): 124 | kp = kp - kp.mean(axis=0, keepdims=True) 125 | area = ConvexHull(kp[:, :2]).volume 126 | area = np.sqrt(area) 127 | kp[:, :2] = kp[:, :2] / area 128 | return kp 129 | 130 | def get_start_frame(self): 131 | return self.start_frame 132 | 133 | def get_start_frame_kp(self): 134 | return self.start_frame_kp 135 | -------------------------------------------------------------------------------- /afy/predictor_remote.py: -------------------------------------------------------------------------------- 1 | from arguments import opt 2 | from networking import SerializingContext, check_connection 3 | from utils import Logger, TicToc, AccumDict, Once 4 | 5 | import multiprocessing as mp 6 | import queue 7 | 8 | import cv2 9 | import numpy as np 10 | import zmq 11 | import msgpack 12 | import msgpack_numpy as m 13 | m.patch() 14 | 15 | 16 | PUT_TIMEOUT = 0.1 # s 17 | GET_TIMEOUT = 0.1 # s 18 | RECV_TIMEOUT = 1000 # ms 19 | QUEUE_SIZE = 100 20 | 21 | 22 | class PredictorRemote: 23 | def __init__(self, *args, in_addr=None, out_addr=None, **kwargs): 24 | self.in_addr = in_addr 25 | self.out_addr = out_addr 26 | self.predictor_args = (args, kwargs) 27 | self.timing = AccumDict() 28 | self.log = Logger('./var/log/predictor_remote.log', verbose=opt.verbose) 29 | 30 | self.send_queue = mp.Queue(QUEUE_SIZE) 31 | self.recv_queue = mp.Queue(QUEUE_SIZE) 32 | 33 | self.worker_alive = mp.Value('i', 0) 34 | 35 | self.send_process = mp.Process( 36 | target=self.send_worker, 37 | args=(self.in_addr, self.send_queue, self.worker_alive), 38 | name="send_process" 39 | ) 40 | self.recv_process = mp.Process( 41 | target=self.recv_worker, 42 | args=(self.out_addr, self.recv_queue, self.worker_alive), 43 | name="recv_process" 44 | ) 45 | 46 | self._i_msg = -1 47 | 48 | def start(self): 49 | self.worker_alive.value = 1 50 | self.send_process.start() 51 | self.recv_process.start() 52 | 53 | self.init_remote_worker() 54 | 55 | def stop(self): 56 | self.worker_alive.value = 0 57 | self.log("join worker processes...") 58 | self.send_process.join(timeout=5) 59 | self.recv_process.join(timeout=5) 60 | self.send_process.terminate() 61 | self.recv_process.terminate() 62 | 63 | def init_remote_worker(self): 64 | return self._send_recv_async('__init__', self.predictor_args, critical=True) 65 | 66 | def __getattr__(self, attr): 67 | is_critical = attr != 'predict' 68 | return lambda *args, **kwargs: self._send_recv_async(attr, (args, kwargs), critical=is_critical) 69 | 70 | def _send_recv_async(self, method, args, critical): 71 | self._i_msg += 1 72 | 73 | args, kwargs = args 74 | 75 | tt = TicToc() 76 | tt.tic() 77 | if method == 'predict': 78 | image = args[0] 79 | assert isinstance(image, np.ndarray), 'Expected image' 80 | ret_code, data = cv2.imencode(".jpg", image, [int(cv2.IMWRITE_JPEG_QUALITY), opt.jpg_quality]) 81 | else: 82 | data = msgpack.packb((args, kwargs)) 83 | self.timing.add('PACK', tt.toc()) 84 | 85 | meta = { 86 | 'name': method, 87 | 'critical': critical, 88 | 'id': self._i_msg 89 | } 90 | 91 | self.log("send", meta) 92 | 93 | if critical: 94 | self.send_queue.put((meta, data)) 95 | 96 | while True: 97 | meta_recv, data_recv = self.recv_queue.get() 98 | if meta_recv == meta: 99 | break 100 | else: 101 | try: 102 | # TODO: find good timeout 103 | self.send_queue.put((meta, data), timeout=PUT_TIMEOUT) 104 | except queue.Full: 105 | self.log('send_queue is full') 106 | 107 | try: 108 | meta_recv, data_recv = self.recv_queue.get(timeout=GET_TIMEOUT) 109 | except queue.Empty: 110 | self.log('recv_queue is empty') 111 | return None 112 | 113 | self.log("recv", meta_recv) 114 | 115 | tt.tic() 116 | if meta_recv['name'] == 'predict': 117 | result = cv2.imdecode(np.frombuffer(data_recv, dtype='uint8'), -1) 118 | else: 119 | result = msgpack.unpackb(data_recv) 120 | self.timing.add('UNPACK', tt.toc()) 121 | 122 | if opt.verbose: 123 | Once(self.timing, per=1) 124 | 125 | return result 126 | 127 | @staticmethod 128 | def send_worker(address, send_queue, worker_alive): 129 | timing = AccumDict() 130 | log = Logger('./var/log/send_worker.log', opt.verbose) 131 | 132 | ctx = SerializingContext() 133 | sender = ctx.socket(zmq.PUSH) 134 | sender.connect(address) 135 | 136 | log(f"Sending to {address}") 137 | 138 | try: 139 | while worker_alive.value: 140 | tt = TicToc() 141 | 142 | try: 143 | msg = send_queue.get(timeout=GET_TIMEOUT) 144 | except queue.Empty: 145 | continue 146 | 147 | tt.tic() 148 | sender.send_data(*msg) 149 | timing.add('SEND', tt.toc()) 150 | 151 | if opt.verbose: 152 | Once(timing, log, per=1) 153 | except KeyboardInterrupt: 154 | log("send_worker: user interrupt") 155 | finally: 156 | worker_alive.value = 0 157 | 158 | sender.disconnect(address) 159 | sender.close() 160 | ctx.destroy() 161 | log("send_worker exit") 162 | 163 | @staticmethod 164 | def recv_worker(address, recv_queue, worker_alive): 165 | timing = AccumDict() 166 | log = Logger('./var/log/recv_worker.log') 167 | 168 | ctx = SerializingContext() 169 | receiver = ctx.socket(zmq.PULL) 170 | receiver.connect(address) 171 | receiver.RCVTIMEO = RECV_TIMEOUT 172 | 173 | log(f"Receiving from {address}") 174 | 175 | try: 176 | while worker_alive.value: 177 | tt = TicToc() 178 | 179 | try: 180 | tt.tic() 181 | msg = receiver.recv_data() 182 | timing.add('RECV', tt.toc()) 183 | except zmq.error.Again: 184 | continue 185 | 186 | try: 187 | recv_queue.put(msg, timeout=PUT_TIMEOUT) 188 | except queue.Full: 189 | log('recv_queue full') 190 | continue 191 | 192 | if opt.verbose: 193 | Once(timing, log, per=1) 194 | except KeyboardInterrupt: 195 | log("recv_worker: user interrupt") 196 | finally: 197 | worker_alive.value = 0 198 | 199 | receiver.disconnect(address) 200 | receiver.close() 201 | ctx.destroy() 202 | log("recv_worker exit") 203 | -------------------------------------------------------------------------------- /afy/predictor_worker.py: -------------------------------------------------------------------------------- 1 | from predictor_local import PredictorLocal 2 | from arguments import opt 3 | from networking import SerializingContext, check_connection 4 | from utils import Logger, TicToc, AccumDict, Once 5 | 6 | import cv2 7 | import numpy as np 8 | import zmq 9 | import msgpack 10 | import msgpack_numpy as m 11 | m.patch() 12 | 13 | import queue 14 | import multiprocessing as mp 15 | import traceback 16 | import time 17 | 18 | 19 | PUT_TIMEOUT = 1 # s 20 | GET_TIMEOUT = 1 # s 21 | RECV_TIMEOUT = 1000 # ms 22 | QUEUE_SIZE = 100 23 | 24 | 25 | # class PredictorLocal(): 26 | # def __init__(self, *args, **kwargs): 27 | # pass 28 | 29 | # def __getattr__(self, *args, **kwargs): 30 | # return lambda *args, **kwargs: None 31 | 32 | 33 | class PredictorWorker(): 34 | def __init__(self, in_port=None, out_port=None): 35 | self.recv_queue = mp.Queue(QUEUE_SIZE) 36 | self.send_queue = mp.Queue(QUEUE_SIZE) 37 | 38 | self.worker_alive = mp.Value('i', 0) 39 | 40 | self.recv_process = mp.Process(target=self.recv_worker, args=(in_port, self.recv_queue, self.worker_alive)) 41 | self.predictor_process = mp.Process(target=self.predictor_worker, args=(self.recv_queue, self.send_queue, self.worker_alive)) 42 | self.send_process = mp.Process(target=self.send_worker, args=(out_port, self.send_queue, self.worker_alive)) 43 | 44 | def run(self): 45 | self.worker_alive.value = 1 46 | 47 | self.recv_process.start() 48 | self.predictor_process.start() 49 | self.send_process.start() 50 | 51 | try: 52 | self.recv_process.join() 53 | self.predictor_process.join() 54 | self.send_process.join() 55 | except KeyboardInterrupt: 56 | pass 57 | 58 | @staticmethod 59 | def recv_worker(port, recv_queue, worker_alive): 60 | timing = AccumDict() 61 | log = Logger('./var/log/recv_worker.log', verbose=opt.verbose) 62 | 63 | ctx = SerializingContext() 64 | socket = ctx.socket(zmq.PULL) 65 | socket.bind(f"tcp://*:{port}") 66 | socket.RCVTIMEO = RECV_TIMEOUT 67 | 68 | log(f'Receiving on port {port}', important=True) 69 | 70 | try: 71 | while worker_alive.value: 72 | tt = TicToc() 73 | 74 | try: 75 | tt.tic() 76 | msg = socket.recv_data() 77 | timing.add('RECV', tt.toc()) 78 | except zmq.error.Again: 79 | log("recv timeout") 80 | continue 81 | 82 | #log('recv', msg[0]) 83 | 84 | method, data = msg 85 | if method['critical']: 86 | recv_queue.put(msg) 87 | else: 88 | try: 89 | recv_queue.put(msg, block=False) 90 | except queue.Full: 91 | log('recv_queue full') 92 | 93 | Once(timing, log, per=1) 94 | except KeyboardInterrupt: 95 | log("recv_worker: user interrupt", important=True) 96 | 97 | worker_alive.value = 0 98 | log("recv_worker exit", important=True) 99 | 100 | @staticmethod 101 | def predictor_worker(recv_queue, send_queue, worker_alive): 102 | predictor = None 103 | predictor_args = () 104 | timing = AccumDict() 105 | log = Logger('./var/log/predictor_worker.log', verbose=opt.verbose) 106 | 107 | try: 108 | while worker_alive.value: 109 | tt = TicToc() 110 | 111 | try: 112 | method, data = recv_queue.get(timeout=GET_TIMEOUT) 113 | except queue.Empty: 114 | continue 115 | 116 | # get the latest non-critical request from the queue 117 | # don't skip critical request 118 | while not recv_queue.empty() and not method['critical']: 119 | log(f"skip {method}") 120 | method, data = recv_queue.get() 121 | 122 | log("working on", method) 123 | 124 | try: 125 | tt.tic() 126 | if method['name'] == 'predict': 127 | image = cv2.imdecode(np.frombuffer(data, dtype='uint8'), -1) 128 | else: 129 | args = msgpack.unpackb(data) 130 | timing.add('UNPACK', tt.toc()) 131 | except ValueError: 132 | log("Invalid Message", important=True) 133 | continue 134 | 135 | tt.tic() 136 | if method['name'] == "hello": 137 | result = "OK" 138 | elif method['name'] == "__init__": 139 | if args == predictor_args: 140 | log("Same config as before... reusing previous predictor") 141 | else: 142 | del predictor 143 | predictor_args = args 144 | predictor = PredictorLocal(*predictor_args[0], **predictor_args[1]) 145 | log("Initialized predictor with:", predictor_args, important=True) 146 | result = True 147 | tt.tic() # don't account for init 148 | elif method['name'] == 'predict': 149 | assert predictor is not None, "Predictor was not initialized" 150 | result = getattr(predictor, method['name'])(image) 151 | else: 152 | assert predictor is not None, "Predictor was not initialized" 153 | result = getattr(predictor, method['name'])(*args[0], **args[1]) 154 | timing.add('CALL', tt.toc()) 155 | 156 | tt.tic() 157 | if method['name'] == 'predict': 158 | assert isinstance(result, np.ndarray), f'Expected np.ndarray, got {result.__class__}' 159 | ret_code, data_send = cv2.imencode(".jpg", result, [int(cv2.IMWRITE_JPEG_QUALITY), opt.jpg_quality]) 160 | else: 161 | data_send = msgpack.packb(result) 162 | timing.add('PACK', tt.toc()) 163 | 164 | if method['critical']: 165 | send_queue.put((method, data_send)) 166 | else: 167 | try: 168 | send_queue.put((method, data_send), block=False) 169 | except queue.Full: 170 | log("send_queue full") 171 | pass 172 | 173 | Once(timing, log, per=1) 174 | except KeyboardInterrupt: 175 | log("predictor_worker: user interrupt", important=True) 176 | except Exception as e: 177 | log("predictor_worker error", important=True) 178 | traceback.print_exc() 179 | 180 | worker_alive.value = 0 181 | log("predictor_worker exit", important=True) 182 | 183 | @staticmethod 184 | def send_worker(port, send_queue, worker_alive): 185 | timing = AccumDict() 186 | log = Logger('./var/log/send_worker.log', verbose=opt.verbose) 187 | 188 | ctx = SerializingContext() 189 | socket = ctx.socket(zmq.PUSH) 190 | socket.bind(f"tcp://*:{port}") 191 | 192 | log(f'Sending on port {port}', important=True) 193 | 194 | try: 195 | while worker_alive.value: 196 | tt = TicToc() 197 | 198 | try: 199 | method, data = send_queue.get(timeout=GET_TIMEOUT) 200 | except queue.Empty: 201 | log("send queue empty") 202 | continue 203 | 204 | # get the latest non-critical request from the queue 205 | # don't skip critical request 206 | while not send_queue.empty() and not method['critical']: 207 | log(f"skip {method}") 208 | method, data = send_queue.get() 209 | 210 | log("sending", method) 211 | 212 | tt.tic() 213 | socket.send_data(method, data) 214 | timing.add('SEND', tt.toc()) 215 | 216 | Once(timing, log, per=1) 217 | except KeyboardInterrupt: 218 | log("predictor_worker: user interrupt", important=True) 219 | 220 | worker_alive.value = 0 221 | log("send_worker exit", important=True) 222 | 223 | 224 | def run_worker(in_port=None, out_port=None): 225 | worker = PredictorWorker(in_port=in_port, out_port=out_port) 226 | worker.run() 227 | -------------------------------------------------------------------------------- /afy/utils.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import time 3 | from collections import defaultdict 4 | 5 | import numpy as np 6 | import cv2 7 | 8 | 9 | def log(*args, file=sys.stderr, **kwargs): 10 | time_str = f'{time.time():.6f}' 11 | print(f'[{time_str}]', *args, file=file, **kwargs) 12 | 13 | 14 | def info(*args, file=sys.stdout, **kwargs): 15 | print(*args, file=file, **kwargs) 16 | 17 | 18 | class Tee(object): 19 | def __init__(self, filename, mode='w', terminal=sys.stderr): 20 | self.file = open(filename, mode, buffering=1) 21 | self.terminal = terminal 22 | 23 | def __del__(self): 24 | self.file.close() 25 | 26 | def write(self, *args, **kwargs): 27 | log(*args, file=self.file, **kwargs) 28 | log(*args, file=self.terminal, **kwargs) 29 | 30 | def __call__(self, *args, **kwargs): 31 | return self.write(*args, **kwargs) 32 | 33 | def flush(self): 34 | self.file.flush() 35 | 36 | 37 | class Logger(): 38 | def __init__(self, filename, verbose=True): 39 | self.tee = Tee(filename) 40 | self.verbose = verbose 41 | 42 | def __call__(self, *args, important=False, **kwargs): 43 | if not self.verbose and not important: 44 | return 45 | 46 | self.tee(*args, **kwargs) 47 | 48 | 49 | class Once(): 50 | _id = {} 51 | 52 | def __init__(self, what, who=log, per=1e12): 53 | """ Do who(what) once per seconds. 54 | what: args for who 55 | who: callable 56 | per: frequency in seconds. 57 | """ 58 | assert callable(who) 59 | now = time.time() 60 | if what not in Once._id or now - Once._id[what] > per: 61 | who(what) 62 | Once._id[what] = now 63 | 64 | 65 | class TicToc: 66 | def __init__(self): 67 | self.t = None 68 | self.t_init = time.time() 69 | 70 | def tic(self): 71 | self.t = time.time() 72 | 73 | def toc(self, total=False): 74 | if total: 75 | return (time.time() - self.t_init) * 1000 76 | 77 | assert self.t, 'You forgot to call tic()' 78 | return (time.time() - self.t) * 1000 79 | 80 | def tocp(self, str): 81 | t = self.toc() 82 | log(f"{str} took {t:.4f}ms") 83 | return t 84 | 85 | 86 | class AccumDict: 87 | def __init__(self, num_f=3): 88 | self.d = defaultdict(list) 89 | self.num_f = num_f 90 | 91 | def add(self, k, v): 92 | self.d[k] += [v] 93 | 94 | def __dict__(self): 95 | return self.d 96 | 97 | def __getitem__(self, key): 98 | return self.d[key] 99 | 100 | def __str__(self): 101 | s = '' 102 | for k in self.d: 103 | if not self.d[k]: 104 | continue 105 | cur = self.d[k][-1] 106 | avg = np.mean(self.d[k]) 107 | format_str = '{:.%df}' % self.num_f 108 | cur_str = format_str.format(cur) 109 | avg_str = format_str.format(avg) 110 | s += f'{k} {cur_str} ({avg_str})\t\t' 111 | return s 112 | 113 | def __repr__(self): 114 | return self.__str__() 115 | 116 | 117 | def clamp(value, min_value, max_value): 118 | return max(min(value, max_value), min_value) 119 | 120 | 121 | def crop(img, p=0.7, offset_x=0, offset_y=0): 122 | h, w = img.shape[:2] 123 | x = int(min(w, h) * p) 124 | l = (w - x) // 2 125 | r = w - l 126 | u = (h - x) // 2 127 | d = h - u 128 | 129 | offset_x = clamp(offset_x, -l, w - r) 130 | offset_y = clamp(offset_y, -u, h - d) 131 | 132 | l += offset_x 133 | r += offset_x 134 | u += offset_y 135 | d += offset_y 136 | 137 | return img[u:d, l:r], (offset_x, offset_y) 138 | 139 | 140 | def pad_img(img, target_size, default_pad=0): 141 | sh, sw = img.shape[:2] 142 | w, h = target_size 143 | pad_w, pad_h = default_pad, default_pad 144 | if w / h > 1: 145 | pad_w += int(sw * (w / h) - sw) // 2 146 | else: 147 | pad_h += int(sh * (h / w) - sh) // 2 148 | out = np.pad(img, [[pad_h, pad_h], [pad_w, pad_w], [0,0]], 'constant') 149 | return out 150 | 151 | 152 | def resize(img, size, version='cv'): 153 | return cv2.resize(img, size) 154 | -------------------------------------------------------------------------------- /afy/videocaptureasync.py: -------------------------------------------------------------------------------- 1 | # https://github.com/gilbertfrancois/video-capture-async 2 | 3 | import threading 4 | import cv2 5 | import time 6 | 7 | 8 | WARMUP_TIMEOUT = 10.0 9 | 10 | 11 | class VideoCaptureAsync: 12 | def __init__(self, src=0, width=640, height=480): 13 | self.src = src 14 | 15 | self.cap = cv2.VideoCapture(self.src) 16 | if not self.cap.isOpened(): 17 | raise RuntimeError("Cannot open camera") 18 | 19 | self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, width) 20 | self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height) 21 | self.grabbed, self.frame = self.cap.read() 22 | self.started = False 23 | self.read_lock = threading.Lock() 24 | 25 | def set(self, var1, var2): 26 | self.cap.set(var1, var2) 27 | 28 | def isOpened(self): 29 | return self.cap.isOpened() 30 | 31 | def start(self): 32 | if self.started: 33 | print('[!] Asynchronous video capturing has already been started.') 34 | return None 35 | self.started = True 36 | self.thread = threading.Thread(target=self.update, args=(), daemon=True) 37 | self.thread.start() 38 | 39 | # (warmup) wait for the first successfully grabbed frame 40 | warmup_start_time = time.time() 41 | while not self.grabbed: 42 | warmup_elapsed_time = (time.time() - warmup_start_time) 43 | if warmup_elapsed_time > WARMUP_TIMEOUT: 44 | raise RuntimeError(f"Failed to succesfully grab frame from the camera (timeout={WARMUP_TIMEOUT}s). Try to restart.") 45 | 46 | time.sleep(0.5) 47 | 48 | return self 49 | 50 | def update(self): 51 | while self.started: 52 | grabbed, frame = self.cap.read() 53 | if not grabbed or frame is None or frame.size == 0: 54 | continue 55 | with self.read_lock: 56 | self.grabbed = grabbed 57 | self.frame = frame 58 | 59 | def read(self): 60 | while True: 61 | with self.read_lock: 62 | frame = self.frame.copy() 63 | grabbed = self.grabbed 64 | break 65 | return grabbed, frame 66 | 67 | def stop(self): 68 | self.started = False 69 | self.thread.join() 70 | 71 | def __exit__(self, exec_type, exc_value, traceback): 72 | self.cap.release() 73 | -------------------------------------------------------------------------------- /avatarify.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Avatarify", 7 | "provenance": [], 8 | "collapsed_sections": [], 9 | "authorship_tag": "ABX9TyMqm8Y/SbrWZMeNGn+BbnaN", 10 | "include_colab_link": true 11 | }, 12 | "kernelspec": { 13 | "name": "python3", 14 | "display_name": "Python 3" 15 | }, 16 | "accelerator": "GPU" 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "metadata": { 22 | "id": "view-in-github", 23 | "colab_type": "text" 24 | }, 25 | "source": [ 26 | "\"Open" 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "metadata": { 32 | "id": "pEl-Q4OpsLb9", 33 | "colab_type": "text" 34 | }, 35 | "source": [ 36 | "# Avatarify Colab Server\n", 37 | "\n", 38 | "This Colab notebook is for running Avatarify rendering server. It allows you to run Avatarify on your computer **without GPU** in this way:\n", 39 | "\n", 40 | "1. When this notebook is executed, it starts listening for incoming requests from your computer;\n", 41 | "1. You start the client on your computer and it connects to the notebook and starts sending requests;\n", 42 | "1. This notebooks receives the requests from your computer, renders avatar images and sends them back;\n", 43 | "\n", 44 | "To this end, all the heavy work is offloaded from your computer to this notebook so you don't need to have a beafy hardware on your PC anymore.\n" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": { 50 | "id": "wRpMPl7VyeoD", 51 | "colab_type": "text" 52 | }, 53 | "source": [ 54 | "## Start the server\n", 55 | "Run the cells below (Shift+Enter) sequentially and pay attention to the hints and instructions included in this notebook.\n", 56 | "\n", 57 | "At the end you will get a command for running the client on your computer.\n", 58 | "\n", 59 | "## Start the client\n", 60 | "\n", 61 | "Make sure you have installed the latest version of Avatarify on your computer. Refer to the [README](https://github.com/alievk/avatarify#install) for the instructions.\n", 62 | "\n", 63 | "When it's ready execute this notebook and get the command for running the client on your computer.\n" 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": { 69 | "id": "8h0f5WQEjgbH", 70 | "colab_type": "text" 71 | }, 72 | "source": [ 73 | "### Technical details\n", 74 | "\n", 75 | "The client on your computer connects to the server via `ngrok` TCP tunnel or a reverse `ssh` tunnel.\n", 76 | "\n", 77 | "`ngrok`, while easy to use, can induce a considerable network lag ranging from dozens of milliseconds to a second. This can lead to a poor experience.\n", 78 | "\n", 79 | "A more stable connection could be established using a reverse `ssh` tunnel to a host with a public IP, like an AWS `t3.micro` (free) instance. This notebook provides a script for creating a tunnel, but launching an instance in a cloud is on your own (find the manual below)." 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": { 85 | "id": "6ZI4EvKaNUhL", 86 | "colab_type": "text" 87 | }, 88 | "source": [ 89 | "# Install" 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": { 95 | "id": "9fYm9X3X125H", 96 | "colab_type": "text" 97 | }, 98 | "source": [ 99 | "### Avatarify\n", 100 | "Follow the steps below to clone Avatarify and install the dependencies." 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "metadata": { 106 | "id": "LC1q-hdat-JP", 107 | "colab_type": "code", 108 | "colab": {} 109 | }, 110 | "source": [ 111 | "!cd /content\n", 112 | "!rm -rf *" 113 | ], 114 | "execution_count": 0, 115 | "outputs": [] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "metadata": { 120 | "id": "kE4_YSbsiX_O", 121 | "colab_type": "code", 122 | "colab": {} 123 | }, 124 | "source": [ 125 | "!git clone https://github.com/alievk/avatarify.git" 126 | ], 127 | "execution_count": 0, 128 | "outputs": [] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "metadata": { 133 | "id": "I8fTBhOEUM_c", 134 | "colab_type": "code", 135 | "colab": {} 136 | }, 137 | "source": [ 138 | "cd avatarify" 139 | ], 140 | "execution_count": 0, 141 | "outputs": [] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "metadata": { 146 | "id": "FONInDgZUmcZ", 147 | "colab_type": "code", 148 | "colab": {} 149 | }, 150 | "source": [ 151 | "!git clone https://github.com/alievk/first-order-model.git fomm\n", 152 | "!pip install face-alignment==1.0.0 msgpack_numpy pyyaml==5.1" 153 | ], 154 | "execution_count": 0, 155 | "outputs": [] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "metadata": { 160 | "id": "hPgXoqE_gAyD", 161 | "colab_type": "code", 162 | "colab": {} 163 | }, 164 | "source": [ 165 | "!scripts/download_data.sh" 166 | ], 167 | "execution_count": 0, 168 | "outputs": [] 169 | }, 170 | { 171 | "cell_type": "markdown", 172 | "metadata": { 173 | "id": "j1soT4zEEFzp", 174 | "colab_type": "text" 175 | }, 176 | "source": [ 177 | "### ngrok\n", 178 | "Follow the steps below to setup ngrok. You will also need to sign up on the ngrok site and get your authtoken (free).\n" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "metadata": { 184 | "id": "bbptNwHL1s61", 185 | "colab_type": "code", 186 | "colab": {} 187 | }, 188 | "source": [ 189 | "# Download ngrok\n", 190 | "!scripts/get_ngrok.sh" 191 | ], 192 | "execution_count": 0, 193 | "outputs": [] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": { 198 | "id": "4qk1FCeeaviZ", 199 | "colab_type": "text" 200 | }, 201 | "source": [ 202 | "# Run\n", 203 | "Start here if the runtime was restarted after installation." 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "metadata": { 209 | "id": "_f2iYcQVI2ss", 210 | "colab_type": "code", 211 | "colab": {} 212 | }, 213 | "source": [ 214 | "cd /content/avatarify" 215 | ], 216 | "execution_count": 0, 217 | "outputs": [] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "metadata": { 222 | "id": "qxK_ZZjPz_Rr", 223 | "colab_type": "code", 224 | "colab": {} 225 | }, 226 | "source": [ 227 | "#!git pull origin" 228 | ], 229 | "execution_count": 0, 230 | "outputs": [] 231 | }, 232 | { 233 | "cell_type": "code", 234 | "metadata": { 235 | "id": "MA-h22jF6-ks", 236 | "colab_type": "code", 237 | "colab": {} 238 | }, 239 | "source": [ 240 | "from subprocess import Popen, PIPE\n", 241 | "import shlex\n", 242 | "import json\n", 243 | "import time\n", 244 | "\n", 245 | "\n", 246 | "def run_with_pipe(command):\n", 247 | " commands = list(map(shlex.split,command.split(\"|\")))\n", 248 | " ps = Popen(commands[0], stdout=PIPE, stderr=PIPE)\n", 249 | " for command in commands[1:]:\n", 250 | " ps = Popen(command, stdin=ps.stdout, stdout=PIPE, stderr=PIPE)\n", 251 | " return ps.stdout.readlines()\n", 252 | "\n", 253 | "\n", 254 | "def get_tunnel_adresses():\n", 255 | " info = run_with_pipe(\"curl http://localhost:4040/api/tunnels\")\n", 256 | " assert info\n", 257 | "\n", 258 | " info = json.loads(info[0])\n", 259 | " for tunnel in info['tunnels']:\n", 260 | " url = tunnel['public_url']\n", 261 | " port = url.split(':')[-1]\n", 262 | " local_port = tunnel['config']['addr'].split(':')[-1]\n", 263 | " print(f'{url} -> {local_port} [{tunnel[\"name\"]}]')\n", 264 | " if tunnel['name'] == 'input':\n", 265 | " in_addr = url\n", 266 | " elif tunnel['name'] == 'output':\n", 267 | " out_addr = url\n", 268 | " else:\n", 269 | " print(f'unknown tunnel: {tunnel[\"name\"]}')\n", 270 | "\n", 271 | " return in_addr, out_addr" 272 | ], 273 | "execution_count": 0, 274 | "outputs": [] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "metadata": { 279 | "id": "RfHa02CBWoNN", 280 | "colab_type": "code", 281 | "colab": {} 282 | }, 283 | "source": [ 284 | "# Input and output ports for communication\n", 285 | "local_in_port = 5557\n", 286 | "local_out_port = 5558" 287 | ], 288 | "execution_count": 0, 289 | "outputs": [] 290 | }, 291 | { 292 | "cell_type": "markdown", 293 | "metadata": { 294 | "id": "BcULGnhGJJjC", 295 | "colab_type": "text" 296 | }, 297 | "source": [ 298 | "# Start the worker\n" 299 | ] 300 | }, 301 | { 302 | "cell_type": "code", 303 | "metadata": { 304 | "id": "8PnArK75mRqx", 305 | "colab_type": "code", 306 | "colab": {} 307 | }, 308 | "source": [ 309 | "# (Re)Start the worker\n", 310 | "with open('/tmp/run.txt', 'w') as f:\n", 311 | " ps = Popen(\n", 312 | " shlex.split(f'./run.sh --is-worker --in-port {local_in_port} --out-port {local_out_port} --no-vcam --no-conda'),\n", 313 | " stdout=f, stderr=f)\n", 314 | " time.sleep(3)" 315 | ], 316 | "execution_count": 0, 317 | "outputs": [] 318 | }, 319 | { 320 | "cell_type": "markdown", 321 | "metadata": { 322 | "id": "XfUqQxMtRSvc", 323 | "colab_type": "text" 324 | }, 325 | "source": [ 326 | "This command should print lines if the worker is successfully started" 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "metadata": { 332 | "id": "W0eY8gkBqUJG", 333 | "colab_type": "code", 334 | "colab": {} 335 | }, 336 | "source": [ 337 | "!ps aux | grep 'python3 afy/cam_fomm.py' | grep -v grep | tee /tmp/ps_run\n", 338 | "!if [[ $(cat /tmp/ps_run | wc -l) == \"0\" ]]; then echo \"Worker failed to start\"; cat /tmp/run.txt; else echo \"Worker started\"; fi" 339 | ], 340 | "execution_count": 0, 341 | "outputs": [] 342 | }, 343 | { 344 | "cell_type": "markdown", 345 | "metadata": { 346 | "id": "tz9gpLD0IsCL", 347 | "colab_type": "text" 348 | }, 349 | "source": [ 350 | "# Open ngrok tunnel" 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": { 356 | "id": "gyB7XIxL0XpD", 357 | "colab_type": "text" 358 | }, 359 | "source": [ 360 | "#### Get ngrok token\n", 361 | "Go to https://dashboard.ngrok.com/auth/your-authtoken (sign up if required), copy your authtoken and put it below." 362 | ] 363 | }, 364 | { 365 | "cell_type": "code", 366 | "metadata": { 367 | "id": "YDtPpi77AkQ1", 368 | "colab_type": "code", 369 | "colab": {} 370 | }, 371 | "source": [ 372 | "# Paste your authtoken here in quotes\n", 373 | "authtoken = \"1cBzFFwzSlaLhlRPXIHJiVLqtiQ_2cVsonJXe52B6DDyp8su7\"" 374 | ], 375 | "execution_count": 0, 376 | "outputs": [] 377 | }, 378 | { 379 | "cell_type": "markdown", 380 | "metadata": { 381 | "id": "gASaDrsFXLXA", 382 | "colab_type": "text" 383 | }, 384 | "source": [ 385 | "Set your region\n", 386 | "\n", 387 | "Code | Region\n", 388 | "--- | ---\n", 389 | "us | United States\n", 390 | "eu | Europe\n", 391 | "ap | Asia/Pacific\n", 392 | "au | Australia\n", 393 | "sa | South America\n", 394 | "jp | Japan\n", 395 | "in | India" 396 | ] 397 | }, 398 | { 399 | "cell_type": "code", 400 | "metadata": { 401 | "id": "r5e9VR9NYckJ", 402 | "colab_type": "code", 403 | "colab": {} 404 | }, 405 | "source": [ 406 | "# Set your region here in quotes\n", 407 | "region = \"eu\"" 408 | ], 409 | "execution_count": 0, 410 | "outputs": [] 411 | }, 412 | { 413 | "cell_type": "code", 414 | "metadata": { 415 | "id": "jZ5_PE_EHpCg", 416 | "colab_type": "code", 417 | "colab": {} 418 | }, 419 | "source": [ 420 | "config =\\\n", 421 | "f\"\"\"\n", 422 | "version: 2\n", 423 | "authtoken: {authtoken}\n", 424 | "region: {region}\n", 425 | "console_ui: False\n", 426 | "tunnels:\n", 427 | " input:\n", 428 | " addr: {local_in_port}\n", 429 | " proto: tcp \n", 430 | " output:\n", 431 | " addr: {local_out_port}\n", 432 | " proto: tcp\n", 433 | "\"\"\"\n", 434 | "\n", 435 | "with open('ngrok.conf', 'w') as f:\n", 436 | " f.write(config)" 437 | ], 438 | "execution_count": 0, 439 | "outputs": [] 440 | }, 441 | { 442 | "cell_type": "code", 443 | "metadata": { 444 | "id": "Z49OEhAdDI7Y", 445 | "colab_type": "code", 446 | "colab": {} 447 | }, 448 | "source": [ 449 | "# (Re)Open tunnel\n", 450 | "ps = Popen('./scripts/open_tunnel_ngrok.sh', stdout=PIPE, stderr=PIPE)\n", 451 | "time.sleep(3)" 452 | ], 453 | "execution_count": 0, 454 | "outputs": [] 455 | }, 456 | { 457 | "cell_type": "code", 458 | "metadata": { 459 | "id": "JAyPH2t2C64H", 460 | "colab_type": "code", 461 | "colab": {} 462 | }, 463 | "source": [ 464 | "# Get tunnel addresses\n", 465 | "try:\n", 466 | " in_addr, out_addr = get_tunnel_adresses()\n", 467 | " print(\"Tunnel opened\")\n", 468 | "except Exception as e:\n", 469 | " [print(l.decode(), end='') for l in ps.stdout.readlines()]\n", 470 | " print(\"Something went wrong, reopen the tunnel\")" 471 | ], 472 | "execution_count": 0, 473 | "outputs": [] 474 | }, 475 | { 476 | "cell_type": "markdown", 477 | "metadata": { 478 | "id": "dc6rg2HQAocK", 479 | "colab_type": "text" 480 | }, 481 | "source": [ 482 | "### [Optional] AWS proxy\n", 483 | "Alternatively you can create a ssh reverse tunnel to an AWS `t3.micro` instance (it's free). It has lower latency than ngrok.\n", 484 | "\n", 485 | "1. In your AWS console go to Services -> EC2 -> Instances -> Launch Instance;\n", 486 | "1. Choose `Ubuntu Server 18.04 LTS` AMI;\n", 487 | "1. Choose `t3.micro` instance type and press Review and launch;\n", 488 | "1. Confirm your key pair and press Launch instances;\n", 489 | "1. Go to the security group of this instance and edit inbound rules. Add TCP ports 5557 and 5558 and set Source to Anywhere. Press Save rules;\n", 490 | "1. ssh into the instance (you can find the command in the Instances if you click on the Connect button) and add this line in the end of `/etc/ssh/sshd_config`:\n", 491 | "```\n", 492 | "GatewayPorts yes\n", 493 | "```\n", 494 | "then restart `sshd`\n", 495 | "```\n", 496 | "sudo service sshd restart\n", 497 | "```\n", 498 | "1. Copy your `key_pair.pem` by dragging and dropping it into avatarify folder in this notebook;\n", 499 | "1. Use the command below to open the tunnel;\n", 500 | "1. Start client with a command (substitute `run_mac.sh` with `run_windows.bat` or `run.sh`)\n", 501 | "```\n", 502 | "./run_mac.sh --is-client --in-addr tcp://instace.compute.amazonaws.com:5557 --out-addr tcp://instance.compute.amazonaws.com:5558\n", 503 | "```" 504 | ] 505 | }, 506 | { 507 | "cell_type": "code", 508 | "metadata": { 509 | "id": "zdN5Qj2BCYsr", 510 | "colab_type": "code", 511 | "colab": {} 512 | }, 513 | "source": [ 514 | "# Open reverse ssh tunnel (uncomment line below)\n", 515 | "# !./scripts/open_tunnel_ssh.sh key_pair.pem ubuntu@instance.compute.amazonaws.com" 516 | ], 517 | "execution_count": 0, 518 | "outputs": [] 519 | }, 520 | { 521 | "cell_type": "markdown", 522 | "metadata": { 523 | "id": "ccZ24BT4Jdis", 524 | "colab_type": "text" 525 | }, 526 | "source": [ 527 | "# Start the client\n", 528 | "When you run the cell below it will print a command. Run this command on your computer:\n", 529 | "\n", 530 | "1. Open a terminal (in Windows open `Anaconda Prompt`);\n", 531 | "2. Change working directory to the `avatarify` directory:
\n", 532 | "* Windows (change `C:\\path\\to\\avatarify` to your path)
\n", 533 | "`cd C:\\path\\to\\avatarify`

\n", 534 | "* Mac/Linux (change `/path/to/avatarify` to your path)
\n", 535 | "`cd /path/to/avatarify`\n", 536 | "3. Copy-paste to the terminal the command below and run;\n", 537 | "4. It can take some time to connect (usually up to 10 seconds). If the preview window doesn't appear in a minute or two, look for the errors above in this notebook and report in the [issues](https://github.com/alievk/avatarify/issues) or [Slack](https://join.slack.com/t/avatarify/shared_invite/zt-dyoqy8tc-~4U2ObQ6WoxuwSaWKKVOgg)." 538 | ] 539 | }, 540 | { 541 | "cell_type": "code", 542 | "metadata": { 543 | "id": "4gaqS0mZWF1V", 544 | "colab_type": "code", 545 | "colab": {} 546 | }, 547 | "source": [ 548 | "print('Copy-paste to the terminal the command below and run (press Enter)\\n')\n", 549 | "print('Mac:')\n", 550 | "print(f'./run_mac.sh --is-client --in-addr {in_addr} --out-addr {out_addr}')\n", 551 | "print('\\nWindows:')\n", 552 | "print(f'run_windows.bat --is-client --in-addr {in_addr} --out-addr {out_addr}')\n", 553 | "print('\\nLinux:')\n", 554 | "print(f'./run.sh --is-client --in-addr {in_addr} --out-addr {out_addr}')" 555 | ], 556 | "execution_count": 0, 557 | "outputs": [] 558 | }, 559 | { 560 | "cell_type": "markdown", 561 | "metadata": { 562 | "id": "M3h92xQ9KA-R", 563 | "colab_type": "text" 564 | }, 565 | "source": [ 566 | "# Logs" 567 | ] 568 | }, 569 | { 570 | "cell_type": "markdown", 571 | "metadata": { 572 | "id": "SvodbjapKBQi", 573 | "colab_type": "text" 574 | }, 575 | "source": [ 576 | "If something doesn't work as expected, please run the cells below and include the logs in your report." 577 | ] 578 | }, 579 | { 580 | "cell_type": "code", 581 | "metadata": { 582 | "id": "0GeT7KxON0Ke", 583 | "colab_type": "code", 584 | "colab": {}, 585 | "cellView": "form" 586 | }, 587 | "source": [ 588 | "#@title\n", 589 | "!cat ./var/log/cam_fomm.log | head -100" 590 | ], 591 | "execution_count": 0, 592 | "outputs": [] 593 | }, 594 | { 595 | "cell_type": "code", 596 | "metadata": { 597 | "id": "P1FQcdzwqdce", 598 | "colab_type": "code", 599 | "colab": {}, 600 | "cellView": "form" 601 | }, 602 | "source": [ 603 | "#@title\n", 604 | "!cat ./var/log/recv_worker.log | tail -100" 605 | ], 606 | "execution_count": 0, 607 | "outputs": [] 608 | }, 609 | { 610 | "cell_type": "code", 611 | "metadata": { 612 | "id": "YThWBXCf_yzI", 613 | "colab_type": "code", 614 | "colab": {}, 615 | "cellView": "form" 616 | }, 617 | "source": [ 618 | "#@title\n", 619 | "!cat ./var/log/predictor_worker.log | tail -100" 620 | ], 621 | "execution_count": 0, 622 | "outputs": [] 623 | }, 624 | { 625 | "cell_type": "code", 626 | "metadata": { 627 | "id": "zhJzygCP_6p3", 628 | "colab_type": "code", 629 | "colab": {}, 630 | "cellView": "form" 631 | }, 632 | "source": [ 633 | "#@title\n", 634 | "!cat ./var/log/send_worker.log | tail -100" 635 | ], 636 | "execution_count": 0, 637 | "outputs": [] 638 | }, 639 | { 640 | "cell_type": "code", 641 | "metadata": { 642 | "id": "nrzNffhR_8HQ", 643 | "colab_type": "code", 644 | "colab": {} 645 | }, 646 | "source": [ 647 | "" 648 | ], 649 | "execution_count": 0, 650 | "outputs": [] 651 | } 652 | ] 653 | } 654 | -------------------------------------------------------------------------------- /avatars/.geetkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/avatars/.geetkeep -------------------------------------------------------------------------------- /avatars/einstein.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/avatars/einstein.jpg -------------------------------------------------------------------------------- /avatars/eminem.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/avatars/eminem.jpg -------------------------------------------------------------------------------- /avatars/jobs.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/avatars/jobs.jpg -------------------------------------------------------------------------------- /avatars/mona.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/avatars/mona.jpg -------------------------------------------------------------------------------- /avatars/obama.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/avatars/obama.jpg -------------------------------------------------------------------------------- /avatars/potter.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/avatars/potter.jpg -------------------------------------------------------------------------------- /avatars/ronaldo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/avatars/ronaldo.png -------------------------------------------------------------------------------- /avatars/schwarzenegger.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/avatars/schwarzenegger.png -------------------------------------------------------------------------------- /config.yaml: -------------------------------------------------------------------------------- 1 | # how many cameras to query at the first start 2 | query_n_cams: 4 3 | 4 | # camera configuration path 5 | cam_config: ./cam.yaml 6 | -------------------------------------------------------------------------------- /docs/README.md: -------------------------------------------------------------------------------- 1 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/alievk/avatarify/blob/master/avatarify.ipynb) 2 | 3 | :arrow_forward: [Demo](https://youtu.be/Q7LFDT-FRzs) 4 | 5 | :arrow_forward: [AI-generated Elon Musk](https://youtu.be/lONuXGNqLO0) 6 | 7 | ## Table of Contents 8 | - [Requirements](#requirements) 9 | - [Install](#install) 10 | - [Download network weights](#download-network-weights) 11 | - [Linux](#linux) 12 | - [Mac](#mac) 13 | - [Windows](#windows) 14 | - [Remote GPU](#remote-gpu) 15 | - [Docker](#docker) 16 | - [Setup avatars](#setup-avatars) 17 | - [Run](#run) 18 | - [Linux](#linux-1) 19 | - [Mac](#mac-1) 20 | - [Windows](#windows-1) 21 | - [Controls](#controls) 22 | - [Driving your avatar](#driving-your-avatar) 23 | - [Configure video meeting app](#configure-video-meeting-app) 24 | - [Skype](#skype) 25 | - [Zoom](#zoom) 26 | - [Teams](#teams) 27 | - [Slack](#slack) 28 | - [Uninstall](#uninstall) 29 | - [Contribution](#contribution) 30 | - [FAQ](#faq) 31 | - [Troubleshooting](#troubleshooting) 32 | 33 | ## Requirements 34 | 35 | You can run Avatarify in two modes: *locally* and *remotely*. 36 | 37 | To run Avatarify *locally* you need a CUDA-enabled (NVIDIA) video card. Otherwise it will fallback to the central processor and run very slowly. These are performance metrics for some hardware: 38 | 39 | - GeForce GTX 1080 Ti: **33 frames per second** 40 | - GeForce GTX 1070: **15 frames per second** 41 | - GeForce GTX 950: **9 frames per second** 42 | 43 | You can also run Avatarify *remotely* on [Google Colab](https://colab.research.google.com/github/alievk/avatarify/blob/master/avatarify.ipynb) (easy) or on a [dedicated server](https://github.com/alievk/avatarify-python/wiki/Remote-GPU) with a GPU (harder). There are no special PC requirements for this mode, only a stable internet connection. 44 | 45 | Of course, you also need a webcam! 46 | 47 | 49 | 50 | ## Install 51 | 52 | #### Download network weights 53 | Download model's weights from [here](https://openavatarify.s3-avatarify.com/weights/vox-adv-cpk.pth.tar) or [here](https://yadi.sk/d/M0FWpz2ExBfgAA) or [here](https://drive.google.com/file/d/1coUCdyRXDbpWnEkA99NLNY60mb9dQ_n3/view?usp=sharing) [228 MB, md5sum `8a45a24037871c045fbb8a6a8aa95ebc`] 54 | 55 | #### Linux 56 | Linux uses `v4l2loopback` to create virtual camera. 57 | 58 | 59 | 1. Download [Miniconda Python 3.7](https://docs.conda.io/en/latest/miniconda.html#linux-installers) and install using command: 60 | ```bash 61 | bash Miniconda3-latest-Linux-x86_64.sh 62 | ``` 63 | 2. Clone `avatarify` and install its dependencies (sudo privelege is required): 64 | ```bash 65 | git clone https://github.com/alievk/avatarify-python.git 66 | cd avatarify-python 67 | bash scripts/install.sh 68 | ``` 69 | 3. [Download network weights](#download-network-weights) and place `vox-adv-cpk.pth.tar` file in the `avatarify-python` directory (don't unpack it). 70 | 71 | 72 | #### Mac 73 | 81 | 82 | 83 | We will use [CamTwist](http://camtwiststudio.com) to create virtual camera for Mac. 84 | 85 | 1. Install [Miniconda Python 3.7](https://docs.conda.io/en/latest/miniconda.html#macosx-installers) or use *Homebrew Cask*: `brew install --cask miniconda`. 86 | 2. [Download](https://github.com/alievk/avatarify-python/archive/master.zip) and unpack the repository or use `git`: 87 | ```bash 88 | git clone https://github.com/alievk/avatarify-python.git 89 | cd avatarify-python 90 | bash scripts/install_mac.sh 91 | ``` 92 | 3. Download and install [CamTwist](http://camtwiststudio.com) from [here](http://camtwiststudio.com/download). It's easy. 93 | 94 | #### Windows 95 | 96 | 97 | 98 | :arrow_forward: [Video tutorial](https://youtu.be/lym9ANVb120) 99 | 100 | This guide is tested for Windows 10. 101 | 102 | 103 | 1. Install [Miniconda Python 3.8](https://docs.conda.io/en/latest/miniconda.html#windows-installers). 104 | 2. Install [Git](https://git-scm.com/download/win). 105 | 3. Press Windows button and type "miniconda". Run suggested Anaconda Prompt. 106 | 4. Download and install Avatarify (please copy-paste these commands and don't change them): 107 | ```bash 108 | git clone https://github.com/alievk/avatarify-python.git 109 | cd avatarify-python 110 | scripts\install_windows.bat 111 | ``` 112 | 5. [Download network weights](#download-network-weights) and place `vox-adv-cpk.pth.tar` file in the `avatarify-python` directory (don't unpack it). 113 | 6. Run `run_windows.bat`. If installation was successful, two windows "cam" and "avatarify" will appear. Leave these windows open for the next installation steps. 114 | 7. Install [OBS Studio](https://obsproject.com/) for capturing Avatarify output. 115 | 8. Install [VirtualCam plugin](https://obsproject.com/forum/resources/obs-virtualcam.539/). Choose `Install and register only 1 virtual camera`. 116 | 9. Run OBS Studio. 117 | 10. In the Sources section, press on Add button ("+" sign), select Windows Capture and press OK. In the appeared window, choose "[python.exe]: avatarify" in Window drop-down menu and press OK. Then select Edit -> Transform -> Fit to screen. 118 | 11. In OBS Studio, go to Tools -> VirtualCam. Check AutoStart, set Buffered Frames to 0 and press Start. 119 | 12. Now `OBS-Camera` camera should be available in Zoom (or other videoconferencing software). 120 | 121 | The steps 10-11 are required only once during setup. 122 | 123 | #### Remote GPU 124 | 125 | You can offload the heavy work to [Google Colab](https://colab.research.google.com/github/alievk/avatarify/blob/master/avatarify.ipynb) or a [server with a GPU](https://github.com/alievk/avatarify-python/wiki/Remote-GPU) and use your laptop just to communicate the video stream. The server and client software are native and dockerized available. 126 | 127 | ### Docker 128 | Docker images are only availabe on Linux. 129 | 130 | 1. Install Docker following the [Documentation](https://docs.docker.com/engine/install/). Then run this [step](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user) to make docker available for your user. 131 | 2. For using the gpu (hardly recommended): Install nvidia drivers and [nvidia docker](https://github.com/NVIDIA/nvidia-docker#quickstart). 132 | 3. Clone `avatarify-python` and install its dependencies (v4l2loopback kernel module): 133 | ```bash 134 | git clone https://github.com/alievk/avatarify-python.git 135 | cd avatarify-python 136 | bash scripts/install_docker.sh 137 | ``` 138 | 4. Build the Dockerfile: 139 | ```bash 140 | cd avatarify-python 141 | docker build -t avatarify . 142 | ``` 143 | ## Setup avatars 144 | Avatarify comes with a standard set of avatars of famous people, but you can extend this set simply copying your avatars into `avatars` folder. 145 | 146 | Follow these advices for better visual quality: 147 | * Make square crop of your avatar picture. 148 | * Crop avatar's face so that it's not too close not too far. Use standard avatars as reference. 149 | * Prefer pictures with uniform background. It will diminish visual artifacts. 150 | 151 | ## Run 152 | Your web cam must be plugged-in. 153 | 154 | **Note:** run your video-conferencing app only after Avatarify is started. 155 | 156 | #### Linux 157 | The run script will create virtual camera `/dev/video9`. You can change these settings in `scripts/settings.sh`. 158 | 159 | 160 | You can use command `v4l2-ctl --list-devices` to list all devices in your system. 161 | 162 | Run: 163 | ```bash 164 | bash run.sh 165 | ``` 166 | If you haven't installed a GPU add the `--no-gpus` flag. In order to use Docker add the `--docker` flag. 167 | 168 | `cam` and `avatarify` windows will pop-up. The `cam` window is for controlling your face position and `avatarify` is for the avatar animation preview. Please follow these [recommendations](#driving-your-avatar) to drive your avatars. 169 | 170 | #### Mac 171 | *Note*: On Mac Avatarify runs only with [Google Colab](https://colab.research.google.com/github/alievk/avatarify/blob/master/avatarify.ipynb) or a [dedicated server](https://github.com/alievk/avatarify-python/wiki/Remote-GPU) with GPU. 172 | 173 | Please find where you downloaded `avatarify` and substitute path `/path/to/avatarify` below. 174 | 175 | 180 | 1. To run Avatarify please follow instructions for [Google Colab](https://colab.research.google.com/github/alievk/avatarify/blob/master/avatarify.ipynb) or a [dedicated server](https://github.com/alievk/avatarify-python/wiki/Remote-GPU). 181 | 2. Go to [CamTwist](http://camtwiststudio.com). 182 | 3. Choose `Desktop+` and press `Select`. 183 | 4. In the `Settings` section choose `Confine to Application Window` and select `python (avatarify)` from the drop-down menu. 184 | 185 | `cam` and `avatarify` windows will pop-up. The `cam` window is for controlling your face position and `avatarify` is for the avatar animation preview. Please follow these [recommendations](#driving-your-avatar) to drive your avatars. 186 | 187 | #### Windows 188 | 189 | 192 | 193 | 1. In Anaconda Prompt: 194 | ``` 195 | cd C:\path\to\avatarify 196 | run_windows.bat 197 | ``` 198 | 2. Run OBS Studio. It should automaitcally start streaming video from Avatarify to `OBS-Camera`. 199 | 200 | `cam` and `avatarify` windows will pop-up. The `cam` window is for controlling your face position and `avatarify` is for the avatar animation preview. Please follow these [recommendations](#driving-your-avatar) to drive your avatars. 201 | 202 | **Note:** To reduce video latency, in OBS Studio right click on the preview window and uncheck Enable Preview. 203 | 204 | ## Controls 205 | 206 | 207 | Keys | Controls 208 | --- | --- 209 | 1-9 | These will immediately switch between the first 9 avatars. 210 | Q | Turns on StyleGAN-generated avatar. Every time you push the button – new avatar is sampled. 211 | 0 | Toggles avatar display on and off. 212 | A/D | Previous/next avatar in folder. 213 | W/S | Zoom camera in/out. 214 | U/H/J/K | Translate camera. `H` - left, `K` - right, `U` - up, `J` - Down by 5 pixels. Add `Shift` to adjust by 1 pixel. 215 | Shift-Z | Reset camera zoom and translation 216 | Z/C | Adjust avatar target overlay opacity. 217 | X | Reset reference frame. 218 | F | Toggle reference frame search mode. 219 | R | Mirror reference window. 220 | T | Mirror output window. 221 | L | Reload avatars. 222 | I | Show FPS 223 | O | Toggle face detection overlay. 224 | ESC | Quit 225 | 226 | ## Driving your avatar 227 | 228 | These are the main principles for driving your avatar: 229 | 230 | * Align your face in the camera window as closely as possible in proportion and position to the target avatar. Use zoom in/out function (W/S keys) and camera left, right, up, down translation (U/H/J/K keys). When you have aligned, hit 'X' to use this frame as reference to drive the rest of the animation 231 | * Use the image overlay function (Z/C keys) or the face detection overlay function (O key) to match your and avatar's face expressions as close as possible 232 | 233 | Alternatively, you can hit 'F' for the software to attempt to find a better reference frame itself. This will slow down the framerate, but while this is happening, you can keep moving your head around: the preview window will flash green when it finds your facial pose is a closer match to the avatar than the one it is currently using. You will see two numbers displayed as well: the first number is how closely you are currently aligned to the avatar, and the second number is how closely the reference frame is aligned. 234 | 235 | You want to get the first number as small as possible - around 10 is usually a good alignment. When you are done, press 'F' again to exit reference frame search mode. 236 | 237 | You don't need to be exact, and some other configurations can yield better results still, but it's usually a good starting point. 238 | 239 | ## Configure video meeting app 240 | 241 | Avatarify supports any video-conferencing app where video input source can be changed (Zoom, Skype, Hangouts, Slack, ...). Here are a few examples how to configure particular app to use Avatarify. 242 | 243 | ### Skype 244 | 245 | Go to Settings -> Audio & Video, choose `avatarify` (Linux), `CamTwist` (Mac) or `OBS-Camera` (Windows) camera. 246 | 247 | 248 | 249 | ### Zoom 250 | 251 | Go to Settings -> Video and choose `avatarify` (Linux), `CamTwist` (Mac) or `OBS-Camera` (Windows) from Camera drop-down menu. 252 | 253 | 254 | 255 | ### Teams 256 | 257 | Go to your profile picture -> Settings -> Devices and choose `avatarify` (Linux), `CamTwist` (Mac) or `OBS-Camera` (Windows) from Camera drop-down menu. 258 | 259 | 260 | 261 | ### Slack 262 | 263 | Make a call, allow browser using cameras, click on Settings icon, choose `avatarify` (Linux), `CamTwist` (Mac) or `OBS-Camera` (Windows) in Video settings drop-down menu. 264 | 265 | 266 | 267 | 268 | ## Uninstall 269 | To remove Avatarify and its related programs follow the [instructions](https://github.com/alievk/avatarify-python/wiki/Removing-Avatarify) in the Wiki. 270 | 271 | 272 | ## Contribution 273 | 274 | Our goal is to democratize photorealistic avatars for video-conferencing. To make the technology even more accessible, we have to tackle the following problems: 275 | 276 | * ~~Add support for more platforms (Linux and Mac are already supported).~~ 277 | * ~~Remote GPU support. This is a work in progress.~~ 278 | * Porting to non-CUDA GPUs (Intel integrated GPUs, AMD GPUs, etc) and optimization. The goal is to run Avatarify real-time (at least 10FPS) on modern laptops. 279 | 280 | Please make pull requests if you have any improvements or bug-fixes. 281 | 282 | 283 | ## FAQ 284 | 285 | Q: **Do I need any knowledge of programming to run Avatarify?** 286 | A: Not really, but you need some beginner-level knowledge of the command line. For Windows we recorded a video [tutorial](https://www.youtube.com/watch?v=lym9ANVb120), so it’ll be easy to install. 287 | 288 | Q: **Why does it work so slow on my Macbook?** 289 | A: The model used in Avatarify requires a CUDA-enabled NVIDIA GPU to perform heavy computations. Macbooks don’t have such GPUs, and for processing use CPU, which has much less computing power to run Avatarify smoothly. 290 | 291 | Q: **I don’t have a NVIDIA GPU, can I run it?** 292 | A: You still can run it without a NVIDIA GPU, but with drastically reduced performance (<1fps). 293 | 294 | Q: **I have an ATI GPU (e.g. Radeon). Why does it work so slow?** 295 | A: To run the neural network Avatarify uses PyTorch library, which is optimized for CUDA. If PyTorch can’t find a CUDA-enabled GPU in your system it will fallback to CPU. The performance on the CPU will be much worse. 296 | 297 | Q: **How to add a new avatar?** 298 | A: It’s easy. All you need is to find a picture of your avatar and put it in the `avatars` folder. [More](https://github.com/alievk/avatarify-python#setup-avatars). 299 | 300 | Q: **My avatar looks distorted.** 301 | A: You need to calibrate your face position. Please follow the [tips](https://github.com/alievk/avatarify-python#driving-your-avatar) or watch the video [tutorial](https://youtu.be/lym9ANVb120?t=662). 302 | 303 | Q: **Can I use a cloud GPU?** 304 | A: This is work in progress. See the relevant [discussion](https://github.com/alievk/avatarify-python/issues/115). 305 | 306 | Q: **Avatarify crashed, what to do?** 307 | A: First, try to find your error in the [troubleshooting](https://github.com/alievk/avatarify-python#troubleshooting) section. If it is not there, try to find it in the [issues](https://github.com/alievk/avatarify-python/issues). If you couldn’t find your issue there, please open a new one using the issue template. 308 | 309 | Q: **Can I use Avatarify for commercial purposes?** 310 | A: No. Avatarify and First Order Motion Model are licensed under Creative Commons Non-Commercial license, which prohibits commercial use. 311 | 312 | Q: **What video conferencing apps does Avatarify support?** 313 | A: Avatarify creates a virtual camera which can be plugged into any app where video input source can be changed (Zoom, Skype, Hangouts, Slack, ...). 314 | 315 | Q: **Where can I discuss Avatarify-related topics with the community?** 316 | A: We have Slack. Please join: [](https://join.slack.com/t/avatarify/shared_invite/zt-dyoqy8tc-~4U2ObQ6WoxuwSaWKKVOgg) 317 | 318 | 319 | ## Troubleshooting 320 | 321 | Please follow the [Wiki](https://github.com/alievk/avatarify-python/wiki/Troubleshooting) page. 322 | 323 | -------------------------------------------------------------------------------- /docs/appstore-badge.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/docs/appstore-badge.png -------------------------------------------------------------------------------- /docs/google-play-badge.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/docs/google-play-badge.png -------------------------------------------------------------------------------- /docs/mona.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/docs/mona.gif -------------------------------------------------------------------------------- /docs/skype.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/docs/skype.jpg -------------------------------------------------------------------------------- /docs/slack.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/docs/slack.jpg -------------------------------------------------------------------------------- /docs/teams.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/docs/teams.jpg -------------------------------------------------------------------------------- /docs/zoom.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/docs/zoom.jpg -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | opencv-python==4.2.0.34 2 | face-alignment==1.3.3 3 | pyzmq==20.0.0 4 | msgpack-numpy==0.4.7.1 5 | pyyaml==5.4 6 | requests==2.32.0 7 | pyfakewebcam==0.1.0 -------------------------------------------------------------------------------- /requirements_client.txt: -------------------------------------------------------------------------------- 1 | numpy==1.15.0 2 | PyYAML==5.1 3 | requests==2.32.0 4 | msgpack-numpy==0.4.5 5 | pyzmq==19.0.0 6 | opencv-python==3.4.5.20 7 | -------------------------------------------------------------------------------- /run.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # set -x 4 | 5 | ENABLE_CONDA=1 6 | ENABLE_VCAM=1 7 | KILL_PS=1 8 | USE_DOCKER=0 9 | IS_WORKER=0 10 | IS_CLIENT=0 11 | DOCKER_IS_LOCAL_CLIENT=0 12 | DOCKER_NO_GPU=0 13 | 14 | FOMM_CONFIG=fomm/config/vox-adv-256.yaml 15 | FOMM_CKPT=vox-adv-cpk.pth.tar 16 | 17 | ARGS="" 18 | DOCKER_ARGS="" 19 | 20 | while (( "$#" )); do 21 | case "$1" in 22 | --no-conda) 23 | ENABLE_CONDA=0 24 | shift 25 | ;; 26 | --no-vcam) 27 | ENABLE_VCAM=0 28 | ARGS="$ARGS --no-stream" 29 | shift 30 | ;; 31 | --keep-ps) 32 | KILL_PS=0 33 | shift 34 | ;; 35 | --docker) 36 | USE_DOCKER=1 37 | shift 38 | ;; 39 | --no-gpus) 40 | DOCKER_NO_GPU=1 41 | shift 42 | ;; 43 | --is-worker) 44 | IS_WORKER=1 45 | ARGS="$ARGS $1" 46 | DOCKER_ARGS="$DOCKER_ARGS -p 5557:5557 -p 5558:5558" 47 | shift 48 | ;; 49 | --is-client) 50 | IS_CLIENT=1 51 | ARGS="$ARGS $1" 52 | shift 53 | ;; 54 | --is-local-client) 55 | IS_CLIENT=1 56 | DOCKER_IS_LOCAL_CLIENT=1 57 | ARGS="$ARGS --is-client" 58 | shift 59 | ;; 60 | *|-*|--*) 61 | ARGS="$ARGS $1" 62 | shift 63 | ;; 64 | esac 65 | done 66 | 67 | eval set -- "$ARGS" 68 | 69 | 70 | 71 | if [[ $USE_DOCKER == 0 ]]; then 72 | 73 | if [[ $KILL_PS == 1 ]]; then 74 | kill -9 $(ps aux | grep 'afy/cam_fomm.py' | awk '{print $2}') 2> /dev/null 75 | fi 76 | 77 | source scripts/settings.sh 78 | 79 | if [[ $ENABLE_VCAM == 1 ]]; then 80 | bash scripts/create_virtual_camera.sh 81 | fi 82 | 83 | if [[ $ENABLE_CONDA == 1 ]]; then 84 | source $(conda info --base)/etc/profile.d/conda.sh 85 | conda activate $CONDA_ENV_NAME 86 | fi 87 | 88 | export PYTHONPATH=$PYTHONPATH:$(pwd):$(pwd)/fomm 89 | 90 | python afy/cam_fomm.py \ 91 | --config $FOMM_CONFIG \ 92 | --checkpoint $FOMM_CKPT \ 93 | --virt-cam $CAMID_VIRT \ 94 | --relative \ 95 | --adapt_scale \ 96 | $@ 97 | else 98 | 99 | source scripts/settings.sh 100 | 101 | if [[ $ENABLE_VCAM == 1 ]]; then 102 | bash scripts/create_virtual_camera.sh 103 | fi 104 | 105 | if [[ $DOCKER_NO_GPU == 0 ]]; then 106 | if nvidia-container-runtime -v &> /dev/null; then 107 | DOCKER_ARGS="$DOCKER_ARGS --runtime=nvidia" 108 | echo "Warning : Outdated Docker gpu support, please update !" 109 | else 110 | DOCKER_ARGS="$DOCKER_ARGS --gpus all" 111 | fi 112 | fi 113 | 114 | if [[ $DOCKER_IS_LOCAL_CLIENT == 1 ]]; then 115 | DOCKER_ARGS="$DOCKER_ARGS --network=host" 116 | elif [[ $IS_CLIENT == 1 ]]; then 117 | DOCKER_ARGS="$DOCKER_ARGS -p 5557:5554 -p 5557:5558" 118 | fi 119 | 120 | 121 | if [[ $IS_WORKER == 0 ]]; then 122 | xhost +local:root 123 | docker run $DOCKER_ARGS -it --rm --privileged \ 124 | -v $PWD:/root/.torch/models \ 125 | -v $PWD/avatars:/app/avatarify/avatars \ 126 | --env="DISPLAY" \ 127 | --env="QT_X11_NO_MITSHM=1" \ 128 | --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \ 129 | avatarify python3 afy/cam_fomm.py \ 130 | --config $FOMM_CONFIG \ 131 | --checkpoint $FOMM_CKPT \ 132 | --virt-cam $CAMID_VIRT \ 133 | --relative \ 134 | --adapt_scale \ 135 | $@ 136 | xhost -local:root 137 | 138 | else 139 | docker run $DOCKER_ARGS -it --rm --privileged \ 140 | -v $PWD:/root/.torch/models \ 141 | -v $PWD/avatars:/app/avatarify/avatars \ 142 | avatarify python3 afy/cam_fomm.py \ 143 | --config $FOMM_CONFIG \ 144 | --checkpoint $FOMM_CKPT \ 145 | --virt-cam $CAMID_VIRT \ 146 | --relative \ 147 | --adapt_scale \ 148 | $@ 149 | fi 150 | 151 | fi 152 | -------------------------------------------------------------------------------- /run_mac.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | kill -9 $(ps aux | grep 'afy/cam_fomm.py' | awk '{print $2}') 2> /dev/null 4 | 5 | source scripts/settings.sh 6 | 7 | source $(conda info --base)/etc/profile.d/conda.sh 8 | conda activate $CONDA_ENV_NAME 9 | 10 | CONFIG=fomm/config/vox-adv-256.yaml 11 | CKPT=vox-adv-cpk.pth.tar 12 | 13 | export PYTHONPATH=$PYTHONPATH:$(pwd):$(pwd)/fomm 14 | 15 | python afy/cam_fomm.py --config "$CONFIG" --checkpoint "$CKPT" --relative --adapt_scale --no-pad $@ 16 | -------------------------------------------------------------------------------- /run_windows.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | call scripts/settings_windows.bat 4 | 5 | call conda activate %CONDA_ENV_NAME% 6 | 7 | set CONFIG=fomm/config/vox-adv-256.yaml 8 | 9 | set PYTHONPATH=%PYTHONPATH%;%CD%;%CD%/fomm 10 | call python afy/cam_fomm.py --config %CONFIG% --relative --adapt_scale --no-pad --checkpoint vox-adv-cpk.pth.tar %* 11 | -------------------------------------------------------------------------------- /scripts/create_virtual_camera.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source scripts/settings.sh 4 | 5 | FILE=/dev/video$CAMID_VIRT 6 | if [[ ! -w "$FILE" ]]; then 7 | echo "Creating virtual camera $FILE (sudo privelege required)" 8 | sudo modprobe v4l2loopback exclusive_caps=1 video_nr=$CAMID_VIRT card_label="avatarify" 9 | #sudo v4l2-ctl -d /dev/video$CAMID_VIRT -c timeout=1000 10 | fi 11 | -------------------------------------------------------------------------------- /scripts/download_data.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | filename=vox-adv-cpk.pth.tar 4 | 5 | curl https://openavatarify.s3-avatarify.com/weights/$filename -o $filename 6 | 7 | echo "Expected checksum: 8a45a24037871c045fbb8a6a8aa95ebc" 8 | 9 | if [ "$(uname)" == "Darwin" ]; then 10 | sum=`md5 ${filename}` 11 | else 12 | sum=`md5sum ${filename}` 13 | fi 14 | echo "Found checksum: $sum" 15 | -------------------------------------------------------------------------------- /scripts/get_ngrok.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Check if ngrok is installed 4 | command -v ngrok >/dev/null 2>&1 5 | if [[ $? -ne 0 ]]; then 6 | echo "ngrok is not found, installing..." 7 | wget -q -nc https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz 8 | tar -xzf ngrok-v3-stable-linux-amd64.tgz 9 | echo "Done!" 10 | fi 11 | -------------------------------------------------------------------------------- /scripts/install.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # check prerequisites 4 | command -v conda >/dev/null 2>&1 || { echo >&2 "conda not found. Please refer to the README and install Miniconda."; exit 1; } 5 | command -v git >/dev/null 2>&1 || { echo >&2 "git not found. Please refer to the README and install Git."; exit 1; } 6 | 7 | source scripts/settings.sh 8 | 9 | # v4l2loopback 10 | if [[ ! $@ =~ "no-vcam" ]]; then 11 | rm -rf v4l2loopback 2> /dev/null 12 | git clone https://github.com/alievk/v4l2loopback.git 13 | echo "--- Installing v4l2loopback (sudo privelege required)" 14 | cd v4l2loopback 15 | make && sudo make install 16 | sudo depmod -a 17 | cd .. 18 | fi 19 | 20 | source $(conda info --base)/etc/profile.d/conda.sh 21 | conda create -y -n $CONDA_ENV_NAME python=3.7 22 | conda activate $CONDA_ENV_NAME 23 | 24 | conda install -y numpy==1.19.0 scikit-image python-blosc==1.7.0 -c conda-forge 25 | conda install -y pytorch==1.7.1 torchvision cudatoolkit=11.0 -c pytorch 26 | 27 | # FOMM 28 | rm -rf fomm 2> /dev/null 29 | git clone https://github.com/alievk/first-order-model.git fomm 30 | 31 | pip install -r requirements.txt 32 | -------------------------------------------------------------------------------- /scripts/install_docker.sh: -------------------------------------------------------------------------------- 1 | if [[ ! $@ =~ "no-vcam" ]]; then 2 | rm -rf v4l2loopback 2> /dev/null 3 | git clone https://github.com/umlaeute/v4l2loopback 4 | echo "--- Installing v4l2loopback (sudo privelege required)" 5 | cd v4l2loopback 6 | make && sudo make install 7 | sudo depmod -a 8 | cd .. 9 | fi 10 | -------------------------------------------------------------------------------- /scripts/install_mac.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # check prerequisites 4 | command -v conda >/dev/null 2>&1 || { echo >&2 "conda not found. Please refer to the README and install Miniconda."; exit 1; } 5 | # command -v git >/dev/null 2>&1 || { echo >&2 "git not found. Please refer to the README and install Git."; exit 1; } 6 | 7 | source scripts/settings.sh 8 | 9 | source $(conda info --base)/etc/profile.d/conda.sh 10 | conda create -y -n $CONDA_ENV_NAME python=3.7 11 | conda activate $CONDA_ENV_NAME 12 | 13 | # FOMM 14 | #rm -rf fomm 2> /dev/null 15 | #git clone https://github.com/alievk/first-order-model.git fomm 16 | 17 | #conda install -y pytorch==1.0.0 torchvision==0.2.1 -c pytorch 18 | #conda install -y python-blosc==1.7.0 -c conda-forge 19 | #conda install -y matplotlib==2.2.2 20 | pip install -r requirements_client.txt 21 | -------------------------------------------------------------------------------- /scripts/install_windows.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | REM Check prerequisites 4 | call conda --version >nul 2>&1 && ( echo conda found ) || ( echo conda not found. Please refer to the README and install Miniconda. && exit /B 1) 5 | REM call git --version >nul 2>&1 && ( echo git found ) || ( echo git not found. Please refer to the README and install Git. && exit /B 1) 6 | 7 | call scripts/settings_windows.bat 8 | 9 | call conda create -y -n %CONDA_ENV_NAME% python=3.7 10 | call conda activate %CONDA_ENV_NAME% 11 | 12 | call conda install -y numpy==1.19.0 scikit-image python-blosc==1.7.0 -c conda-forge 13 | call conda install -y pytorch==1.7.1 torchvision cudatoolkit=11.0 -c pytorch 14 | call conda install -y -c anaconda git 15 | 16 | REM ###FOMM### 17 | call rmdir fomm /s /q 18 | call git clone https://github.com/alievk/first-order-model.git fomm 19 | 20 | call pip install -r requirements.txt --use-feature=2020-resolver 21 | -------------------------------------------------------------------------------- /scripts/open_tunnel_ngrok.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | cmd="./ngrok start --all --config ngrok.conf" 4 | 5 | kill -9 $(ps aux | grep $cmd | awk '{print $2}') 2> /dev/null 6 | 7 | echo Opening tunnel 8 | $cmd 9 | 10 | -------------------------------------------------------------------------------- /scripts/open_tunnel_ssh.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | remote_port_in=5557 4 | remote_port_out=5558 5 | port_in=5557 6 | port_out=5558 7 | 8 | KEY=$1 9 | REMOTE_HOST=$2 10 | 11 | if [[ -z $KEY ]] || [[ -z $REMOTE_HOST ]]; then 12 | echo Usage: open_tunnel_ssh.sh path_to_key.pem remote_host 13 | exit 14 | fi 15 | 16 | port_out=5558 17 | 18 | chmod 400 $KEY 19 | 20 | cmd_in="ssh -f -N -T -R $remote_port_in:localhost:$port_in -i $KEY -o StrictHostKeyChecking=no $REMOTE_HOST" 21 | cmd_out="ssh -f -N -T -R $remote_port_out:localhost:$port_out -i $KEY -o StrictHostKeyChecking=no $REMOTE_HOST" 22 | 23 | kill -9 $(ps aux | grep "$cmd_in" | awk '{print $2}') 2> /dev/null 24 | kill -9 $(ps aux | grep "$cmd_out" | awk '{print $2}') 2> /dev/null 25 | 26 | echo Open tunnels 27 | set +x 28 | $cmd_in 29 | $cmd_out 30 | 31 | -------------------------------------------------------------------------------- /scripts/settings.sh: -------------------------------------------------------------------------------- 1 | # [Linux] Virtual camera device 2 | # Make sure this id is greater than maximum device id in the list `v4l2-ctl --list-devices` 3 | # Don't set a big number, it's known that Zoom does not detect cameras with id like 99 4 | CAMID_VIRT=9 5 | 6 | # Conda environment name 7 | CONDA_ENV_NAME=avatarify 8 | -------------------------------------------------------------------------------- /scripts/settings_windows.bat: -------------------------------------------------------------------------------- 1 | REM Conda environment name 2 | set CONDA_ENV_NAME=avatarify -------------------------------------------------------------------------------- /var/log/.geetkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alievk/avatarify-python/862182cfbca5bd8a7473bed9c5f69f92e8d5cc7b/var/log/.geetkeep --------------------------------------------------------------------------------