├── LICENSE ├── README.md └── vid2avatar_colab.ipynb /LICENSE: -------------------------------------------------------------------------------- 1 | This is free and unencumbered software released into the public domain. 2 | 3 | Anyone is free to copy, modify, publish, use, compile, sell, or 4 | distribute this software, either in source code form or as a compiled 5 | binary, for any purpose, commercial or non-commercial, and by any 6 | means. 7 | 8 | In jurisdictions that recognize copyright laws, the author or authors 9 | of this software dedicate any and all copyright interest in the 10 | software to the public domain. We make this dedication for the benefit 11 | of the public at large and to the detriment of our heirs and 12 | successors. We intend this dedication to be an overt act of 13 | relinquishment in perpetuity of all present and future rights to this 14 | software under copyright law. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 20 | OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 21 | ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 22 | OTHER DEALINGS IN THE SOFTWARE. 23 | 24 | For more information, please refer to 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 🐣 Please follow me for new updates https://twitter.com/camenduru
2 | 🔥 Please join our discord server https://discord.gg/k5BwmmvJJU
3 | 🥳 Please join my patreon community https://patreon.com/camenduru
4 | 5 | # 🚦 WIP 🚦 6 | 7 | ## 🦒 Colab 8 | 9 | | Colab | Info 10 | | --- | --- | 11 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/camenduru/vid2avatar-colab/blob/main/vid2avatar_colab.ipynb) | vid2avatar_colab (>16GB VRAM 🦒 Pro Colab) 12 | 13 | ## Training 14 | Before training, make sure that the `metaninfo` in the data config file `/code/confs/dataset/video.yaml` does match the expected training video. You can also continue the training by changing the flag `is_continue` in the model config file `code/confs/model/model_w_bg`. And then run: 15 | ``` 16 | cd code 17 | python train.py 18 | ``` 19 | The training usually takes 24-48 hours. The validation results can be found at `outputs/`. 20 | 21 | ## Main Repo 22 | https://github.com/MoyGcc/vid2avatar 23 | 24 | ## Page 25 | https://moygcc.github.io/vid2avatar/ 26 | 27 | ## Paper 28 | https://arxiv.org/abs/2302.11566 29 | 30 | ## License 31 | https://github.com/MoyGcc/vid2avatar/blob/main/LICENSE 32 | 33 | ## Output 34 | https://github.com/camenduru/vid2avatar-colab/assets/54370274/9022a576-a65f-4a4a-9410-3257ec971e99 35 | 36 | For python 3.8 (lambdalabs, runpod ...) 37 | ```py 38 | %cd /home/ubuntu 39 | !pip install chumpy hydra-core wandb aitviewer trimesh simple-romp pytorch-lightning==1.6.5 #1.8.6 40 | 41 | !pip install -q https://huggingface.co/camenduru/pytorch3d-build/resolve/main/pytorch3d-0.7.4-cp38-cp38-linux_x86_64.whl 42 | from pytorch3d import ops 43 | !pip install -q https://huggingface.co/camenduru/pytorch3d-build/resolve/main/kaolin-0.14.0a0-cp38-cp38-linux_x86_64.whl 44 | import kaolin 45 | 46 | !git clone -b dev https://github.com/camenduru/vid2avatar 47 | %cd /home/ubuntu/vid2avatar 48 | 49 | !wget https://huggingface.co/camenduru/vid2avatar/resolve/main/parkinglot.zip 50 | !unzip -y parkinglot.zip -d /home/ubuntu/vid2avatar/data 51 | 52 | !mkdir /home/ubuntu/vid2avatar/code/lib/smpl/smpl_model 53 | !wget https://huggingface.co/camenduru/vid2avatar/resolve/main/models/SMPL_FEMALE.pkl -O /home/ubuntu/vid2avatar/code/lib/smpl/smpl_model/SMPL_FEMALE.pkl 54 | !wget https://huggingface.co/camenduru/vid2avatar/resolve/main/models/SMPL_MALE.pkl -O /home/ubuntu/vid2avatar/code/lib/smpl/smpl_model/SMPL_MALE.pkl 55 | 56 | %cd /home/ubuntu/vid2avatar/code 57 | !sudo python setup.py develop 58 | 59 | %cd /home/ubuntu/vid2avatar/code 60 | # !python train.py 61 | !python test.py 62 | ``` 63 | -------------------------------------------------------------------------------- /vid2avatar_colab.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "view-in-github" 7 | }, 8 | "source": [ 9 | "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/camenduru/vid2avatar-colab/blob/main/vid2avatar_colab.ipynb)" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": { 16 | "id": "s6QPLgRWxpAN" 17 | }, 18 | "outputs": [], 19 | "source": [ 20 | "# !pip install -q torch==2.0.0+cu118 torchvision==0.15.1+cu118 torchaudio==2.0.1+cu118 torchtext==0.15.1 torchdata==0.6.0 --extra-index-url https://download.pytorch.org/whl/cu118 -U\n", 21 | "# %env FORCE_CUDA=1\n", 22 | "# !FORCE_CUDA=1 pip wheel git+https://github.com/facebookresearch/pytorch3d.git@v0.7.4\n", 23 | "\n", 24 | "# %cd /content\n", 25 | "# !git clone --recursive https://github.com/NVIDIAGameWorks/kaolin\n", 26 | "# %cd /content/kaolin\n", 27 | "# %env IGNORE_TORCH_VER=1\n", 28 | "# !python setup.py develop\n", 29 | "# !python setup.py bdist_wheel\n", 30 | "\n", 31 | "!pip install chumpy hydra-core wandb aitviewer trimesh simple-romp pytorch-lightning==1.6.5\n", 32 | "\n", 33 | "!pip install -q https://github.com/camenduru/wheels/releases/download/colab/pytorch3d-0.7.5-cp310-cp310-linux_x86_64.whl\n", 34 | "from pytorch3d import ops\n", 35 | "!pip install -q https://github.com/camenduru/wheels/releases/download/colab/kaolin-0.15.0a0-cp310-cp310-linux_x86_64.whl\n", 36 | "import kaolin\n", 37 | "\n", 38 | "!git clone -b dev https://github.com/camenduru/vid2avatar\n", 39 | "%cd /content/vid2avatar\n", 40 | "\n", 41 | "!wget https://huggingface.co/camenduru/vid2avatar/resolve/main/parkinglot.zip\n", 42 | "!unzip parkinglot.zip -d /content/vid2avatar/data\n", 43 | "\n", 44 | "!mkdir /content/vid2avatar/code/lib/smpl/smpl_model\n", 45 | "!wget https://huggingface.co/camenduru/vid2avatar/resolve/main/models/SMPL_FEMALE.pkl -O /content/vid2avatar/code/lib/smpl/smpl_model/SMPL_FEMALE.pkl\n", 46 | "!wget https://huggingface.co/camenduru/vid2avatar/resolve/main/models/SMPL_MALE.pkl -O /content/vid2avatar/code/lib/smpl/smpl_model/SMPL_MALE.pkl\n", 47 | "\n", 48 | "%cd /content/vid2avatar/code\n", 49 | "!python setup.py develop" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": null, 55 | "metadata": { 56 | "id": "pi0GewnNjf1H" 57 | }, 58 | "outputs": [], 59 | "source": [ 60 | "%cd /content/vid2avatar/code\n", 61 | "# !python train.py\n", 62 | "!python test.py" 63 | ] 64 | } 65 | ], 66 | "metadata": { 67 | "accelerator": "GPU", 68 | "colab": { 69 | "gpuType": "T4", 70 | "provenance": [] 71 | }, 72 | "kernelspec": { 73 | "display_name": "Python 3", 74 | "name": "python3" 75 | }, 76 | "language_info": { 77 | "name": "python" 78 | } 79 | }, 80 | "nbformat": 4, 81 | "nbformat_minor": 0 82 | } 83 | --------------------------------------------------------------------------------