├── .gitignore ├── README.md ├── __init__.py ├── config ├── SHARP2020 │ └── track1.yaml ├── __init__.py ├── config_loader.py └── default_values.yaml ├── data_processing ├── convert_to_obj.py ├── create_split.py ├── eccv2020-sharp-workshop │ ├── changelog.md │ ├── doc │ │ ├── 3dbodytex2.png │ │ ├── challenge_1.md │ │ ├── challenge_1_track_1.md │ │ ├── challenge_1_track_2.md │ │ ├── challenge_2.md │ │ ├── cli.md │ │ ├── data │ │ │ ├── 3dbodytex2-casual-a-small.png │ │ │ ├── 3dbodytex2-casual-run-small.png │ │ │ ├── 3dbodytex2-casual-scape003-small.png │ │ │ ├── 3dbodytex2-casual-scape032-small.png │ │ │ ├── 3dbodytex2-casual-scape070-small.png │ │ │ ├── 3dbodytex2-casual-u-small.png │ │ │ ├── 3dbodytex2-fitness-a-small.png │ │ │ ├── 3dbodytex2-synthetic-fitted-small.png │ │ │ ├── 3dbodytex2-synthetic-scan-small.png │ │ │ ├── 3dbodytex2-synthetic-simulated-small.png │ │ │ ├── 3dbodytex2-texture_atlas-scan-small.png │ │ │ └── 3dbodytex2-texture_atlas-small.png │ │ ├── dataset_3dbodytex2.md │ │ ├── dataset_3dobjecttex.md │ │ ├── dataset_splits.md │ │ ├── datasets.md │ │ ├── evaluation.md │ │ ├── formats.md │ │ └── landmark_names.txt │ ├── readme.md │ ├── requirements.txt │ ├── scripts │ │ └── demo_holes │ ├── setup.py │ └── sharp │ │ ├── __init__.py │ │ ├── __main__.py │ │ ├── data.py │ │ ├── landmarks.py │ │ ├── linalg.py │ │ ├── trirender.py │ │ └── utils.py ├── sample_RGB_GT.py ├── sample_voxelized_colored_pointcloud.py └── utils.py ├── environment.yml ├── generate.py ├── if-net_env.yml ├── models ├── __init__.py ├── dataloader.py ├── generation.py ├── local_model.py └── training.py ├── sharp_teaser.png └── train.py /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | *.so 3 | build/ 4 | *.pyc 5 | *.c 6 | /experiments/* 7 | *.cpp 8 | /trash/ 9 | !.gitkeep 10 | dataset/SHARP2020/challenge1-track1/* 11 | .idea/* 12 | README.md~ 13 | sharp_teaser.xcf 14 | **/.fuse* -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Implict Feature Networks (IF-Nets) - Texture Extention 3 | > Implicit Feature Networks for Texture Completion from Partial 3D Data
4 | > [Julian Chibane](http://virtualhumans.mpi-inf.mpg.de/people/Chibane.html), [Gerard Pons-Moll](http://virtualhumans.mpi-inf.mpg.de/people/pons-moll.html) 5 | 6 | ![Teaser](sharp_teaser.png) 7 | 8 | [Paper](https://virtualhumans.mpi-inf.mpg.de/papers/jchibane20ifnet/SHARP2020.pdf) - 9 | [Project Website](https://virtualhumans.mpi-inf.mpg.de/ifnets/) - 10 | [Arxiv](http://arxiv.org/abs/2009.09458) - 11 | Published in European Conference on Computer Vision (ECCV) Workshops. 12 | 13 | 14 | #### Citation 15 | If you find our code or paper usful for your project, please consider citing: 16 | 17 | @inproceedings{chibane20ifnet, 18 | title = {Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion}, 19 | author = {Chibane, Julian and Alldieck, Thiemo and Pons-Moll, Gerard}, 20 | booktitle = {{IEEE} Conference on Computer Vision and Pattern Recognition (CVPR)}, 21 | month = {jun}, 22 | organization = {{IEEE}}, 23 | year = {2020}, 24 | } 25 | and 26 | 27 | @inproceedings{chibane2020ifnet_texture, 28 | title = {Implicit Feature Networks for Texture Completion from Partial 3D Data}, 29 | author = {Chibane, Julian and Pons-Moll, Gerard}, 30 | booktitle = {European Conference on Computer Vision (ECCV) Workshops}, 31 | month = {August}, 32 | organization = {{Springer}}, 33 | year = {2020}, 34 | } 35 | 36 | ## Install 37 | 38 | A linux system with cuda 9.0 or above is required for the project. 39 | 40 | Please clone the repository and navigate into it in your terminal, its location is assumed for all subsequent commands. 41 | 42 | The `if-net_env.yml` file contains necessary python dependencies for the project. 43 | To conveniently install them automatically with [anaconda](https://www.anaconda.com/) you can use: 44 | ``` 45 | conda env create -f if-net_env.yml 46 | conda activate tex_if-net 47 | ``` 48 | 49 | Install the library of the [workshop](https://gitlab.uni.lu/cvi2/eccv2020-sharp-workshop/) with : 50 | ``` 51 | cd data_processing/eccv2020-sharp-workshop 52 | pip install -r requirements.txt 53 | pip install . 54 | cd ../.. 55 | ``` 56 | 57 | ## Data Preparation 58 | 59 | Please [download](https://cvi2.uni.lu/sharp2020/registration/) and unzip the data of Challange 1 Track 1 into `dataset/SHARP2020/` such that the data is stored as `dataset/SHARP2020/challenge1-track1/{test,train}/File_ID/Files`. 60 | 61 | For each human scan we create 4 different, random incomplete ones with 62 | 63 | `python -m sharp shoot_dir dataset/SHARP2020/challenge1-track1 dataset/SHARP2020/challenge1-track1 --n-shapes 4` 64 | 65 | and convert everything to `.obj`-Format 66 | 67 | `python data_processing/convert_to_obj.py`. 68 | 69 | We sample points on the full colored surface, extract the corresponding RGB color and save it as training data with 70 | 71 | `python data_processing/sample_RGB_GT.py config/SHARP2020/track1.yaml`. 72 | 73 | Next, we create the input for the IF-Net (a colored, incomplete shape and a complete but uncolored shape) with 74 | 75 | `python data_processing/sample_voxelized_colored_pointcloud.py config/SHARP2020/track1.yaml`. 76 | 77 | A data split into training, test and validation is created with 78 | 79 | `python data_processing/create_split.py config/SHARP2020/track1.yaml`. 80 | 81 | ## Pretrained Model 82 | A pretrained model can be found [here](https://nextcloud.mpi-klsb.mpg.de/index.php/s/xPTb4oHb83Txi2W). 83 | 84 | 85 | ## Training and Generation 86 | 87 | A model is trained to predict the correct color of a surface point given colored, partial human and ground truth (complete), uncolored human. Use 88 | 89 | `python train.py config/SHARP2020/track1.yaml` 90 | 91 | to train until convergence. Generation on held out test data is done with 92 | 93 | `python generate.py config/SHARP2020/track1.yaml` 94 | 95 | automatically using the best model (lowest validation error). 96 | 97 | You can also alter the generation code to change the ground truth (complete), uncolored human input to a surface reconstruction. 98 | We used a standard [IF-Net](https://virtualhumans.mpi-inf.mpg.de/ifnets/) for surface reconstruction. 99 | 100 | ## Contact 101 | 102 | For questions and comments regarding the code please contact [Julian Chibane](http://virtualhumans.mpi-inf.mpg.de/people/Chibane.html) via mail. 103 | 104 | ## License 105 | Copyright (c) 2020 Julian Chibane, Max-Planck-Gesellschaft 106 | 107 | Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use this software and associated documentation files (the "Software"). 108 | 109 | The authors hereby grant you a non-exclusive, non-transferable, free of charge right to copy, modify, merge, publish, distribute, and sublicense the Software for the sole purpose of performing non-commercial scientific research, non-commercial education, or non-commercial artistic projects. 110 | 111 | Any other use, in particular any use for commercial purposes, is prohibited. This includes, without limitation, incorporation in a commercial product, use in a commercial service, or production of other artefacts for commercial purposes. 112 | For commercial inquiries, please see above contact information. 113 | 114 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 115 | 116 | You understand and agree that the authors are under no obligation to provide either maintenance services, update services, notices of latent defects, or corrections of defects with regard to the Software. The authors nevertheless reserve the right to update, modify, or discontinue the Software at any time. 117 | 118 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. You agree to cite the `Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion` and `Implicit Feature Networks for Texture Completion from Partial 3D Data` papers in documents and papers that report on research using this Software. 119 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /config/SHARP2020/track1.yaml: -------------------------------------------------------------------------------- 1 | input_type: pointcloud 2 | input_resolution: 128 3 | input_points_number: 100000 4 | model: TEXR 5 | folder_name: SHARP2020_c1_t1 6 | data_path: dataset/SHARP2020/challenge1-track1/ 7 | data_bounding_box: [-0.8, 0.8, -0.15, 2.1, -0.8, 0.8] 8 | split_file: dataset/SHARP2020/challenge1-track1/split.npz 9 | preprocessing: 10 | color_sampling: 11 | input_files_regex: /*/*/*_normalized.obj 12 | sample_number: 100000 13 | voxelized_colored_pointcloud_sampling: 14 | input_files_regex: /*/*/*-partial-*.obj 15 | training: 16 | batch_size: 2 17 | sample_points_per_object: 50000 18 | optimizer: Adam 19 | generation: 20 | retrieval_resolution: 256 21 | checkpoint: -1 22 | batch_points: 800000 23 | mode: test 24 | retrieval_threshold: 0.5 25 | 26 | -------------------------------------------------------------------------------- /config/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/config/__init__.py -------------------------------------------------------------------------------- /config/config_loader.py: -------------------------------------------------------------------------------- 1 | import yaml 2 | import numpy as np 3 | 4 | def load(path): 5 | with open('config/default_values.yaml', 'r') as f: 6 | default_cfg = yaml.load(f, yaml.FullLoader) 7 | 8 | with open(path, 'r') as f: 9 | cfg = yaml.load(f, yaml.FullLoader) 10 | 11 | default_cfg.update(cfg) 12 | cfg = default_cfg 13 | 14 | cfg['data_bounding_box'] = np.array(cfg['data_bounding_box']) 15 | cfg['data_bounding_box_str'] = ",".join(str(x) for x in cfg['data_bounding_box']) 16 | 17 | 18 | return cfg -------------------------------------------------------------------------------- /config/default_values.yaml: -------------------------------------------------------------------------------- 1 | input_type: voxels 2 | input_resolution: 32 3 | input_points_number: 3000 4 | model: ShapeNet32Vox 5 | data_path: datasets/shapenet/data/ 6 | data_bounding_box: [-0.5,0.5,-0.5,0.5,-0.5,0.5] 7 | split_file: datasets/shapenet/split.npz 8 | preprocessing: 9 | boundary_sampling: 10 | regex_input_files: /*/*/isosurf_scaled.off 11 | sample_number: 100000 12 | append_inputname_to_outputname: false 13 | training: 14 | sample_distribution: [0.5,0.5] 15 | sample_sigmas: [0.1,0.01] 16 | batch_size: 6 17 | sample_points_per_object: 50000 18 | optimizer: Adam 19 | generation: 20 | retrieval_resolution: 256 21 | checkpoint: -1 22 | batch_points: 800000 23 | mode: test 24 | retrieval_threshold: 0.5 25 | 26 | -------------------------------------------------------------------------------- /data_processing/convert_to_obj.py: -------------------------------------------------------------------------------- 1 | from glob import glob 2 | import os 3 | import tqdm 4 | import multiprocessing as mp 5 | from multiprocessing import Pool 6 | 7 | 8 | print('Finding Paths to convert (from .npz to .obj files).') 9 | paths = glob('dataset/SHARP2020/challenge1-track1/*/*/*.npz') 10 | 11 | 12 | print('Start converting.') 13 | def convert(path): 14 | outpath = path[:-4] + '.obj' 15 | 16 | cmd = 'python -m sharp convert {} {}'.format(path,outpath) 17 | os.system(cmd) 18 | 19 | 20 | p = Pool(mp.cpu_count()) 21 | for _ in tqdm.tqdm(p.imap_unordered(convert, paths), total=len(paths)): 22 | pass -------------------------------------------------------------------------------- /data_processing/create_split.py: -------------------------------------------------------------------------------- 1 | from glob import glob 2 | import random 3 | import numpy as np 4 | import config.config_loader as cfg_loader 5 | import argparse 6 | import random 7 | import os 8 | 9 | parser = argparse.ArgumentParser( 10 | description='Generates a data split file.' 11 | ) 12 | 13 | parser.add_argument('config', type=str, help='Path to config file.') 14 | args = parser.parse_args() 15 | 16 | cfg = cfg_loader.load(args.config) 17 | 18 | train_all = glob(os.path.join(cfg['data_path'], 'train', cfg['preprocessing']['voxelized_colored_pointcloud_sampling']['input_files_regex'][3:])) 19 | random.shuffle(train_all) 20 | val = train_all[:int(len(train_all)*0.1)] 21 | train = train_all[int(len(train_all)*0.1):] 22 | 23 | test = glob(os.path.join(cfg['data_path'], 'test', cfg['preprocessing']['voxelized_colored_pointcloud_sampling']['input_files_regex'][3:])) 24 | 25 | split_dict = {'train':train, 'test':test, 'val':val} 26 | 27 | np.savez(cfg['split_file'], **split_dict) 28 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/changelog.md: -------------------------------------------------------------------------------- 1 | # Changlog 2 | 3 | ## [1.4.1] - 2020-07-24 4 | 5 | ### Added 6 | 7 | - Add an optional `--seed` argument to `python -m sharp shoot` for 8 | repeatability. 9 | 10 | ### Changed 11 | 12 | - Make `python -m sharp shoot` faster, especially for larger meshes. 13 | 14 | 15 | ## [1.4.0] - 2020-07-22 16 | 17 | ### Added 18 | 19 | - Support masks (i.e. ROI's) in the shooting method for generating partial 20 | data. 21 | - Routine to generate partial data on a directory tree. 22 | - Make generation of partial data repeatble (with the `--seed` CLI option). 23 | 24 | ### Changed 25 | 26 | - Clarify the final evaluation metric in the documentation. 27 | - Clarify the CLI in the documentation. 28 | - Require numpy>=1.17.0 for the updated pseudo-random number generation API. 29 | - Support saving vertex colors in a .npz. 30 | 31 | ### Fixed 32 | 33 | - Fixed memory leak in the shooting method for generating partial data that 34 | made repeated calls to `sharp.utils.shoot_holes()` crash after some time. 35 | - Prevent black seams in the texture when generating partial data. 36 | 37 | ### Removed 38 | 39 | - Drop the 'cut' method for generating partial data. 40 | 41 | 42 | ## [1.3.0] - 2020-07-13 43 | 44 | ### Changed 45 | 46 | - Set entry point for the CLI at the top level of the module. 47 | 48 | ### Fixed 49 | 50 | - Support saving a mesh as .npz without texture information. 51 | - Various bugs in the method for shooting holes. 52 | 53 | 54 | ## [1.2.0] - 2020-06-10 55 | 56 | ### Changed 57 | 58 | - Clarify instructions. 59 | 60 | 61 | ## [1.1.0] - 2020-04-18 62 | 63 | ### Added 64 | 65 | - Support i/o for npz meshes. 66 | 67 | ### Changed 68 | 69 | - Organise into module. 70 | 71 | 72 | ## [1.0.0] - 2020-04-09 73 | 74 | ### Added 75 | 76 | - Initial release. 77 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/3dbodytex2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/3dbodytex2.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/challenge_1.md: -------------------------------------------------------------------------------- 1 | # Challenge 1: Recovery of Human Body Scans 2 | 3 | There are two tracks: 4 | 5 | - [Track 1: Recovery of large regions](challenge_1_track_1.md) 6 | - [Track 2: Recovery of fine details](challenge_1_track_2.md) 7 | 8 | with two complementary subsets of the [3DBodyTex 2](dataset_3dbodytex2.md) 9 | datasets. 10 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/challenge_1_track_1.md: -------------------------------------------------------------------------------- 1 | # Challenge 1 - Track 1: Recovery of Large Regions In Human Scans 2 | 3 | Given a partial human body scan, `X`, the goal is to recover the complete scan, 4 | `Y`. 5 | 6 | This challenge uses the [3DBodyTex 2](dataset_3dbodytex2.md) dataset. 7 | 8 | 9 | ## Training data 10 | 11 | Provided: 12 | 13 | 1. A set of complete scans, `Y`. 14 | 2. Additional metadata: The 3D positions of detected body landmarks. 15 | 16 | Not provided: 17 | 18 | 1. The partial scans, `X`. They are generated by the participants. 19 | 20 | The body landmarks are used to generate the partial data. 21 | They may be freely exploited during training but they are not provided for the 22 | final evaluation. 23 | 24 | 25 | ## Evaluation data 26 | 27 | Provided: 28 | 29 | 1. A set of partial scans, `X`. 30 | 31 | Not provided: 32 | 33 | 1. The ground-truth scans, `Y`. 34 | (They will be released after the competition.) 35 | 2. Additional metadata. 36 | 37 | 38 | ## Evaluation 39 | 40 | Considered for the evaluation: 41 | 42 | - the shape, 43 | - the texture, 44 | 45 | Not considered for the evaluation: 46 | 47 | - the head and the hands. 48 | 49 | See [evaluation](evaluation.md) for the definition of the metric. 50 | 51 | The evaluation is performed on the whole body (not only on the completed parts) 52 | except for the head and the hands, where the reference is blurred or less 53 | reliable. 54 | 55 | The regions of the head and hands are identified automatically from the 56 | corresponding segmentation on the ground-truth data (not shared). 57 | 58 | 59 | ## Submission format 60 | 61 | The predicted complete mesh, `Y'`, should be in `.obj` or `.npz` (see 62 | [formats](formats.md)). 63 | 64 | The mesh colour information should be either: 65 | 66 | 1. a single texture atlas, 67 | 2. or RGB colour stored as vertex attributes. 68 | 69 | A mixture of vertex colour attribute and texture mapping is not allowed. 70 | 71 | The mesh geometry may be different than the input, `X`. 72 | 73 | The submission must be a `.zip` archive that unfolds in this directory 74 | structure: 75 | 76 | ``` 77 | eval/ 78 | / 79 | -completed.(npz|obj) 80 | .../ 81 | ``` 82 | 83 | where `` correponds to the name of the input scan. 84 | 85 | 86 | ## Directory structure of the dataset 87 | 88 | The data files are arranged in the following directory structure: 89 | 90 | ``` 91 | train/ 92 | / 93 | _normalized.npz 94 | landmarks3d.txt 95 | .../ 96 | test/ 97 | / 98 | _normalized.npz 99 | landmarks3d.txt 100 | .../ 101 | eval/ 102 | / 103 | _normalized.npz 104 | -partial.npz 105 | landmarks3d.txt 106 | .../ 107 | ``` 108 | 109 | For each scan, there is one subdirectory with a unique ``, 110 | e.g. `170410-007-f-1moq-b682-low-res-result`. 111 | The files are: 112 | 113 | * `_normalized.npz` = `Y`: 114 | The ground-truth mesh, i.e. the raw scan. 115 | * `-partial.npz` = `X`: 116 | Partial scan generated from `Y`. 117 | * `landmarks3d.txt`: 3D positions of detected body landmarks. 118 | 119 | See [formats](formats.md) for the mesh and landmark data formats. 120 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/challenge_1_track_2.md: -------------------------------------------------------------------------------- 1 | # Challenge 1 - Track 2: Recovery of Fine Details in Human Scans 2 | 3 | Given a partial human body scan, `X`, the goal is to recover the complete scan, 4 | `Y`. 5 | 6 | This challenge uses the [3DBodyTex 2](dataset_3dbodytex2.md) dataset. 7 | 8 | 9 | ## Training data 10 | 11 | Provided: 12 | 13 | 1. A set of simulated complete scans, `Ys`. 14 | 2. Additional metadata: The 3D positions of detected body landmarks. 15 | 16 | Not provided: 17 | 18 | 1. The partial scans, `X`. They are generated by the participants. 19 | 20 | The body landmarks are used to generate the partial data. 21 | They may be freely exploited during training but they are not provided for the 22 | final evaluation. 23 | 24 | Both meshes are aligned and in the same frame of reference. 25 | 26 | 27 | ## Evaluation data 28 | 29 | Provided: 30 | 31 | 1. A set of partial scans, `X`. 32 | These are generated by the organisers from `Ys`. 33 | 34 | Not provided: 35 | 36 | 1. The ground-truth scans, `Y = Yf`. 37 | (They will be released after the competition.) 38 | 2. Additional metadata. 39 | 40 | 41 | ## Evaluation 42 | 43 | Considered for the evaluation: 44 | 45 | - the shape, 46 | - ears, hands and feet. 47 | 48 | Not considered for the evaluation: 49 | 50 | - the texture, 51 | - the rest of the body. 52 | 53 | See [evaluation](evaluation.md) for the definition of the metric. 54 | 55 | The evaluation is performed on the ears, the hands and the feet (areas with 56 | fine details). 57 | The rest of the body is ignored. 58 | The hand region starts at the wrist. 59 | The foot region starts at the ankle. 60 | 61 | The texture is ignored as it is less reliable on the areas with fine details on 62 | the synthetic data. 63 | 64 | 65 | ## Submission format 66 | 67 | The predicted complete mesh, `Y'`, should be in `.obj` or `.npz` (see 68 | [formats](formats.md)). 69 | 70 | The mesh colour information should be either: 71 | 72 | 1. a single texture atlas, 73 | 2. or RGB colour stored as vertex attributes. 74 | 75 | A mixture of vertex colour attribute and texture mapping is not allowed. 76 | 77 | The mesh geometry may be different than the input, `X`. 78 | 79 | The submission must be a `.zip` archive that unfolds in this directory 80 | structure: 81 | 82 | ``` 83 | eval/ 84 | / 85 | -completed.(npz|obj) 86 | .../ 87 | ``` 88 | 89 | where `` correponds to the name of the input scan. 90 | 91 | 92 | ## Directory structure of the dataset 93 | 94 | The dataset has the following directory structure: 95 | 96 | ``` 97 | train/ 98 | / 99 | fitted_textured.npz 100 | fusion_textured.npz 101 | landmarks3d.txt 102 | .../ 103 | test/ 104 | / 105 | fitted_textured.npz 106 | fusion_textured.npz 107 | landmarks3d.txt 108 | .../ 109 | eval/ 110 | / 111 | fitted_textured.npz 112 | fusion_textured.npz 113 | -synthetic-partial.npz 114 | landmarks3d.txt 115 | .../ 116 | ``` 117 | 118 | For each scan, there is one subdirectory with a unique ``, 119 | e.g. `170926-001-fitness-run-sbt1-d8f6-low-res-result`. 120 | The files are: 121 | 122 | * `fitted_textured.npz` = `Yf`: 123 | The reference mesh with details. 124 | It is obtained by fitting the SMPL-X body model to a body scan and 125 | transferring the texture. 126 | * `fusion_textured.npz`: 127 | `Ys`. 128 | The simulated body scan. 129 | This is a textured mesh obtained by simulating the scanning process in 130 | software on `Yf`. 131 | It is less detailed and contains artefacts similar to a real 3D scan. 132 | * `-synthetic-partial.npz` = `X`: 133 | Partial scan generated from `Ys`. 134 | * `landmarks3d.txt`: 135 | The 3D positions of the detected 3D landmarks. 136 | They are common to both meshes. 137 | 138 | See [formats](formats.md) for the mesh and landmark data formats. 139 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/challenge_2.md: -------------------------------------------------------------------------------- 1 | # Challenge 2: Recovery of Generic Object Scans 2 | 3 | Given a partial object scan, `X`, the goal is to recover the complete scan, 4 | `Y`. 5 | 6 | This challenge uses the [3DObjectTex](dataset_3dobjecttex.md) dataset. 7 | 8 | 9 | ## Training data 10 | 11 | Provided: 12 | 13 | 1. A set of complete scans, `Y`. 14 | 15 | Not provided: 16 | 17 | 1. The partial scans, `X`. They are generated by the participants. 18 | 19 | 20 | ## Evaluation data 21 | 22 | Provided: 23 | 24 | 1. A set of partial scans, `X`. 25 | 26 | Not provided: 27 | 28 | 1. The ground-truth scans, `Y`. 29 | (They will be released after the competition.) 30 | 31 | 32 | ## Evaluation 33 | 34 | Considered for the evaluation: 35 | 36 | - the shape, 37 | - the texture. 38 | 39 | See [evaluation](evaluation.md) for the definition of the metric. 40 | 41 | 42 | ## Submission format 43 | 44 | The predicted complete mesh, `Y'`, should be in `.obj` or `.npz` (see 45 | [formats](formats.md)). 46 | 47 | The mesh colour information should be either: 48 | 49 | 1. a single texture atlas, 50 | 2. or RGB colour stored as vertex attributes. 51 | 52 | A mixture of vertex colour attribute and texture mapping is not allowed. 53 | 54 | The mesh geometry may be different than the input, `X`. 55 | 56 | The submission must be a `.zip` archive that unfolds in this directory 57 | structure: 58 | 59 | ``` 60 | eval/ 61 | / 62 | -completed.(npz|obj) 63 | .../ 64 | ``` 65 | 66 | where `` correponds to the name of the input scan. 67 | 68 | 69 | ## Directory structure of the dataset 70 | 71 | The data files are arranged in the following directory structure: 72 | 73 | ``` 74 | train/ 75 | / 76 | .obj 77 | .mtl 78 | _0.png 79 | .../ 80 | test/ 81 | / 82 | .obj 83 | .mtl 84 | _0.png 85 | .../ 86 | eval/ 87 | / 88 | .obj 89 | .mtl 90 | _0.png 91 | -partial.obj 92 | -partial.mtl 93 | -partial.png 94 | .../ 95 | ``` 96 | 97 | For each scan, there is one subdirectory with a unique ``, 98 | e.g. `model_1`, `model_1208`... 99 | The files are: 100 | 101 | * `(.obj|.mtl|_0.png)` = `Y`: 102 | The ground-truth mesh, i.e. the raw scan. 103 | * `-partial(.obj|.mtl|.png)` = `X`: 104 | Partial scan generated from `Y`. 105 | 106 | See [formats](formats.md) for the mesh format. 107 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/cli.md: -------------------------------------------------------------------------------- 1 | # Command line interface and tools 2 | 3 | Display help on available commands: 4 | 5 | ```bash 6 | $ python -m sharp 7 | ``` 8 | 9 | Display help on a specific command, e.g. `convert`: 10 | 11 | ```bash 12 | $ python -m sharp convert -h 13 | ``` 14 | 15 | 16 | ## Convert between mesh formats 17 | 18 | Supported formats: `.obj`, `.npz`. 19 | 20 | ```bash 21 | $ python -m sharp convert shoot path/to/input.obj path/to/output.npz 22 | $ python -m sharp convert path/to/input.npz path/to/output.obj 23 | ``` 24 | 25 | 26 | ## Generate partial data 27 | 28 | Supported formats: `.obj`, `.npz`. 29 | 30 | 31 | ### Holes shooting on a single mesh 32 | 33 | Usage example: 34 | 35 | ```bash 36 | # Shoot 40 holes with each hole removing 2% of the points of the mesh. 37 | $ python -m sharp shoot path/to/input.(npz|obj) path/to/output.(npz|obj) --holes 40 --dropout 0.02 [--mask path/to/mask.npy] 38 | ``` 39 | 40 | --mask: (optional) path to the mask (.npy) to generate holes only on regions considered for evaluation (only challenge 1). 41 | As mentioned in the [evaluation doc](https://gitlab.uni.lu/asaint/eccv2020-sharp-workshop/-/blob/update-instructions/doc/evaluation.md#challenge-specific-criteria), 42 | challenge 1 is evaluated on specific regions of the body mesh: 43 | 44 | - Track 1: head and hands are ignored, rest of the body is considered 45 | - Track 2: hands, ears, and feet are considered, rest of the body is ignored 46 | 47 | A mask is defined per face as boolean information: 0 if the face is to be ignored, and 1 if the face is to be kept. 48 | 49 | 50 | ### Holes shooting on a directory tree of meshes 51 | 52 | Usage examples: 53 | 54 | ```bash 55 | # Shoot 40 holes with each hole removing 2% of the points of the mesh. 56 | $ python -m sharp shoot_dir path/to/input_directory path/to/output_directory --holes 40 --dropout 0.02 [--mask-dir path/to/mask_directory] [--seed seed_value] [--n-workers n_workers] [--n-shapes n_shapes] 57 | ``` 58 | 59 | --mask-dir: (optional) Directory tree with the masks (.npy). If defined, the partial data is created only on the non-masked faces of the meshes (only challenge 1). 60 | 61 | --seed: Initial state for the pseudo random number generator. If not set, the initial state is not set explicitly. 62 | 63 | --n-workers: Number of parallel processes. By default, the number of available processors. 64 | 65 | -n: (or --n-shapes) Number of partial shapes to generate per mesh. Default is 1. 66 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-a-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-a-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-run-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-run-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-scape003-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-scape003-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-scape032-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-scape032-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-scape070-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-scape070-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-u-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-casual-u-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-fitness-a-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-fitness-a-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-synthetic-fitted-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-synthetic-fitted-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-synthetic-scan-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-synthetic-scan-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-synthetic-simulated-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-synthetic-simulated-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-texture_atlas-scan-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-texture_atlas-scan-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-texture_atlas-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/doc/data/3dbodytex2-texture_atlas-small.png -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/dataset_3dbodytex2.md: -------------------------------------------------------------------------------- 1 | # 3DBodyTex 2 2 | 3 | 3DBodyTex 2 contains thousands of 3D textured human scans. 4 | It is an extension of [3DBodyTex](https://cvi2.uni.lu/datasets/). 5 | 6 | ![Sample scans from the 3DBodyTex 2 dataset](3dbodytex2.png) 7 | *Subset of 3DBodyTex 2.* 8 | 9 | ## Content 10 | 11 | - 3D human scans (`Y`): 12 | - acquired with the [Shapify Booth][shapify booth], 13 | - many subjects, 14 | - varied poses, 15 | - varied casual clothing, 16 | - standard tight-fitting clothing, 17 | - 3D watertight meshes with a high-resolution texture atlas; 18 | - 3D landmarks: 19 | - body joints and landmark points on the surface, 20 | - estimated automatically; 21 | - Fitted textured SMPL-X model (`Yf`): 22 | - SMPL-X is fitted to all the raw scans `Y` with close-fitting clothing to 23 | obtain `Yf`, 24 | - pose and shape parameters, 25 | - fixed template mesh (SMPL-X), 26 | - texture mapped from the raw scans; 27 | - Simulated 3D scan acquisitions (`Ys`): 28 | - the process of the [Shapify Booth][shapify booth] is simulated in software, 29 | - the acquisition is simulated on the fitted SMPL-X models `Yf` to obtain 30 | `Ys`; 31 | - Synthetic pairs `(Yf, Ys)` of `(detailed ground truth, acquisition)`: 32 | - `Yf` is a synthetic groud truth with detailed shape and texture, 33 | - `Ys` is a synthetic scan acquisition with typical artefacts, 34 | - for scans with close-fitting clothing only. 35 | 36 | [shapify booth]: https://www.artec3d.com/portable-3d-scanners/shapifybooth 37 | 38 | 39 | ## Usage in the SHARP [Challenge 1](challenge_1.md) 40 | 41 | ### [Track 1](challenge_1_track_1.md): Recovery of large regions 42 | 43 | - The ground truth is `Y`. 44 | - The partial scans are generated from `Y`. 45 | - The 3D landmarks are provided at training time only, not at evaluation time. 46 | 47 | ### [Track 2](challenge_1_track_2.md): Recovery of fine details 48 | 49 | - The ground truth is `Yf`. 50 | - The partial scans are generated from `Ys`. 51 | - The 3D landmarks are provided at training time only, not at evaluation time. 52 | 53 | 54 | ## Detailed description and statistics 55 | 56 | ### Clothing 57 | 58 | casual | fitness 59 | -|- 60 | ![casual][img-casual] | ![fitness][img-fitness] 61 | 62 | [img-casual]: data/3dbodytex2-casual-a-small.png "casual" 63 | [img-fitness]: data/3dbodytex2-fitness-a-small.png "fitness" 64 | 65 | ### Poses 66 | 67 | A | U | run | scape | free 68 | -|-|-|-|- 69 | ![][img-pose-a] | ![][img-pose-u] | ![][img-pose-run] | ![][img-pose-scape-0] ![][img-pose-scape-1] ![][img-pose-scape-2] ... | unconstrained 70 | 71 | [img-pose-a]: data/3dbodytex2-casual-a-small.png "pose-a" 72 | [img-pose-u]: data/3dbodytex2-casual-u-small.png "pose-u" 73 | [img-pose-run]: data/3dbodytex2-casual-run-small.png "pose-run" 74 | [img-pose-scape-0]: data/3dbodytex2-casual-scape003-small.png "pose-scape-003" 75 | [img-pose-scape-1]: data/3dbodytex2-casual-scape032-small.png "pose-scape-032" 76 | [img-pose-scape-2]: data/3dbodytex2-casual-scape070-small.png "pose-scape-070" 77 | 78 | ### Synthetic detailed ground-truth and simulated acquisitions 79 | 80 | The goal is to obtain a ground-truth shape preserving the details of 81 | smaller-scale regions (e.g. fingers and feet). 82 | This cannot be acquired systematically with a body scanner. 83 | Instead, the template mesh of a body model (SMPL-X) is used as a reference for 84 | the fine-scale shape details. 85 | 86 | The procedure to synthesise the data is: 87 | 88 | 1. `Y` = input: raw scan in close-fitting clothing, 89 | 2. `Yf` = fit `Y`: fit the body model to `Y` and transfer the texture to obtain 90 | `Yf`, 91 | 3. `Ys` = acquire `Yf`: simulate the acquisition process on `Yf` to obtain 92 | `Ys`. 93 | 94 | The pair `(Yf, Ys)` is a synthetic pair of `(acquired, ground-truth)` shape 95 | that can be used for supervision. 96 | 97 | scan (`Y`) | fitted (`Yf`) | simulated acquisition (`Ys`) 98 | -|-|- 99 | ![fitness][img-synthetic-scan] | ![fitted][img-synthetic-fitted] | ![simulated][img-synthetic-simulated] 100 | 101 | [img-synthetic-scan]: data/3dbodytex2-synthetic-scan-small.png "scan" 102 | [img-synthetic-fitted]: data/3dbodytex2-synthetic-fitted-small.png "fitted" 103 | [img-synthetic-simulated]: data/3dbodytex2-synthetic-simulated-small.png "simulated acquisition" 104 | 105 | 106 | ## Encoding formats and conventions 107 | 108 | ### File formats 109 | 110 | Meshes are in the `.npz` format. 111 | Landmark positions are in a tabular text format. 112 | See [formats](formats.md). 113 | 114 | ### Orientation 115 | 116 | The up direction is the y axis. 117 | The ground is the x-z plane. 118 | Most scans are facing the -z direction, but not all. 119 | 120 | ### Units 121 | 122 | Vertex positions are in metres. 123 | 124 | 125 | ## 3D body landmarks 126 | 127 | There are 67 body landmarks detected automatically on each scan. 128 | They are provided to generate the partial data but may also be used for 129 | training. 130 | They comprise standard body joints and other keypoints on the body (eyes, nose, 131 | ears...). 132 | The detection of most landmarks is stable except for the finger joints and 133 | finger tips. 134 | 135 | Below, `` is either `left` or `right`. 136 | 137 | ### Body joints (20) 138 | 139 | | landmark name | 140 | | - | 141 | | ankle_\ | 142 | | elbow_\ | 143 | | heel_\ | 144 | | hip_\ | 145 | | hip_middle | 146 | | knee_\ | 147 | | neck | 148 | | shoulder_\ | 149 | | toe_1_\ | 150 | | toe_5_\ | 151 | | wrist_\ | 152 | 153 | ### Face landmarks (5) 154 | 155 | | landmark name | 156 | | - | 157 | | ear_\ | 158 | | eye_\ | 159 | | nose | 160 | 161 | ### Finger joints and tips (42) 162 | 163 | `` is one of `base`, `middle`, `top`, `tip`. 164 | 165 | | landmark name | 166 | | - | 167 | | finger_baby_\_\ | 168 | | finger_index_\_\ | 169 | | finger_middle_\_\ | 170 | | finger_ring_\_\ | 171 | | finger_thumb_\_\ | 172 | | hand_base_\ | 173 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/dataset_3dobjecttex.md: -------------------------------------------------------------------------------- 1 | # 3DObjectTex 2 | 3 | 3DObjectTex contains about 1200 scans of generic objects. 4 | 5 | ## Encoding formats and conventions 6 | 7 | ### File formats 8 | 9 | Meshes are in the `.obj` format. 10 | See [formats](formats.md). 11 | 12 | ### Units 13 | 14 | Vertex positions are in millimetres. 15 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/dataset_splits.md: -------------------------------------------------------------------------------- 1 | # Dataset splits 2 | 3 | In both challenges, the data is split into `train/test/eval` sets. 4 | 5 | In the first stage of the the competition, the `train/test` sets are provided 6 | for training. 7 | They contain only the ground-truth shapes, `Y`. 8 | The partial data, `X`, is generated by the participants. 9 | Some [example routines](../sharp/preprocess.py) are provided. 10 | Custom routines may be used if reported. 11 | 12 | At a later stage, the `eval` set is provided to generate the submission for the 13 | final evaluation. 14 | In this set, the partial data, `X`, is generated by the organisers and provided 15 | to the participants. 16 | The corresponding ground-truth shapes, `Y`, are kept secret. 17 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/datasets.md: -------------------------------------------------------------------------------- 1 | # Datasets 2 | 3 | Two new datasets with thousands of 3D textured scans are released as part of 4 | the SHARP competition: 5 | 6 | 1. [3DBodyTex 2](dataset_3dbodytex2.md): textured human body scans, 7 | 2. [3DObjectTex](dataset_3dobjecttex.md): textured object scans. 8 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/evaluation.md: -------------------------------------------------------------------------------- 1 | # Evaluation 2 | 3 | ## Quantitative 4 | 5 | Notation: 6 | 7 | - $`Y`$: the ground truth complete reference mesh, 8 | - $`Y'`$: the estimated complete mesh. 9 | 10 | The quality of the estimation, $`Y'`$, is evaluated quantitatively with respect 11 | to the ground truth, $`Y`$, using three criterions: 12 | 13 | ### 1. Surface-to-surface distances 14 | 15 | Consist of two directed distances: 16 | 17 | 1. $`d_{ER}`$ is computed from the estimation to the reference 18 | 2. $`d_{RE}`$ is computed from the reference to the estimation. 19 | 20 | These distances are inspired from [1] but have been adpated to fit the problem at hand. 21 | The directed distance $`d_{AB}`$ between meshes $`A`$ and $`B`$ is 22 | approximated in practice by sampling points on $`A`$ and computing their 23 | distances to the nearest triangles in mesh $`B`$. 24 | 25 | The directed distances $`d_{RE}`$ and $`d_{ER}`$ are given by, 26 | ```math 27 | d_{ER}(Y',Y) = \sum_{y' \in Y'} d(y', Y) ,\\ 28 | d_{RE}(Y,Y') = \sum_{y \in Y} d(y, Y') , 29 | ``` 30 | where $`y'`$ are the sampled points on the estimated surface $`Y'`$ and $`y`$ are the sampled points on the reference surface $`Y`$. 31 | 32 | In the two directions, the shape and texture reconstruction errors are measured separately. 33 | For the shape error, the distance, 34 | ```math 35 | d(a, B) = d_{shape}(a, B) , 36 | ``` 37 | operates on the 3D positions directly and computes a point-to-triangle distance between the sampled point $`a`$ on the source surface $`A`$ 38 | and its nearest triangle on the target surface $`B`$. 39 | For the texture error, the distance, 40 | ```math 41 | d(a, B) = d_{tex}(a, B) , 42 | ``` 43 | operates on the interpolated texture values at the source and target 3D positions used to compute the shape distance. 44 | 45 | This results in two shape distance values ($`d_{ER}^{shape}`$, $`d_{RE}^{shape}`$) and two texture distance values ($`d_{ER}^{tex}`$, $`d_{RE}^{tex}`$). 46 | Good estimations are expected to have low shape and texture distance values. 47 | 48 | ### 2. Surface hit-rates 49 | ith the point-to-triangle distance used above. 50 | Consist of two rates that are computed in two directions: 51 | 52 | 1. $`h_{ER}`$ computed from estimation to reference 53 | 2. $`h_{RE}`$ computed from reference to estimation. 54 | 55 | The hit-rate $`h_{AB}`$ indicates the amount of points sampled on the surface of a source mesh $`A`$ that have 56 | a correspondence on the target mesh $`B`$. A point in mesh $`A`$ has a correspondence (hit) in mesh $`B`$ if 57 | its projection on the plane of the nearest triangle in $`B`$ intersects the triangle. 58 | 59 | Let us consider: 60 | 61 | - $`H_{AB}`$: number of points of the source mesh $`A`$ that hit the target $`B`$ 62 | - $`M_{AB}`$: number of points of the source mesh $`A`$ that miss the target $`B`$. 63 | 64 | The hit-rate from $`A`$ to $`B`ith the point-to-triangle distance used above.$ is then given by, 65 | ```math 66 | h_{AB} = \frac{H_{AB}}{H_{AB} + M_{AB}} . 67 | ``` 68 | In the two directions, the hit-rate is a score with a value in [0,1]. Good estimations are expected to have high hit-rates. 69 | 70 | 71 | ### 3. Surface area score 72 | 73 | Consists of a score that quantifies the similarity between 74 | the surface area of the estimation and that of the reference. The surface area of the estimated mesh and the reference mesh 75 | denoted as $`A_{E}`$ and $`A_{R}`$, respectively, are computed by summing over the areas of the triangles of each mesh. 76 | These areas are then normalized as follows, 77 | ```math 78 | \bar{A_{R}} = \frac{A_{R}}{A_{R} + A_{E}} , \\ 79 | \bar{A_{E}} = \frac{A_{E}}{A_{R} + A_{E}} . 80 | ``` 81 | 82 | The area score $`S_a`$ is then given by, 83 | 84 | ```math 85 | S_a = 1 - | \bar{A_{R}} - \bar{A_{E}} | . 86 | ``` 87 | 88 | This score results in a value in [0,1]. Good estimations are expected to have high area scores. 89 | 90 | 91 | ### Final score 92 | 93 | Consists of a combination of the three measures explained above. 94 | 95 | The shape and texture scores are computed as follows, 96 | 97 | ```math 98 | S_s = \frac{1}{2} [ \Phi_{k_1}(d_{ER}^{shape}(Y',Y)) h_{ER} + \Phi_{k_2}(d_{RE}^{shape}(Y,Y')) h_{RE} ] , \\ 99 | S_t = \frac{1}{2} [ \Phi_{k_3}(d_{ER}^{tex}(Y',Y)) h_{ER} + \Phi_{k_4}(d_{RE}^{tex}(Y,Y')) h_{RE} ] , 100 | ``` 101 | 102 | where $`\Phi_{k_i}(d) = e^{-k_id^2}`$ maps a distance $`d`$ to a score in [0,1]. The parameters $`k_i`$ are chosen 103 | according to some conducted baselines. 104 | 105 | The final score is finally given by, 106 | 107 | ```math 108 | S = \frac{1}{2} S_a(S_s + S_t) . 109 | ``` 110 | 111 | ### Challenge-specific criteria 112 | 113 | | challenge(/track) | shape | texture | note | 114 | | - | - | - | - | 115 | | 1/1 | Yes | Yes | hands and head ignored | 116 | | 1/2 | Yes | No | only hands, feet and ears | 117 | | 2 | Yes | Yes | - | 118 | 119 | 120 | ## References 121 | 122 | [1] Jensen, Rasmus, et al. 123 | "Large scale multi-view stereopsis evaluation." 124 | Proceedings of the IEEE Conference on Computer Vision and Pattern 125 | Recognition. 126 | 2014. 127 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/formats.md: -------------------------------------------------------------------------------- 1 | # Data formats 2 | 3 | ## Meshes 4 | 5 | The meshes of generic objects are stored as [.obj](#wavefront-obj-mesh). 6 | 7 | The body scans are textured 3D meshes stored in [.npz](#npz-mesh) archives. 8 | 9 | The texture is encoded as a 10 | [texture atlas](https://en.wikipedia.org/wiki/Texture_atlas). 11 | 12 | | scan | texture atlas | 13 | | - | - | 14 | | ![][texture-scan] | ![][texture] | 15 | 16 | [texture-scan]: data/3dbodytex2-texture_atlas-scan-small.png 17 | [texture]: data/3dbodytex2-texture_atlas-small.png 18 | 19 | ### Wavefront OBJ mesh 20 | 21 | See [Wavefront `.obj`](https://en.wikipedia.org/wiki/Wavefront_.obj_file). 22 | 23 | ### Npz mesh 24 | 25 | This format stores the numpy arrays defining the mesh inside a 26 | [(compressed)](https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html) 27 | [numpy `.npz`](https://numpy.org/doc/stable/reference/generated/numpy.savez.html) 28 | archive. 29 | 30 | The following arrays inside the `.npz` define a mesh: 31 | 32 | * `vertices`, float (N, 3): 33 | The 3D positions of the vertices. 34 | N varies across the meshes. 35 | * `faces`, int (20000, 3): 36 | The vertex indices defining the faces in 3D space (i.e. triplets of indices 37 | into the `vertices` array). Fixed number of faces (20000) for all meshes. 38 | * `texcoords`, float (Nt, 2): 39 | The 2D positions of the vertices in the texture atlas (Nt > N). 40 | * `texcoords_indices`, int (20000, 3): 41 | The vertex indices defining the faces in the UV space (2D texture image) 42 | (i.e. triplets of indices into the `texcoords` array). Fixed number of 43 | faces (20000) for all meshes 44 | * `texture`, uint8 (2048, 2048, 3): 45 | The RGB texture image. 46 | 47 | Fields not described above should not be relied upon. 48 | 49 | The mesh can be loaded with [`numpy.load`][np.load]. 50 | For example: 51 | 52 | ```python 53 | import numpy as np 54 | mesh = np.load("name.npz", allow_pickle=True) 55 | mesh["vertices"] 56 | mesh["faces"] 57 | # ... 58 | ``` 59 | 60 | [np.load]: https://numpy.org/doc/stable/reference/generated/numpy.load.html 61 | 62 | ## Body landmarks 63 | 64 | 3D positions of detected body landmarks are provided 65 | in the training data of [Challenge 1](challenge_1.md). 66 | They are stored in files with name `landmarks3d.txt`. 67 | The format is plain text and tabular, with one landmark per row and one space 68 | between columns: 69 | 70 | ``` 71 | name x y z 72 | ``` 73 | 74 | with `name`, the name of the landmark, and `(x, y, z)`, its 3D position in 75 | the frame of reference of the scan or mesh. 76 | If the landmark was not detected, the coordinates are `nan`. 77 | 78 | For example, 79 | 80 | ``` 81 | elbow_left 1.234 0.123 0.389 82 | finger_thumb_top_left nan nan nan 83 | ... 84 | ``` 85 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/doc/landmark_names.txt: -------------------------------------------------------------------------------- 1 | ankle_left 2 | ankle_right 3 | ear_left 4 | ear_right 5 | elbow_left 6 | elbow_right 7 | eye_left 8 | eye_right 9 | finger_baby_base_left 10 | finger_baby_base_right 11 | finger_baby_middle_left 12 | finger_baby_middle_right 13 | finger_baby_tip_left 14 | finger_baby_tip_right 15 | finger_baby_top_left 16 | finger_baby_top_right 17 | finger_index_base_left 18 | finger_index_base_right 19 | finger_index_middle_left 20 | finger_index_middle_right 21 | finger_index_tip_left 22 | finger_index_tip_right 23 | finger_index_top_left 24 | finger_index_top_right 25 | finger_mean_base_knuckle_left 26 | finger_mean_base_knuckle_right 27 | finger_middle_base_left 28 | finger_middle_base_right 29 | finger_middle_middle_left 30 | finger_middle_middle_right 31 | finger_middle_tip_left 32 | finger_middle_tip_right 33 | finger_middle_top_left 34 | finger_middle_top_right 35 | finger_ring_base_left 36 | finger_ring_base_right 37 | finger_ring_middle_left 38 | finger_ring_middle_right 39 | finger_ring_tip_left 40 | finger_ring_tip_right 41 | finger_ring_top_left 42 | finger_ring_top_right 43 | finger_thumb_base_left 44 | finger_thumb_base_right 45 | finger_thumb_middle_left 46 | finger_thumb_middle_right 47 | finger_thumb_tip_left 48 | finger_thumb_tip_right 49 | finger_thumb_top_left 50 | finger_thumb_top_right 51 | hand_base_left 52 | hand_base_right 53 | heel_left 54 | heel_right 55 | hip_left 56 | hip_middle 57 | hip_right 58 | knee_left 59 | knee_right 60 | neck 61 | nose 62 | shoulder_left 63 | shoulder_right 64 | toe_1_left 65 | toe_1_right 66 | toe_5_left 67 | toe_5_right 68 | wrist_left 69 | wrist_right 70 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/readme.md: -------------------------------------------------------------------------------- 1 | # SHARP: SHApe Recovery from Partial textured 3D scans 2 | 3 | [SHARP 2020 website](https://cvi2.uni.lu/sharp2020/) 4 | 5 | A workshop and challenge in conjunction with 6 | [ECCV 2020](https://eccv2020.eu/workshops/). 7 | 8 | Organised by [CVI²](https://cvi2.uni.lu) & [Artec3D](https://www.artec3d.com). 9 | 10 | ## Overview 11 | 12 | The goal of the competition is to recover a reference textured scan `Y` from a 13 | partial version of it `X`. 14 | This is a *completion task* of 3D shape and texture. 15 | 16 | The data consists in pairs `(X, Y)` generated from two 17 | [datasets](doc/datasets.md) of 3D textured scans: 18 | 19 | 1. [3DBodyTex 2](doc/dataset_3dbodytex2.md): a dataset of 3D human scans 20 | (extending [3DBodyTex](https://cvi2.uni.lu/datasets/)). 21 | 2. [3DObjectTex](doc/dataset_3dobjecttex.md): a dataset of 3D scans of generic 22 | objects. 23 | 24 | There are two challenges, one per dataset. 25 | 26 | The *ground-truth shapes*, `Y`, are the raw scans (or the fitted body models in 27 | a variant of the challenge on human scans). 28 | 29 | The *partial scans*, `X`, are generated synthetically from the ground-truth 30 | shapes, `Y`. 31 | The idea is to simulate the partial acquisition produced by a 3D scanner, e.g. 32 | a hand-held scanner. 33 | 34 | For *training*, only the ground-truth shapes, `Y`, are provided. 35 | The partial scans, `X`, are generated by the participants. 36 | Some [example routines](sharp/preprocess.py) are provided. 37 | Custom routines may be used if reported. 38 | 39 | For *evaluation*, a set of partial shapes, `X`, is provided to the 40 | participants. 41 | The corresponding ground-truth shapes, `Y`, are kept secret. 42 | The participants estimate a complete shape, `Y'`, from `X`. 43 | The submissions are evaluated quantitatively by computing the 44 | surface-to-surface distance between `Y` and `Y'`. 45 | 46 | 47 | ## Detailed instructions 48 | 49 | - [Challenge 1: Recovery of Human Body Scans](doc/challenge_1.md) 50 | - [Track 1: Recovery of large regions](doc/challenge_1_track_1.md) 51 | - [Track 2: Recovery of fine details](doc/challenge_1_track_2.md) 52 | - [Challenge 2: Recovery of Generic Object Scans](doc/challenge_2.md) 53 | - [Evaluation and metrics](doc/evaluation.md) 54 | - [Datasets](doc/datasets.md) 55 | - [Dataset splits](doc/dataset_splits.md) 56 | - [Data formats](doc/formats.md) 57 | - [Command line interface (CLI)](doc/cli.md) 58 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/requirements.txt: -------------------------------------------------------------------------------- 1 | argparse 2 | numpy 3 | pathlib 4 | moderngl 5 | opencv-python 6 | scipy 7 | nptyping -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/scripts/demo_holes: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Generate partial data with the hole method with different configurations. 3 | 4 | (( $# != 1 )) && { echo "usage: $0 MESH"; exit 1; } 5 | 6 | mesh=$1 7 | 8 | out=${mesh%.*}-holes-h_20-d_1.npz 9 | python -m sharp shoot $mesh $out --holes 20 --dropout 0.01 10 | printf . 11 | out=${mesh%.*}-holes-h_40-d_1.npz 12 | python -m sharp shoot $mesh $out --holes 40 --dropout 0.01 13 | printf . 14 | out=${mesh%.*}-holes-h_60-d_1.npz 15 | python -m sharp shoot $mesh $out --holes 60 --dropout 0.01 16 | printf . 17 | 18 | out=${mesh%.*}-holes-h_20-d_2.npz 19 | python -m sharp shoot $mesh $out --holes 20 --dropout 0.02 20 | printf . 21 | out=${mesh%.*}-holes-h_40-d_2.npz 22 | python -m sharp shoot $mesh $out --holes 40 --dropout 0.02 23 | printf . 24 | out=${mesh%.*}-holes-h_60-d_2.npz 25 | python -m sharp shoot $mesh $out --holes 60 --dropout 0.02 26 | printf . 27 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/setup.py: -------------------------------------------------------------------------------- 1 | import setuptools 2 | 3 | with open("readme.md", "r") as fh: 4 | long_description = fh.read() 5 | 6 | setuptools.setup( 7 | name="sharpcvi2", 8 | version="1.0.0", 9 | author="CVI2: Computer Vision, Imaging and Machine Intelligence Research Group", 10 | author_email="shapify3D@uni.lu", 11 | description="Routines for the SHARP Challenge, ECCV 2020", 12 | long_description=long_description, 13 | long_description_content_type="text/markdown", 14 | url="https://cvi2.uni.lu/sharp2020/", 15 | packages=setuptools.find_packages(), 16 | classifiers=[ 17 | "Programming Language :: Python :: 3", 18 | "Operating System :: OS Independent", 19 | ], 20 | python_requires='>=3.6', 21 | ) 22 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/sharp/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/data_processing/eccv2020-sharp-workshop/sharp/__init__.py -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/sharp/__main__.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import concurrent.futures 3 | import itertools 4 | import logging 5 | import pathlib 6 | import sys 7 | 8 | import numpy as np 9 | 10 | from . import data 11 | from . import utils 12 | 13 | 14 | logger = logging.getLogger(__name__) 15 | 16 | 17 | def _do_convert(args): 18 | mesh = data.load_mesh(args.input) 19 | data.save_mesh(args.output, mesh) 20 | 21 | 22 | def _do_shoot(args): 23 | mesh = data.load_mesh(str(args.input)) 24 | 25 | has_min_holes = args.min_holes is not None 26 | has_max_holes = args.max_holes is not None 27 | if has_min_holes != has_max_holes: 28 | raise ValueError("--min-holes and --max-holes must be set together") 29 | n_holes = ((args.min_holes, args.max_holes) 30 | if has_min_holes 31 | else args.holes) 32 | 33 | has_min_dropout = args.min_dropout is not None 34 | has_max_dropout = args.max_dropout is not None 35 | if has_min_dropout != has_max_dropout: 36 | raise ValueError( 37 | "--min-dropout and --max-dropout must be set together") 38 | dropout = ((args.min_dropout, args.max_dropout) 39 | if has_min_dropout 40 | else args.dropout) 41 | 42 | mask_faces = (np.load(args.mask) if args.mask is not None 43 | else None) 44 | faces = None if mask_faces is None else mesh.faces 45 | point_indices = utils.shoot_holes(mesh.vertices, 46 | n_holes, 47 | dropout, 48 | mask_faces=mask_faces, 49 | faces=faces) 50 | shot = utils.remove_points(mesh, point_indices) 51 | shot.save(str(args.output)) 52 | 53 | 54 | def identify_meshes(dir_): 55 | """List meshes and identify the challenge/track in a directory tree.""" 56 | meshes_track1 = list(dir_.glob("**/*_normalized.npz")) 57 | meshes_track2 = list(dir_.glob("**/fusion_textured.npz")) 58 | meshes_challenge2 = list(dir_.glob("**/model_*.obj")) 59 | if meshes_track1: 60 | meshes = sorted(meshes_track1) 61 | challenge = 1 62 | track = 1 63 | elif meshes_track2: 64 | meshes = sorted(meshes_track2) 65 | challenge = 1 66 | track = 2 67 | elif meshes_challenge2: 68 | meshes = sorted(meshes_challenge2) 69 | challenge = 2 70 | track = None 71 | else: 72 | meshes = [] 73 | challenge = None 74 | track = None 75 | 76 | return meshes, challenge, track 77 | 78 | 79 | def shoot_helper(mesh_index, 80 | path, 81 | n_holes, 82 | dropout, 83 | shape_seeds, 84 | n_meshes, 85 | input_dir, 86 | output_dir, 87 | mask_dir=None, 88 | ): 89 | logger.info(f"{mesh_index + 1}/{n_meshes} -------- TEST ------------ processing {path}") 90 | 91 | rel_path = path.relative_to(input_dir) 92 | 93 | # Lazily load the mesh and mask when it is sure they are needed. 94 | mesh_cached = None 95 | mask_cached = None 96 | 97 | def make_name_suffix(shape_index, n_shapes): 98 | if n_shapes == 1: 99 | return "partial" 100 | else: 101 | # Assuming 0 <= shape_index <= 99. 102 | return f"partial-{shape_index:02d}" 103 | 104 | def load_mask(rel_path): 105 | mask_name = f"{rel_path.stem}-mask.npy" 106 | mask_rel_path = rel_path.with_name(mask_name) 107 | mask_path = mask_dir / mask_rel_path 108 | logger.info(f"loading mask {mask_path}") 109 | mask_faces = np.load(mask_path) 110 | return mask_faces 111 | 112 | n_shapes = len(shape_seeds) 113 | for shape_index, shape_seed in enumerate(shape_seeds): 114 | out_name_suffix = make_name_suffix(shape_index, n_shapes) 115 | out_name = f"{rel_path.stem}-{out_name_suffix}.npz" 116 | out_rel_path = rel_path.with_name(out_name) 117 | out_path = output_dir / out_rel_path 118 | 119 | if out_path.exists(): 120 | logger.warning(f"shape exists, skipping {out_path}") 121 | continue 122 | 123 | logger.info(f"generating shape {shape_index + 1}/{n_shapes}") 124 | 125 | out_path.parent.mkdir(parents=True, exist_ok=True) 126 | 127 | if mesh_cached is None: 128 | mesh_cached = data.load_mesh(str(path)) 129 | if mask_cached is None and mask_dir is not None: 130 | mask_cached = load_mask(rel_path) 131 | 132 | mesh = mesh_cached 133 | mask = mask_cached 134 | 135 | logger.info(f"TEST shape seed = {shape_seed}") 136 | shape_rng = np.random.default_rng(shape_seed) 137 | point_indices = utils.shoot_holes(mesh.vertices, 138 | n_holes, 139 | dropout, 140 | mask_faces=mask, 141 | faces=mesh.faces, 142 | rng=shape_rng) 143 | partial = utils.remove_points(mesh, point_indices) 144 | 145 | logger.info(f"{shape_index + 1}/{n_shapes} saving {out_path}") 146 | partial.save(str(out_path)) 147 | 148 | 149 | def _do_shoot_dir(args): 150 | """Generate partial data in a directory tree of meshes. 151 | 152 | An independent PRNG is used for each partial shape. Independent seeds are 153 | created in advance for each partial shape. The process is thus reproducible 154 | and can also be interrupted and resumed without generating all the previous 155 | shapes. 156 | """ 157 | input_dir = args.input_dir 158 | output_dir = args.output_dir 159 | mask_dir = args.mask_dir 160 | seed = args.seed 161 | n_holes = args.holes 162 | dropout = args.dropout 163 | n_shapes = args.n_shapes 164 | n_workers = args.n_workers 165 | 166 | logger.info("generating partial data in directory tree") 167 | logger.info(f"input dir = {input_dir}") 168 | logger.info(f"output dir = {output_dir}") 169 | logger.info(f"mask dir = {mask_dir}") 170 | logger.info(f"seed = {seed}") 171 | logger.info(f"holes = {n_holes}") 172 | logger.info(f"dropout = {dropout}") 173 | logger.info(f"n_shapes = {n_shapes}") 174 | logger.info(f"n_workers = {n_workers}") 175 | 176 | mesh_paths, challenge, track = identify_meshes(input_dir) 177 | if challenge is None: 178 | raise ValueError(f"could not identify meshes in {input_dir}") 179 | logger.info(f"detected challenge {challenge} track {track}") 180 | n_meshes = len(mesh_paths) 181 | logger.info(f"found {n_meshes} meshes") 182 | 183 | logger.info(f"setting random seed {seed}") 184 | rng = np.random.default_rng(seed) 185 | 186 | # Create in advance the individual initial random states for each partial 187 | # shape to be generated so that the process of generation can be 188 | # - resumed without regenerating the previous shapes, 189 | # - parallelised. 190 | seeds = rng.integers(1e12, size=(n_meshes, n_shapes)) 191 | 192 | mesh_indices = range(n_meshes) 193 | with concurrent.futures.ProcessPoolExecutor(max_workers=n_workers) as executor: 194 | executor.map( 195 | shoot_helper, 196 | mesh_indices, 197 | mesh_paths, 198 | itertools.repeat(n_holes), 199 | itertools.repeat(dropout), 200 | seeds.tolist(), 201 | itertools.repeat(n_meshes), 202 | itertools.repeat(input_dir), 203 | itertools.repeat(output_dir), 204 | itertools.repeat(mask_dir), 205 | ) 206 | 207 | 208 | def _parse_args(): 209 | parser = argparse.ArgumentParser() 210 | subparsers = parser.add_subparsers() 211 | 212 | parser_convert = subparsers.add_parser( 213 | "convert", 214 | help="Convert between mesh formats.", 215 | ) 216 | parser_convert.add_argument("input", type=pathlib.Path) 217 | parser_convert.add_argument("output", type=pathlib.Path) 218 | parser_convert.set_defaults(func=_do_convert) 219 | 220 | parser_shoot = subparsers.add_parser( 221 | "shoot", 222 | help="Generate partial data with the shooting method.", 223 | ) 224 | parser_shoot.add_argument("input", type=pathlib.Path) 225 | parser_shoot.add_argument("output", type=pathlib.Path) 226 | parser_shoot.add_argument( 227 | "--holes", type=int, default=40, 228 | help="Number of holes to shoot.", 229 | ) 230 | parser_shoot.add_argument( 231 | "--min-holes", type=int, default=None, 232 | help="Minimum number of holes to generate." 233 | " (Supersedes --holes and requires --max-holes.)", 234 | ) 235 | parser_shoot.add_argument( 236 | "--max-holes", type=int, default=None, 237 | help="Maximum number of holes to generate." 238 | " (Supersedes --holes and requires --min-holes.)", 239 | ) 240 | parser_shoot.add_argument( 241 | "--dropout", type=float, default=2e-2, 242 | help="Proportion of points of the mesh to remove in a single hole.", 243 | ) 244 | parser_shoot.add_argument( 245 | "--min-dropout", type=float, default=None, 246 | help="Minimum proportion of points of the mesh to remove in a single " 247 | "hole." 248 | " (Supersedes --dropout and requires --max-dropout.)", 249 | ) 250 | parser_shoot.add_argument( 251 | "--max-dropout", type=float, default=None, 252 | help="Maximum proportion of points of the mesh to remove in a single " 253 | "hole." 254 | " (Supersedes --dropout and requires --min-dropout.)", 255 | ) 256 | parser_shoot.add_argument( 257 | "--mask", type=pathlib.Path, 258 | help=" (optional) Path to the mask (.npy) to generate holes only on" 259 | " regions considered for evaluation.", 260 | ) 261 | parser_shoot.set_defaults(func=_do_shoot) 262 | 263 | parser_shoot_dir = subparsers.add_parser( 264 | "shoot_dir", 265 | help="Generate partial data with the shooting method for a directory" 266 | " tree of meshes." 267 | ) 268 | parser_shoot_dir.add_argument("input_dir", type=pathlib.Path) 269 | parser_shoot_dir.add_argument("output_dir", type=pathlib.Path) 270 | parser_shoot_dir.add_argument( 271 | "--mask-dir", 272 | type=pathlib.Path, 273 | help=" (optional) Directory tree with the masks (.npy). If defined," 274 | " the partial data is created only on the non-masked faces of the" 275 | " meshes. (Only valid for challenge 1.)", 276 | ) 277 | parser_shoot_dir.add_argument( 278 | "--holes", 279 | type=int, 280 | default=40, 281 | help="Number of holes to shoot.", 282 | ) 283 | parser_shoot_dir.add_argument( 284 | "--dropout", 285 | type=float, 286 | default=2e-2, 287 | help="Proportion of points of the mesh to remove in a single hole.", 288 | ) 289 | parser_shoot_dir.add_argument( 290 | "--seed", 291 | type=int, 292 | default=None, 293 | help="Initial state for the pseudo random number generator." 294 | " If not set, the initial state is not set explicitly.", 295 | ) 296 | parser_shoot_dir.add_argument( 297 | "-n", "--n-shapes", 298 | type=int, 299 | default=1, 300 | help="Number of partial shapes to generate per mesh." 301 | " If n = 1 (default), the shape is saved as" 302 | " '-partial.npz'." 303 | " If n > 1, the shapes are saved as '-partial-XY.npz'" 304 | ", with XY as 00, 01, 02... (assuming n <= 99).", 305 | ) 306 | parser_shoot_dir.add_argument( 307 | "--n-workers", 308 | type=int, 309 | default=None, 310 | help="Number of parallel processes. By default, the number of" 311 | " available processors.", 312 | ) 313 | parser_shoot_dir.set_defaults(func=_do_shoot_dir) 314 | 315 | args = parser.parse_args() 316 | 317 | # Ensure the help message is displayed when no command is provided. 318 | if "func" not in args: 319 | parser.print_help() 320 | sys.exit(1) 321 | 322 | return args 323 | 324 | 325 | def main(): 326 | args = _parse_args() 327 | args.func(args) 328 | 329 | 330 | if __name__ == "__main__": 331 | logging.basicConfig(level=logging.INFO) 332 | main() 333 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/sharp/data.py: -------------------------------------------------------------------------------- 1 | import csv 2 | import pathlib 3 | import re 4 | 5 | import cv2 6 | import numpy as np 7 | 8 | 9 | def read_3d_landmarks(inpath): 10 | """Read the 3d landmarks locations and names. 11 | 12 | Landmarks are stored in a CSV file with rows of the form: 13 | landmark_name x y z 14 | 15 | Parameters 16 | ---------- 17 | inpath : string 18 | path to the file 19 | 20 | Returns 21 | ------- 22 | landmarks: dict 23 | A dictionary of (landmark_name, 3d_location) pairs. 24 | """ 25 | landmarks = {} 26 | with open(inpath) as f: 27 | reader = csv.reader(f, delimiter=' ') 28 | for row in reader: 29 | name = row[0] 30 | location = np.array([float(v) for v in row[1:4]]) 31 | landmarks[name] = location 32 | return landmarks 33 | 34 | 35 | def imread(path): 36 | """Read a color image. 37 | 38 | Return an image in RGB format with float values into [0, 1]. 39 | """ 40 | img = cv2.imread(str(path)) 41 | # Convert to RGB. OpenCV returns BGR. 42 | img = img[..., ::-1] 43 | return img.astype(float) / np.iinfo(img.dtype).max 44 | 45 | 46 | def imwrite(path, img, dtype=np.uint8): 47 | """Save an RGB image to a file. 48 | 49 | Expect float values into [0, 1]. 50 | """ 51 | img = (img * np.iinfo(dtype).max).astype(dtype) 52 | # OpenCV expects BGR. 53 | img = img[..., ::-1] 54 | cv2.imwrite(str(path), img) 55 | 56 | 57 | def load_mesh(path): 58 | if str(path).endswith(".obj"): 59 | return load_obj(path) 60 | elif str(path).endswith(".npz"): 61 | return load_npz(path) 62 | raise ValueError(f"unknown mesh format {path}") 63 | 64 | 65 | def save_mesh(path, mesh): 66 | if str(path).endswith(".obj"): 67 | return save_obj(path, mesh) 68 | elif str(path).endswith(".npz"): 69 | return save_npz(path, mesh) 70 | raise ValueError(f"unknown mesh format for {path}") 71 | 72 | 73 | class Mesh: 74 | 75 | def __init__(self, path=None, 76 | vertices=None, vertex_normals=None, vertex_colors=None, 77 | faces=None, face_normals=None, faces_normal_indices=None, 78 | normals=None, 79 | texcoords=None, texture_indices=None, texture=None, 80 | material=None, mask_faces=None): 81 | self.path = path 82 | self.vertices = vertices 83 | self.vertex_normals = vertex_normals 84 | self.vertex_colors = vertex_colors 85 | self.faces = faces 86 | self.face_normals = face_normals 87 | self.faces_normal_indices = faces_normal_indices 88 | self.normals = normals 89 | self.texcoords = texcoords 90 | self.texture_indices = texture_indices 91 | self.texture = texture 92 | self.material = material 93 | self.mask_faces = mask_faces 94 | 95 | @staticmethod 96 | def load(path): 97 | return load_mesh(path) 98 | 99 | def save(self, path): 100 | save_mesh(path, self) 101 | 102 | 103 | def _read_mtl(mtl_path): 104 | if not mtl_path.exists(): 105 | return None 106 | with open(mtl_path) as f: 107 | for line in f: 108 | tokens = line.strip().split(maxsplit=1) 109 | if not tokens: 110 | continue 111 | if tokens[0] == 'map_Kd': 112 | texture_name = tokens[1] 113 | return texture_name 114 | return None 115 | 116 | 117 | def _parse_faces(obj_faces): 118 | """Parse the OBJ encoding of the face attribute index buffers. 119 | 120 | A value of -1 in the index array of the texture means the index is 121 | undefined (i.e. no texture is mapped to this face corner). 122 | 123 | Returns: 124 | faces 125 | faces_texture 126 | faces_normals 127 | """ 128 | face_pattern = re.compile(r'(\d+)/?(\d*)?/?(\d*)?') 129 | faces = [] 130 | faces_normals = [] 131 | faces_texture = [] 132 | for obj_face in obj_faces: 133 | fv, ft, fn = [], [], [] 134 | arrs = (fv, ft, fn) 135 | for element in obj_face: 136 | match = face_pattern.match(element) 137 | if match is not None: 138 | for i, group in enumerate(match.groups()): 139 | if len(group) > 0: 140 | arrs[i].append(int(group)) 141 | if len(fv) > 0: 142 | faces.append(fv) 143 | if len(ft) > 0: 144 | faces_texture.append(ft) 145 | else: 146 | el = [-1 for x in range(len(fv))] 147 | faces_texture.append(el) 148 | if len(fn) > 0: 149 | faces_normals.append(fn) 150 | # else: 151 | # el = ['' for x in range(len(fv))] 152 | # faces_normals.append(el) 153 | 154 | faces = np.array(faces, dtype=int) 155 | faces_texture = np.array(faces_texture, dtype=int) 156 | faces_normals = np.array(faces_normals, dtype=int) 157 | 158 | # Change to zero-based indexing. 159 | faces -= 1 160 | faces_texture -= 1 161 | faces_normals -= 1 162 | 163 | return faces, faces_texture, faces_normals 164 | 165 | 166 | def _complete_texcoords(texcoords, texture_indices): 167 | """Ensure all faces reference some texture coordinates. 168 | 169 | Make untextured faces reference a new dummy texture coordinate. 170 | The new texture coordinate is placed at `(u, v) = (0, 0)`, assuming no 171 | texture exists there (i.e. black color). 172 | """ 173 | no_texcoords = texture_indices < 0 174 | if np.any(no_texcoords): 175 | texcoords = np.append(texcoords, [[.0, .0]], axis=0) 176 | new_index = len(texcoords) - 1 177 | texture_indices[no_texcoords] = new_index 178 | return texcoords, texture_indices 179 | 180 | 181 | def _load_texture(path): 182 | return imread(path) 183 | 184 | 185 | def _save_texture(path, texture): 186 | imwrite(path, texture) 187 | 188 | 189 | def load_obj(path): 190 | path = pathlib.Path(path) 191 | 192 | vertices = [] 193 | vertex_colors = [] 194 | normals = [] 195 | faces = [] 196 | texcoords = [] 197 | material_name = None 198 | mtl_filename = None 199 | texture_filename = None 200 | texture = None 201 | with open(path) as f: 202 | for line in f: 203 | line = line.strip() 204 | 205 | if not line: 206 | # blank line 207 | continue 208 | if line.startswith('#'): 209 | # comment 210 | continue 211 | 212 | tokens = line.split() 213 | if tokens[0] == 'v': 214 | vertices.append([float(x) for x in tokens[1:]]) 215 | elif tokens[0] == 'vn': 216 | normals.append([float(x) for x in tokens[1:]]) 217 | elif tokens[0] == 'f': 218 | faces.append(tokens[1:]) 219 | elif tokens[0] == 'vt': 220 | texcoords.append([float(tokens[1]), float(tokens[2])]) 221 | elif tokens[0] in ('usemtl', 'usemat'): 222 | material_name = tokens[1] 223 | elif tokens[0] == 'mtllib': 224 | if len(tokens) > 1: 225 | mtl_filename = tokens[1] 226 | mtl_path = path.parent / mtl_filename 227 | texture_filename = _read_mtl(mtl_path) 228 | 229 | vertices = np.array(vertices) 230 | if vertices.shape[1] == 6: 231 | vertex_colors = vertices[:, 3:] 232 | vertices = vertices[:, :3] 233 | 234 | faces, texture_indices, faces_normal_indices = _parse_faces(faces) 235 | 236 | if texcoords: 237 | texcoords = np.asarray(texcoords, dtype=float) 238 | texture_indices = np.asarray(texture_indices, dtype=int) 239 | texcoords, texture_indices = _complete_texcoords(texcoords, 240 | texture_indices) 241 | else: 242 | texture_indices = [] 243 | 244 | if normals: 245 | normals = np.array(normals) 246 | 247 | if texture_filename is not None: 248 | texture_path = path.parent / texture_filename 249 | if texture_path.exists(): 250 | texture = _load_texture(texture_path) 251 | 252 | return Mesh( 253 | path=path, 254 | vertices=vertices, 255 | vertex_colors=vertex_colors if len(vertex_colors) > 0 else None, 256 | faces=faces, 257 | faces_normal_indices=(faces_normal_indices 258 | if len(faces_normal_indices) > 0 259 | else None), 260 | normals=normals if len(normals) > 0 else None, 261 | texcoords=texcoords if len(texcoords) > 0 else None, 262 | texture=texture, 263 | texture_indices=(texture_indices 264 | if len(texture_indices) > 0 265 | else None), 266 | material=material_name, 267 | ) 268 | 269 | 270 | def _save_mtl(path, texture_name=None): 271 | if texture_name is not None: 272 | texture_command = f"map_Kd {texture_name}" 273 | else: 274 | texture_command = "" 275 | 276 | content = f"""\ 277 | newmtl material_0 278 | Ka 1 1 1 279 | Kd 1 1 1 280 | Ks 1 1 1 281 | Ns 1000 282 | {texture_command} 283 | newmtl material_1 284 | Ka 1 1 1 285 | Kd 1 1 1 286 | Ks 1 1 1 287 | Ns 1000 288 | """ 289 | 290 | with open(path, 'w') as mtl_file: 291 | mtl_file.write(content) 292 | 293 | 294 | def save_obj(path, mesh, save_texture=True): 295 | path = pathlib.Path(path) 296 | out_dir = pathlib.Path(path).parent 297 | out_dir.mkdir(parents=True, exist_ok=True) 298 | 299 | # Write the texture image file. 300 | if save_texture and mesh.texture is not None: 301 | texture_path = path.with_suffix(".png") 302 | texture_name = texture_path.name 303 | _save_texture(texture_path, mesh.texture) 304 | else: 305 | texture_path = None 306 | texture_name = None 307 | 308 | # Write the mtl file. 309 | mtlpath = path.with_suffix(".mtl") 310 | mtlname = mtlpath.name 311 | _save_mtl(mtlpath, texture_name=texture_name) 312 | 313 | # Write the obj file. 314 | with open(path, 'w') as f: 315 | f.write(f"mtllib {mtlname}\n") 316 | f.write("usemtl material_0\n") 317 | 318 | if mesh.vertex_colors is not None: 319 | for vertex, color in zip(mesh.vertices, mesh.vertex_colors): 320 | f.write("v {} {} {} {} {} {}\n".format(*vertex, *color)) 321 | else: 322 | for vertex in mesh.vertices: 323 | f.write("v {} {} {}\n".format(*vertex)) 324 | 325 | if mesh.vertex_normals is not None: 326 | for normal in mesh.vertex_normals: 327 | f.write("vn {} {} {}\n".format(*normal)) 328 | 329 | if mesh.texcoords is not None: 330 | for coords in mesh.texcoords: 331 | f.write("vt {} {}\n".format(*coords)) 332 | 333 | def _interleave_columns(*arrays): 334 | n_rows = len(arrays[0]) 335 | return np.dstack(arrays).reshape(n_rows, -1) 336 | 337 | faces = mesh.faces + 1 338 | texture_indices = (mesh.texture_indices + 1 339 | if mesh.texture_indices is not None 340 | else None) 341 | normal_indices = (mesh.faces_normal_indices + 1 342 | if mesh.faces_normal_indices is not None 343 | else None) 344 | 345 | if texture_indices is None and normal_indices is None: 346 | for face in faces: 347 | f.write("f {} {} {}".format(*face)) 348 | elif normal_indices is None: 349 | indices = _interleave_columns(faces, texture_indices) 350 | for indices_ in indices: 351 | f.write("f {}/{} {}/{} {}/{}\n".format(*indices_)) 352 | elif texture_indices is None: 353 | indices = _interleave_columns(faces, normal_indices) 354 | for indices_ in indices: 355 | f.write("f {}//{} {}//{} {}//{}\n".format(*indices_)) 356 | 357 | 358 | def astype_or_none(array, type_): 359 | if array is None: 360 | return None 361 | return array.astype(type_) 362 | 363 | 364 | def load_npz(path): 365 | data = np.load(path) 366 | 367 | assert data.f.texture.dtype == np.uint8 368 | texture = data.f.texture.astype(float) / 255 369 | 370 | return Mesh( 371 | path=path, 372 | vertices=data.f.vertices.astype(float), 373 | faces=data.f.faces.astype(int), 374 | texcoords=data.f.texcoords.astype(float), 375 | texture_indices=data.f.texcoords_indices.astype(int), 376 | texture=texture, 377 | ) 378 | 379 | 380 | def save_npz(path, mesh): 381 | np.savez_compressed( 382 | path, 383 | vertices=mesh.vertices.astype("float32"), 384 | faces=mesh.faces.astype("uint32"), 385 | texcoords=astype_or_none(mesh.texcoords, "float32"), 386 | texcoords_indices=astype_or_none(mesh.texture_indices, "uint32"), 387 | texture=((255 * mesh.texture).astype("uint8") 388 | if mesh.texture is not None 389 | else None), 390 | mask_faces=astype_or_none(mesh.mask_faces, "uint32"), 391 | vertex_colors=astype_or_none(mesh.vertex_colors, "float32"), 392 | ) 393 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/sharp/landmarks.py: -------------------------------------------------------------------------------- 1 | NAMES = [ 2 | 'nose', 3 | 'neck', 4 | 'shoulder_right', 5 | 'elbow_right', 6 | 'wrist_right', 7 | 'shoulder_left', 8 | 'elbow_left', 9 | 'wrist_left', 10 | 'hip_middle', 11 | 'hip_right', 12 | 'knee_right', 13 | 'ankle_right', 14 | 'hip_left', 15 | 'knee_left', 16 | 'ankle_left', 17 | 'eye_right', 18 | 'eye_left', 19 | 'ear_right', 20 | 'ear_left', 21 | 'toe_1_left', 22 | 'toe_5_left', 23 | 'heel_left', 24 | 'toe_1_right', 25 | 'toe_5_right', 26 | 'heel_right', 27 | 'hand_base_left', 28 | 'finger_thumb_base_left', 29 | 'finger_thumb_middle_left', 30 | 'finger_thumb_top_left', 31 | 'finger_thumb_tip_left', 32 | 'finger_index_base_left', 33 | 'finger_index_middle_left', 34 | 'finger_index_top_left', 35 | 'finger_index_tip_left', 36 | 'finger_middle_base_left', 37 | 'finger_middle_middle_left', 38 | 'finger_middle_top_left', 39 | 'finger_middle_tip_left', 40 | 'finger_ring_base_left', 41 | 'finger_ring_middle_left', 42 | 'finger_ring_top_left', 43 | 'finger_ring_tip_left', 44 | 'finger_baby_base_left', 45 | 'finger_baby_middle_left', 46 | 'finger_baby_top_left', 47 | 'finger_baby_tip_left', 48 | 'hand_base_right', 49 | 'finger_thumb_base_right', 50 | 'finger_thumb_middle_right', 51 | 'finger_thumb_top_right', 52 | 'finger_thumb_tip_right', 53 | 'finger_index_base_right', 54 | 'finger_index_middle_right', 55 | 'finger_index_top_right', 56 | 'finger_index_tip_right', 57 | 'finger_middle_base_right', 58 | 'finger_middle_middle_right', 59 | 'finger_middle_top_right', 60 | 'finger_middle_tip_right', 61 | 'finger_ring_base_right', 62 | 'finger_ring_middle_right', 63 | 'finger_ring_top_right', 64 | 'finger_ring_tip_right', 65 | 'finger_baby_base_right', 66 | 'finger_baby_middle_right', 67 | 'finger_baby_top_right', 68 | 'finger_baby_tip_right', 69 | 'finger_mean_base_knuckle_left', 70 | 'finger_mean_base_knuckle_right', 71 | ] 72 | 73 | 74 | NAMES_HAND_RIGHT = [ 75 | 'wrist_right', 76 | 'hand_base_right', 77 | 'finger_thumb_base_right', 78 | 'finger_thumb_middle_right', 79 | 'finger_thumb_top_right', 80 | 'finger_thumb_tip_right', 81 | 'finger_index_base_right', 82 | 'finger_index_middle_right', 83 | 'finger_index_top_right', 84 | 'finger_index_tip_right', 85 | 'finger_middle_base_right', 86 | 'finger_middle_middle_right', 87 | 'finger_middle_top_right', 88 | 'finger_middle_tip_right', 89 | 'finger_ring_base_right', 90 | 'finger_ring_middle_right', 91 | 'finger_ring_top_right', 92 | 'finger_ring_tip_right', 93 | 'finger_baby_base_right', 94 | 'finger_baby_middle_right', 95 | 'finger_baby_top_right', 96 | 'finger_baby_tip_right', 97 | 'finger_mean_base_knuckle_right', 98 | 'finger_mean_base_knuckle_right', 99 | ] 100 | 101 | 102 | NAMES_HAND_LEFT = [ 103 | 'wrist_left', 104 | 'hand_base_left', 105 | 'finger_thumb_base_left', 106 | 'finger_thumb_middle_left', 107 | 'finger_thumb_top_left', 108 | 'finger_thumb_tip_left', 109 | 'finger_index_base_left', 110 | 'finger_index_middle_left', 111 | 'finger_index_top_left', 112 | 'finger_index_tip_left', 113 | 'finger_middle_base_left', 114 | 'finger_middle_middle_left', 115 | 'finger_middle_top_left', 116 | 'finger_middle_tip_left', 117 | 'finger_ring_base_left', 118 | 'finger_ring_middle_left', 119 | 'finger_ring_top_left', 120 | 'finger_ring_tip_left', 121 | 'finger_baby_base_left', 122 | 'finger_baby_middle_left', 123 | 'finger_baby_top_left', 124 | 'finger_baby_tip_left', 125 | 'finger_mean_base_knuckle_left', 126 | 'finger_mean_base_knuckle_left', 127 | ] 128 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/sharp/linalg.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | 4 | def squared_norm(x, axis=None, keepdims=False): 5 | """Squared Euclidean norm of an array.""" 6 | return (x ** 2).sum(axis=axis, keepdims=keepdims) 7 | 8 | 9 | def norm(x, axis=None, keepdims=False): 10 | """Euclidean norm of an array.""" 11 | return np.sqrt(squared_norm(x, axis=axis, keepdims=keepdims)) 12 | 13 | 14 | def normed(x, axis=None, keepdims=False): 15 | """Normalize an array.""" 16 | eps = np.finfo(x.dtype).eps 17 | return x / (norm(x, axis=axis, keepdims=True) + eps) 18 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/sharp/trirender.py: -------------------------------------------------------------------------------- 1 | from typing import Any, Tuple 2 | import numpy as np 3 | from nptyping import NDArray 4 | import moderngl as MGL 5 | 6 | 7 | VERTEX_SHADER = """ 8 | #version 330 9 | 10 | in vec2 point_uv; 11 | out vec2 fragment_uv; 12 | 13 | void main() { 14 | gl_Position = vec4(point_uv.x * 2.0 - 1.0, point_uv.y * 2.0 - 1.0, 0.0, 1.0); 15 | fragment_uv = point_uv; 16 | } 17 | """ 18 | 19 | FRAGMENT_SHADER = """ 20 | #version 330 21 | 22 | in vec2 fragment_uv; 23 | out vec4 fragment_color; 24 | 25 | uniform sampler2D texture_color; 26 | 27 | void main() { 28 | vec3 rgb = texture(texture_color, fragment_uv).rgb; 29 | fragment_color = vec4(rgb, 1.0); 30 | } 31 | 32 | """ 33 | 34 | 35 | class UVTrianglesRenderer: 36 | def __init__(self, ctx: MGL.Context, output_size: Tuple[int, int]): 37 | self.ctx = ctx 38 | self.output_size = output_size 39 | self.shader = self.ctx.program( 40 | vertex_shader=VERTEX_SHADER, fragment_shader=FRAGMENT_SHADER 41 | ) 42 | self.fbo = self.ctx.framebuffer( 43 | self.ctx.renderbuffer(self.output_size, dtype="f4") 44 | ) 45 | self.background_color = (0, 0, 0) 46 | 47 | def __del__(self): 48 | self.ctx.release() 49 | self.fbo.release() 50 | self.shader.release() 51 | 52 | @staticmethod 53 | def with_standalone_ctx(output_size: Tuple[int, int]) -> "UVTrianglesRenderer": 54 | return UVTrianglesRenderer( 55 | MGL.create_standalone_context(require=330), output_size 56 | ) 57 | 58 | @staticmethod 59 | def with_window_ctx(output_size: Tuple[int, int]) -> "UVTrianglesRenderer": 60 | return UVTrianglesRenderer(MGL.create_context(require=330), output_size) 61 | 62 | def _init_ctx_object(self, tex_coords, tri_indices, texture): 63 | resources = [] 64 | tex_coords_buffer = self.ctx.buffer(tex_coords.astype("f4").tobytes()) 65 | resources.append(tex_coords_buffer) 66 | tri_indices_buffer = self.ctx.buffer(tri_indices.astype("i4").tobytes()) 67 | resources.append(tri_indices_buffer) 68 | 69 | texture_height = texture.shape[0] 70 | texture_width = texture.shape[1] 71 | if len(texture.shape) == 3: 72 | components = texture.shape[2] 73 | else: 74 | components = 1 75 | 76 | texture_object = self.ctx.texture( 77 | (texture_width, texture_height), 78 | components, 79 | texture.astype("f4").tobytes(), 80 | dtype="f4", 81 | ) 82 | 83 | resources.append(texture_object) 84 | 85 | content = (tex_coords_buffer, "2f4", "point_uv") 86 | self.shader["texture_color"] = 0 87 | texture_object.use(0) 88 | vao = self.ctx.vertex_array(self.shader, [content], tri_indices_buffer) 89 | resources.append(vao) 90 | 91 | return vao, resources 92 | 93 | def _render(self, vao): 94 | self.fbo.use() 95 | self.ctx.clear(*self.background_color) 96 | vao.render() 97 | self.ctx.finish() 98 | 99 | def _get_fbo_image(self): 100 | fbo_image = self.fbo.read(dtype="f4") 101 | fbo_image = np.frombuffer(fbo_image, dtype="f4").reshape( 102 | self.output_size[1], self.output_size[0], 3 103 | ) 104 | fbo_image = np.flipud(fbo_image) 105 | return fbo_image 106 | 107 | def render( 108 | self, 109 | tex_coords: NDArray[(Any, 2), float], 110 | tri_indices: NDArray[(Any, 3), int], 111 | texture: NDArray[(Any, Any, 3), float], 112 | flip_y: bool = True, 113 | ) -> NDArray[(Any, Any, 3), float]: 114 | assert isinstance(tex_coords, NDArray[(Any, 2), float]) 115 | assert isinstance(tri_indices, NDArray[(Any, 3), int]) 116 | assert isinstance(texture, NDArray[(Any, Any, 3), float]) 117 | 118 | if flip_y: 119 | texture = np.flipud(texture) 120 | 121 | resources = [] 122 | 123 | try: 124 | vao, resources = self._init_ctx_object(tex_coords, tri_indices, texture) 125 | self._render(vao) 126 | result = self._get_fbo_image() 127 | 128 | return result 129 | 130 | finally: 131 | for resource in resources: 132 | resource.release() 133 | -------------------------------------------------------------------------------- /data_processing/eccv2020-sharp-workshop/sharp/utils.py: -------------------------------------------------------------------------------- 1 | import copy 2 | import numbers 3 | 4 | import cv2 5 | import numpy as np 6 | try: 7 | from scipy.spatial import cKDTree as KDTree 8 | except ImportError: 9 | from scipy.spatial import KDTree 10 | 11 | from .trirender import UVTrianglesRenderer 12 | 13 | 14 | def slice_by_plane(mesh, center, n): 15 | c = np.dot(center, n) 16 | plane_side = lambda x: np.dot(x, n) >= c 17 | split = np.asarray([plane_side(v) for v in mesh.vertices]) 18 | slice1_indices = np.argwhere(split == True) 19 | slice2_indices = np.argwhere(split == False) 20 | return slice1_indices, slice2_indices 21 | 22 | 23 | def remove_points(mesh, indices, blackoutTexture=True): 24 | cpy = copy.deepcopy(mesh) 25 | cpy.vertices = np.delete(mesh.vertices, indices, axis=0) 26 | if mesh.vertex_colors is not None: 27 | cpy.vertex_colors = np.delete(mesh.vertex_colors, indices, axis=0) 28 | if mesh.vertex_normals is not None: 29 | cpy.vertex_normals = np.delete( 30 | mesh.vertex_normals, indices, axis=0) 31 | 32 | if mesh.faces is not None: 33 | face_indices = np.where( 34 | np.any(np.isin(mesh.faces[:], indices, assume_unique=False), 35 | axis=1) 36 | )[0] 37 | cpy.faces = np.delete(mesh.faces, face_indices, axis=0) 38 | fix_indices = np.vectorize( 39 | lambda x: np.sum(x >= indices))(cpy.faces) 40 | cpy.faces -= fix_indices 41 | 42 | if mesh.face_normals is not None: 43 | cpy.face_normals = np.delete( 44 | mesh.face_normals, face_indices, axis=0) 45 | unused_uv = None 46 | if mesh.texture_indices is not None: 47 | cpy.texture_indices = np.delete( 48 | mesh.texture_indices, face_indices, axis=0) 49 | used_uv = np.unique(cpy.texture_indices.flatten()) 50 | all_uv = np.arange(len(mesh.texcoords)) 51 | unused_uv = np.setdiff1d(all_uv, used_uv, assume_unique=True) 52 | fix_uv_idx = np.vectorize( 53 | lambda x: np.sum(x >= unused_uv))(cpy.texture_indices) 54 | cpy.texture_indices -= fix_uv_idx 55 | cpy.texcoords = np.delete(mesh.texcoords, unused_uv, axis=0) 56 | 57 | # render texture 58 | if blackoutTexture: 59 | tri_indices = cpy.texture_indices 60 | tex_coords = cpy.texcoords 61 | img = render_texture(mesh.texture, tex_coords, tri_indices) 62 | # dilate the result to remove sewing 63 | kernel = np.ones((3, 3), np.uint8) 64 | texture_f32 = cv2.dilate(img, kernel, iterations=1) 65 | cpy.texture = texture_f32.astype(np.float64) 66 | 67 | if mesh.faces_normal_indices is not None: 68 | cpy.faces_normal_indices = np.delete( 69 | mesh.faces_normal_indices, face_indices, axis=0) 70 | used_ni = np.unique(cpy.faces_normal_indices.flatten()) 71 | all_ni = np.arange(len(mesh.face_normals)) 72 | unused_ni = np.setdiff1d(all_ni, used_ni, assume_unique=True) 73 | fix_ni_idx = np.vectorize(lambda x: np.sum( 74 | x > unused_ni))(cpy.faces_normal_indices) 75 | cpy.faces_normal_indices -= fix_ni_idx 76 | cpy.face_normals = np.delete( 77 | mesh.face_normals, unused_ni, axis=0) 78 | 79 | return cpy 80 | 81 | 82 | def render_texture(texture, tex_coords, tri_indices): 83 | if len(texture.shape) == 3 and texture.shape[2] == 4: 84 | texture = texture[:, :, 0:3] 85 | elif len(texture.shape) == 2: 86 | texture = np.concatenate([texture, texture, texture], axis=2) 87 | 88 | renderer = UVTrianglesRenderer.with_standalone_ctx( 89 | (texture.shape[1], texture.shape[0]) 90 | ) 91 | 92 | return renderer.render(tex_coords, tri_indices, texture, True) 93 | 94 | 95 | def estimate_plane(a, b, c): 96 | """Estimate the parameters of the plane passing by three points. 97 | 98 | Returns: 99 | center(float): The center point of the three input points. 100 | normal(float): The normal to the plane. 101 | """ 102 | center = (a + b + c) / 3 103 | normal = np.cross(b - a, c - a) 104 | assert(np.isclose(np.dot(b - a, normal), np.dot(c - a, normal))) 105 | return center, normal 106 | 107 | 108 | def shoot_holes(vertices, n_holes, dropout, mask_faces=None, faces=None, 109 | rng=None): 110 | """Generate a partial shape by cutting holes of random location and size. 111 | 112 | Each hole is created by selecting a random point as the center and removing 113 | the k nearest-neighboring points around it. 114 | 115 | Args: 116 | vertices: The array of vertices of the mesh. 117 | n_holes (int or (int, int)): Number of holes to create, or bounds from 118 | which to randomly draw the number of holes. 119 | dropout (float or (float, float)): Proportion of points (with respect 120 | to the total number of points) in each hole, or bounds from which 121 | to randomly draw the proportions (a different proportion is drawn 122 | for each hole). 123 | mask_faces: A boolean mask on the faces. 1 to keep, 0 to ignore. If 124 | set, the centers of the holes are sampled only on the 125 | non-masked regions. 126 | faces: The array of faces of the mesh. Required only when `mask_faces` 127 | is set. 128 | rng: (optional) An initialised np.random.Generator object. If None, a 129 | default Generator is created. 130 | 131 | Returns: 132 | array: Indices of the points defining the holes. 133 | """ 134 | if rng is None: 135 | rng = np.random.default_rng() 136 | 137 | if not isinstance(n_holes, numbers.Integral): 138 | n_holes_min, n_holes_max = n_holes 139 | n_holes = rng.integers(n_holes_min, n_holes_max) 140 | 141 | if mask_faces is not None: 142 | valid_vertex_indices = np.unique(faces[mask_faces > 0]) 143 | valid_vertices = vertices[valid_vertex_indices] 144 | else: 145 | valid_vertices = vertices 146 | 147 | # Select random hole centers. 148 | center_indices = rng.choice(len(valid_vertices), size=n_holes) 149 | centers = valid_vertices[center_indices] 150 | 151 | n_vertices = len(valid_vertices) 152 | if isinstance(dropout, numbers.Number): 153 | hole_size = n_vertices * dropout 154 | hole_sizes = [hole_size] * n_holes 155 | else: 156 | hole_size_bounds = n_vertices * np.asarray(dropout) 157 | hole_sizes = rng.integers(*hole_size_bounds, size=n_holes) 158 | 159 | # Identify the points indices making up the holes. 160 | kdtree = KDTree(vertices, leafsize=200) 161 | to_crop = [] 162 | for center, size in zip(centers, hole_sizes): 163 | _, indices = kdtree.query(center, k=size) 164 | to_crop.append(indices) 165 | to_crop = np.unique(np.concatenate(to_crop)) 166 | 167 | return to_crop 168 | -------------------------------------------------------------------------------- /data_processing/sample_RGB_GT.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import trimesh 3 | from glob import glob 4 | import os 5 | import multiprocessing as mp 6 | from multiprocessing import Pool 7 | import argparse 8 | import config.config_loader as cfg_loader 9 | import utils 10 | import traceback 11 | import tqdm 12 | 13 | 14 | 15 | 16 | def sample_colors(gt_mesh_path): 17 | try: 18 | path = os.path.normpath(gt_mesh_path) 19 | challange = path.split(os.sep)[-4] 20 | split = path.split(os.sep)[-3] 21 | gt_file_name = path.split(os.sep)[-2] 22 | full_file_name = path.split(os.sep)[-1][:-4] 23 | 24 | out_file = cfg['data_path'] + '/{}/{}/{}_color_samples{}_bbox{}.npz' \ 25 | .format(split, gt_file_name, full_file_name, num_points, cfg['data_bounding_box_str']) 26 | 27 | if os.path.exists(out_file): 28 | print('File exists. Done.') 29 | return 30 | 31 | gt_mesh = utils.as_mesh(trimesh.load(gt_mesh_path)) 32 | sample_points, face_idxs = gt_mesh.sample(num_points, return_index = True) 33 | 34 | triangles = gt_mesh.triangles[face_idxs] 35 | face_vertices = gt_mesh.faces[face_idxs] 36 | faces_uvs = gt_mesh.visual.uv[face_vertices] 37 | 38 | q = triangles[:, 0] 39 | u = triangles[:, 1] 40 | v = triangles[:, 2] 41 | 42 | uvs = [] 43 | 44 | for i, p in enumerate(sample_points): 45 | barycentric_weights = utils.barycentric_coordinates(p, q[i], u[i], v[i]) 46 | uv = np.average(faces_uvs[i], 0, barycentric_weights) 47 | uvs.append(uv) 48 | 49 | texture = gt_mesh.visual.material.image 50 | 51 | colors = trimesh.visual.color.uv_to_color(np.array(uvs), texture) 52 | 53 | np.savez(out_file, points = sample_points, grid_coords = utils.to_grid_sample_coords(sample_points, bbox), colors = colors[:,:3]) 54 | except Exception as err: 55 | print('Error with {}: {}'.format(out_file, traceback.format_exc())) 56 | 57 | if __name__ == '__main__': 58 | parser = argparse.ArgumentParser( 59 | description='Run color sampling. Samples surface points on the GT objects, and saves their coordinates along with the RGB color at their location.' 60 | ) 61 | 62 | parser.add_argument('config', type=str, help='Path to config file.') 63 | args = parser.parse_args() 64 | 65 | cfg = cfg_loader.load(args.config) 66 | 67 | num_points = cfg['preprocessing']['color_sampling']['sample_number'] 68 | bbox = cfg['data_bounding_box'] 69 | 70 | print('Fining all gt object paths for point and RGB sampling.') 71 | paths = glob(cfg['data_path'] + cfg['preprocessing']['color_sampling']['input_files_regex']) 72 | 73 | print('Start sampling.') 74 | p = Pool(mp.cpu_count()) 75 | for _ in tqdm.tqdm(p.imap_unordered(sample_colors, paths), total=len(paths)): 76 | pass 77 | p.close() 78 | p.join() 79 | -------------------------------------------------------------------------------- /data_processing/sample_voxelized_colored_pointcloud.py: -------------------------------------------------------------------------------- 1 | import utils 2 | from scipy.spatial import cKDTree as KDTree 3 | import numpy as np 4 | import trimesh 5 | from glob import glob 6 | import os 7 | import multiprocessing as mp 8 | from multiprocessing import Pool 9 | import argparse 10 | import random 11 | import config.config_loader as cfg_loader 12 | import traceback 13 | import tqdm 14 | import utils 15 | 16 | 17 | def voxelized_colored_pointcloud_sampling(partial_mesh_path): 18 | try: 19 | path = os.path.normpath(partial_mesh_path) 20 | gt_file_name = path.split(os.sep)[-2] 21 | full_file_name = path.split(os.sep)[-1][:-4] 22 | 23 | out_file = os.path.dirname(partial_mesh_path) + '/{}_voxelized_colored_point_cloud_res{}_points{}_bbox{}.npz'\ 24 | .format(full_file_name, res, num_points, bbox_str) 25 | 26 | if os.path.exists(out_file): 27 | print('File exists. Done.') 28 | return 29 | 30 | # color from partial input 31 | partial_mesh = utils.as_mesh(trimesh.load(partial_mesh_path)) 32 | colored_point_cloud, face_idxs = partial_mesh.sample(num_points, return_index = True) 33 | 34 | triangles = partial_mesh.triangles[face_idxs] 35 | face_vertices = partial_mesh.faces[face_idxs] 36 | faces_uvs = partial_mesh.visual.uv[face_vertices] 37 | 38 | q = triangles[:, 0] 39 | u = triangles[:, 1] 40 | v = triangles[:, 2] 41 | 42 | uvs = [] 43 | 44 | for i, p in enumerate(colored_point_cloud): 45 | barycentric_weights = utils.barycentric_coordinates(p, q[i], u[i], v[i]) 46 | uv = np.average(faces_uvs[i], 0, barycentric_weights) 47 | uvs.append(uv) 48 | 49 | partial_texture = partial_mesh.visual.material.image 50 | 51 | colors = trimesh.visual.color.uv_to_color(np.array(uvs), partial_texture) 52 | 53 | R = - 1 * np.ones(len(grid_points), dtype=np.int16) 54 | G = - 1 * np.ones(len(grid_points), dtype=np.int16) 55 | B = - 1 * np.ones(len(grid_points), dtype=np.int16) 56 | 57 | _, idx = kdtree.query(colored_point_cloud) 58 | R[idx] = colors[:,0] 59 | G[idx] = colors[:,1] 60 | B[idx] = colors[:,2] 61 | 62 | # encode uncolorized, complete shape of object (at inference time obtained from IF-Nets surface reconstruction) 63 | # encoding is done by sampling a pointcloud and voxelizing it (into discrete grid for 3D CNN usage) 64 | full_shape = trimesh.load(os.path.join(os.path.dirname(partial_mesh_path) , gt_file_name +'_normalized.obj')) 65 | shape_point_cloud = full_shape.sample(num_points) 66 | S = np.zeros(len(grid_points), dtype=np.int8) 67 | 68 | _, idx = kdtree.query(shape_point_cloud) 69 | S[idx] = 1 70 | 71 | np.savez(out_file, R=R, G=G,B=B, S=S, colored_point_cloud=colored_point_cloud, bbox = bbox, res = res) 72 | 73 | except Exception as err: 74 | print('Error with {}: {}'.format(partial_mesh_path, traceback.format_exc())) 75 | 76 | if __name__ == '__main__': 77 | parser = argparse.ArgumentParser( 78 | description='Generates the input for the network: a partial colored shape and a uncolorized, but completed shape. \ 79 | Both encoded as 3D voxel grids for usage with a 3D CNN.' 80 | ) 81 | 82 | parser.add_argument('config', type=str, help='Path to config file.') 83 | args = parser.parse_args() 84 | 85 | cfg = cfg_loader.load(args.config) 86 | 87 | # shorthands 88 | bbox = cfg['data_bounding_box'] 89 | res = cfg['input_resolution'] 90 | num_points = cfg['input_points_number'] 91 | bbox_str = cfg['data_bounding_box_str'] 92 | 93 | grid_points = utils.create_grid_points_from_xyz_bounds(*bbox, res) 94 | kdtree = KDTree(grid_points) 95 | 96 | print('Fining all input partial paths for voxelization.') 97 | paths = glob(cfg['data_path'] + cfg['preprocessing']['voxelized_colored_pointcloud_sampling']['input_files_regex']) 98 | 99 | print('Start voxelization.') 100 | p = Pool(mp.cpu_count()) 101 | for _ in tqdm.tqdm(p.imap_unordered(voxelized_colored_pointcloud_sampling, paths), total=len(paths)): 102 | pass 103 | p.close() 104 | p.join() 105 | -------------------------------------------------------------------------------- /data_processing/utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import trimesh 3 | 4 | def create_grid_points_from_xyz_bounds(min_x, max_x, min_y, max_y ,min_z, max_z, res): 5 | x = np.linspace(min_x, max_x, res) 6 | y = np.linspace(min_y, max_y, res) 7 | z = np.linspace(min_z, max_z, res) 8 | X, Y, Z = np.meshgrid(x, y, z, indexing='ij', sparse=False) 9 | X = X.reshape((np.prod(X.shape),)) 10 | Y = Y.reshape((np.prod(Y.shape),)) 11 | Z = Z.reshape((np.prod(Z.shape),)) 12 | 13 | points_list = np.column_stack((X, Y, Z)) 14 | del X, Y, Z, x 15 | return points_list 16 | 17 | 18 | # conversion of points into coordinates understood by pytorch's grid_sample function 19 | # bbox = [min_x,max_x,min_y,max_y,min_z,max_z] 20 | def to_grid_sample_coords(points, bbox): 21 | min_values, max_values = bbox[::2], bbox[1::2] 22 | points = 2 * 1 / (max_values - min_values) * (points - min_values) - 1 23 | grid_coords = points.copy() 24 | grid_coords[:, 0], grid_coords[:, 2] = points[:, 2], points[:, 0] 25 | return grid_coords 26 | 27 | def barycentric_coordinates(p, q, u, v): 28 | """ 29 | Calculate barycentric coordinates of the given point 30 | :param p: a given point 31 | :param q: triangle vertex 32 | :param u: triangle vertex 33 | :param v: triangle vertex 34 | :return: 1X3 ndarray with the barycentric coordinates of p 35 | """ 36 | v0 = u - q 37 | v1 = v - q 38 | v2 = p - q 39 | d00 = v0.dot(v0) 40 | d01 = v0.dot(v1) 41 | d11 = v1.dot(v1) 42 | d20 = v2.dot(v0) 43 | d21 = v2.dot(v1) 44 | denom = d00 * d11 - d01 * d01 45 | y = (d11 * d20 - d01 * d21) / denom 46 | z = (d00 * d21 - d01 * d20) / denom 47 | x = 1.0 - z - y 48 | return np.array([x, y, z]) 49 | 50 | def as_mesh(scene_or_mesh): 51 | if isinstance(scene_or_mesh, trimesh.Scene): 52 | mesh = trimesh.util.concatenate([ 53 | trimesh.Trimesh(vertices=m.vertices, faces=m.faces) 54 | for m in scene_or_mesh.geometry.values()]) 55 | else: 56 | mesh = scene_or_mesh 57 | return mesh -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | name: TexIF-Net 2 | channels: 3 | - pytorch 4 | - anaconda 5 | - conda-forge 6 | - defaults 7 | dependencies: 8 | - _libgcc_mutex=0.1=main 9 | - bzip2=1.0.8=h516909a_3 10 | - ca-certificates=2020.7.22=0 11 | - cairo=1.16.0=h18b612c_1001 12 | - certifi=2020.6.20=py36_0 13 | - cudatoolkit=9.2=0 14 | - dbus=1.13.6=he372182_0 15 | - expat=2.2.9=he1b5a44_2 16 | - ffmpeg=4.1.3=h167e202_0 17 | - fontconfig=2.13.1=he4413a7_1000 18 | - freetype=2.10.2=he06d7ca_0 19 | - giflib=5.2.1=h516909a_2 20 | - glib=2.65.0=h3eb4bd4_0 21 | - gmp=6.2.0=he1b5a44_2 22 | - gnutls=3.6.13=h79a8f9a_0 23 | - graphite2=1.3.13=he1b5a44_1001 24 | - gst-plugins-base=1.14.0=hbbd80ab_1 25 | - gstreamer=1.14.0=hb31296c_0 26 | - harfbuzz=2.4.0=h37c48d4_1 27 | - hdf5=1.10.5=nompi_h3c11f04_1104 28 | - icu=58.2=hf484d3e_1000 29 | - intel-openmp=2020.2=254 30 | - jasper=1.900.1=h07fcdf6_1006 31 | - jpeg=9d=h516909a_0 32 | - lame=3.100=h14c3975_1001 33 | - lcms2=2.11=h396b838_0 34 | - ld_impl_linux-64=2.33.1=h53a641e_7 35 | - libblas=3.8.0=17_openblas 36 | - libcblas=3.8.0=17_openblas 37 | - libedit=3.1.20191231=h14c3975_1 38 | - libffi=3.3=he6710b0_2 39 | - libgcc-ng=9.1.0=hdf63c60_0 40 | - libgfortran-ng=7.5.0=hdf63c60_16 41 | - libiconv=1.16=h516909a_0 42 | - liblapack=3.8.0=17_openblas 43 | - liblapacke=3.8.0=17_openblas 44 | - libopenblas=0.3.10=pthreads_hb3c22a3_4 45 | - libpng=1.6.37=hed695b0_2 46 | - libstdcxx-ng=9.1.0=hdf63c60_0 47 | - libtiff=4.1.0=hc3755c2_3 48 | - libuuid=2.32.1=h14c3975_1000 49 | - libwebp=1.0.2=h56121f0_5 50 | - libxcb=1.13=h14c3975_1002 51 | - libxml2=2.9.9=h13577e0_2 52 | - lz4-c=1.9.2=he1b5a44_3 53 | - mkl=2020.2=256 54 | - ncurses=6.2=he6710b0_1 55 | - nettle=3.4.1=h1bed415_1002 56 | - ninja=1.10.1=py36hfd86e86_0 57 | - olefile=0.46=py36_0 58 | - opencv=4.1.1=py36ha799480_1 59 | - openh264=1.8.0=hdbcaa40_1000 60 | - openssl=1.1.1h=h7b6447c_0 61 | - pcre=8.44=he1b5a44_0 62 | - pillow=7.2.0=py36hb39fc2d_0 63 | - pip=20.2.2=py36_0 64 | - pixman=0.38.0=h516909a_1003 65 | - pthread-stubs=0.4=h14c3975_1001 66 | - python=3.6.12=hcff3b4d_2 67 | - python_abi=3.6=1_cp36m 68 | - pytorch=1.4.0=py3.6_cuda9.2.148_cudnn7.6.3_0 69 | - qt=5.9.7=h5867ecd_1 70 | - readline=8.0=h7b6447c_0 71 | - setuptools=49.6.0=py36_0 72 | - six=1.15.0=py_0 73 | - sqlite=3.33.0=h62c20be_0 74 | - tk=8.6.10=hbc83047_0 75 | - torchvision=0.5.0=py36_cu92 76 | - tqdm=4.49.0=pyh9f0ad1d_0 77 | - wheel=0.35.1=py_0 78 | - x264=1!152.20180806=h14c3975_0 79 | - xorg-kbproto=1.0.7=h14c3975_1002 80 | - xorg-libice=1.0.10=h516909a_0 81 | - xorg-libsm=1.2.3=h84519dc_1000 82 | - xorg-libx11=1.6.12=h516909a_0 83 | - xorg-libxau=1.0.9=h14c3975_0 84 | - xorg-libxdmcp=1.1.3=h516909a_0 85 | - xorg-libxext=1.3.4=h516909a_0 86 | - xorg-libxrender=0.9.10=h516909a_1002 87 | - xorg-renderproto=0.11.1=h14c3975_1002 88 | - xorg-xextproto=7.3.0=h14c3975_1002 89 | - xorg-xproto=7.0.31=h14c3975_1007 90 | - xz=5.2.5=h7b6447c_0 91 | - yaml=0.2.5=h7b6447c_0 92 | - zlib=1.2.11=h7b6447c_3 93 | - zstd=1.4.5=h6597ccf_2 94 | - pip: 95 | - argparse==1.4.0 96 | - glcontext==2.2.0 97 | - moderngl==5.6.2 98 | - nptyping==1.3.0 99 | - numpy==1.19.2 100 | - opencv-python==4.4.0.42 101 | - pathlib==1.0.1 102 | - pyyaml==5.3.1 103 | - scipy==1.5.2 104 | - sharp-cvi2==1.0.0 105 | - trimesh==3.8.8 106 | - typish==1.7.0 107 | prefix: /BS/chiban/work/anaconda3/envs/TexIF-Net 108 | 109 | -------------------------------------------------------------------------------- /generate.py: -------------------------------------------------------------------------------- 1 | import models.local_model as model 2 | import models.dataloader as dataloader 3 | import numpy as np 4 | import argparse 5 | from models.generation import Generator 6 | import config.config_loader as cfg_loader 7 | import os 8 | import trimesh 9 | import torch 10 | from data_processing import utils 11 | from tqdm import tqdm 12 | 13 | 14 | 15 | if __name__ == "__main__": 16 | parser = argparse.ArgumentParser( 17 | description='Generation Model' 18 | ) 19 | 20 | parser.add_argument('config', type=str, help='Path to config file.') 21 | args = parser.parse_args() 22 | 23 | cfg = cfg_loader.load(args.config) 24 | 25 | net = model.get_models()[cfg['model']]() 26 | 27 | dataloader = dataloader.VoxelizedDataset('test', cfg, generation = True, num_workers=0).get_loader() 28 | 29 | gen = Generator(net, cfg) 30 | 31 | 32 | out_path = 'experiments/{}/evaluation_{}/'.format(cfg['folder_name'], gen.checkpoint) 33 | 34 | 35 | for data in tqdm(dataloader): 36 | 37 | 38 | try: 39 | inputs = data['inputs'] 40 | path = data['path'][0] 41 | except: 42 | print('none') 43 | continue 44 | 45 | 46 | path = os.path.normpath(path) 47 | challange = path.split(os.sep)[-4] 48 | split = path.split(os.sep)[-3] 49 | gt_file_name = path.split(os.sep)[-2] 50 | basename = path.split(os.sep)[-1] 51 | filename_partial = os.path.splitext(path.split(os.sep)[-1])[0] 52 | 53 | file_out_path = out_path + '/{}/'.format(gt_file_name) 54 | os.makedirs(file_out_path, exist_ok=True) 55 | 56 | if os.path.exists(file_out_path + 'colored_surface_reconstuction.obj'): 57 | continue 58 | 59 | 60 | path_surface = os.path.join(cfg['data_path'], split, gt_file_name, gt_file_name + '_normalized.obj') 61 | 62 | mesh = trimesh.load(path_surface) 63 | 64 | # create new uncolored mesh for color prediction 65 | pred_mesh = trimesh.Trimesh(mesh.vertices, mesh.faces) 66 | 67 | # colors will be attached per vertex 68 | # subdivide in order to have high enough number of vertices for good texture representation 69 | pred_mesh = pred_mesh.subdivide().subdivide() 70 | 71 | pred_verts_gird_coords = utils.to_grid_sample_coords( pred_mesh.vertices, cfg['data_bounding_box']) 72 | pred_verts_gird_coords = torch.tensor(pred_verts_gird_coords).unsqueeze(0) 73 | 74 | 75 | colors_pred_surface = gen.generate_colors(inputs, pred_verts_gird_coords) 76 | 77 | # attach predicted colors to the mesh 78 | pred_mesh.visual.vertex_colors = colors_pred_surface 79 | 80 | pred_mesh.export( file_out_path + f'{filename_partial}_color_reconstruction.obj') -------------------------------------------------------------------------------- /if-net_env.yml: -------------------------------------------------------------------------------- 1 | name: tex_if-net 2 | channels: 3 | - conda-forge 4 | - pytorch 5 | - defaults 6 | dependencies: 7 | - trimesh=3.6.18 8 | - pytorch==1.1.0 9 | - cudatoolkit=9.0 10 | - tensorboard=1.14.0 11 | - numpy=1.17 12 | - pyyaml 13 | - pillow 14 | - tqdm -------------------------------------------------------------------------------- /models/__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /models/dataloader.py: -------------------------------------------------------------------------------- 1 | from __future__ import division 2 | from torch.utils.data import Dataset 3 | import os 4 | import numpy as np 5 | import pickle 6 | import imp 7 | import trimesh 8 | import torch 9 | 10 | 11 | 12 | class VoxelizedDataset(Dataset): 13 | 14 | 15 | def __init__(self, mode, cfg, generation = False, num_workers = 12): 16 | 17 | self.path = cfg['data_path'] 18 | self.mode = mode 19 | self.data = np.load(cfg['split_file'])[mode] 20 | self.res = cfg['input_resolution'] 21 | self.bbox_str = cfg['data_bounding_box_str'] 22 | self.bbox = cfg['data_bounding_box'] 23 | self.num_gt_rgb_samples = cfg['preprocessing']['color_sampling']['sample_number'] 24 | 25 | self.sample_points_per_object = cfg['training']['sample_points_per_object'] 26 | if generation: 27 | self.batch_size = 1 28 | else: 29 | self.batch_size = cfg['training']['batch_size'] 30 | self.num_workers = num_workers 31 | if cfg['input_type'] == 'voxels': 32 | self.voxelized_pointcloud = False 33 | else: 34 | self.voxelized_pointcloud = True 35 | self.pointcloud_samples = cfg['input_points_number'] 36 | 37 | 38 | 39 | def __len__(self): 40 | return len(self.data) 41 | 42 | def __getitem__(self, idx): 43 | path = self.data[idx] 44 | 45 | path = os.path.normpath(path) 46 | challange = path.split(os.sep)[-4] 47 | split = path.split(os.sep)[-3] 48 | gt_file_name = path.split(os.sep)[-2] 49 | full_file_name = os.path.splitext(path.split(os.sep)[-1])[0] 50 | 51 | voxel_path = os.path.join(self.path, split, gt_file_name,\ 52 | '{}_voxelized_colored_point_cloud_res{}_points{}_bbox{}.npz'\ 53 | .format(full_file_name, self.res, self.pointcloud_samples, self.bbox_str)) 54 | 55 | 56 | R = np.load(voxel_path)['R'] 57 | G = np.load(voxel_path)['G'] 58 | B = np.load(voxel_path)['B'] 59 | S = np.load(voxel_path)['S'] 60 | 61 | R = np.reshape(R, (self.res,)*3) 62 | G = np.reshape(G, (self.res,)*3) 63 | B = np.reshape(B, (self.res,)*3) 64 | S = np.reshape(S, (self.res,)*3) 65 | input = np.array([R,G,B,S]) 66 | 67 | if self.mode == 'test': 68 | return { 'inputs': np.array(input, dtype=np.float32), 'path' : path} 69 | 70 | 71 | rgb_samples_path = os.path.join(self.path, split, gt_file_name,\ 72 | '{}_normalized_color_samples{}_bbox{}.npz' \ 73 | .format(gt_file_name, self.num_gt_rgb_samples, self.bbox_str)) 74 | 75 | rgb_samples_npz = np.load(rgb_samples_path) 76 | rgb_coords = rgb_samples_npz['grid_coords'] 77 | rgb_values = rgb_samples_npz['colors'] 78 | subsample_indices = np.random.randint(0, len(rgb_values), self.sample_points_per_object) 79 | rgb_coords = rgb_coords[subsample_indices] 80 | rgb_values = rgb_values[subsample_indices] 81 | 82 | 83 | return {'grid_coords':np.array(rgb_coords, dtype=np.float32),'rgb': np.array(rgb_values, dtype=np.float32), 'inputs': np.array(input, dtype=np.float32), 'path' : path} 84 | 85 | def get_loader(self, shuffle =True): 86 | 87 | return torch.utils.data.DataLoader( 88 | self, batch_size=self.batch_size, num_workers=self.num_workers, shuffle=shuffle, 89 | worker_init_fn=self.worker_init_fn) 90 | 91 | def worker_init_fn(self, worker_id): 92 | random_data = os.urandom(4) 93 | base_seed = int.from_bytes(random_data, byteorder="big") 94 | np.random.seed(base_seed + worker_id) -------------------------------------------------------------------------------- /models/generation.py: -------------------------------------------------------------------------------- 1 | import data_processing.utils as utils 2 | import trimesh 3 | import torch 4 | import os 5 | from glob import glob 6 | import numpy as np 7 | 8 | class Generator(object): 9 | def __init__(self, model, cfg, device = torch.device("cuda")): 10 | self.model = model.to(device) 11 | self.model.eval() 12 | 13 | 14 | self.checkpoint_path = os.path.dirname(__file__) + '/../experiments/{}/checkpoints/'.format( 15 | cfg['folder_name']) 16 | self.exp_folder_name = cfg['folder_name'] 17 | self.checkpoint = self.load_checkpoint(cfg['generation']['checkpoint']) 18 | self.threshold = cfg['generation']['retrieval_threshold'] 19 | 20 | self.device = device 21 | self.resolution = cfg['generation']['retrieval_resolution'] 22 | self.batch_points = cfg['generation']['batch_points'] 23 | 24 | self.bbox = cfg['data_bounding_box'] 25 | self.min = self.bbox[::2] 26 | self.max = self.bbox[1::2] 27 | 28 | grid_points = utils.create_grid_points_from_xyz_bounds(*cfg['data_bounding_box'], self.resolution) 29 | 30 | grid_coords = utils.to_grid_sample_coords(grid_points, self.bbox) 31 | 32 | grid_coords = torch.from_numpy(grid_coords).to(self.device, dtype=torch.float) 33 | grid_coords = torch.reshape(grid_coords, (1, len(grid_points), 3)).to(self.device) 34 | self.grid_points_split = torch.split(grid_coords, self.batch_points, dim=1) 35 | 36 | 37 | 38 | def generate_colors(self, inputs,points): 39 | 40 | p = points.to(self.device).float() 41 | # p.shape is [1, n_verts, 3] 693016 -> > 21gb gram 42 | 43 | i = inputs.to(self.device).float() 44 | full_pred = [] 45 | 46 | p_batches = torch.split(p, 200000, dim=1) 47 | 48 | for p_batch in p_batches: 49 | with torch.no_grad(): 50 | pred_rgb = self.model(p_batch,i) 51 | full_pred.append(pred_rgb.squeeze(0).detach().cpu().transpose(0,1)) 52 | 53 | pred_rgb = torch.cat(full_pred, dim=0).numpy() 54 | pred_rgb.astype(np.int)[0] 55 | pred_rgb = np.clip(pred_rgb, 0, 255) 56 | 57 | return pred_rgb 58 | 59 | 60 | 61 | 62 | def load_checkpoint(self, checkpoint): 63 | if checkpoint == -1: 64 | val_min_npy = os.path.dirname(__file__) + '/../experiments/{}/val_min.npy'.format( 65 | self.exp_folder_name) 66 | checkpoint = int(np.load(val_min_npy)[0]) 67 | path = self.checkpoint_path + 'checkpoint_epoch_{}.tar'.format(checkpoint) 68 | else: 69 | path = self.checkpoint_path + 'checkpoint_epoch_{}.tar'.format(checkpoint) 70 | print('Loaded checkpoint from: {}'.format(path)) 71 | torch_checkpoint = torch.load(path) 72 | self.model.load_state_dict(torch_checkpoint['model_state_dict']) 73 | return checkpoint -------------------------------------------------------------------------------- /models/local_model.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | 5 | 6 | 7 | def get_models(): 8 | return {'ShapeNet32Vox': ShapeNet32Vox, 'ShapeNet128Vox': ShapeNet128Vox, 'ShapeNetPoints': ShapeNetPoints, 'TEXR':TEXR} 9 | 10 | 11 | # 1D convolution is used for the decoder. It acts as a standard FC, but allows to use a batch of point samples features, 12 | # additionally to the batch over the input objects. 13 | # The dimensions are used as follows: 14 | # batch_size (N) = #3D objects , channels = features, signal_lengt (L) (convolution dimension) = #point samples 15 | # kernel_size = 1 i.e. every convolution is done over only all features of one point sample, this makes it a FC. 16 | 17 | 18 | # ShapeNet Voxel Super-Resolution -------------------------------------------------------------------- 19 | # ---------------------------------------------------------------------------------------------------- 20 | class ShapeNet32Vox(nn.Module): 21 | 22 | def __init__(self, hidden_dim=256): 23 | super(ShapeNet32Vox, self).__init__() 24 | 25 | self.conv_1 = nn.Conv3d(1, 32, 3, padding=1) # out: 32 26 | self.conv_1_1 = nn.Conv3d(32, 64, 3, padding=1) # out: 32 27 | self.conv_2 = nn.Conv3d(64, 128, 3, padding=1) # out: 16 28 | self.conv_2_1 = nn.Conv3d(128, 128, 3, padding=1) # out: 16 29 | self.conv_3 = nn.Conv3d(128, 128, 3, padding=1) # out: 8 30 | self.conv_3_1 = nn.Conv3d(128, 128, 3, padding=1) # out: 8 31 | 32 | feature_size = (1 + 64 + 128 + 128 ) * 7 33 | self.fc_0 = nn.Conv1d(feature_size, hidden_dim*2, 1) 34 | self.fc_1 = nn.Conv1d(hidden_dim*2, hidden_dim, 1) 35 | self.fc_2 = nn.Conv1d(hidden_dim, hidden_dim, 1) 36 | self.fc_out = nn.Conv1d(hidden_dim, 1, 1) 37 | self.actvn = nn.ReLU() 38 | 39 | self.maxpool = nn.MaxPool3d(2) 40 | 41 | self.conv1_1_bn = nn.BatchNorm3d(64) 42 | self.conv2_1_bn = nn.BatchNorm3d(128) 43 | self.conv3_1_bn = nn.BatchNorm3d(128) 44 | 45 | 46 | displacment = 0.035 47 | displacments = [] 48 | displacments.append([0, 0, 0]) 49 | for x in range(3): 50 | for y in [-1, 1]: 51 | input = [0, 0, 0] 52 | input[x] = y * displacment 53 | displacments.append(input) 54 | 55 | self.displacments = torch.Tensor(displacments).cuda() 56 | 57 | def forward(self, p, x): 58 | x = x.unsqueeze(1) 59 | 60 | p_features = p.transpose(1, -1) 61 | p = p.unsqueeze(1).unsqueeze(1) 62 | p = torch.cat([p + d for d in self.displacments], dim=2) # (B,1,7,num_samples,3) 63 | feature_0 = F.grid_sample(x, p) # out : (B,C (of x), 1,1,sample_num) 64 | 65 | net = self.actvn(self.conv_1(x)) 66 | net = self.actvn(self.conv_1_1(net)) 67 | net = self.conv1_1_bn(net) 68 | feature_1 = F.grid_sample(net, p) # out : (B,C (of x), 1,1,sample_num) 69 | net = self.maxpool(net) 70 | 71 | net = self.actvn(self.conv_2(net)) 72 | net = self.actvn(self.conv_2_1(net)) 73 | net = self.conv2_1_bn(net) 74 | feature_2 = F.grid_sample(net, p) 75 | net = self.maxpool(net) 76 | 77 | net = self.actvn(self.conv_3(net)) 78 | net = self.actvn(self.conv_3_1(net)) 79 | net = self.conv3_1_bn(net) 80 | feature_3 = F.grid_sample(net, p) 81 | 82 | # here every channel corresponse to one feature. 83 | 84 | features = torch.cat((feature_0, feature_1, feature_2, feature_3), 85 | dim=1) # (B, features, 1,7,sample_num) 86 | shape = features.shape 87 | features = torch.reshape(features, 88 | (shape[0], shape[1] * shape[3], shape[4])) # (B, featues_per_sample, samples_num) 89 | #features = torch.cat((features, p_features), dim=1) # (B, featue_size, samples_num) 90 | 91 | net = self.actvn(self.fc_0(features)) 92 | net = self.actvn(self.fc_1(net)) 93 | net = self.actvn(self.fc_2(net)) 94 | net = self.fc_out(net) 95 | out = net.squeeze(1) 96 | 97 | return out 98 | 99 | class ShapeNet128Vox(nn.Module): 100 | 101 | def __init__(self, hidden_dim=256): 102 | super(ShapeNet128Vox, self).__init__() 103 | # accepts 128**3 res input 104 | self.conv_in = nn.Conv3d(1, 16, 3, padding=1) # out: 128 105 | self.conv_0 = nn.Conv3d(16, 32, 3, padding=1) # out: 64 106 | self.conv_0_1 = nn.Conv3d(32, 32, 3, padding=1) # out: 64 107 | self.conv_1 = nn.Conv3d(32, 64, 3, padding=1) # out: 32 108 | self.conv_1_1 = nn.Conv3d(64, 64, 3, padding=1) # out: 32 109 | self.conv_2 = nn.Conv3d(64, 128, 3, padding=1) # out: 16 110 | self.conv_2_1 = nn.Conv3d(128, 128, 3, padding=1) # out: 16 111 | self.conv_3 = nn.Conv3d(128, 128, 3, padding=1) # out: 8 112 | self.conv_3_1 = nn.Conv3d(128, 128, 3, padding=1) # out: 8 113 | 114 | feature_size = (1 + 16 + 32 + 64 + 128 + 128 ) * 7 115 | self.fc_0 = nn.Conv1d(feature_size, hidden_dim, 1) 116 | self.fc_1 = nn.Conv1d(hidden_dim, hidden_dim, 1) 117 | self.fc_2 = nn.Conv1d(hidden_dim, hidden_dim, 1) 118 | self.fc_out = nn.Conv1d(hidden_dim, 1, 1) 119 | self.actvn = nn.ReLU() 120 | 121 | self.maxpool = nn.MaxPool3d(2) 122 | 123 | self.conv_in_bn = nn.BatchNorm3d(16) 124 | self.conv0_1_bn = nn.BatchNorm3d(32) 125 | self.conv1_1_bn = nn.BatchNorm3d(64) 126 | self.conv2_1_bn = nn.BatchNorm3d(128) 127 | self.conv3_1_bn = nn.BatchNorm3d(128) 128 | 129 | 130 | displacment = 0.0722 131 | displacments = [] 132 | displacments.append([0, 0, 0]) 133 | for x in range(3): 134 | for y in [-1, 1]: 135 | input = [0, 0, 0] 136 | input[x] = y * displacment 137 | displacments.append(input) 138 | 139 | self.displacments = torch.Tensor(displacments).cuda() 140 | 141 | def forward(self, p, x): 142 | x = x.unsqueeze(1) 143 | 144 | p_features = p.transpose(1, -1) 145 | p = p.unsqueeze(1).unsqueeze(1) 146 | p = torch.cat([p + d for d in self.displacments], dim=2) # (B,1,7,num_samples,3) 147 | feature_0 = F.grid_sample(x, p) # out : (B,C (of x), 1,1,sample_num) 148 | 149 | net = self.actvn(self.conv_in(x)) 150 | net = self.conv_in_bn(net) 151 | feature_1 = F.grid_sample(net, p) # out : (B,C (of x), 1,1,sample_num) 152 | net = self.maxpool(net) 153 | 154 | net = self.actvn(self.conv_0(net)) 155 | net = self.actvn(self.conv_0_1(net)) 156 | net = self.conv0_1_bn(net) 157 | feature_2 = F.grid_sample(net, p) # out : (B,C (of x), 1,1,sample_num) 158 | net = self.maxpool(net) 159 | 160 | net = self.actvn(self.conv_1(net)) 161 | net = self.actvn(self.conv_1_1(net)) 162 | net = self.conv1_1_bn(net) 163 | feature_3 = F.grid_sample(net, p) # out : (B,C (of x), 1,1,sample_num) 164 | net = self.maxpool(net) 165 | 166 | net = self.actvn(self.conv_2(net)) 167 | net = self.actvn(self.conv_2_1(net)) 168 | net = self.conv2_1_bn(net) 169 | feature_4 = F.grid_sample(net, p) 170 | net = self.maxpool(net) 171 | 172 | net = self.actvn(self.conv_3(net)) 173 | net = self.actvn(self.conv_3_1(net)) 174 | net = self.conv3_1_bn(net) 175 | feature_5 = F.grid_sample(net, p) 176 | 177 | # here every channel corresponse to one feature. 178 | 179 | features = torch.cat((feature_0, feature_1, feature_2, feature_3, feature_4, feature_5), 180 | dim=1) # (B, features, 1,7,sample_num) 181 | shape = features.shape 182 | features = torch.reshape(features, 183 | (shape[0], shape[1] * shape[3], shape[4])) # (B, featues_per_sample, samples_num) 184 | #features = torch.cat((features, p_features), dim=1) # (B, featue_size, samples_num) 185 | 186 | net = self.actvn(self.fc_0(features)) 187 | net = self.actvn(self.fc_1(net)) 188 | net = self.actvn(self.fc_2(net)) 189 | net = self.fc_out(net) 190 | out = net.squeeze(1) 191 | 192 | return out 193 | 194 | 195 | 196 | 197 | # ShapeNet Pointcloud Completion --------------------------------------------------------------------- 198 | # ---------------------------------------------------------------------------------------------------- 199 | 200 | class ShapeNetPoints(nn.Module): 201 | 202 | def __init__(self, hidden_dim=256): 203 | super(ShapeNetPoints, self).__init__() 204 | # 128**3 res input 205 | self.conv_in = nn.Conv3d(1, 16, 3, padding=1, padding_mode='border') 206 | self.conv_0 = nn.Conv3d(16, 32, 3, padding=1, padding_mode='border') 207 | self.conv_0_1 = nn.Conv3d(32, 32, 3, padding=1, padding_mode='border') 208 | self.conv_1 = nn.Conv3d(32, 64, 3, padding=1, padding_mode='border') 209 | self.conv_1_1 = nn.Conv3d(64, 64, 3, padding=1, padding_mode='border') 210 | self.conv_2 = nn.Conv3d(64, 128, 3, padding=1, padding_mode='border') 211 | self.conv_2_1 = nn.Conv3d(128, 128, 3, padding=1, padding_mode='border') 212 | self.conv_3 = nn.Conv3d(128, 128, 3, padding=1, padding_mode='border') 213 | self.conv_3_1 = nn.Conv3d(128, 128, 3, padding=1, padding_mode='border') 214 | 215 | feature_size = (1 + 16 + 32 + 64 + 128 + 128 ) * 7 216 | self.fc_0 = nn.Conv1d(feature_size, hidden_dim, 1) 217 | self.fc_1 = nn.Conv1d(hidden_dim, hidden_dim, 1) 218 | self.fc_2 = nn.Conv1d(hidden_dim, hidden_dim, 1) 219 | self.fc_out = nn.Conv1d(hidden_dim, 1, 1) 220 | self.actvn = nn.ReLU() 221 | 222 | self.maxpool = nn.MaxPool3d(2) 223 | 224 | self.conv_in_bn = nn.BatchNorm3d(16) 225 | self.conv0_1_bn = nn.BatchNorm3d(32) 226 | self.conv1_1_bn = nn.BatchNorm3d(64) 227 | self.conv2_1_bn = nn.BatchNorm3d(128) 228 | self.conv3_1_bn = nn.BatchNorm3d(128) 229 | 230 | 231 | displacment = 0.0722 232 | displacments = [] 233 | displacments.append([0, 0, 0]) 234 | for x in range(3): 235 | for y in [-1, 1]: 236 | input = [0, 0, 0] 237 | input[x] = y * displacment 238 | displacments.append(input) 239 | 240 | self.displacments = torch.Tensor(displacments).cuda() 241 | 242 | def forward(self, p, x): 243 | x = x.unsqueeze(1) 244 | 245 | p_features = p.transpose(1, -1) 246 | p = p.unsqueeze(1).unsqueeze(1) 247 | p = torch.cat([p + d for d in self.displacments], dim=2) # (B,1,7,num_samples,3) 248 | feature_0 = F.grid_sample(x, p, padding_mode='border') # out : (B,C (of x), 1,1,sample_num) 249 | 250 | net = self.actvn(self.conv_in(x)) 251 | net = self.conv_in_bn(net) 252 | feature_1 = F.grid_sample(net, p, padding_mode='border') # out : (B,C (of x), 1,1,sample_num) 253 | net = self.maxpool(net) 254 | 255 | net = self.actvn(self.conv_0(net)) 256 | net = self.actvn(self.conv_0_1(net)) 257 | net = self.conv0_1_bn(net) 258 | feature_2 = F.grid_sample(net, p, padding_mode='border') # out : (B,C (of x), 1,1,sample_num) 259 | net = self.maxpool(net) 260 | 261 | net = self.actvn(self.conv_1(net)) 262 | net = self.actvn(self.conv_1_1(net)) 263 | net = self.conv1_1_bn(net) 264 | feature_3 = F.grid_sample(net, p, padding_mode='border') # out : (B,C (of x), 1,1,sample_num) 265 | net = self.maxpool(net) 266 | 267 | net = self.actvn(self.conv_2(net)) 268 | net = self.actvn(self.conv_2_1(net)) 269 | net = self.conv2_1_bn(net) 270 | feature_4 = F.grid_sample(net, p, padding_mode='border') 271 | net = self.maxpool(net) 272 | 273 | net = self.actvn(self.conv_3(net)) 274 | net = self.actvn(self.conv_3_1(net)) 275 | net = self.conv3_1_bn(net) 276 | feature_5 = F.grid_sample(net, p, padding_mode='border') 277 | 278 | # here every channel corresponds to one feature. 279 | 280 | features = torch.cat((feature_0, feature_1, feature_2, feature_3, feature_4, feature_5), 281 | dim=1) # (B, features, 1,7,sample_num) 282 | shape = features.shape 283 | features = torch.reshape(features, 284 | (shape[0], shape[1] * shape[3], shape[4])) # (B, featues_per_sample, samples_num) 285 | #features = torch.cat((features, p_features), dim=1) # (B, featue_size, samples_num) 286 | 287 | net = self.actvn(self.fc_0(features)) 288 | net = self.actvn(self.fc_1(net)) 289 | net = self.actvn(self.fc_2(net)) 290 | net = self.fc_out(net) 291 | out = net.squeeze(1) 292 | 293 | return out 294 | 295 | 296 | 297 | 298 | # 3D Single View Reconsturction (for 256**3 input voxelization) -------------------------------------- 299 | # ---------------------------------------------------------------------------------------------------- 300 | 301 | class TEXR(nn.Module): 302 | 303 | 304 | def __init__(self, hidden_dim=256): 305 | super(TEXR, self).__init__() 306 | 307 | self.conv_in = nn.Conv3d(4, 16, 3, padding=1, padding_mode='border') # out: 256 ->m.p. 128 308 | self.conv_0 = nn.Conv3d(16, 32, 3, padding=1, padding_mode='border') # out: 128 309 | self.conv_0_1 = nn.Conv3d(32, 32, 3, padding=1, padding_mode='border') # out: 128 ->m.p. 64 310 | self.conv_1 = nn.Conv3d(32, 64, 3, padding=1, padding_mode='border') # out: 64 311 | self.conv_1_1 = nn.Conv3d(64, 64, 3, padding=1, padding_mode='border') # out: 64 -> mp 32 312 | self.conv_2 = nn.Conv3d(64, 128, 3, padding=1, padding_mode='border') # out: 32 313 | self.conv_2_1 = nn.Conv3d(128, 128, 3, padding=1, padding_mode='border') # out: 32 -> mp 16 314 | self.conv_3 = nn.Conv3d(128, 128, 3, padding=1, padding_mode='border') # out: 16 315 | self.conv_3_1 = nn.Conv3d(128, 128, 3, padding=1, padding_mode='border') # out: 16 -> mp 8 316 | self.conv_4 = nn.Conv3d(128, 128, 3, padding=1, padding_mode='border') # out: 8 317 | self.conv_4_1 = nn.Conv3d(128, 128, 3, padding=1, padding_mode='border') # out: 8 318 | 319 | feature_size = (4 + 16 + 32 + 64 + 128 + 128 + 128) * 7 + 3 320 | self.fc_0 = nn.Conv1d(feature_size, hidden_dim * 2, 1) 321 | self.fc_1 = nn.Conv1d(hidden_dim *2, hidden_dim, 1) 322 | self.fc_2 = nn.Conv1d(hidden_dim , hidden_dim, 1) 323 | self.fc_out = nn.Conv1d(hidden_dim, 3, 1) 324 | self.actvn = nn.ReLU() 325 | 326 | self.maxpool = nn.MaxPool3d(2) 327 | 328 | self.conv_in_bn = nn.BatchNorm3d(16) 329 | self.conv0_1_bn = nn.BatchNorm3d(32) 330 | self.conv1_1_bn = nn.BatchNorm3d(64) 331 | self.conv2_1_bn = nn.BatchNorm3d(128) 332 | self.conv3_1_bn = nn.BatchNorm3d(128) 333 | self.conv4_1_bn = nn.BatchNorm3d(128) 334 | 335 | 336 | displacment = 0.0722 337 | displacments = [] 338 | displacments.append([0, 0, 0]) 339 | for x in range(3): 340 | for y in [-1, 1]: 341 | input = [0, 0, 0] 342 | input[x] = y * displacment 343 | displacments.append(input) 344 | 345 | self.displacments = torch.Tensor(displacments).cuda() 346 | 347 | def forward(self, p, x): 348 | # x = x.unsqueeze(1) 349 | 350 | p_features = p.transpose(1, -1) 351 | p = p.unsqueeze(1).unsqueeze(1) 352 | p = torch.cat([p + d for d in self.displacments], dim=2) 353 | feature_0 = F.grid_sample(x, p, padding_mode='border') 354 | # print(feature_0.shape) 355 | # print(feature_0[:,:,:,0,0]) 356 | 357 | net = self.actvn(self.conv_in(x)) 358 | net = self.conv_in_bn(net) 359 | feature_1 = F.grid_sample(net, p, padding_mode='border') 360 | net = self.maxpool(net) #out 128 361 | 362 | net = self.actvn(self.conv_0(net)) 363 | net = self.actvn(self.conv_0_1(net)) 364 | net = self.conv0_1_bn(net) 365 | feature_2 = F.grid_sample(net, p, padding_mode='border') 366 | net = self.maxpool(net) #out 64 367 | 368 | net = self.actvn(self.conv_1(net)) 369 | net = self.actvn(self.conv_1_1(net)) 370 | net = self.conv1_1_bn(net) 371 | feature_3 = F.grid_sample(net, p, padding_mode='border') 372 | net = self.maxpool(net) 373 | 374 | net = self.actvn(self.conv_2(net)) 375 | net = self.actvn(self.conv_2_1(net)) 376 | net = self.conv2_1_bn(net) 377 | feature_4 = F.grid_sample(net, p, padding_mode='border') 378 | net = self.maxpool(net) 379 | 380 | net = self.actvn(self.conv_3(net)) 381 | net = self.actvn(self.conv_3_1(net)) 382 | net = self.conv3_1_bn(net) 383 | feature_5 = F.grid_sample(net, p, padding_mode='border') 384 | net = self.maxpool(net) 385 | 386 | net = self.actvn(self.conv_4(net)) 387 | net = self.actvn(self.conv_4_1(net)) 388 | net = self.conv4_1_bn(net) 389 | feature_6 = F.grid_sample(net, p, padding_mode='border') 390 | 391 | # here every channel corresponse to one feature. 392 | 393 | features = torch.cat((feature_0, feature_1, feature_2, feature_3, feature_4, feature_5, feature_6), 394 | dim=1) # (B, features, 1,7,sample_num) 395 | shape = features.shape 396 | features = torch.reshape(features,(shape[0], shape[1] * shape[3], shape[4])) # (B, featues_per_sample, samples_num) 397 | features = torch.cat((features, p_features), dim=1) # (B, featue_size, samples_num) samples_num 0->0,...,N->N 398 | # print('features: ', features[:,:,:3]) 399 | 400 | net = self.actvn(self.fc_0(features)) 401 | net = self.actvn(self.fc_1(net)) 402 | net = self.actvn(self.fc_2(net)) 403 | out = self.fc_out(net) 404 | # out = net.squeeze(1) 405 | 406 | return out -------------------------------------------------------------------------------- /models/training.py: -------------------------------------------------------------------------------- 1 | from __future__ import division 2 | import torch 3 | import torch.optim as optim 4 | from torch.nn import functional as F 5 | import os 6 | from glob import glob 7 | import numpy as np 8 | 9 | 10 | 11 | 12 | class Trainer(object): 13 | 14 | def __init__(self, model, device, train_dataset, val_dataset, exp_name, optimizer='Adam'): 15 | self.model = model.to(device) 16 | self.device = device 17 | if optimizer == 'Adam': 18 | self.optimizer = optim.Adam(self.model.parameters(), lr=1e-4) 19 | if optimizer == 'Adadelta': 20 | self.optimizer = optim.Adadelta(self.model.parameters()) 21 | if optimizer == 'RMSprop': 22 | self.optimizer = optim.RMSprop(self.model.parameters(), momentum=0.9) 23 | 24 | self.train_dataset = train_dataset 25 | self.val_dataset = val_dataset 26 | self.exp_path = os.path.dirname(__file__) + '/../experiments/{}/'.format( exp_name) 27 | self.checkpoint_path = self.exp_path + 'checkpoints/' 28 | if not os.path.exists(self.checkpoint_path): 29 | print(self.checkpoint_path) 30 | os.makedirs(self.checkpoint_path) 31 | self.val_min = None 32 | 33 | 34 | def train_step(self,batch): 35 | self.model.train() 36 | self.optimizer.zero_grad() 37 | loss = self.compute_loss(batch) 38 | loss.backward() 39 | self.optimizer.step() 40 | 41 | return loss.item() 42 | 43 | def compute_loss(self,batch): 44 | device = self.device 45 | 46 | p = batch.get('grid_coords').to(device) 47 | gt_rgb = batch.get('rgb').to(device) 48 | inputs = batch.get('inputs').to(device) 49 | 50 | # print(gt_rgb) 51 | # p = p.unsqueeze(1).unsqueeze(1) 52 | # print(F.grid_sample(inputs, p, padding_mode='border')) 53 | # General points 54 | 55 | # print(p[:,:3]) 56 | pred_rgb = self.model(p,inputs) 57 | pred_rgb = pred_rgb.transpose(-1,-2) 58 | # print(gt_rgb.shape) 59 | loss_i = torch.nn.L1Loss(reduction='none')(pred_rgb, gt_rgb) # out = (B,num_points,3) 60 | 61 | loss = loss_i.sum(-1).mean() # loss_i summed 3 rgb channels for all #num_samples samples -> out = (B,1) and mean over batch -> out = (1) 62 | 63 | return loss 64 | 65 | def train_model(self, epochs): 66 | loss = 0 67 | start = self.load_checkpoint() 68 | 69 | for epoch in range(start, epochs): 70 | sum_loss = 0 71 | print('Start epoch {}'.format(epoch)) 72 | train_data_loader = self.train_dataset.get_loader() 73 | 74 | if epoch % 1 == 0: 75 | self.save_checkpoint(epoch) 76 | val_loss = self.compute_val_loss() 77 | 78 | if self.val_min is None: 79 | self.val_min = val_loss 80 | 81 | if val_loss < self.val_min: 82 | self.val_min = val_loss 83 | # for path in glob(self.exp_path + 'val_min=*'): 84 | # os.remove(path) 85 | np.save(self.exp_path + 'val_min',[epoch,val_loss]) 86 | 87 | 88 | for batch in train_data_loader: 89 | loss = self.train_step(batch) 90 | print("Current loss: {}".format(loss)) 91 | sum_loss += loss 92 | 93 | 94 | 95 | 96 | def save_checkpoint(self, epoch): 97 | path = self.checkpoint_path + 'checkpoint_epoch_{}.tar'.format(epoch) 98 | if not os.path.exists(path): 99 | torch.save({'epoch':epoch,'model_state_dict': self.model.state_dict(), 100 | 'optimizer_state_dict': self.optimizer.state_dict()}, path) 101 | 102 | def load_checkpoint(self): 103 | checkpoints = glob(self.checkpoint_path+'/*') 104 | if len(checkpoints) == 0: 105 | print('No checkpoints found at {}'.format(self.checkpoint_path)) 106 | return 0 107 | 108 | checkpoints = [os.path.splitext(os.path.basename(path))[0][17:] for path in checkpoints] 109 | checkpoints = np.array(checkpoints, dtype=int) 110 | checkpoints = np.sort(checkpoints) 111 | path = self.checkpoint_path + 'checkpoint_epoch_{}.tar'.format(checkpoints[-1]) 112 | 113 | print('Loaded checkpoint from: {}'.format(path)) 114 | checkpoint = torch.load(path) 115 | self.model.load_state_dict(checkpoint['model_state_dict']) 116 | self.optimizer.load_state_dict(checkpoint['optimizer_state_dict']) 117 | epoch = checkpoint['epoch'] 118 | return epoch 119 | 120 | def compute_val_loss(self): 121 | self.model.eval() 122 | 123 | sum_val_loss = 0 124 | num_batches = 15 125 | for _ in range(num_batches): 126 | try: 127 | val_batch = self.val_data_iterator.next() 128 | except: 129 | self.val_data_iterator = self.val_dataset.get_loader().__iter__() 130 | val_batch = self.val_data_iterator.next() 131 | 132 | sum_val_loss += self.compute_loss( val_batch).item() 133 | 134 | return sum_val_loss / num_batches -------------------------------------------------------------------------------- /sharp_teaser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jchibane/if-net_texture/989ffdc9afc0296d92a55a7a1f155c1cc3e6e5e4/sharp_teaser.png -------------------------------------------------------------------------------- /train.py: -------------------------------------------------------------------------------- 1 | import models.local_model as model 2 | import models.dataloader as dataloader 3 | from models import training 4 | import argparse 5 | import torch 6 | import config.config_loader as cfg_loader 7 | 8 | parser = argparse.ArgumentParser( 9 | description='Train Model' 10 | ) 11 | 12 | parser.add_argument('config', type=str, help='Path to config file.') 13 | args = parser.parse_args() 14 | 15 | 16 | cfg = cfg_loader.load(args.config) 17 | 18 | net = model.get_models()[cfg['model']]() 19 | 20 | train_dataset = dataloader.VoxelizedDataset('train', cfg) 21 | 22 | val_dataset = dataloader.VoxelizedDataset('val', cfg) 23 | 24 | trainer = training.Trainer(net,torch.device("cuda"),train_dataset, val_dataset, cfg['folder_name'], optimizer=cfg['training']['optimizer']) 25 | trainer.train_model(1500) 26 | --------------------------------------------------------------------------------