├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── apps ├── batch_openpose.py ├── clean_mesh.py ├── recon.py ├── render_turntable.py └── simple_test.py ├── data ├── RenderPeople_all.csv └── RenderPeople_test.csv ├── lib ├── __init__.py ├── colab_util.py ├── data │ ├── EvalDataset.py │ ├── EvalWPoseDataset.py │ └── __init__.py ├── evaluator.py ├── geometry.py ├── mesh_util.py ├── model │ ├── BasePIFuNet.py │ ├── DepthNormalizer.py │ ├── HGFilters.py │ ├── HGPIFuMRNet.py │ ├── HGPIFuNetwNML.py │ ├── MLP.py │ └── __init__.py ├── net_util.py ├── networks.py ├── options.py ├── render │ ├── __init__.py │ ├── camera.py │ ├── gl │ │ ├── __init__.py │ │ ├── cam_render.py │ │ ├── color_render.py │ │ ├── data │ │ │ ├── color.fs │ │ │ ├── color.vs │ │ │ ├── geo.fs │ │ │ ├── geo.vs │ │ │ ├── normal.fs │ │ │ ├── normal.vs │ │ │ ├── quad.fs │ │ │ └── quad.vs │ │ ├── framework.py │ │ ├── geo_render.py │ │ ├── normal_render.py │ │ └── render.py │ ├── glm.py │ └── mesh.py └── sdf.py ├── requirements.txt ├── sample_images ├── test.png └── test_keypoints.json └── scripts ├── demo.sh └── download_trained_model.sh /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to make participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies within all project spaces, and it also applies when 49 | an individual is representing the project or its community in public spaces. 50 | Examples of representing a project or community include using an official 51 | project e-mail address, posting via an official social media account, or acting 52 | as an appointed representative at an online or offline event. Representation of 53 | a project may be further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at . All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | 75 | For answers to common questions about this code of conduct, see 76 | https://www.contributor-covenant.org/faq 77 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to pifuhd 2 | We want to make contributing to this project as easy and transparent as 3 | possible. 4 | 5 | ## Pull Requests 6 | We actively welcome your pull requests. 7 | 8 | 1. Fork the repo and create your branch from `master`. 9 | 2. If you've added code that should be tested, add tests. 10 | 3. If you've changed APIs, update the documentation. 11 | 4. Ensure the test suite passes. 12 | 5. Make sure your code lints. 13 | 6. If you haven't already, complete the Contributor License Agreement ("CLA"). 14 | 15 | ## Contributor License Agreement ("CLA") 16 | In order to accept your pull request, we need you to submit a CLA. You only need 17 | to do this once to work on any of Facebook's open source projects. 18 | 19 | Complete your CLA here: 20 | 21 | ## Issues 22 | We use GitHub issues to track public bugs. Please ensure your description is 23 | clear and has sufficient instructions to be able to reproduce the issue. 24 | 25 | Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe 26 | disclosure of security bugs. In those cases, please go through the process 27 | outlined on that page and do not file a public issue. 28 | 29 | ## License 30 | By contributing to pifuhd, you agree that your contributions will be licensed 31 | under the LICENSE file in the root directory of this source tree. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization (CVPR 2020)](https://shunsukesaito.github.io/PIFuHD/) 2 | 3 | [![report](https://img.shields.io/badge/arxiv-report-red)](https://arxiv.org/pdf/2004.00452.pdf) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/11z58bl3meSzo6kFqkahMa35G5jmh2Wgt?usp=sharing) 4 | 5 | News: 6 | * \[2020/06/15\] Demo with Google Colab (incl. visualization) is available! Please check out [#pifuhd on Twitter](https://twitter.com/search?q=%23pifuhd&src=recent_search_click&f=live) for many results tested by users! 7 | 8 | This repository contains a pytorch implementation of "Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization". 9 | 10 | ![Teaser Image](https://shunsukesaito.github.io/PIFuHD/resources/images/pifuhd.gif) 11 | 12 | This codebase provides: 13 | - test code 14 | - visualization code 15 | 16 | See our [blog post](https://ai.facebook.com/blog/facebook-research-at-cvpr-2020/) to learn more about our work at CVPR2020! 17 | 18 | ## Demo on Google Colab 19 | In case you don't have an environment with GPUs to run PIFuHD, we offer Google Colab demo. You can also upload your own images and reconstruct 3D geometry together with visualization. Try our Colab demo using the following notebook: \ 20 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/11z58bl3meSzo6kFqkahMa35G5jmh2Wgt) 21 | 22 | ## Requirements 23 | - Python 3 24 | - [PyTorch](https://pytorch.org/) tested on 1.4.0, 1.5.0 25 | - json 26 | - PIL 27 | - skimage 28 | - tqdm 29 | - cv2 30 | 31 | For visualization 32 | - [trimesh](https://trimsh.org/) with pyembree 33 | - PyOpenGL 34 | - freeglut (use `sudo apt-get install freeglut3-dev` for ubuntu users) 35 | - ffmpeg 36 | 37 | Note: At least 8GB GPU memory is recommended to run PIFuHD model. 38 | 39 | Run the following code to install all pip packages: 40 | ```sh 41 | pip install -r requirements.txt 42 | ``` 43 | 44 | 45 | 46 | ## Download Pre-trained model 47 | 48 | Run the following script to download the pretrained model. The checkpoint is saved under `./checkpoints/`. 49 | ``` 50 | sh ./scripts/download_trained_model.sh 51 | ``` 52 | 53 | ## A Quick Testing 54 | To process images under `./sample_images`, run the following code: 55 | ``` 56 | sh ./scripts/demo.sh 57 | ``` 58 | The resulting obj files and rendering will be saved in `./results`. You may use meshlab (http://www.meshlab.net/) to visualize the 3D mesh output (obj file). 59 | 60 | 61 | ## Testing 62 | 63 | 1. run the following script to get joints for each image for testing (joints are used for image cropping only.). Make sure you correctly set the location of OpenPose binary. Alternatively [colab demo](https://colab.research.google.com/drive/11z58bl3meSzo6kFqkahMa35G5jmh2Wgt) provides more light-weight cropping rectange estimation without requiring openpose. 64 | ``` 65 | python apps/batch_openpose.py -d {openpose_root_path} -i {path_of_images} -o {path_of_images} 66 | ``` 67 | 68 | 2. run the following script to run reconstruction code. Make sure to set `--input_path` to `path_of_images`, `--out_path` to where you want to dump out results, and `--ckpt_path` to the checkpoint. Note that unlike PIFu, PIFuHD doesn't require segmentation mask as input. But if you observe severe artifacts, you may try removing background with off-the-shelf tools such as [removebg](https://www.remove.bg/). If you have `{image_name}_rect.txt` instead of `{image_name}_keypoints.json`, add `--use_rect` flag. For reference, you can take a look at [colab demo](https://colab.research.google.com/drive/11z58bl3meSzo6kFqkahMa35G5jmh2Wgt). 69 | ``` 70 | python -m apps.simple_test 71 | ``` 72 | 73 | 3. optionally, you can also remove artifacts by keeping only the biggest connected component from the mesh reconstruction with the following script. (Warning: the script will overwrite the original obj files.) 74 | ``` 75 | python apps/clean_mesh.py -f {path_of_objs} 76 | ``` 77 | 78 | ## Visualization 79 | To render results with turn-table, run the following code. The rendered animation (.mp4) will be stored under `{path_of_objs}`. 80 | ``` 81 | python -m apps.render_turntable -f {path_of_objs} -ww {rendering_width} -hh {rendering_height} 82 | # add -g for geometry rendering. default is normal visualization. 83 | ``` 84 | 85 | ## Citation 86 | ``` 87 | @inproceedings{saito2020pifuhd, 88 | title={PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization}, 89 | author={Saito, Shunsuke and Simon, Tomas and Saragih, Jason and Joo, Hanbyul}, 90 | booktitle={CVPR}, 91 | year={2020} 92 | } 93 | ``` 94 | 95 | ## Relevant Projects 96 | **[Monocular Real-Time Volumetric Performance Capture (ECCV 2020)](https://project-splinter.github.io/)** 97 | *Ruilong Li\*, Yuliang Xiu\*, Shunsuke Saito, Zeng Huang, Kyle Olszewski, Hao Li* 98 | 99 | The first real-time PIFu by accelerating reconstruction and rendering!! 100 | 101 | **[PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization (ICCV 2019)](https://shunsukesaito.github.io/PIFu/)** 102 | *Shunsuke Saito\*, Zeng Huang\*, Ryota Natsume\*, Shigeo Morishima, Angjoo Kanazawa, Hao Li* 103 | 104 | The original work of Pixel-Aligned Implicit Function for geometry and texture reconstruction, unifying sigle-view and multi-view methods. 105 | 106 | **[Learning to Infer Implicit Surfaces without 3d Supervision (NeurIPS 2019)](http://papers.nips.cc/paper/9039-learning-to-infer-implicit-surfaces-without-3d-supervision.pdf)** 107 | *Shichen Liu, Shunsuke Saito, Weikai Chen, Hao Li* 108 | 109 | We answer to the question of "how can we learn implicit function if we don't have 3D ground truth?" 110 | 111 | **[SiCloPe: Silhouette-Based Clothed People (CVPR 2019, best paper finalist)](https://arxiv.org/pdf/1901.00049.pdf)** 112 | *Ryota Natsume\*, Shunsuke Saito\*, Zeng Huang, Weikai Chen, Chongyang Ma, Hao Li, Shigeo Morishima* 113 | 114 | Our first attempt to reconstruct 3D clothed human body with texture from a single image! 115 | 116 | ## Other Relevant Works 117 | **[ARCH: Animatable Reconstruction of Clothed Humans (CVPR 2020)](https://arxiv.org/pdf/2004.04572.pdf)** 118 | *Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, Tony Tung* 119 | 120 | Learning PIFu in canonical space for animatable avatar generation! 121 | 122 | **[Robust 3D Self-portraits in Seconds (CVPR 2020)](http://www.liuyebin.com/portrait/portrait.html)** 123 | *Zhe Li, Tao Yu, Chuanyu Pan, Zerong Zheng, Yebin Liu* 124 | 125 | They extend PIFu to RGBD + introduce "PIFusion" utilizing PIFu reconstruction for non-rigid fusion. 126 | 127 | **[Deep Volumetric Video from Very Sparse Multi-view Performance Capture (ECCV 2018)](http://openaccess.thecvf.com/content_ECCV_2018/papers/Zeng_Huang_Deep_Volumetric_Video_ECCV_2018_paper.pdf)** 128 | *Zeng Huang, Tianye Li, Weikai Chen, Yajie Zhao, Jun Xing, Chloe LeGendre, Linjie Luo, Chongyang Ma, Hao Li* 129 | 130 | Implict surface learning for sparse view human performance capture! 131 | 132 | ## License 133 | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode). 134 | See the [LICENSE](LICENSE) file. 135 | -------------------------------------------------------------------------------- /apps/batch_openpose.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import numpy as np 4 | 5 | import os 6 | import argparse 7 | 8 | parser = argparse.ArgumentParser() 9 | parser.add_argument('-d', '--openpose_dir', type=str, required=True) 10 | parser.add_argument('-i', '--input_root', type=str, required=True) 11 | parser.add_argument('-o', '--out_path', type=str, required=True) 12 | args = parser.parse_args() 13 | 14 | op_dir = args.openpose_dir 15 | input_path = args.input_root 16 | out_json_path = args.out_path 17 | 18 | os.makedirs(out_json_path, exist_ok=True) 19 | 20 | cmd = "cd {0}; ./build/examples/openpose/openpose.bin --image_dir {1} --write_json {2} --render_pose 2 --face --face_render 2 --hand --hand_render 2".format(op_dir, input_path, out_json_path) 21 | print(cmd) 22 | os.system(cmd) 23 | -------------------------------------------------------------------------------- /apps/clean_mesh.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import os 4 | import argparse 5 | import trimesh 6 | 7 | 8 | def meshcleaning(file_dir): 9 | files = sorted([f for f in os.listdir(file_dir) if '.obj' in f]) 10 | for i, file in enumerate(files): 11 | obj_path = os.path.join(file_dir, file) 12 | 13 | print(f"Processing: {obj_path}") 14 | 15 | mesh = trimesh.load(obj_path) 16 | cc = mesh.split(only_watertight=False) 17 | 18 | out_mesh = cc[0] 19 | bbox = out_mesh.bounds 20 | height = bbox[1,0] - bbox[0,0] 21 | for c in cc: 22 | bbox = c.bounds 23 | if height < bbox[1,0] - bbox[0,0]: 24 | height = bbox[1,0] - bbox[0,0] 25 | out_mesh = c 26 | 27 | out_mesh.export(obj_path) 28 | 29 | 30 | if __name__ == '__main__': 31 | parser = argparse.ArgumentParser() 32 | parser.add_argument('-f', '--file_dir', type=str, required=True) 33 | args = parser.parse_args() 34 | 35 | meshcleaning(args.file_dir) -------------------------------------------------------------------------------- /apps/recon.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import sys 4 | import os 5 | 6 | sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) 7 | ROOT_PATH = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 8 | 9 | import time 10 | import json 11 | import numpy as np 12 | import cv2 13 | import random 14 | import torch 15 | import torch.nn as nn 16 | from tqdm import tqdm 17 | from torch.utils.data import DataLoader 18 | import matplotlib.pyplot as plt 19 | import matplotlib.cm as cm 20 | import matplotlib 21 | from numpy.linalg import inv 22 | 23 | from lib.options import BaseOptions 24 | from lib.mesh_util import save_obj_mesh_with_color, reconstruction 25 | from lib.data import EvalWPoseDataset, EvalDataset 26 | from lib.model import HGPIFuNetwNML, HGPIFuMRNet 27 | from lib.geometry import index 28 | 29 | from PIL import Image 30 | 31 | parser = BaseOptions() 32 | 33 | def gen_mesh(res, net, cuda, data, save_path, thresh=0.5, use_octree=True, components=False): 34 | image_tensor_global = data['img_512'].to(device=cuda) 35 | image_tensor = data['img'].to(device=cuda) 36 | calib_tensor = data['calib'].to(device=cuda) 37 | 38 | net.filter_global(image_tensor_global) 39 | net.filter_local(image_tensor[:,None]) 40 | 41 | try: 42 | if net.netG.netF is not None: 43 | image_tensor_global = torch.cat([image_tensor_global, net.netG.nmlF], 0) 44 | if net.netG.netB is not None: 45 | image_tensor_global = torch.cat([image_tensor_global, net.netG.nmlB], 0) 46 | except: 47 | pass 48 | 49 | b_min = data['b_min'] 50 | b_max = data['b_max'] 51 | try: 52 | save_img_path = save_path[:-4] + '.png' 53 | save_img_list = [] 54 | for v in range(image_tensor_global.shape[0]): 55 | save_img = (np.transpose(image_tensor_global[v].detach().cpu().numpy(), (1, 2, 0)) * 0.5 + 0.5)[:, :, ::-1] * 255.0 56 | save_img_list.append(save_img) 57 | save_img = np.concatenate(save_img_list, axis=1) 58 | cv2.imwrite(save_img_path, save_img) 59 | 60 | verts, faces, _, _ = reconstruction( 61 | net, cuda, calib_tensor, res, b_min, b_max, thresh, use_octree=use_octree, num_samples=50000) 62 | verts_tensor = torch.from_numpy(verts.T).unsqueeze(0).to(device=cuda).float() 63 | # if 'calib_world' in data: 64 | # calib_world = data['calib_world'].numpy()[0] 65 | # verts = np.matmul(np.concatenate([verts, np.ones_like(verts[:,:1])],1), inv(calib_world).T)[:,:3] 66 | 67 | color = np.zeros(verts.shape) 68 | interval = 50000 69 | for i in range(len(color) // interval + 1): 70 | left = i * interval 71 | if i == len(color) // interval: 72 | right = -1 73 | else: 74 | right = (i + 1) * interval 75 | net.calc_normal(verts_tensor[:, None, :, left:right], calib_tensor[:,None], calib_tensor) 76 | nml = net.nmls.detach().cpu().numpy()[0] * 0.5 + 0.5 77 | color[left:right] = nml.T 78 | 79 | save_obj_mesh_with_color(save_path, verts, faces, color) 80 | except Exception as e: 81 | print(e) 82 | 83 | 84 | def gen_mesh_imgColor(res, net, cuda, data, save_path, thresh=0.5, use_octree=True, components=False): 85 | image_tensor_global = data['img_512'].to(device=cuda) 86 | image_tensor = data['img'].to(device=cuda) 87 | calib_tensor = data['calib'].to(device=cuda) 88 | 89 | net.filter_global(image_tensor_global) 90 | net.filter_local(image_tensor[:,None]) 91 | 92 | try: 93 | if net.netG.netF is not None: 94 | image_tensor_global = torch.cat([image_tensor_global, net.netG.nmlF], 0) 95 | if net.netG.netB is not None: 96 | image_tensor_global = torch.cat([image_tensor_global, net.netG.nmlB], 0) 97 | except: 98 | pass 99 | 100 | b_min = data['b_min'] 101 | b_max = data['b_max'] 102 | try: 103 | save_img_path = save_path[:-4] + '.png' 104 | save_img_list = [] 105 | for v in range(image_tensor_global.shape[0]): 106 | save_img = (np.transpose(image_tensor_global[v].detach().cpu().numpy(), (1, 2, 0)) * 0.5 + 0.5)[:, :, ::-1] * 255.0 107 | save_img_list.append(save_img) 108 | save_img = np.concatenate(save_img_list, axis=1) 109 | cv2.imwrite(save_img_path, save_img) 110 | 111 | verts, faces, _, _ = reconstruction( 112 | net, cuda, calib_tensor, res, b_min, b_max, thresh, use_octree=use_octree, num_samples=100000) 113 | verts_tensor = torch.from_numpy(verts.T).unsqueeze(0).to(device=cuda).float() 114 | 115 | # if this returns error, projection must be defined somewhere else 116 | xyz_tensor = net.projection(verts_tensor, calib_tensor[:1]) 117 | uv = xyz_tensor[:, :2, :] 118 | color = index(image_tensor[:1], uv).detach().cpu().numpy()[0].T 119 | color = color * 0.5 + 0.5 120 | 121 | if 'calib_world' in data: 122 | calib_world = data['calib_world'].numpy()[0] 123 | verts = np.matmul(np.concatenate([verts, np.ones_like(verts[:,:1])],1), inv(calib_world).T)[:,:3] 124 | 125 | save_obj_mesh_with_color(save_path, verts, faces, color) 126 | 127 | except Exception as e: 128 | print(e) 129 | 130 | 131 | def recon(opt, use_rect=False): 132 | # load checkpoints 133 | state_dict_path = None 134 | if opt.load_netMR_checkpoint_path is not None: 135 | state_dict_path = opt.load_netMR_checkpoint_path 136 | elif opt.resume_epoch < 0: 137 | state_dict_path = '%s/%s_train_latest' % (opt.checkpoints_path, opt.name) 138 | opt.resume_epoch = 0 139 | else: 140 | state_dict_path = '%s/%s_train_epoch_%d' % (opt.checkpoints_path, opt.name, opt.resume_epoch) 141 | 142 | start_id = opt.start_id 143 | end_id = opt.end_id 144 | 145 | cuda = torch.device('cuda:%d' % opt.gpu_id if torch.cuda.is_available() else 'cpu') 146 | 147 | state_dict = None 148 | if state_dict_path is not None and os.path.exists(state_dict_path): 149 | print('Resuming from ', state_dict_path) 150 | state_dict = torch.load(state_dict_path, map_location=cuda) 151 | print('Warning: opt is overwritten.') 152 | dataroot = opt.dataroot 153 | resolution = opt.resolution 154 | results_path = opt.results_path 155 | loadSize = opt.loadSize 156 | 157 | opt = state_dict['opt'] 158 | opt.dataroot = dataroot 159 | opt.resolution = resolution 160 | opt.results_path = results_path 161 | opt.loadSize = loadSize 162 | else: 163 | raise Exception('failed loading state dict!', state_dict_path) 164 | 165 | # parser.print_options(opt) 166 | 167 | if use_rect: 168 | test_dataset = EvalDataset(opt) 169 | else: 170 | test_dataset = EvalWPoseDataset(opt) 171 | 172 | print('test data size: ', len(test_dataset)) 173 | projection_mode = test_dataset.projection_mode 174 | 175 | opt_netG = state_dict['opt_netG'] 176 | netG = HGPIFuNetwNML(opt_netG, projection_mode).to(device=cuda) 177 | netMR = HGPIFuMRNet(opt, netG, projection_mode).to(device=cuda) 178 | 179 | def set_eval(): 180 | netG.eval() 181 | 182 | # load checkpoints 183 | netMR.load_state_dict(state_dict['model_state_dict']) 184 | 185 | os.makedirs(opt.checkpoints_path, exist_ok=True) 186 | os.makedirs(opt.results_path, exist_ok=True) 187 | os.makedirs('%s/%s/recon' % (opt.results_path, opt.name), exist_ok=True) 188 | 189 | if start_id < 0: 190 | start_id = 0 191 | if end_id < 0: 192 | end_id = len(test_dataset) 193 | 194 | ## test 195 | with torch.no_grad(): 196 | set_eval() 197 | 198 | print('generate mesh (test) ...') 199 | for i in tqdm(range(start_id, end_id)): 200 | if i >= len(test_dataset): 201 | break 202 | 203 | # for multi-person processing, set it to False 204 | if True: 205 | test_data = test_dataset[i] 206 | 207 | save_path = '%s/%s/recon/result_%s_%d.obj' % (opt.results_path, opt.name, test_data['name'], opt.resolution) 208 | 209 | print(save_path) 210 | gen_mesh(opt.resolution, netMR, cuda, test_data, save_path, components=opt.use_compose) 211 | else: 212 | for j in range(test_dataset.get_n_person(i)): 213 | test_dataset.person_id = j 214 | test_data = test_dataset[i] 215 | save_path = '%s/%s/recon/result_%s_%d.obj' % (opt.results_path, opt.name, test_data['name'], j) 216 | gen_mesh(opt.resolution, netMR, cuda, test_data, save_path, components=opt.use_compose) 217 | 218 | def reconWrapper(args=None, use_rect=False): 219 | opt = parser.parse(args) 220 | recon(opt, use_rect) 221 | 222 | if __name__ == '__main__': 223 | reconWrapper() 224 | 225 | -------------------------------------------------------------------------------- /apps/render_turntable.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import math 4 | import numpy as np 5 | import sys 6 | import os 7 | 8 | sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) 9 | ROOT_PATH = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 10 | 11 | from lib.render.mesh import load_obj_mesh, compute_normal 12 | from lib.render.camera import Camera 13 | from lib.render.gl.geo_render import GeoRender 14 | from lib.render.gl.color_render import ColorRender 15 | import trimesh 16 | 17 | import cv2 18 | import os 19 | import argparse 20 | 21 | width = 512 22 | height = 512 23 | 24 | def make_rotate(rx, ry, rz): 25 | 26 | sinX = np.sin(rx) 27 | sinY = np.sin(ry) 28 | sinZ = np.sin(rz) 29 | 30 | cosX = np.cos(rx) 31 | cosY = np.cos(ry) 32 | cosZ = np.cos(rz) 33 | 34 | Rx = np.zeros((3,3)) 35 | Rx[0, 0] = 1.0 36 | Rx[1, 1] = cosX 37 | Rx[1, 2] = -sinX 38 | Rx[2, 1] = sinX 39 | Rx[2, 2] = cosX 40 | 41 | Ry = np.zeros((3,3)) 42 | Ry[0, 0] = cosY 43 | Ry[0, 2] = sinY 44 | Ry[1, 1] = 1.0 45 | Ry[2, 0] = -sinY 46 | Ry[2, 2] = cosY 47 | 48 | Rz = np.zeros((3,3)) 49 | Rz[0, 0] = cosZ 50 | Rz[0, 1] = -sinZ 51 | Rz[1, 0] = sinZ 52 | Rz[1, 1] = cosZ 53 | Rz[2, 2] = 1.0 54 | 55 | R = np.matmul(np.matmul(Rz,Ry),Rx) 56 | return R 57 | 58 | parser = argparse.ArgumentParser() 59 | parser.add_argument('-f', '--file_dir', type=str, required=True) 60 | parser.add_argument('-ww', '--width', type=int, default=512) 61 | parser.add_argument('-hh', '--height', type=int, default=512) 62 | parser.add_argument('-g', '--geo_render', action='store_true', help='default is normal rendering') 63 | 64 | args = parser.parse_args() 65 | 66 | if args.geo_render: 67 | renderer = GeoRender(width=args.width, height=args.height) 68 | else: 69 | renderer = ColorRender(width=args.width, height=args.height) 70 | cam = Camera(width=1.0, height=args.height/args.width) 71 | cam.ortho_ratio = 1.2 72 | cam.near = -100 73 | cam.far = 10 74 | 75 | obj_files = [] 76 | for (root,dirs,files) in os.walk(args.file_dir, topdown=True): 77 | for file in files: 78 | if '.obj' in file: 79 | obj_files.append(os.path.join(root, file)) 80 | print(obj_files) 81 | 82 | R = make_rotate(math.radians(180),0,0) 83 | 84 | for i, obj_path in enumerate(obj_files): 85 | 86 | print(obj_path) 87 | obj_file = obj_path.split('/')[-1] 88 | obj_root = obj_path.replace(obj_file,'') 89 | file_name = obj_file[:-4] 90 | 91 | if not os.path.exists(obj_path): 92 | continue 93 | mesh = trimesh.load(obj_path) 94 | vertices = mesh.vertices 95 | faces = mesh.faces 96 | 97 | # vertices = np.matmul(vertices, R.T) 98 | bbox_max = vertices.max(0) 99 | bbox_min = vertices.min(0) 100 | 101 | # notice that original scale is discarded to render with the same size 102 | vertices -= 0.5 * (bbox_max + bbox_min)[None,:] 103 | vertices /= bbox_max[1] - bbox_min[1] 104 | 105 | normals = compute_normal(vertices, faces) 106 | 107 | if args.geo_render: 108 | renderer.set_mesh(vertices, faces, normals, faces) 109 | else: 110 | renderer.set_mesh(vertices, faces, 0.5*normals+0.5, faces) 111 | 112 | cnt = 0 113 | for j in range(0, 361, 2): 114 | cam.center = np.array([0, 0, 0]) 115 | cam.eye = np.array([2.0*math.sin(math.radians(j)), 0, 2.0*math.cos(math.radians(j))]) + cam.center 116 | 117 | renderer.set_camera(cam) 118 | renderer.display() 119 | 120 | img = renderer.get_color(0) 121 | img = cv2.cvtColor(img, cv2.COLOR_RGBA2BGRA) 122 | 123 | cv2.imwrite(os.path.join(obj_root, 'rot_%04d.png' % cnt), 255*img) 124 | cnt += 1 125 | 126 | cmd = 'ffmpeg -framerate 30 -i ' + obj_root + '/rot_%04d.png -vcodec libx264 -y -pix_fmt yuv420p -refs 16 ' + os.path.join(obj_root, file_name + '.mp4') 127 | os.system(cmd) 128 | cmd = 'rm %s/rot_*.png' % obj_root 129 | os.system(cmd) -------------------------------------------------------------------------------- /apps/simple_test.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | 4 | from .recon import reconWrapper 5 | import argparse 6 | 7 | 8 | ############################################################################################### 9 | ## Setting 10 | ############################################################################################### 11 | parser = argparse.ArgumentParser() 12 | parser.add_argument('-i', '--input_path', type=str, default='./sample_images') 13 | parser.add_argument('-o', '--out_path', type=str, default='./results') 14 | parser.add_argument('-c', '--ckpt_path', type=str, default='./checkpoints/pifuhd.pt') 15 | parser.add_argument('-r', '--resolution', type=int, default=512) 16 | parser.add_argument('--use_rect', action='store_true', help='use rectangle for cropping') 17 | args = parser.parse_args() 18 | ############################################################################################### 19 | ## Upper PIFu 20 | ############################################################################################### 21 | 22 | resolution = str(args.resolution) 23 | 24 | start_id = -1 25 | end_id = -1 26 | cmd = ['--dataroot', args.input_path, '--results_path', args.out_path,\ 27 | '--loadSize', '1024', '--resolution', resolution, '--load_netMR_checkpoint_path', \ 28 | args.ckpt_path,\ 29 | '--start_id', '%d' % start_id, '--end_id', '%d' % end_id] 30 | reconWrapper(cmd, args.use_rect) 31 | 32 | -------------------------------------------------------------------------------- /data/RenderPeople_all.csv: -------------------------------------------------------------------------------- 1 | rp_aaron_posed_001 2 | rp_aaron_posed_005 3 | rp_aaron_posed_012 4 | rp_aaron_posed_019 5 | rp_alisha_posed_001 6 | rp_alison_posed_018 7 | rp_alison_posed_027 8 | rp_alison_posed_034 9 | rp_alison_posed_035 10 | rp_alvin_posed_002 11 | rp_alvin_posed_007 12 | rp_alvin_posed_011 13 | rp_alvin_posed_014 14 | rp_amber_posed_003 15 | rp_amber_posed_006 16 | rp_amber_posed_011 17 | rp_amber_posed_018 18 | rp_amber_posed_023 19 | rp_amber_posed_028 20 | rp_amir_posed_002 21 | rp_amir_posed_003 22 | rp_amir_posed_004 23 | rp_andrew_posed_001 24 | rp_andrew_posed_002 25 | rp_andrew_posed_004 26 | rp_aneko_posed_001 27 | rp_aneko_posed_004 28 | rp_aneko_posed_022 29 | rp_anna_posed_001 30 | rp_anna_posed_007 31 | rp_anna_posed_008 32 | rp_antonia_posed_003 33 | rp_antonia_posed_005 34 | rp_antonia_posed_020 35 | rp_ashley_posed_001 36 | rp_ashley_posed_003 37 | rp_ashley_posed_005 38 | rp_beatrice_posed_007 39 | rp_beatrice_posed_017 40 | rp_beatrice_posed_024 41 | rp_beatrice_posed_030 42 | rp_beatrice_posed_036 43 | rp_belle_posed_004 44 | rp_belle_posed_005 45 | rp_belle_posed_006 46 | rp_ben_posed_001 47 | rp_ben_posed_003 48 | rp_ben_posed_007 49 | rp_berkan_posed_001 50 | rp_berkan_posed_002 51 | rp_berkan_posed_003 52 | rp_bianca_posed_010 53 | rp_bianca_posed_019 54 | rp_bianca_posed_025 55 | rp_bob_posed_001 56 | rp_bob_posed_002 57 | rp_brad_posed_002 58 | rp_brad_posed_005 59 | rp_brad_posed_006 60 | rp_brandon_posed_006 61 | rp_brandon_posed_013 62 | rp_brandon_posed_019 63 | rp_brandon_posed_024 64 | rp_caren_posed_008 65 | rp_caren_posed_009 66 | rp_caren_posed_011 67 | rp_caren_posed_020 68 | rp_carina_posed_006 69 | rp_carina_posed_007 70 | rp_carla_posed_003 71 | rp_carla_posed_012 72 | rp_carla_posed_018 73 | rp_carla_posed_026 74 | rp_celina_posed_002 75 | rp_celina_posed_003 76 | rp_celina_posed_004 77 | rp_chen_posed_001 78 | rp_chen_posed_003 79 | rp_chen_posed_006 80 | rp_chloe_posed_002 81 | rp_chloe_posed_003 82 | rp_christine_posed_001 83 | rp_christine_posed_014 84 | rp_christine_posed_025 85 | rp_christopher_posed_001 86 | rp_christopher_posed_003 87 | rp_christopher_posed_005 88 | rp_christopher_posed_008 89 | rp_cindy_posed_004 90 | rp_cindy_posed_009 91 | rp_cindy_posed_019 92 | rp_claudia_posed_003 93 | rp_claudia_posed_017 94 | rp_claudia_posed_019 95 | rp_claudia_posed_026 96 | rp_corey_posed_004 97 | rp_corey_posed_010 98 | rp_corey_posed_017 99 | rp_corey_posed_026 100 | rp_cornell_posed_003 101 | rp_cornell_posed_009 102 | rp_cornell_posed_014 103 | rp_dana_posed_001 104 | rp_daniel_posed_001 105 | rp_daniel_posed_003 106 | rp_daniel_posed_004 107 | rp_dave_posed_004 108 | rp_dave_posed_005 109 | rp_debra_posed_002 110 | rp_debra_posed_013 111 | rp_debra_posed_017 112 | rp_dennis_posed_018 113 | rp_dennis_posed_024 114 | rp_dennis_posed_031 115 | rp_dennis_posed_035 116 | rp_elena_posed_006 117 | rp_elena_posed_010 118 | rp_elena_posed_013 119 | rp_elias_posed_001 120 | rp_elias_posed_005 121 | rp_elias_posed_012 122 | rp_elizabeth_posed_001 123 | rp_elizabeth_posed_002 124 | rp_elizabeth_posed_004 125 | rp_ellie_posed_005 126 | rp_ellie_posed_014 127 | rp_ellie_posed_016 128 | rp_emily_posed_008 129 | rp_emily_posed_009 130 | rp_emily_posed_010 131 | rp_emily_posed_016 132 | rp_emily_posed_023 133 | rp_emily_posed_027 134 | rp_emma_posed_006 135 | rp_emma_posed_010 136 | rp_emma_posed_012 137 | rp_emma_posed_020 138 | rp_emma_posed_025 139 | rp_emma_posed_032 140 | rp_eric_posed_013 141 | rp_eric_posed_027 142 | rp_eric_posed_029 143 | rp_eric_posed_039 144 | rp_ethan_posed_001 145 | rp_ethan_posed_004 146 | rp_ethan_posed_006 147 | rp_ethan_posed_016 148 | rp_ethan_posed_021 149 | rp_ethan_posed_023 150 | rp_eve_posed_001 151 | rp_fabienne_posed_003 152 | rp_fabienne_posed_011 153 | rp_fabienne_posed_014 154 | rp_felice_posed_002 155 | rp_felice_posed_004 156 | rp_felice_posed_021 157 | rp_felice_posed_024 158 | rp_felix_posed_001 159 | rp_fernanda_posed_016 160 | rp_fernanda_posed_026 161 | rp_fernanda_posed_032 162 | rp_fernanda_posed_035 163 | rp_fiona_posed_001 164 | rp_fiona_posed_003 165 | rp_fiona_posed_013 166 | rp_fiona_posed_017 167 | rp_frank_posed_002 168 | rp_frank_posed_015 169 | rp_frank_posed_020 170 | rp_frank_posed_023 171 | rp_george_posed_002 172 | rp_george_posed_003 173 | rp_grace_posed_001 174 | rp_grace_posed_002 175 | rp_grace_posed_004 176 | rp_hannah_posed_007 177 | rp_hannah_posed_009 178 | rp_helen_posed_010 179 | rp_helen_posed_012 180 | rp_helen_posed_027 181 | rp_helen_posed_034 182 | rp_helen_posed_035 183 | rp_henry_posed_001 184 | rp_henry_posed_002 185 | rp_henry_posed_010 186 | rp_henry_posed_015 187 | rp_holly_posed_002 188 | rp_holly_posed_004 189 | rp_holly_posed_006 190 | rp_holly_posed_010 191 | rp_holly_posed_014 192 | rp_jacob_posed_001 193 | rp_jacob_posed_003 194 | rp_janett_posed_001 195 | rp_janett_posed_007 196 | rp_janett_posed_019 197 | rp_janett_posed_029 198 | rp_janna_posed_019 199 | rp_janna_posed_026 200 | rp_janna_posed_046 201 | rp_janna_posed_048 202 | rp_jason_posed_002 203 | rp_jason_posed_008 204 | rp_jason_posed_011 205 | rp_jasper_posed_007 206 | rp_jasper_posed_010 207 | rp_jennifer_posed_003 208 | rp_jennifer_posed_004 209 | rp_jeremy_posed_001 210 | rp_jessica_posed_020 211 | rp_jessica_posed_034 212 | rp_jessica_posed_040 213 | rp_jessica_posed_053 214 | rp_joel_posed_003 215 | rp_joel_posed_009 216 | rp_joel_posed_011 217 | rp_joel_posed_016 218 | rp_johnny_posed_001 219 | rp_johnny_posed_002 220 | rp_john_posed_001 221 | rp_john_posed_002 222 | rp_joko_posed_001 223 | rp_joko_posed_009 224 | rp_joko_posed_013 225 | rp_joko_posed_016 226 | rp_joscha_posed_003 227 | rp_joscha_posed_004 228 | rp_joscha_posed_006 229 | rp_joyce_posed_010 230 | rp_joyce_posed_015 231 | rp_joyce_posed_029 232 | rp_judy_posed_001 233 | rp_judy_posed_002 234 | rp_julia_posed_037 235 | rp_julia_posed_041 236 | rp_julia_posed_043 237 | rp_julia_posed_050 238 | rp_kai_posed_009 239 | rp_kai_posed_014 240 | rp_kai_posed_023 241 | rp_kai_posed_024 242 | rp_kati_posed_003 243 | rp_kati_posed_006 244 | rp_kati_posed_012 245 | rp_kelly_posed_003 246 | rp_kent_posed_002 247 | rp_kent_posed_007 248 | rp_kent_posed_009 249 | rp_kim_posed_001 250 | rp_kim_posed_003 251 | rp_kim_posed_004 252 | rp_koji_posed_001 253 | rp_koji_posed_003 254 | rp_koji_posed_010 255 | rp_koji_posed_014 256 | rp_kumar_posed_001 257 | rp_kumar_posed_005 258 | rp_kumar_posed_009 259 | rp_kumar_posed_020 260 | rp_kylie_posed_010 261 | rp_kylie_posed_012 262 | rp_kylie_posed_014 263 | rp_kylie_posed_016 264 | rp_kylie_posed_017 265 | rp_kylie_posed_018 266 | rp_lars_posed_001 267 | rp_laura_posed_002 268 | rp_lee_posed_001 269 | rp_lee_posed_002 270 | rp_lee_posed_005 271 | rp_lee_posed_006 272 | rp_lena_posed_001 273 | rp_leo_posed_002 274 | rp_liam_posed_001 275 | rp_lina_posed_002 276 | rp_lina_posed_003 277 | rp_lina_posed_006 278 | rp_liz_posed_002 279 | rp_liz_posed_003 280 | rp_liz_posed_005 281 | rp_liz_posed_006 282 | rp_luisa_posed_001 283 | rp_luisa_posed_011 284 | rp_luisa_posed_020 285 | rp_luisa_posed_023 286 | rp_luisa_posed_024 287 | rp_luke_posed_002 288 | rp_luke_posed_006 289 | rp_luke_posed_009 290 | rp_maria_posed_005 291 | rp_maria_posed_007 292 | rp_maria_posed_008 293 | rp_marie_posed_001 294 | rp_mark_posed_002 295 | rp_mark_posed_003 296 | rp_mark_posed_004 297 | rp_mark_posed_005 298 | rp_marleen_posed_001 299 | rp_marleen_posed_002 300 | rp_marleen_posed_003 301 | rp_martha_posed_002 302 | rp_maurice_posed_001 303 | rp_maurice_posed_004 304 | rp_maurice_posed_006 305 | rp_maurice_posed_010 306 | rp_maurice_posed_012 307 | rp_maurice_posed_015 308 | rp_max_posed_005 309 | rp_max_posed_009 310 | rp_maya_posed_006 311 | rp_maya_posed_014 312 | rp_maya_posed_019 313 | rp_maya_posed_027 314 | rp_mei_posed_002 315 | rp_mei_posed_009 316 | rp_mei_posed_011 317 | rp_melanie_posed_004 318 | rp_melanie_posed_011 319 | rp_melanie_posed_012 320 | rp_melanie_posed_023 321 | rp_melinda_posed_003 322 | rp_melinda_posed_006 323 | rp_melinda_posed_008 324 | rp_mia_posed_001 325 | rp_michael_posed_006 326 | rp_michael_posed_009 327 | rp_michael_posed_018 328 | rp_michael_posed_025 329 | rp_mika_posed_001 330 | rp_mira_posed_006 331 | rp_mira_posed_016 332 | rp_mira_posed_019 333 | rp_mira_posed_024 334 | rp_nagy_posed_001 335 | rp_naomi_posed_001 336 | rp_naomi_posed_005 337 | rp_naomi_posed_009 338 | rp_naomi_posed_030 339 | rp_nathan_posed_002 340 | rp_nathan_posed_003 341 | rp_nathan_posed_011 342 | rp_nathan_posed_018 343 | rp_nathan_posed_024 344 | rp_nathan_posed_026 345 | rp_nina_posed_002 346 | rp_nina_posed_004 347 | rp_nina_posed_009 348 | rp_noah_posed_005 349 | rp_noah_posed_011 350 | rp_noah_posed_012 351 | rp_oliver_posed_003 352 | rp_oliver_posed_012 353 | rp_oliver_posed_023 354 | rp_oliver_posed_027 355 | rp_olivia_posed_002 356 | rp_olivia_posed_014 357 | rp_olivia_posed_022 358 | rp_olivia_posed_025 359 | rp_paige_posed_005 360 | rp_paige_posed_009 361 | rp_paige_posed_011 362 | rp_pamela_posed_003 363 | rp_pamela_posed_011 364 | rp_pamela_posed_012 365 | rp_patrick_posed_001 366 | rp_percy_posed_003 367 | rp_percy_posed_010 368 | rp_percy_posed_011 369 | rp_peter_posed_001 370 | rp_peter_posed_002 371 | rp_petra_posed_006 372 | rp_petra_posed_007 373 | rp_petra_posed_009 374 | rp_petra_posed_020 375 | rp_philip_posed_001 376 | rp_philip_posed_009 377 | rp_philip_posed_028 378 | rp_philip_posed_030 379 | rp_philip_posed_032 380 | rp_pierre_posed_002 381 | rp_pierre_posed_005 382 | rp_pierre_posed_006 383 | rp_ralph_posed_009 384 | rp_ralph_posed_010 385 | rp_ralph_posed_013 386 | rp_ralph_posed_015 387 | rp_ramon_posed_003 388 | rp_ramon_posed_005 389 | rp_ramon_posed_007 390 | rp_ray_posed_001 391 | rp_ray_posed_003 392 | rp_ray_posed_004 393 | rp_ray_posed_011 394 | rp_ricarda_posed_004 395 | rp_ricarda_posed_009 396 | rp_ricarda_posed_014 397 | rp_ricarda_posed_029 398 | rp_richard_posed_005 399 | rp_richard_posed_008 400 | rp_richard_posed_031 401 | rp_richard_posed_037 402 | rp_rick_posed_002 403 | rp_rick_posed_009 404 | rp_rick_posed_011 405 | rp_rick_posed_014 406 | rp_roberta_posed_005 407 | rp_roberta_posed_011 408 | rp_roberta_posed_023 409 | rp_roberta_posed_032 410 | rp_rosy_posed_009 411 | rp_rosy_posed_013 412 | rp_rosy_posed_016 413 | rp_ryo_posed_005 414 | rp_ryo_posed_009 415 | rp_ryo_posed_015 416 | rp_ryo_posed_016 417 | rp_saki_posed_012 418 | rp_saki_posed_020 419 | rp_saki_posed_024 420 | rp_saki_posed_025 421 | rp_saki_posed_030 422 | rp_samantha_posed_004 423 | rp_samantha_posed_006 424 | rp_samantha_posed_007 425 | rp_samantha_posed_009 426 | rp_samantha_posed_011 427 | rp_sarah_posed_001 428 | rp_sarah_posed_003 429 | rp_sarah_posed_004 430 | rp_scott_posed_006 431 | rp_scott_posed_013 432 | rp_scott_posed_029 433 | rp_scott_posed_036 434 | rp_seiko_posed_002 435 | rp_seiko_posed_016 436 | rp_seiko_posed_019 437 | rp_serena_posed_002 438 | rp_serena_posed_005 439 | rp_serena_posed_011 440 | rp_serena_posed_022 441 | rp_shawn_posed_002 442 | rp_shawn_posed_011 443 | rp_shawn_posed_026 444 | rp_shawn_posed_032 445 | rp_shawn_posed_037 446 | rp_sheila_posed_011 447 | rp_sheila_posed_012 448 | rp_sheila_posed_019 449 | rp_sheila_posed_032 450 | rp_sophia_posed_012 451 | rp_sophia_posed_022 452 | rp_sophia_posed_032 453 | rp_sophia_posed_039 454 | rp_stephanie_posed_003 455 | rp_stephanie_posed_009 456 | rp_stephanie_posed_012 457 | rp_stephen_posed_015 458 | rp_stephen_posed_031 459 | rp_stephen_posed_034 460 | rp_stephen_posed_045 461 | rp_steve_posed_001 462 | rp_sydney_posed_004 463 | rp_sydney_posed_014 464 | rp_sydney_posed_024 465 | rp_sydney_posed_030 466 | rp_tanja_posed_003 467 | rp_tanja_posed_007 468 | rp_tanja_posed_012 469 | rp_tanja_posed_017 470 | rp_thomas_posed_005 471 | rp_thomas_posed_009 472 | rp_thomas_posed_017 473 | rp_thomas_posed_021 474 | rp_tilda_posed_001 475 | rp_tilda_posed_002 476 | rp_tilda_posed_003 477 | rp_tim_posed_001 478 | rp_tony_posed_002 479 | rp_toshiro_posed_006 480 | rp_toshiro_posed_012 481 | rp_toshiro_posed_015 482 | rp_toshiro_posed_036 483 | rp_toshiro_posed_038 484 | rp_tyler_posed_003 485 | rp_tyler_posed_008 486 | rp_tyler_posed_009 487 | rp_tyler_posed_013 488 | rp_tyrone_posed_002 489 | rp_tyrone_posed_003 490 | rp_tyrone_posed_005 491 | rp_vanessa_posed_001 492 | rp_victoria_posed_003 493 | rp_victoria_posed_004 494 | rp_victoria_posed_008 495 | rp_wendy_posed_001 496 | rp_wendy_posed_002 497 | rp_yasmin_posed_002 498 | rp_yasmin_posed_015 499 | rp_yasmin_posed_030 500 | rp_zoey_posed_001 501 | -------------------------------------------------------------------------------- /data/RenderPeople_test.csv: -------------------------------------------------------------------------------- 1 | rp_ben_posed_001 2 | rp_ben_posed_003 3 | rp_ben_posed_007 4 | rp_caren_posed_008 5 | rp_caren_posed_009 6 | rp_caren_posed_011 7 | rp_caren_posed_020 8 | rp_christopher_posed_001 9 | rp_christopher_posed_003 10 | rp_christopher_posed_005 11 | rp_christopher_posed_008 12 | rp_emily_posed_008 13 | rp_emily_posed_009 14 | rp_emily_posed_010 15 | rp_emily_posed_016 16 | rp_emily_posed_023 17 | rp_emily_posed_027 18 | rp_eve_posed_001 19 | rp_george_posed_002 20 | rp_george_posed_003 21 | rp_laura_posed_002 22 | rp_mira_posed_006 23 | rp_mira_posed_016 24 | rp_mira_posed_019 25 | rp_mira_posed_024 26 | rp_oliver_posed_003 27 | rp_oliver_posed_012 28 | rp_oliver_posed_023 29 | rp_oliver_posed_027 30 | rp_pamela_posed_003 31 | rp_pamela_posed_011 32 | rp_pamela_posed_012 33 | rp_saki_posed_012 34 | rp_saki_posed_020 35 | rp_saki_posed_024 36 | rp_saki_posed_025 37 | rp_saki_posed_030 38 | rp_serena_posed_002 39 | rp_serena_posed_005 40 | rp_serena_posed_011 41 | rp_serena_posed_022 42 | rp_sheila_posed_011 43 | rp_sheila_posed_012 44 | rp_sheila_posed_019 45 | rp_sheila_posed_032 46 | rp_stephen_posed_015 47 | rp_stephen_posed_031 48 | rp_stephen_posed_034 49 | rp_stephen_posed_045 50 | rp_zoey_posed_001 -------------------------------------------------------------------------------- /lib/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. -------------------------------------------------------------------------------- /lib/colab_util.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import io 25 | import os 26 | import torch 27 | from skimage.io import imread 28 | import numpy as np 29 | import cv2 30 | from tqdm import tqdm_notebook as tqdm 31 | import base64 32 | from IPython.display import HTML 33 | 34 | # Util function for loading meshes 35 | from pytorch3d.io import load_objs_as_meshes 36 | 37 | from IPython.display import HTML 38 | from base64 import b64encode 39 | 40 | # Data structures and functions for rendering 41 | from pytorch3d.structures import Meshes 42 | from pytorch3d.renderer import ( 43 | look_at_view_transform, 44 | OpenGLOrthographicCameras, 45 | PointLights, 46 | DirectionalLights, 47 | Materials, 48 | RasterizationSettings, 49 | MeshRenderer, 50 | MeshRasterizer, 51 | HardPhongShader, 52 | TexturesVertex 53 | ) 54 | 55 | def set_renderer(): 56 | # Setup 57 | device = torch.device("cuda:0") 58 | torch.cuda.set_device(device) 59 | 60 | # Initialize an OpenGL perspective camera. 61 | R, T = look_at_view_transform(2.0, 0, 180) 62 | cameras = OpenGLOrthographicCameras(device=device, R=R, T=T) 63 | 64 | raster_settings = RasterizationSettings( 65 | image_size=512, 66 | blur_radius=0.0, 67 | faces_per_pixel=1, 68 | bin_size = None, 69 | max_faces_per_bin = None 70 | ) 71 | 72 | lights = PointLights(device=device, location=((2.0, 2.0, 2.0),)) 73 | 74 | renderer = MeshRenderer( 75 | rasterizer=MeshRasterizer( 76 | cameras=cameras, 77 | raster_settings=raster_settings 78 | ), 79 | shader=HardPhongShader( 80 | device=device, 81 | cameras=cameras, 82 | lights=lights 83 | ) 84 | ) 85 | return renderer 86 | 87 | def get_verts_rgb_colors(obj_path): 88 | rgb_colors = [] 89 | 90 | f = open(obj_path) 91 | lines = f.readlines() 92 | for line in lines: 93 | ls = line.split(' ') 94 | if len(ls) == 7: 95 | rgb_colors.append(ls[-3:]) 96 | 97 | return np.array(rgb_colors, dtype='float32')[None, :, :] 98 | 99 | def generate_video_from_obj(obj_path, image_path, video_path, renderer): 100 | input_image = cv2.imread(image_path) 101 | input_image = input_image[:,:input_image.shape[1]//3] 102 | input_image = cv2.resize(input_image, (512,512)) 103 | 104 | # Setup 105 | device = torch.device("cuda:0") 106 | torch.cuda.set_device(device) 107 | 108 | # Load obj file 109 | verts_rgb_colors = get_verts_rgb_colors(obj_path) 110 | verts_rgb_colors = torch.from_numpy(verts_rgb_colors).to(device) 111 | textures = TexturesVertex(verts_features=verts_rgb_colors) 112 | # wo_textures = TexturesVertex(verts_features=torch.ones_like(verts_rgb_colors)*0.75) 113 | 114 | # Load obj 115 | mesh = load_objs_as_meshes([obj_path], device=device) 116 | 117 | # Set mesh 118 | vers = mesh._verts_list 119 | faces = mesh._faces_list 120 | mesh_w_tex = Meshes(vers, faces, textures) 121 | # mesh_wo_tex = Meshes(vers, faces, wo_textures) 122 | 123 | # create VideoWriter 124 | fourcc = cv2. VideoWriter_fourcc(*'MP4V') 125 | out = cv2.VideoWriter(video_path, fourcc, 20.0, (1024,512)) 126 | 127 | for i in tqdm(range(90)): 128 | R, T = look_at_view_transform(1.8, 0, i*4, device=device) 129 | images_w_tex = renderer(mesh_w_tex, R=R, T=T) 130 | images_w_tex = np.clip(images_w_tex[0, ..., :3].cpu().numpy(), 0.0, 1.0)[:, :, ::-1] * 255 131 | # images_wo_tex = renderer(mesh_wo_tex, R=R, T=T) 132 | # images_wo_tex = np.clip(images_wo_tex[0, ..., :3].cpu().numpy(), 0.0, 1.0)[:, :, ::-1] * 255 133 | image = np.concatenate([input_image, images_w_tex], axis=1) 134 | out.write(image.astype('uint8')) 135 | out.release() 136 | 137 | def video(path): 138 | mp4 = open(path,'rb').read() 139 | data_url = "data:video/mp4;base64," + b64encode(mp4).decode() 140 | return HTML('' % data_url) 141 | -------------------------------------------------------------------------------- /lib/data/EvalDataset.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import os 4 | import random 5 | 6 | import numpy as np 7 | from PIL import Image, ImageOps 8 | from PIL.ImageFilter import GaussianBlur 9 | import cv2 10 | import torch 11 | import json 12 | 13 | from torch.utils.data import Dataset 14 | import torchvision.transforms as transforms 15 | 16 | def crop_image(img, rect): 17 | x, y, w, h = rect 18 | 19 | left = abs(x) if x < 0 else 0 20 | top = abs(y) if y < 0 else 0 21 | right = abs(img.shape[1]-(x+w)) if x + w >= img.shape[1] else 0 22 | bottom = abs(img.shape[0]-(y+h)) if y + h >= img.shape[0] else 0 23 | 24 | if img.shape[2] == 4: 25 | color = [0, 0, 0, 0] 26 | else: 27 | color = [0, 0, 0] 28 | new_img = cv2.copyMakeBorder(img, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color) 29 | 30 | x = x + left 31 | y = y + top 32 | 33 | return new_img[y:(y+h),x:(x+w),:] 34 | 35 | class EvalDataset(Dataset): 36 | @staticmethod 37 | def modify_commandline_options(parser, is_train): 38 | return parser 39 | 40 | def __init__(self, opt, projection='orthogonal'): 41 | self.opt = opt 42 | self.projection_mode = projection 43 | 44 | self.root = self.opt.dataroot 45 | self.img_files = sorted([os.path.join(self.root,f) for f in os.listdir(self.root) if f.split('.')[-1] in ['png', 'jpeg', 'jpg', 'PNG', 'JPG', 'JPEG'] and os.path.exists(os.path.join(self.root,f.replace('.%s' % (f.split('.')[-1]), '_rect.txt')))]) 46 | self.IMG = os.path.join(self.root) 47 | 48 | self.phase = 'val' 49 | self.load_size = self.opt.loadSize 50 | 51 | # PIL to tensor 52 | self.to_tensor = transforms.Compose([ 53 | transforms.ToTensor(), 54 | transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) 55 | ]) 56 | 57 | # only used in case of multi-person processing 58 | self.person_id = 0 59 | 60 | def __len__(self): 61 | return len(self.img_files) 62 | 63 | def get_n_person(self, index): 64 | rect_path = self.img_files[index].replace('.%s' % (self.img_files[index].split('.')[-1]), '_rect.txt') 65 | rects = np.loadtxt(rect_path, dtype=np.int32) 66 | 67 | return rects.shape[0] if len(rects.shape) == 2 else 1 68 | 69 | def get_item(self, index): 70 | img_path = self.img_files[index] 71 | rect_path = self.img_files[index].replace('.%s' % (self.img_files[index].split('.')[-1]), '_rect.txt') 72 | # Name 73 | img_name = os.path.splitext(os.path.basename(img_path))[0] 74 | 75 | im = cv2.imread(img_path, cv2.IMREAD_UNCHANGED) 76 | if im.shape[2] == 4: 77 | im = im / 255.0 78 | im[:,:,:3] /= im[:,:,3:] + 1e-8 79 | im = im[:,:,3:] * im[:,:,:3] + 0.5 * (1.0 - im[:,:,3:]) 80 | im = (255.0 * im).astype(np.uint8) 81 | h, w = im.shape[:2] 82 | 83 | intrinsic = np.identity(4) 84 | 85 | trans_mat = np.identity(4) 86 | 87 | rects = np.loadtxt(rect_path, dtype=np.int32) 88 | if len(rects.shape) == 1: 89 | rects = rects[None] 90 | pid = min(rects.shape[0]-1, self.person_id) 91 | 92 | rect = rects[pid].tolist() 93 | im = crop_image(im, rect) 94 | 95 | scale_im2ndc = 1.0 / float(w // 2) 96 | scale = w / rect[2] 97 | trans_mat *= scale 98 | trans_mat[3,3] = 1.0 99 | trans_mat[0, 3] = -scale*(rect[0] + rect[2]//2 - w//2) * scale_im2ndc 100 | trans_mat[1, 3] = scale*(rect[1] + rect[3]//2 - h//2) * scale_im2ndc 101 | 102 | intrinsic = np.matmul(trans_mat, intrinsic) 103 | im_512 = cv2.resize(im, (512, 512)) 104 | im = cv2.resize(im, (self.load_size, self.load_size)) 105 | 106 | image_512 = Image.fromarray(im_512[:,:,::-1]).convert('RGB') 107 | image = Image.fromarray(im[:,:,::-1]).convert('RGB') 108 | 109 | B_MIN = np.array([-1, -1, -1]) 110 | B_MAX = np.array([1, 1, 1]) 111 | projection_matrix = np.identity(4) 112 | projection_matrix[1, 1] = -1 113 | calib = torch.Tensor(projection_matrix).float() 114 | 115 | calib_world = torch.Tensor(intrinsic).float() 116 | 117 | # image 118 | image_512 = self.to_tensor(image_512) 119 | image = self.to_tensor(image) 120 | return { 121 | 'name': img_name, 122 | 'img': image.unsqueeze(0), 123 | 'img_512': image_512.unsqueeze(0), 124 | 'calib': calib.unsqueeze(0), 125 | 'calib_world': calib_world.unsqueeze(0), 126 | 'b_min': B_MIN, 127 | 'b_max': B_MAX, 128 | } 129 | 130 | def __getitem__(self, index): 131 | return self.get_item(index) 132 | -------------------------------------------------------------------------------- /lib/data/EvalWPoseDataset.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import os 4 | import random 5 | 6 | import numpy as np 7 | from PIL import Image, ImageOps 8 | from PIL.ImageFilter import GaussianBlur 9 | import cv2 10 | import torch 11 | import json 12 | 13 | from torch.utils.data import Dataset 14 | import torchvision.transforms as transforms 15 | 16 | def crop_image(img, rect): 17 | x, y, w, h = rect 18 | 19 | left = abs(x) if x < 0 else 0 20 | top = abs(y) if y < 0 else 0 21 | right = abs(img.shape[1]-(x+w)) if x + w >= img.shape[1] else 0 22 | bottom = abs(img.shape[0]-(y+h)) if y + h >= img.shape[0] else 0 23 | 24 | if img.shape[2] == 4: 25 | color = [0, 0, 0, 0] 26 | else: 27 | color = [0, 0, 0] 28 | new_img = cv2.copyMakeBorder(img, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color) 29 | 30 | x = x + left 31 | y = y + top 32 | 33 | return new_img[y:(y+h),x:(x+w),:] 34 | 35 | def face_crop(pts): 36 | flag = pts[:,2] > 0.2 37 | 38 | mshoulder = pts[1,:2] 39 | rear = pts[18,:2] 40 | lear = pts[17,:2] 41 | nose = pts[0,:2] 42 | 43 | center = np.copy(mshoulder) 44 | center[1] = min(nose[1] if flag[0] else 1e8, lear[1] if flag[17] else 1e8, rear[1] if flag[18] else 1e8) 45 | 46 | ps = [] 47 | pts_id = [0, 15, 16, 17, 18] 48 | cnt = 0 49 | for i in pts_id: 50 | if flag[i]: 51 | ps.append(pts[i,:2]) 52 | if i in [17, 18]: 53 | cnt += 1 54 | 55 | ps = np.stack(ps, 0) 56 | if ps.shape[0] <= 1: 57 | raise IOError('key points are not properly set') 58 | if ps.shape[0] <= 3 and cnt != 2: 59 | center = ps[-1] 60 | else: 61 | center = ps.mean(0) 62 | radius = int(1.4*np.max(np.sqrt(((ps - center[None,:])**2).reshape(-1,2).sum(0)))) 63 | 64 | 65 | # radius = np.max(np.sqrt(((center[None] - np.stack([]))**2).sum(0)) 66 | # radius = int(1.0*abs(center[1] - mshoulder[1])) 67 | center = center.astype(np.int) 68 | 69 | x1 = center[0] - radius 70 | x2 = center[0] + radius 71 | y1 = center[1] - radius 72 | y2 = center[1] + radius 73 | 74 | return (x1, y1, x2-x1, y2-y1) 75 | 76 | def upperbody_crop(pts): 77 | flag = pts[:,2] > 0.2 78 | 79 | mshoulder = pts[1,:2] 80 | ps = [] 81 | pts_id = [8] 82 | for i in pts_id: 83 | if flag[i]: 84 | ps.append(pts[i,:2]) 85 | 86 | center = mshoulder 87 | if len(ps) == 1: 88 | ps = np.stack(ps, 0) 89 | radius = int(0.8*np.max(np.sqrt(((ps - center[None,:])**2).reshape(-1,2).sum(1)))) 90 | else: 91 | ps = [] 92 | pts_id = [0, 2, 5] 93 | ratio = [0.4, 0.3, 0.3] 94 | for i in pts_id: 95 | if flag[i]: 96 | ps.append(pts[i,:2]) 97 | ps = np.stack(ps, 0) 98 | radius = int(0.8*np.max(np.sqrt(((ps - center[None,:])**2).reshape(-1,2).sum(1)) / np.array(ratio))) 99 | 100 | center = center.astype(np.int) 101 | 102 | x1 = center[0] - radius 103 | x2 = center[0] + radius 104 | y1 = center[1] - radius 105 | y2 = center[1] + radius 106 | 107 | return (x1, y1, x2-x1, y2-y1) 108 | 109 | def fullbody_crop(pts): 110 | flags = pts[:,2] > 0.5 #openpose 111 | # flags = pts[:,2] > 0.2 #detectron 112 | check_id = [11,19,21,22] 113 | cnt = sum(flags[check_id]) 114 | 115 | if cnt == 0: 116 | center = pts[8,:2].astype(np.int) 117 | pts = pts[pts[:,2] > 0.5][:,:2] 118 | radius = int(1.45*np.sqrt(((center[None,:] - pts)**2).sum(1)).max(0)) 119 | center[1] += int(0.05*radius) 120 | else: 121 | pts = pts[pts[:,2] > 0.2] 122 | pmax = pts.max(0) 123 | pmin = pts.min(0) 124 | 125 | center = (0.5 * (pmax[:2] + pmin[:2])).astype(np.int) 126 | radius = int(0.65 * max(pmax[0]-pmin[0], pmax[1]-pmin[1])) 127 | 128 | x1 = center[0] - radius 129 | x2 = center[0] + radius 130 | y1 = center[1] - radius 131 | y2 = center[1] + radius 132 | 133 | return (x1, y1, x2-x1, y2-y1) 134 | 135 | 136 | class EvalWPoseDataset(Dataset): 137 | @staticmethod 138 | def modify_commandline_options(parser, is_train): 139 | return parser 140 | 141 | def __init__(self, opt, projection='orthogonal'): 142 | self.opt = opt 143 | self.projection_mode = projection 144 | 145 | self.root = self.opt.dataroot 146 | self.img_files = sorted([os.path.join(self.root,f) for f in os.listdir(self.root) if f.split('.')[-1] in ['png', 'jpeg', 'jpg', 'PNG', 'JPG', 'JPEG'] and os.path.exists(os.path.join(self.root,f.replace('.%s' % (f.split('.')[-1]), '_keypoints.json')))]) 147 | self.IMG = os.path.join(self.root) 148 | 149 | self.phase = 'val' 150 | self.load_size = self.opt.loadSize 151 | 152 | if self.opt.crop_type == 'face': 153 | self.crop_func = face_crop 154 | elif self.opt.crop_type == 'upperbody': 155 | self.crop_func = upperbody_crop 156 | else: 157 | self.crop_func = fullbody_crop 158 | 159 | # PIL to tensor 160 | self.to_tensor = transforms.Compose([ 161 | transforms.ToTensor(), 162 | transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) 163 | ]) 164 | 165 | # only used in case of multi-person processing 166 | self.person_id = 0 167 | 168 | def __len__(self): 169 | return len(self.img_files) 170 | 171 | def get_n_person(self, index): 172 | joint_path = self.img_files[index].replace('.%s' % (self.img_files[index].split('.')[-1]), '_keypoints.json') 173 | # Calib 174 | with open(joint_path) as json_file: 175 | data = json.load(json_file) 176 | return len(data['people']) 177 | 178 | def get_item(self, index): 179 | img_path = self.img_files[index] 180 | joint_path = self.img_files[index].replace('.%s' % (self.img_files[index].split('.')[-1]), '_keypoints.json') 181 | # Name 182 | img_name = os.path.splitext(os.path.basename(img_path))[0] 183 | # Calib 184 | with open(joint_path) as json_file: 185 | data = json.load(json_file) 186 | if len(data['people']) == 0: 187 | raise IOError('non human found!!') 188 | 189 | # if True, the person with the largest height will be chosen. 190 | # set to False for multi-person processing 191 | if True: 192 | selected_data = data['people'][0] 193 | height = 0 194 | if len(data['people']) != 1: 195 | for i in range(len(data['people'])): 196 | tmp = data['people'][i] 197 | keypoints = np.array(tmp['pose_keypoints_2d']).reshape(-1,3) 198 | 199 | flags = keypoints[:,2] > 0.5 #openpose 200 | # flags = keypoints[:,2] > 0.2 #detectron 201 | if sum(flags) == 0: 202 | continue 203 | bbox = keypoints[flags] 204 | bbox_max = bbox.max(0) 205 | bbox_min = bbox.min(0) 206 | 207 | if height < bbox_max[1] - bbox_min[1]: 208 | height = bbox_max[1] - bbox_min[1] 209 | selected_data = tmp 210 | else: 211 | pid = min(len(data['people'])-1, self.person_id) 212 | selected_data = data['people'][pid] 213 | 214 | keypoints = np.array(selected_data['pose_keypoints_2d']).reshape(-1,3) 215 | 216 | flags = keypoints[:,2] > 0.5 #openpose 217 | # flags = keypoints[:,2] > 0.2 #detectron 218 | 219 | nflag = flags[0] 220 | mflag = flags[1] 221 | 222 | check_id = [2, 5, 15, 16, 17, 18] 223 | cnt = sum(flags[check_id]) 224 | if self.opt.crop_type == 'face' and (not (nflag and cnt > 3)): 225 | print('Waring: face should not be backfacing.') 226 | if self.opt.crop_type == 'upperbody' and (not (mflag and nflag and cnt > 3)): 227 | print('Waring: upperbody should not be backfacing.') 228 | if self.opt.crop_type == 'fullbody' and sum(flags) < 15: 229 | print('Waring: not sufficient keypoints.') 230 | 231 | im = cv2.imread(img_path, cv2.IMREAD_UNCHANGED) 232 | if im.shape[2] == 4: 233 | im = im / 255.0 234 | im[:,:,:3] /= im[:,:,3:] + 1e-8 235 | im = im[:,:,3:] * im[:,:,:3] + 0.5 * (1.0 - im[:,:,3:]) 236 | im = (255.0 * im).astype(np.uint8) 237 | h, w = im.shape[:2] 238 | 239 | intrinsic = np.identity(4) 240 | 241 | trans_mat = np.identity(4) 242 | rect = self.crop_func(keypoints) 243 | 244 | im = crop_image(im, rect) 245 | 246 | scale_im2ndc = 1.0 / float(w // 2) 247 | scale = w / rect[2] 248 | trans_mat *= scale 249 | trans_mat[3,3] = 1.0 250 | trans_mat[0, 3] = -scale*(rect[0] + rect[2]//2 - w//2) * scale_im2ndc 251 | trans_mat[1, 3] = scale*(rect[1] + rect[3]//2 - h//2) * scale_im2ndc 252 | 253 | intrinsic = np.matmul(trans_mat, intrinsic) 254 | im_512 = cv2.resize(im, (512, 512)) 255 | im = cv2.resize(im, (self.load_size, self.load_size)) 256 | 257 | image_512 = Image.fromarray(im_512[:,:,::-1]).convert('RGB') 258 | image = Image.fromarray(im[:,:,::-1]).convert('RGB') 259 | 260 | B_MIN = np.array([-1, -1, -1]) 261 | B_MAX = np.array([1, 1, 1]) 262 | projection_matrix = np.identity(4) 263 | projection_matrix[1, 1] = -1 264 | calib = torch.Tensor(projection_matrix).float() 265 | 266 | calib_world = torch.Tensor(intrinsic).float() 267 | 268 | # image 269 | image_512 = self.to_tensor(image_512) 270 | image = self.to_tensor(image) 271 | return { 272 | 'name': img_name, 273 | 'img': image.unsqueeze(0), 274 | 'img_512': image_512.unsqueeze(0), 275 | 'calib': calib.unsqueeze(0), 276 | 'calib_world': calib_world.unsqueeze(0), 277 | 'b_min': B_MIN, 278 | 'b_max': B_MAX, 279 | } 280 | 281 | def __getitem__(self, index): 282 | return self.get_item(index) 283 | -------------------------------------------------------------------------------- /lib/data/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | from .EvalWPoseDataset import EvalWPoseDataset 4 | from .EvalDataset import EvalDataset -------------------------------------------------------------------------------- /lib/evaluator.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import trimesh 4 | import trimesh.proximity 5 | import trimesh.sample 6 | import numpy as np 7 | import math 8 | import os 9 | from PIL import Image 10 | 11 | import argparse 12 | 13 | def euler_to_rot_mat(r_x, r_y, r_z): 14 | R_x = np.array([[1, 0, 0], 15 | [0, math.cos(r_x), -math.sin(r_x)], 16 | [0, math.sin(r_x), math.cos(r_x)] 17 | ]) 18 | 19 | R_y = np.array([[math.cos(r_y), 0, math.sin(r_y)], 20 | [0, 1, 0], 21 | [-math.sin(r_y), 0, math.cos(r_y)] 22 | ]) 23 | 24 | R_z = np.array([[math.cos(r_z), -math.sin(r_z), 0], 25 | [math.sin(r_z), math.cos(r_z), 0], 26 | [0, 0, 1] 27 | ]) 28 | 29 | R = np.dot(R_z, np.dot(R_y, R_x)) 30 | 31 | return R 32 | 33 | 34 | class MeshEvaluator: 35 | _normal_render = None 36 | 37 | @staticmethod 38 | def init_gl(): 39 | from .render.gl.normal_render import NormalRender 40 | MeshEvaluator._normal_render = NormalRender(width=512, height=512) 41 | 42 | def __init__(self): 43 | pass 44 | 45 | def set_mesh(self, src_path, tgt_path, scale_factor=1.0, offset=0): 46 | self.src_mesh = trimesh.load(src_path) 47 | self.tgt_mesh = trimesh.load(tgt_path) 48 | 49 | self.scale_factor = scale_factor 50 | self.offset = offset 51 | 52 | 53 | def get_chamfer_dist(self, num_samples=10000): 54 | # Chamfer 55 | src_surf_pts, _ = trimesh.sample.sample_surface(self.src_mesh, num_samples) 56 | tgt_surf_pts, _ = trimesh.sample.sample_surface(self.tgt_mesh, num_samples) 57 | 58 | _, src_tgt_dist, _ = trimesh.proximity.closest_point(self.tgt_mesh, src_surf_pts) 59 | _, tgt_src_dist, _ = trimesh.proximity.closest_point(self.src_mesh, tgt_surf_pts) 60 | 61 | src_tgt_dist[np.isnan(src_tgt_dist)] = 0 62 | tgt_src_dist[np.isnan(tgt_src_dist)] = 0 63 | 64 | src_tgt_dist = src_tgt_dist.mean() 65 | tgt_src_dist = tgt_src_dist.mean() 66 | 67 | chamfer_dist = (src_tgt_dist + tgt_src_dist) / 2 68 | 69 | return chamfer_dist 70 | 71 | def get_surface_dist(self, num_samples=10000): 72 | # P2S 73 | src_surf_pts, _ = trimesh.sample.sample_surface(self.src_mesh, num_samples) 74 | 75 | _, src_tgt_dist, _ = trimesh.proximity.closest_point(self.tgt_mesh, src_surf_pts) 76 | 77 | src_tgt_dist[np.isnan(src_tgt_dist)] = 0 78 | 79 | src_tgt_dist = src_tgt_dist.mean() 80 | 81 | return src_tgt_dist 82 | 83 | def _render_normal(self, mesh, deg): 84 | view_mat = np.identity(4) 85 | view_mat[:3, :3] *= 2 / 256 86 | rz = deg / 180. * np.pi 87 | model_mat = np.identity(4) 88 | model_mat[:3, :3] = euler_to_rot_mat(0, rz, 0) 89 | model_mat[1, 3] = self.offset 90 | view_mat[2, 2] *= -1 91 | 92 | self._normal_render.set_matrices(view_mat, model_mat) 93 | self._normal_render.set_normal_mesh(self.scale_factor*mesh.vertices, mesh.faces, mesh.vertex_normals, mesh.faces) 94 | self._normal_render.draw() 95 | normal_img = self._normal_render.get_color() 96 | return normal_img 97 | 98 | def _get_reproj_normal_error(self, deg): 99 | tgt_normal = self._render_normal(self.tgt_mesh, deg) 100 | src_normal = self._render_normal(self.src_mesh, deg) 101 | 102 | error = ((src_normal[:, :, :3] - tgt_normal[:, :, :3]) ** 2).mean() * 3 103 | 104 | return error, src_normal, tgt_normal 105 | 106 | def get_reproj_normal_error(self, frontal=True, back=True, left=True, right=True, save_demo_img=None): 107 | # reproj error 108 | # if save_demo_img is not None, save a visualization at the given path (etc, "./test.png") 109 | if self._normal_render is None: 110 | print("In order to use normal render, " 111 | "you have to call init_gl() before initialing any evaluator objects.") 112 | return -1 113 | 114 | side_cnt = 0 115 | total_error = 0 116 | demo_list = [] 117 | if frontal: 118 | side_cnt += 1 119 | error, src_normal, tgt_normal = self._get_reproj_normal_error(0) 120 | total_error += error 121 | demo_list.append(np.concatenate([src_normal, tgt_normal], axis=0)) 122 | if back: 123 | side_cnt += 1 124 | error, src_normal, tgt_normal = self._get_reproj_normal_error(180) 125 | total_error += error 126 | demo_list.append(np.concatenate([src_normal, tgt_normal], axis=0)) 127 | if left: 128 | side_cnt += 1 129 | error, src_normal, tgt_normal = self._get_reproj_normal_error(90) 130 | total_error += error 131 | demo_list.append(np.concatenate([src_normal, tgt_normal], axis=0)) 132 | if right: 133 | side_cnt += 1 134 | error, src_normal, tgt_normal = self._get_reproj_normal_error(270) 135 | total_error += error 136 | demo_list.append(np.concatenate([src_normal, tgt_normal], axis=0)) 137 | if save_demo_img is not None: 138 | res_array = np.concatenate(demo_list, axis=1) 139 | res_img = Image.fromarray((res_array * 255).astype(np.uint8)) 140 | res_img.save(save_demo_img) 141 | return total_error / side_cnt 142 | 143 | if __name__ == '__main__': 144 | parser = argparse.ArgumentParser() 145 | parser.add_argument('-r', '--root', type=str, required=True) 146 | parser.add_argument('-t', '--tar_path', type=str, required=True) 147 | args = parser.parse_args() 148 | 149 | evaluator = MeshEvaluator() 150 | evaluator.init_gl() 151 | 152 | def run(root, exp_name, tar_path): 153 | src_path = os.path.join(root, exp_name, 'recon') 154 | rp_path = os.path.join(tar_path, 'RP', 'GEO', 'OBJ') 155 | bf_path = os.path.join(tar_path, 'BUFF', 'GEO', 'PLY') 156 | 157 | buff_files = [f for f in os.listdir(bf_path) if '.ply' in f] 158 | 159 | src_names = ['0_0_00.obj', '90_0_00.obj', '180_0_00.obj', '270_0_00.obj'] 160 | 161 | total_vals = [] 162 | items = [] 163 | for file in buff_files: 164 | tar_name = os.path.join(bf_path, file) 165 | name = tar_name.split('/')[-1][:-4] 166 | 167 | for src in src_names: 168 | src_name = os.path.join(src_path, 'result_%s_%s' % (name, src)) 169 | if not os.path.exists(src_name): 170 | continue 171 | evaluator.set_mesh(src_name, tar_name, 0.13, -40) 172 | 173 | vals = [] 174 | vals.append(0.1 * evaluator.get_chamfer_dist()) 175 | vals.append(0.1 * evaluator.get_surface_dist()) 176 | vals.append(4.0 * evaluator.get_reproj_normal_error(save_demo_img=os.path.join(src_path, '%s_%s.png' % (name, src[:-4])))) 177 | 178 | item = { 179 | 'name': '%s_%s' % (name, src), 180 | 'vals': vals 181 | } 182 | 183 | total_vals.append(vals) 184 | items.append(item) 185 | 186 | vals = np.array(total_vals).mean(0) 187 | buf_val = vals 188 | 189 | np.save(os.path.join(root, exp_name, 'buff-item.npy'), np.array(items)) 190 | np.save(os.path.join(root, exp_name, 'buff-vals.npy'), total_vals) 191 | 192 | rp_files = [f for f in os.listdir(rp_path) if '.obj' in f] 193 | 194 | total_vals = [] 195 | items = [] 196 | for file in rp_files: 197 | tar_name = os.path.join(rp_path, file) 198 | name = tar_name.split('/')[-1][:-9] 199 | 200 | for src in src_names: 201 | src_name = os.path.join(src_path, 'result_%s_%s' % (name, src)) 202 | if not os.path.exists(src_name): 203 | continue 204 | 205 | evaluator.set_mesh(src_name, tar_name, 1.3, -120) 206 | 207 | vals = [] 208 | vals.append(evaluator.get_chamfer_dist()) 209 | vals.append(evaluator.get_surface_dist()) 210 | vals.append(4.0 * evaluator.get_reproj_normal_error(save_demo_img=os.path.join(src_path, '%s_%s.png' % (name, src[:-4])))) 211 | 212 | item = { 213 | 'name': '%s_%s' % (name, src), 214 | 'vals': vals 215 | } 216 | 217 | total_vals.append(vals) 218 | items.append(item) 219 | 220 | np.save(os.path.join(root, exp_name, 'rp-item.npy'), np.array(items)) 221 | np.save(os.path.join(root, exp_name, 'rp-vals.npy'), total_vals) 222 | 223 | vals = np.array(total_vals).mean(0) 224 | print('BUFF - chamfer: %.4f p2s: %.4f nml: %.4f' % (buf_val[0], buf_val[1], buf_val[2])) 225 | print('RP - chamfer: %.4f p2s: %.4f nml: %.4f' % (vals[0], vals[1], vals[2])) 226 | 227 | exp_list = ['pifuhd_final'] 228 | 229 | root = args.root 230 | tar_path = args.tar_path 231 | 232 | for exp in exp_list: 233 | run(root, exp, tar_path) -------------------------------------------------------------------------------- /lib/geometry.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import torch 25 | 26 | def index(feat, uv): 27 | ''' 28 | extract image features at floating coordinates with bilinear interpolation 29 | args: 30 | feat: [B, C, H, W] image features 31 | uv: [B, 2, N] normalized image coordinates ranged in [-1, 1] 32 | return: 33 | [B, C, N] sampled pixel values 34 | ''' 35 | uv = uv.transpose(1, 2) 36 | uv = uv.unsqueeze(2) 37 | samples = torch.nn.functional.grid_sample(feat, uv, align_corners=True) 38 | return samples[:, :, :, 0] 39 | 40 | def orthogonal(points, calib, transform=None): 41 | ''' 42 | project points onto screen space using orthogonal projection 43 | args: 44 | points: [B, 3, N] 3d points in world coordinates 45 | calib: [B, 3, 4] projection matrix 46 | transform: [B, 2, 3] screen space transformation 47 | return: 48 | [B, 3, N] 3d coordinates in screen space 49 | ''' 50 | rot = calib[:, :3, :3] 51 | trans = calib[:, :3, 3:4] 52 | pts = torch.baddbmm(trans, rot, points) 53 | if transform is not None: 54 | scale = transform[:2, :2] 55 | shift = transform[:2, 2:3] 56 | pts[:, :2, :] = torch.baddbmm(shift, scale, pts[:, :2, :]) 57 | return pts 58 | 59 | def perspective(points, calib, transform=None): 60 | ''' 61 | project points onto screen space using perspective projection 62 | args: 63 | points: [B, 3, N] 3d points in world coordinates 64 | calib: [B, 3, 4] projection matrix 65 | transform: [B, 2, 3] screen space trasnformation 66 | return: 67 | [B, 3, N] 3d coordinates in screen space 68 | ''' 69 | rot = calib[:, :3, :3] 70 | trans = calib[:, :3, 3:4] 71 | homo = torch.baddbmm(trans, rot, points) 72 | xy = homo[:, :2, :] / homo[:, 2:3, :] 73 | if transform is not None: 74 | scale = transform[:2, :2] 75 | shift = transform[:2, 2:3] 76 | xy = torch.baddbmm(shift, scale, xy) 77 | 78 | xyz = torch.cat([xy, homo[:, 2:3, :]], 1) 79 | return xyz -------------------------------------------------------------------------------- /lib/mesh_util.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | from skimage import measure 25 | import numpy as np 26 | import torch 27 | from .sdf import create_grid, eval_grid_octree, eval_grid 28 | from skimage import measure 29 | 30 | from numpy.linalg import inv 31 | 32 | def reconstruction(net, cuda, calib_tensor, 33 | resolution, b_min, b_max, thresh=0.5, 34 | use_octree=False, num_samples=10000, transform=None): 35 | ''' 36 | Reconstruct meshes from sdf predicted by the network. 37 | :param net: a BasePixImpNet object. call image filter beforehead. 38 | :param cuda: cuda device 39 | :param calib_tensor: calibration tensor 40 | :param resolution: resolution of the grid cell 41 | :param b_min: bounding box corner [x_min, y_min, z_min] 42 | :param b_max: bounding box corner [x_max, y_max, z_max] 43 | :param use_octree: whether to use octree acceleration 44 | :param num_samples: how many points to query each gpu iteration 45 | :return: marching cubes results. 46 | ''' 47 | # First we create a grid by resolution 48 | # and transforming matrix for grid coordinates to real world xyz 49 | coords, mat = create_grid(resolution, resolution, resolution) 50 | #b_min, b_max, transform=transform) 51 | 52 | calib = calib_tensor[0].cpu().numpy() 53 | 54 | calib_inv = inv(calib) 55 | coords = coords.reshape(3,-1).T 56 | coords = np.matmul(np.concatenate([coords, np.ones((coords.shape[0],1))], 1), calib_inv.T)[:, :3] 57 | coords = coords.T.reshape(3,resolution,resolution,resolution) 58 | 59 | # Then we define the lambda function for cell evaluation 60 | def eval_func(points): 61 | points = np.expand_dims(points, axis=0) 62 | points = np.repeat(points, 1, axis=0) 63 | samples = torch.from_numpy(points).to(device=cuda).float() 64 | 65 | net.query(samples, calib_tensor) 66 | pred = net.get_preds()[0][0] 67 | return pred.detach().cpu().numpy() 68 | 69 | # Then we evaluate the grid 70 | if use_octree: 71 | sdf = eval_grid_octree(coords, eval_func, num_samples=num_samples) 72 | else: 73 | sdf = eval_grid(coords, eval_func, num_samples=num_samples) 74 | 75 | # Finally we do marching cubes 76 | try: 77 | verts, faces, normals, values = measure.marching_cubes(sdf, thresh) 78 | # transform verts into world coordinate system 79 | trans_mat = np.matmul(calib_inv, mat) 80 | verts = np.matmul(trans_mat[:3, :3], verts.T) + trans_mat[:3, 3:4] 81 | verts = verts.T 82 | # in case mesh has flip transformation 83 | if np.linalg.det(trans_mat[:3, :3]) < 0.0: 84 | faces = faces[:,::-1] 85 | return verts, faces, normals, values 86 | except: 87 | print('error cannot marching cubes') 88 | return -1 89 | 90 | 91 | def save_obj_mesh(mesh_path, verts, faces=None): 92 | file = open(mesh_path, 'w') 93 | 94 | for v in verts: 95 | file.write('v %.4f %.4f %.4f\n' % (v[0], v[1], v[2])) 96 | if faces is not None: 97 | for f in faces: 98 | if f[0] == f[1] or f[1] == f[2] or f[0] == f[2]: 99 | continue 100 | f_plus = f + 1 101 | file.write('f %d %d %d\n' % (f_plus[0], f_plus[2], f_plus[1])) 102 | file.close() 103 | 104 | 105 | def save_obj_mesh_with_color(mesh_path, verts, faces, colors): 106 | file = open(mesh_path, 'w') 107 | 108 | for idx, v in enumerate(verts): 109 | c = colors[idx] 110 | file.write('v %.4f %.4f %.4f %.4f %.4f %.4f\n' % (v[0], v[1], v[2], c[0], c[1], c[2])) 111 | for f in faces: 112 | f_plus = f + 1 113 | file.write('f %d %d %d\n' % (f_plus[0], f_plus[2], f_plus[1])) 114 | file.close() 115 | 116 | 117 | def save_obj_mesh_with_uv(mesh_path, verts, faces, uvs): 118 | file = open(mesh_path, 'w') 119 | 120 | for idx, v in enumerate(verts): 121 | vt = uvs[idx] 122 | file.write('v %.4f %.4f %.4f\n' % (v[0], v[1], v[2])) 123 | file.write('vt %.4f %.4f\n' % (vt[0], vt[1])) 124 | 125 | for f in faces: 126 | f_plus = f + 1 127 | file.write('f %d/%d %d/%d %d/%d\n' % (f_plus[0], f_plus[0], 128 | f_plus[2], f_plus[2], 129 | f_plus[1], f_plus[1])) 130 | file.close() 131 | -------------------------------------------------------------------------------- /lib/model/BasePIFuNet.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import torch 4 | import torch.nn as nn 5 | import torch.nn.functional as F 6 | 7 | from ..geometry import index, orthogonal, perspective 8 | 9 | class BasePIFuNet(nn.Module): 10 | def __init__(self, 11 | projection_mode='orthogonal', 12 | criteria={'occ': nn.MSELoss()}, 13 | ): 14 | ''' 15 | args: 16 | projection_mode: orthonal / perspective 17 | error_term: point-wise error term 18 | ''' 19 | super(BasePIFuNet, self).__init__() 20 | self.name = 'base' 21 | 22 | self.criteria = criteria 23 | 24 | self.index = index 25 | self.projection = orthogonal if projection_mode == 'orthogonal' else perspective 26 | 27 | self.preds = None 28 | self.labels = None 29 | self.nmls = None 30 | self.labels_nml = None 31 | self.preds_surface = None # with normal loss only 32 | 33 | def forward(self, points, images, calibs, transforms=None): 34 | ''' 35 | args: 36 | points: [B, 3, N] 3d points in world space 37 | images: [B, C, H, W] input images 38 | calibs: [B, 3, 4] calibration matrices for each image 39 | transforms: [B, 2, 3] image space coordinate transforms 40 | return: 41 | [B, C, N] prediction corresponding to the given points 42 | ''' 43 | self.filter(images) 44 | self.query(points, calibs, transforms) 45 | return self.get_preds() 46 | 47 | def filter(self, images): 48 | ''' 49 | apply a fully convolutional network to images. 50 | the resulting feature will be stored. 51 | args: 52 | images: [B, C, H, W] 53 | ''' 54 | None 55 | 56 | def query(self, points, calibs, trasnforms=None, labels=None): 57 | ''' 58 | given 3d points, we obtain 2d projection of these given the camera matrices. 59 | filter needs to be called beforehand. 60 | the prediction is stored to self.preds 61 | args: 62 | points: [B, 3, N] 3d points in world space 63 | calibs: [B, 3, 4] calibration matrices for each image 64 | transforms: [B, 2, 3] image space coordinate transforms 65 | labels: [B, C, N] ground truth labels (for supervision only) 66 | return: 67 | [B, C, N] prediction 68 | ''' 69 | None 70 | 71 | def calc_normal(self, points, calibs, transforms=None, delta=0.1): 72 | ''' 73 | return surface normal in 'model' space. 74 | it computes normal only in the last stack. 75 | note that the current implementation use forward difference. 76 | args: 77 | points: [B, 3, N] 3d points in world space 78 | calibs: [B, 3, 4] calibration matrices for each image 79 | transforms: [B, 2, 3] image space coordinate transforms 80 | delta: perturbation for finite difference 81 | ''' 82 | None 83 | 84 | def get_preds(self): 85 | ''' 86 | return the current prediction. 87 | return: 88 | [B, C, N] prediction 89 | ''' 90 | return self.preds 91 | 92 | def get_error(self, gamma=None): 93 | ''' 94 | return the loss given the ground truth labels and prediction 95 | ''' 96 | return self.error_term(self.preds, self.labels, gamma) 97 | 98 | -------------------------------------------------------------------------------- /lib/model/DepthNormalizer.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import torch 4 | import torch.nn as nn 5 | import torch.nn.functional as F 6 | 7 | class DepthNormalizer(nn.Module): 8 | def __init__(self, opt): 9 | super(DepthNormalizer, self).__init__() 10 | self.opt = opt 11 | 12 | def forward(self, xyz, calibs=None, index_feat=None): 13 | ''' 14 | normalize depth value 15 | args: 16 | xyz: [B, 3, N] depth value 17 | ''' 18 | z_feat = xyz[:,2:3,:] * (self.opt.loadSize // 2) / self.opt.z_size 19 | 20 | return z_feat -------------------------------------------------------------------------------- /lib/model/HGFilters.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import torch 25 | import torch.nn as nn 26 | import torch.nn.functional as F 27 | from ..net_util import conv3x3 28 | 29 | class ConvBlock(nn.Module): 30 | def __init__(self, in_planes, out_planes, norm='batch'): 31 | super(ConvBlock, self).__init__() 32 | self.conv1 = conv3x3(in_planes, int(out_planes / 2)) 33 | self.conv2 = conv3x3(int(out_planes / 2), int(out_planes / 4)) 34 | self.conv3 = conv3x3(int(out_planes / 4), int(out_planes / 4)) 35 | 36 | if norm == 'batch': 37 | self.bn1 = nn.BatchNorm2d(in_planes) 38 | self.bn2 = nn.BatchNorm2d(int(out_planes / 2)) 39 | self.bn3 = nn.BatchNorm2d(int(out_planes / 4)) 40 | self.bn4 = nn.BatchNorm2d(in_planes) 41 | elif norm == 'group': 42 | self.bn1 = nn.GroupNorm(32, in_planes) 43 | self.bn2 = nn.GroupNorm(32, int(out_planes / 2)) 44 | self.bn3 = nn.GroupNorm(32, int(out_planes / 4)) 45 | self.bn4 = nn.GroupNorm(32, in_planes) 46 | 47 | if in_planes != out_planes: 48 | self.downsample = nn.Sequential( 49 | self.bn4, 50 | nn.ReLU(True), 51 | nn.Conv2d(in_planes, out_planes, 52 | kernel_size=1, stride=1, bias=False), 53 | ) 54 | else: 55 | self.downsample = None 56 | 57 | def forward(self, x): 58 | residual = x 59 | 60 | out1 = self.conv1(F.relu(self.bn1(x), True)) 61 | out2 = self.conv2(F.relu(self.bn2(out1), True)) 62 | out3 = self.conv3(F.relu(self.bn3(out2), True)) 63 | 64 | out3 = torch.cat([out1, out2, out3], 1) 65 | 66 | if self.downsample is not None: 67 | residual = self.downsample(residual) 68 | 69 | out3 += residual 70 | 71 | return out3 72 | 73 | class HourGlass(nn.Module): 74 | def __init__(self, depth, n_features, norm='batch'): 75 | super(HourGlass, self).__init__() 76 | self.depth = depth 77 | self.features = n_features 78 | self.norm = norm 79 | 80 | self._generate_network(self.depth) 81 | 82 | def _generate_network(self, level): 83 | self.add_module('b1_' + str(level), ConvBlock(self.features, self.features, norm=self.norm)) 84 | self.add_module('b2_' + str(level), ConvBlock(self.features, self.features, norm=self.norm)) 85 | 86 | if level > 1: 87 | self._generate_network(level - 1) 88 | else: 89 | self.add_module('b2_plus_' + str(level), ConvBlock(self.features, self.features, norm=self.norm)) 90 | 91 | self.add_module('b3_' + str(level), ConvBlock(self.features, self.features, norm=self.norm)) 92 | 93 | def _forward(self, level, inp): 94 | # upper branch 95 | up1 = inp 96 | up1 = self._modules['b1_' + str(level)](up1) 97 | 98 | # lower branch 99 | low1 = F.avg_pool2d(inp, 2, stride=2) 100 | low1 = self._modules['b2_' + str(level)](low1) 101 | 102 | if level > 1: 103 | low2 = self._forward(level - 1, low1) 104 | else: 105 | low2 = low1 106 | low2 = self._modules['b2_plus_' + str(level)](low2) 107 | 108 | low3 = low2 109 | low3 = self._modules['b3_' + str(level)](low3) 110 | 111 | up2 = F.interpolate(low3, scale_factor=2, mode='bicubic', align_corners=True) 112 | # up2 = F.interpolate(low3, scale_factor=2, mode='bilinear') 113 | 114 | return up1 + up2 115 | 116 | def forward(self, x): 117 | return self._forward(self.depth, x) 118 | 119 | 120 | class HGFilter(nn.Module): 121 | def __init__(self, stack, depth, in_ch, last_ch, norm='batch', down_type='conv64', use_sigmoid=True): 122 | super(HGFilter, self).__init__() 123 | self.n_stack = stack 124 | self.use_sigmoid = use_sigmoid 125 | self.depth = depth 126 | self.last_ch = last_ch 127 | self.norm = norm 128 | self.down_type = down_type 129 | 130 | self.conv1 = nn.Conv2d(in_ch, 64, kernel_size=7, stride=2, padding=3) 131 | 132 | last_ch = self.last_ch 133 | 134 | if self.norm == 'batch': 135 | self.bn1 = nn.BatchNorm2d(64) 136 | elif self.norm == 'group': 137 | self.bn1 = nn.GroupNorm(32, 64) 138 | 139 | if self.down_type == 'conv64': 140 | self.conv2 = ConvBlock(64, 64, self.norm) 141 | self.down_conv2 = nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=1) 142 | elif self.down_type == 'conv128': 143 | self.conv2 = ConvBlock(128, 128, self.norm) 144 | self.down_conv2 = nn.Conv2d(128, 128, kernel_size=3, stride=2, padding=1) 145 | elif self.down_type == 'ave_pool' or self.down_type == 'no_down': 146 | self.conv2 = ConvBlock(64, 128, self.norm) 147 | 148 | self.conv3 = ConvBlock(128, 128, self.norm) 149 | self.conv4 = ConvBlock(128, 256, self.norm) 150 | 151 | # start stacking 152 | for stack in range(self.n_stack): 153 | self.add_module('m' + str(stack), HourGlass(self.depth, 256, self.norm)) 154 | 155 | self.add_module('top_m_' + str(stack), ConvBlock(256, 256, self.norm)) 156 | self.add_module('conv_last' + str(stack), 157 | nn.Conv2d(256, 256, kernel_size=1, stride=1, padding=0)) 158 | if self.norm == 'batch': 159 | self.add_module('bn_end' + str(stack), nn.BatchNorm2d(256)) 160 | elif self.norm == 'group': 161 | self.add_module('bn_end' + str(stack), nn.GroupNorm(32, 256)) 162 | 163 | self.add_module('l' + str(stack), 164 | nn.Conv2d(256, last_ch, 165 | kernel_size=1, stride=1, padding=0)) 166 | 167 | if stack < self.n_stack - 1: 168 | self.add_module( 169 | 'bl' + str(stack), nn.Conv2d(256, 256, kernel_size=1, stride=1, padding=0)) 170 | self.add_module( 171 | 'al' + str(stack), nn.Conv2d(last_ch, 256, kernel_size=1, stride=1, padding=0)) 172 | 173 | def forward(self, x): 174 | x = F.relu(self.bn1(self.conv1(x)), True) 175 | 176 | if self.down_type == 'ave_pool': 177 | x = F.avg_pool2d(self.conv2(x), 2, stride=2) 178 | elif self.down_type == ['conv64', 'conv128']: 179 | x = self.conv2(x) 180 | x = self.down_conv2(x) 181 | elif self.down_type == 'no_down': 182 | x = self.conv2(x) 183 | else: 184 | raise NameError('unknown downsampling type') 185 | 186 | normx = x 187 | 188 | x = self.conv3(x) 189 | x = self.conv4(x) 190 | 191 | previous = x 192 | 193 | outputs = [] 194 | for i in range(self.n_stack): 195 | hg = self._modules['m' + str(i)](previous) 196 | 197 | ll = hg 198 | ll = self._modules['top_m_' + str(i)](ll) 199 | 200 | ll = F.relu(self._modules['bn_end' + str(i)] 201 | (self._modules['conv_last' + str(i)](ll)), True) 202 | 203 | tmp_out = self._modules['l' + str(i)](ll) 204 | 205 | if self.use_sigmoid: 206 | outputs.append(nn.Tanh()(tmp_out)) 207 | else: 208 | outputs.append(tmp_out) 209 | 210 | if i < self.n_stack - 1: 211 | ll = self._modules['bl' + str(i)](ll) 212 | tmp_out_ = self._modules['al' + str(i)](tmp_out) 213 | previous = previous + ll + tmp_out_ 214 | 215 | return outputs, normx 216 | -------------------------------------------------------------------------------- /lib/model/HGPIFuMRNet.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import numpy as np 4 | import torch 5 | import torch.nn as nn 6 | import torch.nn.functional as F 7 | from .BasePIFuNet import BasePIFuNet 8 | from .MLP import MLP 9 | from .DepthNormalizer import DepthNormalizer 10 | from .HGFilters import HGFilter 11 | from ..net_util import init_net 12 | import cv2 13 | 14 | class HGPIFuMRNet(BasePIFuNet): 15 | ''' 16 | HGPIFu uses stacked hourglass as an image encoder. 17 | ''' 18 | 19 | def __init__(self, 20 | opt, 21 | netG, 22 | projection_mode='orthogonal', 23 | criteria={'occ': nn.MSELoss()} 24 | ): 25 | super(HGPIFuMRNet, self).__init__( 26 | projection_mode=projection_mode, 27 | criteria=criteria) 28 | 29 | self.name = 'hg_pifu' 30 | 31 | in_ch = 3 32 | try: 33 | if netG.opt.use_front_normal: 34 | in_ch += 3 35 | if netG.opt.use_back_normal: 36 | in_ch += 3 37 | except: 38 | pass 39 | 40 | self.opt = opt 41 | self.image_filter = HGFilter(opt.num_stack, opt.hg_depth, in_ch, opt.hg_dim, 42 | opt.norm, 'no_down', False) 43 | 44 | self.mlp = MLP( 45 | filter_channels=self.opt.mlp_dim, 46 | merge_layer=-1, 47 | res_layers=self.opt.mlp_res_layers, 48 | norm=self.opt.mlp_norm, 49 | last_op=nn.Sigmoid()) 50 | 51 | self.im_feat_list = [] 52 | self.preds_interm = None 53 | self.preds_low = None 54 | self.w = None 55 | self.gamma = None 56 | 57 | self.intermediate_preds_list = [] 58 | 59 | init_net(self) 60 | 61 | self.netG = netG 62 | 63 | def train(self, mode=True): 64 | r"""Sets the module in training mode.""" 65 | self.training = mode 66 | for module in self.children(): 67 | module.train(mode) 68 | if not self.opt.train_full_pifu: 69 | self.netG.eval() 70 | return self 71 | 72 | def filter_global(self, images): 73 | ''' 74 | apply a fully convolutional network to images. 75 | the resulting feature will be stored. 76 | args: 77 | images: [B1, C, H, W] 78 | ''' 79 | if self.opt.train_full_pifu: 80 | self.netG.filter(images) 81 | else: 82 | with torch.no_grad(): 83 | self.netG.filter(images) 84 | 85 | def filter_local(self, images, rect=None): 86 | ''' 87 | apply a fully convolutional network to images. 88 | the resulting feature will be stored. 89 | args: 90 | images: [B1, B2, C, H, W] 91 | ''' 92 | nmls = [] 93 | try: 94 | if self.netG.opt.use_front_normal: 95 | nmls.append(self.netG.nmlF) 96 | if self.netG.opt.use_back_normal: 97 | nmls.append(self.netG.nmlB) 98 | except: 99 | pass 100 | 101 | if len(nmls): 102 | nmls = nn.Upsample(size=(self.opt.loadSizeBig,self.opt.loadSizeBig), mode='bilinear', align_corners=True)(torch.cat(nmls,1)) 103 | 104 | # it's kind of damn way. 105 | if rect is None: 106 | images = torch.cat([images, nmls[:,None].expand(-1,images.size(1),-1,-1,-1)], 2) 107 | else: 108 | nml = [] 109 | for i in range(rect.size(0)): 110 | for j in range(rect.size(1)): 111 | x1, y1, x2, y2 = rect[i,j] 112 | tmp = nmls[i,:,y1:y2,x1:x2] 113 | nml.append(nmls[i,:,y1:y2,x1:x2]) 114 | nml = torch.stack(nml, 0).view(*rect.shape[:2],*nml[0].size()) 115 | images = torch.cat([images, nml], 2) 116 | 117 | self.im_feat_list, self.normx = self.image_filter(images.view(-1,*images.size()[2:])) 118 | if not self.training: 119 | self.im_feat_list = [self.im_feat_list[-1]] 120 | 121 | def query(self, points, calib_local, calib_global=None, transforms=None, labels=None): 122 | ''' 123 | given 3d points, we obtain 2d projection of these given the camera matrices. 124 | filter needs to be called beforehand. 125 | the prediction is stored to self.preds 126 | args: 127 | points: [B1, B2, 3, N] 3d points in world space 128 | calibs_local: [B1, B2, 4, 4] calibration matrices for each image 129 | calibs_global: [B1, 4, 4] calibration matrices for each image 130 | transforms: [B1, 2, 3] image space coordinate transforms 131 | labels: [B1, B2, C, N] ground truth labels (for supervision only) 132 | return: 133 | [B, C, N] prediction 134 | ''' 135 | if calib_global is not None: 136 | B = calib_local.size(1) 137 | else: 138 | B = 1 139 | points = points[:,None] 140 | calib_global = calib_local 141 | calib_local = calib_local[:,None] 142 | 143 | ws = [] 144 | preds = [] 145 | preds_interm = [] 146 | preds_low = [] 147 | gammas = [] 148 | newlabels = [] 149 | for i in range(B): 150 | xyz = self.projection(points[:,i], calib_local[:,i], transforms) 151 | 152 | xy = xyz[:, :2, :] 153 | 154 | # if the point is outside bounding box, return outside. 155 | in_bb = (xyz >= -1) & (xyz <= 1) 156 | in_bb = in_bb[:, 0, :] & in_bb[:, 1, :] 157 | in_bb = in_bb[:, None, :].detach().float() 158 | 159 | self.netG.query(points=points[:,i], calibs=calib_global) 160 | preds_low.append(torch.stack(self.netG.intermediate_preds_list,0)) 161 | 162 | if labels is not None: 163 | newlabels.append(in_bb * labels[:,i]) 164 | with torch.no_grad(): 165 | ws.append(in_bb.size(2) / in_bb.view(in_bb.size(0),-1).sum(1)) 166 | gammas.append(1 - newlabels[-1].view(newlabels[-1].size(0),-1).sum(1) / in_bb.view(in_bb.size(0),-1).sum(1)) 167 | 168 | z_feat = self.netG.phi 169 | if not self.opt.train_full_pifu: 170 | z_feat = z_feat.detach() 171 | 172 | intermediate_preds_list = [] 173 | for j, im_feat in enumerate(self.im_feat_list): 174 | point_local_feat_list = [self.index(im_feat.view(-1,B,*im_feat.size()[1:])[:,i], xy), z_feat] 175 | point_local_feat = torch.cat(point_local_feat_list, 1) 176 | pred = self.mlp(point_local_feat)[0] 177 | pred = in_bb * pred 178 | intermediate_preds_list.append(pred) 179 | 180 | preds_interm.append(torch.stack(intermediate_preds_list,0)) 181 | preds.append(intermediate_preds_list[-1]) 182 | 183 | self.preds = torch.cat(preds,0) 184 | self.preds_interm = torch.cat(preds_interm, 1) # first dim is for intermediate predictions 185 | self.preds_low = torch.cat(preds_low, 1) # first dim is for intermediate predictions 186 | 187 | if labels is not None: 188 | self.w = torch.cat(ws,0) 189 | self.gamma = torch.cat(gammas,0) 190 | self.labels = torch.cat(newlabels,0) 191 | 192 | def calc_normal(self, points, calib_local, calib_global, transforms=None, labels=None, delta=0.001, fd_type='forward'): 193 | ''' 194 | return surface normal in 'model' space. 195 | it computes normal only in the last stack. 196 | note that the current implementation use forward difference. 197 | args: 198 | points: [B1, B2, 3, N] 3d points in world space 199 | calibs_local: [B1, B2, 4, 4] calibration matrices for each image 200 | calibs_global: [B1, 4, 4] calibration matrices for each image 201 | transforms: [B1, 2, 3] image space coordinate transforms 202 | labels: [B1, B2, 3, N] ground truth normal 203 | delta: perturbation for finite difference 204 | fd_type: finite difference type (forward/backward/central) 205 | ''' 206 | B = calib_local.size(1) 207 | 208 | if labels is not None: 209 | self.labels_nml = labels.view(-1,*labels.size()[2:]) 210 | 211 | im_feat = self.im_feat_list[-1].view(-1,B,*self.im_feat_list[-1].size()[1:]) 212 | 213 | nmls = [] 214 | for i in range(B): 215 | points_sub = points[:,i] 216 | pdx = points_sub.clone() 217 | pdx[:,0,:] += delta 218 | pdy = points_sub.clone() 219 | pdy[:,1,:] += delta 220 | pdz = points_sub.clone() 221 | pdz[:,2,:] += delta 222 | 223 | points_all = torch.stack([points_sub, pdx, pdy, pdz], 3) 224 | points_all = points_all.view(*points_sub.size()[:2],-1) 225 | xyz = self.projection(points_all, calib_local[:,i], transforms) 226 | xy = xyz[:, :2, :] 227 | 228 | 229 | self.netG.query(points=points_all, calibs=calib_global, update_pred=False) 230 | z_feat = self.netG.phi 231 | if not self.opt.train_full_pifu: 232 | z_feat = z_feat.detach() 233 | 234 | point_local_feat_list = [self.index(im_feat[:,i], xy), z_feat] 235 | point_local_feat = torch.cat(point_local_feat_list, 1) 236 | pred = self.mlp(point_local_feat)[0] 237 | 238 | pred = pred.view(*pred.size()[:2],-1,4) # (B, 1, N, 4) 239 | 240 | # divide by delta is omitted since it's normalized anyway 241 | dfdx = pred[:,:,:,1] - pred[:,:,:,0] 242 | dfdy = pred[:,:,:,2] - pred[:,:,:,0] 243 | dfdz = pred[:,:,:,3] - pred[:,:,:,0] 244 | 245 | nml = -torch.cat([dfdx,dfdy,dfdz], 1) 246 | nml = F.normalize(nml, dim=1, eps=1e-8) 247 | 248 | nmls.append(nml) 249 | 250 | self.nmls = torch.stack(nmls,1).view(-1,3,points.size(3)) 251 | 252 | def get_im_feat(self): 253 | ''' 254 | return the image filter in the last stack 255 | return: 256 | [B, C, H, W] 257 | ''' 258 | return self.im_feat_list[-1] 259 | 260 | def get_error(self): 261 | ''' 262 | return the loss given the ground truth labels and prediction 263 | ''' 264 | 265 | error = {} 266 | if self.opt.train_full_pifu: 267 | if not self.opt.no_intermediate_loss: 268 | error['Err(occ)'] = 0.0 269 | for i in range(self.preds_low.size(0)): 270 | error['Err(occ)'] += self.criteria['occ'](self.preds_low[i], self.labels, self.gamma, self.w) 271 | error['Err(occ)'] /= self.preds_low.size(0) 272 | 273 | error['Err(occ:fine)'] = 0.0 274 | for i in range(self.preds_interm.size(0)): 275 | error['Err(occ:fine)'] += self.criteria['occ'](self.preds_interm[i], self.labels, self.gamma, self.w) 276 | error['Err(occ:fine)'] /= self.preds_interm.size(0) 277 | 278 | if self.nmls is not None and self.labels_nml is not None: 279 | error['Err(nml:fine)'] = self.criteria['nml'](self.nmls, self.labels_nml) 280 | else: 281 | error['Err(occ:fine)'] = 0.0 282 | for i in range(self.preds_interm.size(0)): 283 | error['Err(occ:fine)'] += self.criteria['occ'](self.preds_interm[i], self.labels, self.gamma, self.w) 284 | error['Err(occ:fine)'] /= self.preds_interm.size(0) 285 | 286 | if self.nmls is not None and self.labels_nml is not None: 287 | error['Err(nml:fine)'] = self.criteria['nml'](self.nmls, self.labels_nml) 288 | 289 | return error 290 | 291 | 292 | def forward(self, images_local, images_global, points, calib_local, calib_global, labels, points_nml=None, labels_nml=None, rect=None): 293 | self.filter_global(images_global) 294 | self.filter_local(images_local, rect) 295 | self.query(points, calib_local, calib_global, labels=labels) 296 | if points_nml is not None and labels_nml is not None: 297 | self.calc_normal(points_nml, calib_local, calib_global, labels=labels_nml) 298 | res = self.get_preds() 299 | 300 | err = self.get_error() 301 | 302 | return err, res 303 | -------------------------------------------------------------------------------- /lib/model/HGPIFuNetwNML.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import numpy as np 4 | import torch 5 | import torch.nn as nn 6 | import torch.nn.functional as F 7 | from .BasePIFuNet import BasePIFuNet 8 | from .MLP import MLP 9 | from .DepthNormalizer import DepthNormalizer 10 | from .HGFilters import HGFilter 11 | from ..net_util import init_net 12 | from ..networks import define_G 13 | import cv2 14 | 15 | class HGPIFuNetwNML(BasePIFuNet): 16 | ''' 17 | HGPIFu uses stacked hourglass as an image encoder. 18 | ''' 19 | 20 | def __init__(self, 21 | opt, 22 | projection_mode='orthogonal', 23 | criteria={'occ': nn.MSELoss()} 24 | ): 25 | super(HGPIFuNetwNML, self).__init__( 26 | projection_mode=projection_mode, 27 | criteria=criteria) 28 | 29 | self.name = 'hg_pifu' 30 | 31 | in_ch = 3 32 | try: 33 | if opt.use_front_normal: 34 | in_ch += 3 35 | if opt.use_back_normal: 36 | in_ch += 3 37 | except: 38 | pass 39 | self.opt = opt 40 | self.image_filter = HGFilter(opt.num_stack, opt.hg_depth, in_ch, opt.hg_dim, 41 | opt.norm, opt.hg_down, False) 42 | 43 | self.mlp = MLP( 44 | filter_channels=self.opt.mlp_dim, 45 | merge_layer=self.opt.merge_layer, 46 | res_layers=self.opt.mlp_res_layers, 47 | norm=self.opt.mlp_norm, 48 | last_op=nn.Sigmoid()) 49 | 50 | self.spatial_enc = DepthNormalizer(opt) 51 | 52 | self.im_feat_list = [] 53 | self.tmpx = None 54 | self.normx = None 55 | self.phi = None 56 | 57 | self.intermediate_preds_list = [] 58 | 59 | init_net(self) 60 | 61 | self.netF = None 62 | self.netB = None 63 | try: 64 | if opt.use_front_normal: 65 | self.netF = define_G(3, 3, 64, "global", 4, 9, 1, 3, "instance") 66 | if opt.use_back_normal: 67 | self.netB = define_G(3, 3, 64, "global", 4, 9, 1, 3, "instance") 68 | except: 69 | pass 70 | self.nmlF = None 71 | self.nmlB = None 72 | 73 | def loadFromHGHPIFu(self, net): 74 | hgnet = net.image_filter 75 | pretrained_dict = hgnet.state_dict() 76 | model_dict = self.image_filter.state_dict() 77 | 78 | pretrained_dict = {k: v for k, v in hgnet.state_dict().items() if k in model_dict} 79 | 80 | for k, v in pretrained_dict.items(): 81 | if v.size() == model_dict[k].size(): 82 | model_dict[k] = v 83 | 84 | not_initialized = set() 85 | 86 | for k, v in model_dict.items(): 87 | if k not in pretrained_dict or v.size() != pretrained_dict[k].size(): 88 | not_initialized.add(k.split('.')[0]) 89 | 90 | print('not initialized', sorted(not_initialized)) 91 | self.image_filter.load_state_dict(model_dict) 92 | 93 | pretrained_dict = net.mlp.state_dict() 94 | model_dict = self.mlp.state_dict() 95 | 96 | pretrained_dict = {k: v for k, v in net.mlp.state_dict().items() if k in model_dict} 97 | 98 | for k, v in pretrained_dict.items(): 99 | if v.size() == model_dict[k].size(): 100 | model_dict[k] = v 101 | 102 | not_initialized = set() 103 | 104 | for k, v in model_dict.items(): 105 | if k not in pretrained_dict or v.size() != pretrained_dict[k].size(): 106 | not_initialized.add(k.split('.')[0]) 107 | 108 | print('not initialized', sorted(not_initialized)) 109 | self.mlp.load_state_dict(model_dict) 110 | 111 | def filter(self, images): 112 | ''' 113 | apply a fully convolutional network to images. 114 | the resulting feature will be stored. 115 | args: 116 | images: [B, C, H, W] 117 | ''' 118 | nmls = [] 119 | # if you wish to train jointly, remove detach etc. 120 | with torch.no_grad(): 121 | if self.netF is not None: 122 | self.nmlF = self.netF.forward(images).detach() 123 | nmls.append(self.nmlF) 124 | if self.netB is not None: 125 | self.nmlB = self.netB.forward(images).detach() 126 | nmls.append(self.nmlB) 127 | if len(nmls) != 0: 128 | nmls = torch.cat(nmls,1) 129 | if images.size()[2:] != nmls.size()[2:]: 130 | nmls = nn.Upsample(size=images.size()[2:], mode='bilinear', align_corners=True)(nmls) 131 | images = torch.cat([images,nmls],1) 132 | 133 | 134 | self.im_feat_list, self.normx = self.image_filter(images) 135 | 136 | if not self.training: 137 | self.im_feat_list = [self.im_feat_list[-1]] 138 | 139 | def query(self, points, calibs, transforms=None, labels=None, update_pred=True, update_phi=True): 140 | ''' 141 | given 3d points, we obtain 2d projection of these given the camera matrices. 142 | filter needs to be called beforehand. 143 | the prediction is stored to self.preds 144 | args: 145 | points: [B, 3, N] 3d points in world space 146 | calibs: [B, 3, 4] calibration matrices for each image 147 | transforms: [B, 2, 3] image space coordinate transforms 148 | labels: [B, C, N] ground truth labels (for supervision only) 149 | return: 150 | [B, C, N] prediction 151 | ''' 152 | xyz = self.projection(points, calibs, transforms) 153 | xy = xyz[:, :2, :] 154 | 155 | # if the point is outside bounding box, return outside. 156 | in_bb = (xyz >= -1) & (xyz <= 1) 157 | in_bb = in_bb[:, 0, :] & in_bb[:, 1, :] & in_bb[:, 2, :] 158 | in_bb = in_bb[:, None, :].detach().float() 159 | 160 | if labels is not None: 161 | self.labels = in_bb * labels 162 | 163 | sp_feat = self.spatial_enc(xyz, calibs=calibs) 164 | 165 | intermediate_preds_list = [] 166 | 167 | phi = None 168 | for i, im_feat in enumerate(self.im_feat_list): 169 | point_local_feat_list = [self.index(im_feat, xy), sp_feat] 170 | point_local_feat = torch.cat(point_local_feat_list, 1) 171 | pred, phi = self.mlp(point_local_feat) 172 | pred = in_bb * pred 173 | 174 | intermediate_preds_list.append(pred) 175 | 176 | if update_phi: 177 | self.phi = phi 178 | 179 | if update_pred: 180 | self.intermediate_preds_list = intermediate_preds_list 181 | self.preds = self.intermediate_preds_list[-1] 182 | 183 | def calc_normal(self, points, calibs, transforms=None, labels=None, delta=0.01, fd_type='forward'): 184 | ''' 185 | return surface normal in 'model' space. 186 | it computes normal only in the last stack. 187 | note that the current implementation use forward difference. 188 | args: 189 | points: [B, 3, N] 3d points in world space 190 | calibs: [B, 3, 4] calibration matrices for each image 191 | transforms: [B, 2, 3] image space coordinate transforms 192 | delta: perturbation for finite difference 193 | fd_type: finite difference type (forward/backward/central) 194 | ''' 195 | pdx = points.clone() 196 | pdx[:,0,:] += delta 197 | pdy = points.clone() 198 | pdy[:,1,:] += delta 199 | pdz = points.clone() 200 | pdz[:,2,:] += delta 201 | 202 | if labels is not None: 203 | self.labels_nml = labels 204 | 205 | points_all = torch.stack([points, pdx, pdy, pdz], 3) 206 | points_all = points_all.view(*points.size()[:2],-1) 207 | xyz = self.projection(points_all, calibs, transforms) 208 | xy = xyz[:, :2, :] 209 | 210 | im_feat = self.im_feat_list[-1] 211 | sp_feat = self.spatial_enc(xyz, calibs=calibs) 212 | 213 | point_local_feat_list = [self.index(im_feat, xy), sp_feat] 214 | point_local_feat = torch.cat(point_local_feat_list, 1) 215 | 216 | pred = self.mlp(point_local_feat)[0] 217 | 218 | pred = pred.view(*pred.size()[:2],-1,4) # (B, 1, N, 4) 219 | 220 | # divide by delta is omitted since it's normalized anyway 221 | dfdx = pred[:,:,:,1] - pred[:,:,:,0] 222 | dfdy = pred[:,:,:,2] - pred[:,:,:,0] 223 | dfdz = pred[:,:,:,3] - pred[:,:,:,0] 224 | 225 | nml = -torch.cat([dfdx,dfdy,dfdz], 1) 226 | nml = F.normalize(nml, dim=1, eps=1e-8) 227 | 228 | self.nmls = nml 229 | 230 | def get_im_feat(self): 231 | ''' 232 | return the image filter in the last stack 233 | return: 234 | [B, C, H, W] 235 | ''' 236 | return self.im_feat_list[-1] 237 | 238 | 239 | def get_error(self, gamma): 240 | ''' 241 | return the loss given the ground truth labels and prediction 242 | ''' 243 | error = {} 244 | error['Err(occ)'] = 0 245 | for preds in self.intermediate_preds_list: 246 | error['Err(occ)'] += self.criteria['occ'](preds, self.labels, gamma) 247 | 248 | error['Err(occ)'] /= len(self.intermediate_preds_list) 249 | 250 | if self.nmls is not None and self.labels_nml is not None: 251 | error['Err(nml)'] = self.criteria['nml'](self.nmls, self.labels_nml) 252 | 253 | return error 254 | 255 | def forward(self, images, points, calibs, labels, gamma, points_nml=None, labels_nml=None): 256 | self.filter(images) 257 | self.query(points, calibs, labels=labels) 258 | if points_nml is not None and labels_nml is not None: 259 | self.calc_normal(points_nml, calibs, labels=labels_nml) 260 | res = self.get_preds() 261 | 262 | err = self.get_error(gamma) 263 | 264 | return err, res 265 | -------------------------------------------------------------------------------- /lib/model/MLP.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import torch 4 | import torch.nn as nn 5 | import torch.nn.functional as F 6 | 7 | class MLP(nn.Module): 8 | def __init__(self, 9 | filter_channels, 10 | merge_layer=0, 11 | res_layers=[], 12 | norm='group', 13 | last_op=None): 14 | super(MLP, self).__init__() 15 | 16 | self.filters = nn.ModuleList() 17 | self.norms = nn.ModuleList() 18 | self.merge_layer = merge_layer if merge_layer > 0 else len(filter_channels) // 2 19 | self.res_layers = res_layers 20 | self.norm = norm 21 | self.last_op = last_op 22 | 23 | for l in range(0, len(filter_channels)-1): 24 | if l in self.res_layers: 25 | self.filters.append(nn.Conv1d( 26 | filter_channels[l] + filter_channels[0], 27 | filter_channels[l+1], 28 | 1)) 29 | else: 30 | self.filters.append(nn.Conv1d( 31 | filter_channels[l], 32 | filter_channels[l+1], 33 | 1)) 34 | if l != len(filter_channels)-2: 35 | if norm == 'group': 36 | self.norms.append(nn.GroupNorm(32, filter_channels[l+1])) 37 | elif norm == 'batch': 38 | self.norms.append(nn.BatchNorm1d(filter_channels[l+1])) 39 | 40 | def forward(self, feature): 41 | ''' 42 | feature may include multiple view inputs 43 | args: 44 | feature: [B, C_in, N] 45 | return: 46 | [B, C_out, N] prediction 47 | ''' 48 | y = feature 49 | tmpy = feature 50 | phi = None 51 | for i, f in enumerate(self.filters): 52 | y = f( 53 | y if i not in self.res_layers 54 | else torch.cat([y, tmpy], 1) 55 | ) 56 | if i != len(self.filters)-1: 57 | if self.norm not in ['batch', 'group']: 58 | y = F.leaky_relu(y) 59 | else: 60 | y = F.leaky_relu(self.norms[i](y)) 61 | if i == self.merge_layer: 62 | phi = y.clone() 63 | 64 | if self.last_op is not None: 65 | y = self.last_op(y) 66 | 67 | return y, phi 68 | -------------------------------------------------------------------------------- /lib/model/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | from .MLP import MLP 4 | from .HGPIFuMRNet import HGPIFuMRNet 5 | from .HGPIFuNetwNML import HGPIFuNetwNML -------------------------------------------------------------------------------- /lib/net_util.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import torch 25 | from torch.nn import init 26 | import torch.nn as nn 27 | import torch.nn.functional as F 28 | import functools 29 | 30 | def load_state_dict(state_dict, net): 31 | model_dict = net.state_dict() 32 | 33 | pretrained_dict = {k: v for k, v in state_dict.items() if k in model_dict} 34 | 35 | for k, v in pretrained_dict.items(): 36 | if v.size() == model_dict[k].size(): 37 | model_dict[k] = v 38 | 39 | not_initialized = set() 40 | 41 | for k, v in model_dict.items(): 42 | if k not in pretrained_dict or v.size() != pretrained_dict[k].size(): 43 | not_initialized.add(k.split('.')[0]) 44 | 45 | print('not initialized', sorted(not_initialized)) 46 | net.load_state_dict(model_dict) 47 | 48 | return net 49 | 50 | def conv3x3(in_planes, out_planes, strd=1, padding=1, bias=False): 51 | return nn.Conv2d(in_planes, out_planes, kernel_size=3, 52 | stride=strd, padding=padding, bias=bias) 53 | 54 | def init_weights(net, init_type='normal', init_gain=0.02): 55 | def init_func(m): # define the initialization function 56 | classname = m.__class__.__name__ 57 | if hasattr(m, 'weight') and (classname.find('Conv') != -1 or classname.find('Linear') != -1): 58 | if init_type == 'normal': 59 | init.normal_(m.weight.data, 0.0, init_gain) 60 | elif init_type == 'xavier': 61 | init.xavier_normal_(m.weight.data, gain=init_gain) 62 | elif init_type == 'kaiming': 63 | init.kaiming_normal_(m.weight.data, a=0, mode='fan_in') 64 | elif init_type == 'orthogonal': 65 | init.orthogonal_(m.weight.data, gain=init_gain) 66 | else: 67 | raise NotImplementedError('initialization method [%s] is not implemented' % init_type) 68 | if hasattr(m, 'bias') and m.bias is not None: 69 | init.constant_(m.bias.data, 0.0) 70 | elif classname.find( 71 | 'BatchNorm2d') != -1: 72 | init.normal_(m.weight.data, 1.0, init_gain) 73 | init.constant_(m.bias.data, 0.0) 74 | 75 | print('initialize network with %s' % init_type) 76 | net.apply(init_func) 77 | 78 | def init_net(net, init_type='normal', init_gain=0.02, gpu_ids=[]): 79 | if len(gpu_ids) > 0: 80 | assert (torch.cuda.is_available()) 81 | net.to(gpu_ids[0]) 82 | net = torch.nn.DataParallel(net, gpu_ids) # multi-GPUs 83 | init_weights(net, init_type, init_gain=init_gain) 84 | return net 85 | 86 | class CustomBCELoss(nn.Module): 87 | def __init__(self, brock=False, gamma=None): 88 | super(CustomBCELoss, self).__init__() 89 | self.brock = brock 90 | self.gamma = gamma 91 | 92 | def forward(self, pred, gt, gamma, w=None): 93 | x_hat = torch.clamp(pred, 1e-5, 1.0-1e-5) # prevent log(0) from happening 94 | gamma = gamma[:,None,None] if self.gamma is None else self.gamma 95 | if self.brock: 96 | x = 3.0*gt - 1.0 # rescaled to [-1,2] 97 | 98 | loss = -(gamma*x*torch.log(x_hat) + (1.0-gamma)*(1.0-x)*torch.log(1.0-x_hat)) 99 | else: 100 | loss = -(gamma*gt*torch.log(x_hat) + (1.0-gamma)*(1.0-gt)*torch.log(1.0-x_hat)) 101 | 102 | if w is not None: 103 | if len(w.size()) == 1: 104 | w = w[:,None,None] 105 | return (loss * w).mean() 106 | else: 107 | return loss.mean() 108 | 109 | class CustomMSELoss(nn.Module): 110 | def __init__(self, gamma=None): 111 | super(CustomMSELoss, self).__init__() 112 | self.gamma = gamma 113 | 114 | def forward(self, pred, gt, gamma, w=None): 115 | gamma = gamma[:,None,None] if self.gamma is None else self.gamma 116 | weight = gamma * gt + (1.0-gamma) * (1 - gt) 117 | loss = (weight * (pred - gt).pow(2)).mean() 118 | 119 | if w is not None: 120 | return (loss * w).mean() 121 | else: 122 | return loss.mean() 123 | 124 | def createMLP(dims, norm='bn', activation='relu', last_op=nn.Tanh(), dropout=False): 125 | act = None 126 | if activation == 'relu': 127 | act = nn.ReLU() 128 | if activation == 'lrelu': 129 | act = nn.LeakyReLU() 130 | if activation == 'selu': 131 | act = nn.SELU() 132 | if activation == 'elu': 133 | act = nn.ELU() 134 | if activation == 'prelu': 135 | act = nn.PReLU() 136 | 137 | mlp = [] 138 | for i in range(1,len(dims)): 139 | if norm == 'bn': 140 | mlp += [ nn.Linear(dims[i-1], dims[i]), 141 | nn.BatchNorm1d(dims[i])] 142 | if norm == 'in': 143 | mlp += [ nn.Linear(dims[i-1], dims[i]), 144 | nn.InstanceNorm1d(dims[i])] 145 | if norm == 'wn': 146 | mlp += [ nn.utils.weight_norm(nn.Linear(dims[i-1], dims[i]), name='weight')] 147 | if norm == 'none': 148 | mlp += [ nn.Linear(dims[i-1], dims[i])] 149 | 150 | if i != len(dims)-1: 151 | if act is not None: 152 | mlp += [act] 153 | if dropout: 154 | mlp += [nn.Dropout(0.2)] 155 | 156 | if last_op is not None: 157 | mlp += [last_op] 158 | 159 | return mlp -------------------------------------------------------------------------------- /lib/networks.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright (C) 2019 NVIDIA Corporation. Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu. 3 | BSD License. All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | 8 | * Redistributions of source code must retain the above copyright notice, this 9 | list of conditions and the following disclaimer. 10 | 11 | * Redistributions in binary form must reproduce the above copyright notice, 12 | this list of conditions and the following disclaimer in the documentation 13 | and/or other materials provided with the distribution. 14 | 15 | THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL 16 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR ANY PARTICULAR PURPOSE. 17 | IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL 18 | DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, 19 | WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING 20 | OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 21 | ''' 22 | import torch 23 | import torch.nn as nn 24 | import functools 25 | from torch.autograd import Variable 26 | import numpy as np 27 | 28 | ############################################################################### 29 | # Functions 30 | ############################################################################### 31 | def weights_init(m): 32 | classname = m.__class__.__name__ 33 | if classname.find('Conv') != -1: 34 | m.weight.data.normal_(0.0, 0.02) 35 | elif classname.find('BatchNorm2d') != -1: 36 | m.weight.data.normal_(1.0, 0.02) 37 | m.bias.data.fill_(0) 38 | 39 | def get_norm_layer(norm_type='instance'): 40 | if norm_type == 'batch': 41 | norm_layer = functools.partial(nn.BatchNorm2d, affine=True) 42 | elif norm_type == 'instance': 43 | norm_layer = functools.partial(nn.InstanceNorm2d, affine=False) 44 | else: 45 | raise NotImplementedError('normalization layer [%s] is not found' % norm_type) 46 | return norm_layer 47 | 48 | def define_G(input_nc, output_nc, ngf, netG, n_downsample_global=3, n_blocks_global=9, n_local_enhancers=1, 49 | n_blocks_local=3, norm='instance', gpu_ids=[], last_op=nn.Tanh()): 50 | norm_layer = get_norm_layer(norm_type=norm) 51 | if netG == 'global': 52 | netG = GlobalGenerator(input_nc, output_nc, ngf, n_downsample_global, n_blocks_global, norm_layer, last_op=last_op) 53 | elif netG == 'local': 54 | netG = LocalEnhancer(input_nc, output_nc, ngf, n_downsample_global, n_blocks_global, 55 | n_local_enhancers, n_blocks_local, norm_layer) 56 | elif netG == 'encoder': 57 | netG = Encoder(input_nc, output_nc, ngf, n_downsample_global, norm_layer) 58 | else: 59 | raise('generator not implemented!') 60 | # print(netG) 61 | if len(gpu_ids) > 0: 62 | assert(torch.cuda.is_available()) 63 | netG.cuda(gpu_ids[0]) 64 | netG.apply(weights_init) 65 | return netG 66 | 67 | def print_network(net): 68 | if isinstance(net, list): 69 | net = net[0] 70 | num_params = 0 71 | for param in net.parameters(): 72 | num_params += param.numel() 73 | print(net) 74 | print('Total number of parameters: %d' % num_params) 75 | 76 | ############################################################################## 77 | # Generator 78 | ############################################################################## 79 | class LocalEnhancer(nn.Module): 80 | def __init__(self, input_nc, output_nc, ngf=32, n_downsample_global=3, n_blocks_global=9, 81 | n_local_enhancers=1, n_blocks_local=3, norm_layer=nn.BatchNorm2d, padding_type='reflect'): 82 | super(LocalEnhancer, self).__init__() 83 | self.n_local_enhancers = n_local_enhancers 84 | 85 | ###### global generator model ##### 86 | ngf_global = ngf * (2**n_local_enhancers) 87 | model_global = GlobalGenerator(input_nc, output_nc, ngf_global, n_downsample_global, n_blocks_global, norm_layer).model 88 | model_global = [model_global[i] for i in range(len(model_global)-3)] # get rid of final convolution layers 89 | self.model = nn.Sequential(*model_global) 90 | 91 | ###### local enhancer layers ##### 92 | for n in range(1, n_local_enhancers+1): 93 | ### downsample 94 | ngf_global = ngf * (2**(n_local_enhancers-n)) 95 | model_downsample = [nn.ReflectionPad2d(3), nn.Conv2d(input_nc, ngf_global, kernel_size=7, padding=0), 96 | norm_layer(ngf_global), nn.ReLU(True), 97 | nn.Conv2d(ngf_global, ngf_global * 2, kernel_size=3, stride=2, padding=1), 98 | norm_layer(ngf_global * 2), nn.ReLU(True)] 99 | ### residual blocks 100 | model_upsample = [] 101 | for i in range(n_blocks_local): 102 | model_upsample += [ResnetBlock(ngf_global * 2, padding_type=padding_type, norm_layer=norm_layer)] 103 | 104 | ### upsample 105 | model_upsample += [nn.ConvTranspose2d(ngf_global * 2, ngf_global, kernel_size=3, stride=2, padding=1, output_padding=1), 106 | norm_layer(ngf_global), nn.ReLU(True)] 107 | 108 | ### final convolution 109 | if n == n_local_enhancers: 110 | model_upsample += [nn.ReflectionPad2d(3), nn.Conv2d(ngf, output_nc, kernel_size=7, padding=0), nn.Tanh()] 111 | 112 | setattr(self, 'model'+str(n)+'_1', nn.Sequential(*model_downsample)) 113 | setattr(self, 'model'+str(n)+'_2', nn.Sequential(*model_upsample)) 114 | 115 | self.downsample = nn.AvgPool2d(3, stride=2, padding=[1, 1], count_include_pad=False) 116 | 117 | def forward(self, input): 118 | ### create input pyramid 119 | input_downsampled = [input] 120 | for i in range(self.n_local_enhancers): 121 | input_downsampled.append(self.downsample(input_downsampled[-1])) 122 | 123 | ### output at coarest level 124 | output_prev = self.model(input_downsampled[-1]) 125 | ### build up one layer at a time 126 | for n_local_enhancers in range(1, self.n_local_enhancers+1): 127 | model_downsample = getattr(self, 'model'+str(n_local_enhancers)+'_1') 128 | model_upsample = getattr(self, 'model'+str(n_local_enhancers)+'_2') 129 | input_i = input_downsampled[self.n_local_enhancers-n_local_enhancers] 130 | output_prev = model_upsample(model_downsample(input_i) + output_prev) 131 | return output_prev 132 | 133 | class GlobalGenerator(nn.Module): 134 | def __init__(self, input_nc, output_nc, ngf=64, n_downsampling=3, n_blocks=9, norm_layer=nn.BatchNorm2d, 135 | padding_type='reflect', last_op=nn.Tanh()): 136 | assert(n_blocks >= 0) 137 | super(GlobalGenerator, self).__init__() 138 | activation = nn.ReLU(True) 139 | 140 | model = [nn.ReflectionPad2d(3), nn.Conv2d(input_nc, ngf, kernel_size=7, padding=0), norm_layer(ngf), activation] 141 | ### downsample 142 | for i in range(n_downsampling): 143 | mult = 2**i 144 | model += [nn.Conv2d(ngf * mult, ngf * mult * 2, kernel_size=3, stride=2, padding=1), 145 | norm_layer(ngf * mult * 2), activation] 146 | 147 | ### resnet blocks 148 | mult = 2**n_downsampling 149 | for i in range(n_blocks): 150 | model += [ResnetBlock(ngf * mult, padding_type=padding_type, activation=activation, norm_layer=norm_layer)] 151 | 152 | ### upsample 153 | for i in range(n_downsampling): 154 | mult = 2**(n_downsampling - i) 155 | model += [nn.ConvTranspose2d(ngf * mult, int(ngf * mult / 2), kernel_size=3, stride=2, padding=1, output_padding=1), 156 | norm_layer(int(ngf * mult / 2)), activation] 157 | model += [nn.ReflectionPad2d(3), nn.Conv2d(ngf, output_nc, kernel_size=7, padding=0)] 158 | if last_op is not None: 159 | model += [last_op] 160 | self.model = nn.Sequential(*model) 161 | 162 | def forward(self, input): 163 | return self.model(input) 164 | 165 | # Define a resnet block 166 | class ResnetBlock(nn.Module): 167 | def __init__(self, dim, padding_type, norm_layer, activation=nn.ReLU(True), use_dropout=False): 168 | super(ResnetBlock, self).__init__() 169 | self.conv_block = self.build_conv_block(dim, padding_type, norm_layer, activation, use_dropout) 170 | 171 | def build_conv_block(self, dim, padding_type, norm_layer, activation, use_dropout): 172 | conv_block = [] 173 | p = 0 174 | if padding_type == 'reflect': 175 | conv_block += [nn.ReflectionPad2d(1)] 176 | elif padding_type == 'replicate': 177 | conv_block += [nn.ReplicationPad2d(1)] 178 | elif padding_type == 'zero': 179 | p = 1 180 | else: 181 | raise NotImplementedError('padding [%s] is not implemented' % padding_type) 182 | 183 | conv_block += [nn.Conv2d(dim, dim, kernel_size=3, padding=p), 184 | norm_layer(dim), 185 | activation] 186 | if use_dropout: 187 | conv_block += [nn.Dropout(0.5)] 188 | 189 | p = 0 190 | if padding_type == 'reflect': 191 | conv_block += [nn.ReflectionPad2d(1)] 192 | elif padding_type == 'replicate': 193 | conv_block += [nn.ReplicationPad2d(1)] 194 | elif padding_type == 'zero': 195 | p = 1 196 | else: 197 | raise NotImplementedError('padding [%s] is not implemented' % padding_type) 198 | conv_block += [nn.Conv2d(dim, dim, kernel_size=3, padding=p), 199 | norm_layer(dim)] 200 | 201 | return nn.Sequential(*conv_block) 202 | 203 | def forward(self, x): 204 | out = x + self.conv_block(x) 205 | return out 206 | 207 | class Encoder(nn.Module): 208 | def __init__(self, input_nc, output_nc, ngf=32, n_downsampling=4, norm_layer=nn.BatchNorm2d): 209 | super(Encoder, self).__init__() 210 | self.output_nc = output_nc 211 | 212 | model = [nn.ReflectionPad2d(3), nn.Conv2d(input_nc, ngf, kernel_size=7, padding=0), 213 | norm_layer(ngf), nn.ReLU(True)] 214 | ### downsample 215 | for i in range(n_downsampling): 216 | mult = 2**i 217 | model += [nn.Conv2d(ngf * mult, ngf * mult * 2, kernel_size=3, stride=2, padding=1), 218 | norm_layer(ngf * mult * 2), nn.ReLU(True)] 219 | 220 | ### upsample 221 | for i in range(n_downsampling): 222 | mult = 2**(n_downsampling - i) 223 | model += [nn.ConvTranspose2d(ngf * mult, int(ngf * mult / 2), kernel_size=3, stride=2, padding=1, output_padding=1), 224 | norm_layer(int(ngf * mult / 2)), nn.ReLU(True)] 225 | 226 | model += [nn.ReflectionPad2d(3), nn.Conv2d(ngf, output_nc, kernel_size=7, padding=0), nn.Tanh()] 227 | self.model = nn.Sequential(*model) 228 | 229 | def forward(self, input, inst): 230 | outputs = self.model(input) 231 | 232 | # instance-wise average pooling 233 | outputs_mean = outputs.clone() 234 | inst_list = np.unique(inst.cpu().numpy().astype(int)) 235 | for i in inst_list: 236 | for b in range(input.size()[0]): 237 | indices = (inst[b:b+1] == int(i)).nonzero() # n x 4 238 | for j in range(self.output_nc): 239 | output_ins = outputs[indices[:,0] + b, indices[:,1] + j, indices[:,2], indices[:,3]] 240 | mean_feat = torch.mean(output_ins).expand_as(output_ins) 241 | outputs_mean[indices[:,0] + b, indices[:,1] + j, indices[:,2], indices[:,3]] = mean_feat 242 | return outputs_mean 243 | -------------------------------------------------------------------------------- /lib/options.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | import argparse 4 | import os 5 | 6 | class BaseOptions(): 7 | def __init__(self): 8 | self.initialized = False 9 | self.parser = None 10 | 11 | def initialize(self, parser): 12 | # Datasets related 13 | g_data = parser.add_argument_group('Data') 14 | g_data.add_argument('--dataset', type=str, default='renderppl', help='dataset name') 15 | g_data.add_argument('--dataroot', type=str, default='./data', 16 | help='path to images (data folder)') 17 | 18 | g_data.add_argument('--loadSize', type=int, default=512, help='load size of input image') 19 | 20 | # Experiment related 21 | g_exp = parser.add_argument_group('Experiment') 22 | g_exp.add_argument('--name', type=str, default='', 23 | help='name of the experiment. It decides where to store samples and models') 24 | g_exp.add_argument('--debug', action='store_true', help='debug mode or not') 25 | g_exp.add_argument('--mode', type=str, default='inout', help='inout || color') 26 | 27 | # Training related 28 | g_train = parser.add_argument_group('Training') 29 | g_train.add_argument('--tmp_id', type=int, default=0, help='tmp_id') 30 | g_train.add_argument('--gpu_id', type=int, default=0, help='gpu id for cuda') 31 | g_train.add_argument('--batch_size', type=int, default=32, help='input batch size') 32 | g_train.add_argument('--num_threads', default=1, type=int, help='# sthreads for loading data') 33 | g_train.add_argument('--serial_batches', action='store_true', 34 | help='if true, takes images in order to make batches, otherwise takes them randomly') 35 | g_train.add_argument('--pin_memory', action='store_true', help='pin_memory') 36 | g_train.add_argument('--learning_rate', type=float, default=1e-3, help='adam learning rate') 37 | g_train.add_argument('--num_iter', type=int, default=30000, help='num iterations to train') 38 | g_train.add_argument('--freq_plot', type=int, default=100, help='freqency of the error plot') 39 | g_train.add_argument('--freq_mesh', type=int, default=20000, help='freqency of the save_checkpoints') 40 | g_train.add_argument('--freq_eval', type=int, default=5000, help='freqency of the save_checkpoints') 41 | g_train.add_argument('--freq_save_ply', type=int, default=5000, help='freqency of the save ply') 42 | g_train.add_argument('--freq_save_image', type=int, default=100, help='freqency of the save input image') 43 | g_train.add_argument('--resume_epoch', type=int, default=-1, help='epoch resuming the training') 44 | g_train.add_argument('--continue_train', action='store_true', help='continue training: load the latest model') 45 | g_train.add_argument('--finetune', action='store_true', help='fine tuning netG in training C') 46 | 47 | # Testing related 48 | g_test = parser.add_argument_group('Testing') 49 | g_test.add_argument('--resolution', type=int, default=512, help='# of grid in mesh reconstruction') 50 | g_test.add_argument('--no_numel_eval', action='store_true', help='no numerical evaluation') 51 | g_test.add_argument('--no_mesh_recon', action='store_true', help='no mesh reconstruction') 52 | 53 | # Sampling related 54 | g_sample = parser.add_argument_group('Sampling') 55 | g_sample.add_argument('--num_sample_inout', type=int, default=6000, help='# of sampling points') 56 | g_sample.add_argument('--num_sample_surface', type=int, default=0, help='# of sampling points') 57 | g_sample.add_argument('--num_sample_normal', type=int, default=0, help='# of sampling points') 58 | g_sample.add_argument('--num_sample_color', type=int, default=0, help='# of sampling points') 59 | g_sample.add_argument('--num_pts_dic', type=int, default=1, help='# of pts dic you load') 60 | 61 | g_sample.add_argument('--crop_type', type=str, default='fullbody', help='Sampling file name.') 62 | g_sample.add_argument('--uniform_ratio', type=float, default=0.1, help='maximum sigma for sampling') 63 | g_sample.add_argument('--mask_ratio', type=float, default=0.5, help='maximum sigma for sampling') 64 | g_sample.add_argument('--sampling_parts', action='store_true', help='Sampling on the fly') 65 | g_sample.add_argument('--sampling_otf', action='store_true', help='Sampling on the fly') 66 | g_sample.add_argument('--sampling_mode', type=str, default='sigma_uniform', help='Sampling file name.') 67 | g_sample.add_argument('--linear_anneal_sigma', action='store_true', help='linear annealing of sigma') 68 | g_sample.add_argument('--sigma_max', type=float, default=0.0, help='maximum sigma for sampling') 69 | g_sample.add_argument('--sigma_min', type=float, default=0.0, help='minimum sigma for sampling') 70 | g_sample.add_argument('--sigma', type=float, default=1.0, help='sigma for sampling') 71 | g_sample.add_argument('--sigma_surface', type=float, default=1.0, help='sigma for sampling') 72 | 73 | g_sample.add_argument('--z_size', type=float, default=200.0, help='z normalization factor') 74 | 75 | # Model related 76 | g_model = parser.add_argument_group('Model') 77 | # General 78 | g_model.add_argument('--norm', type=str, default='batch', 79 | help='instance normalization or batch normalization or group normalization') 80 | 81 | # Image filter General 82 | g_model.add_argument('--netG', type=str, default='hgpifu', help='piximp | fanimp | hghpifu') 83 | g_model.add_argument('--netC', type=str, default='resblkpifu', help='resblkpifu | resblkhpifu') 84 | 85 | # hgimp specific 86 | g_model.add_argument('--num_stack', type=int, default=4, help='# of hourglass') 87 | g_model.add_argument('--hg_depth', type=int, default=2, help='# of stacked layer of hourglass') 88 | g_model.add_argument('--hg_down', type=str, default='ave_pool', help='ave pool || conv64 || conv128') 89 | g_model.add_argument('--hg_dim', type=int, default=256, help='256 | 512') 90 | 91 | # Classification General 92 | g_model.add_argument('--mlp_norm', type=str, default='group', help='normalization for volume branch') 93 | g_model.add_argument('--mlp_dim', nargs='+', default=[257, 1024, 512, 256, 128, 1], type=int, 94 | help='# of dimensions of mlp. no need to put the first channel') 95 | g_model.add_argument('--mlp_dim_color', nargs='+', default=[1024, 512, 256, 128, 3], type=int, 96 | help='# of dimensions of mlp. no need to put the first channel') 97 | g_model.add_argument('--mlp_res_layers', nargs='+', default=[2,3,4], type=int, 98 | help='leyers that has skip connection. use 0 for no residual pass') 99 | g_model.add_argument('--merge_layer', type=int, default=-1) 100 | 101 | # for train 102 | parser.add_argument('--random_body_chop', action='store_true', help='if random flip') 103 | parser.add_argument('--random_flip', action='store_true', help='if random flip') 104 | parser.add_argument('--random_trans', action='store_true', help='if random flip') 105 | parser.add_argument('--random_scale', action='store_true', help='if random flip') 106 | parser.add_argument('--random_rotate', action='store_true', help='if random flip') 107 | parser.add_argument('--random_bg', action='store_true', help='using random background') 108 | 109 | parser.add_argument('--schedule', type=int, nargs='+', default=[10, 15], 110 | help='Decrease learning rate at these epochs.') 111 | parser.add_argument('--gamma', type=float, default=0.1, help='LR is multiplied by gamma on schedule.') 112 | parser.add_argument('--lambda_nml', type=float, default=0.0, help='weight of normal loss') 113 | parser.add_argument('--lambda_cmp_l1', type=float, default=0.0, help='weight of normal loss') 114 | parser.add_argument('--occ_loss_type', type=str, default='mse', help='bce | brock_bce | mse') 115 | parser.add_argument('--clr_loss_type', type=str, default='mse', help='mse | l1') 116 | parser.add_argument('--nml_loss_type', type=str, default='mse', help='mse | l1') 117 | parser.add_argument('--occ_gamma', type=float, default=None, help='weighting term') 118 | parser.add_argument('--no_finetune', action='store_true', help='fine tuning netG in training C') 119 | 120 | # for eval 121 | parser.add_argument('--val_test_error', action='store_true', help='validate errors of test data') 122 | parser.add_argument('--val_train_error', action='store_true', help='validate errors of train data') 123 | parser.add_argument('--gen_test_mesh', action='store_true', help='generate test mesh') 124 | parser.add_argument('--gen_train_mesh', action='store_true', help='generate train mesh') 125 | parser.add_argument('--all_mesh', action='store_true', help='generate meshs from all hourglass output') 126 | parser.add_argument('--num_gen_mesh_test', type=int, default=4, 127 | help='how many meshes to generate during testing') 128 | 129 | # path 130 | parser.add_argument('--load_netG_checkpoint_path', type=str, help='path to save checkpoints') 131 | parser.add_argument('--load_netC_checkpoint_path', type=str, help='path to save checkpoints') 132 | parser.add_argument('--checkpoints_path', type=str, default='./checkpoints', help='path to save checkpoints') 133 | parser.add_argument('--results_path', type=str, default='./results', help='path to save results ply') 134 | parser.add_argument('--load_checkpoint_path', type=str, help='path to save results ply') 135 | parser.add_argument('--single', type=str, default='', help='single data for training') 136 | 137 | # for single image reconstruction 138 | parser.add_argument('--mask_path', type=str, help='path for input mask') 139 | parser.add_argument('--img_path', type=str, help='path for input image') 140 | 141 | # for multi resolution 142 | parser.add_argument('--load_netMR_checkpoint_path', type=str, help='path to save checkpoints') 143 | parser.add_argument('--loadSizeBig', type=int, default=1024, help='load size of input image') 144 | parser.add_argument('--loadSizeLocal', type=int, default=512, help='load size of input image') 145 | parser.add_argument('--train_full_pifu', action='store_true', help='enable end-to-end training') 146 | parser.add_argument('--num_local', type=int, default=1, help='number of local cropping') 147 | 148 | # for normal condition 149 | parser.add_argument('--load_netFB_checkpoint_path', type=str, help='path to save checkpoints') 150 | parser.add_argument('--load_netF_checkpoint_path', type=str, help='path to save checkpoints') 151 | parser.add_argument('--load_netB_checkpoint_path', type=str, help='path to save checkpoints') 152 | parser.add_argument('--use_aio_normal', action='store_true') 153 | parser.add_argument('--use_front_normal', action='store_true') 154 | parser.add_argument('--use_back_normal', action='store_true') 155 | parser.add_argument('--no_intermediate_loss', action='store_true') 156 | 157 | # aug 158 | group_aug = parser.add_argument_group('aug') 159 | group_aug.add_argument('--aug_alstd', type=float, default=0.0, help='augmentation pca lighting alpha std') 160 | group_aug.add_argument('--aug_bri', type=float, default=0.2, help='augmentation brightness') 161 | group_aug.add_argument('--aug_con', type=float, default=0.2, help='augmentation contrast') 162 | group_aug.add_argument('--aug_sat', type=float, default=0.05, help='augmentation saturation') 163 | group_aug.add_argument('--aug_hue', type=float, default=0.05, help='augmentation hue') 164 | group_aug.add_argument('--aug_gry', type=float, default=0.1, help='augmentation gray scale') 165 | group_aug.add_argument('--aug_blur', type=float, default=0.0, help='augmentation blur') 166 | 167 | # for reconstruction 168 | parser.add_argument('--start_id', type=int, default=-1, help='load size of input image') 169 | parser.add_argument('--end_id', type=int, default=-1, help='load size of input image') 170 | 171 | # special tasks 172 | self.initialized = True 173 | return parser 174 | 175 | def gather_options(self, args=None): 176 | # initialize parser with basic options 177 | if not self.initialized: 178 | parser = argparse.ArgumentParser( 179 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 180 | parser = self.initialize(parser) 181 | self.parser = parser 182 | 183 | if args is None: 184 | return self.parser.parse_args() 185 | else: 186 | return self.parser.parse_args(args) 187 | 188 | def print_options(self, opt): 189 | message = '' 190 | message += '----------------- Options ---------------\n' 191 | for k, v in sorted(vars(opt).items()): 192 | comment = '' 193 | default = self.parser.get_default(k) 194 | if v != default: 195 | comment = '\t[default: %s]' % str(default) 196 | message += '{:>25}: {:<30}{}\n'.format(str(k), str(v), comment) 197 | message += '----------------- End -------------------' 198 | print(message) 199 | 200 | def parse(self, args=None): 201 | opt = self.gather_options(args) 202 | 203 | opt.sigma = opt.sigma_max 204 | 205 | if len(opt.mlp_res_layers) == 1 and opt.mlp_res_layers[0] < 1: 206 | opt.mlp_res_layers = [] 207 | 208 | return opt 209 | -------------------------------------------------------------------------------- /lib/render/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. -------------------------------------------------------------------------------- /lib/render/camera.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import cv2 25 | import numpy as np 26 | 27 | from .glm import ortho 28 | 29 | 30 | class Camera: 31 | def __init__(self, width=1600, height=1200): 32 | # Focal Length 33 | # equivalent 50mm 34 | focal = np.sqrt(width * width + height * height) 35 | self.focal_x = focal 36 | self.focal_y = focal 37 | # Principal Point Offset 38 | self.principal_x = width / 2 39 | self.principal_y = height / 2 40 | # Axis Skew 41 | self.skew = 0 42 | # Image Size 43 | self.width = width 44 | self.height = height 45 | 46 | self.near = 1 47 | self.far = 10 48 | 49 | # Camera Center 50 | self.eye = np.array([0, 0, -3.6]) 51 | self.center = np.array([0, 0, 0]) 52 | self.direction = np.array([0, 0, -1]) 53 | self.right = np.array([1, 0, 0]) 54 | self.up = np.array([0, 1, 0]) 55 | 56 | self.ortho_ratio = None 57 | 58 | def sanity_check(self): 59 | self.center = self.center.reshape([-1]) 60 | self.direction = self.direction.reshape([-1]) 61 | self.right = self.right.reshape([-1]) 62 | self.up = self.up.reshape([-1]) 63 | 64 | assert len(self.center) == 3 65 | assert len(self.direction) == 3 66 | assert len(self.right) == 3 67 | assert len(self.up) == 3 68 | 69 | @staticmethod 70 | def normalize_vector(v): 71 | v_norm = np.linalg.norm(v) 72 | return v if v_norm == 0 else v / v_norm 73 | 74 | def get_real_z_value(self, z): 75 | z_near = self.near 76 | z_far = self.far 77 | z_n = 2.0 * z - 1.0 78 | z_e = 2.0 * z_near * z_far / (z_far + z_near - z_n * (z_far - z_near)) 79 | return z_e 80 | 81 | def get_rotation_matrix(self): 82 | rot_mat = np.eye(3) 83 | d = self.eye - self.center 84 | d = -self.normalize_vector(d) 85 | u = self.up 86 | self.right = -np.cross(u, d) 87 | u = np.cross(d, self.right) 88 | rot_mat[0, :] = self.right 89 | rot_mat[1, :] = u 90 | rot_mat[2, :] = d 91 | 92 | # s = self.right 93 | # s = self.normalize_vector(s) 94 | # rot_mat[0, :] = s 95 | # u = self.up 96 | # u = self.normalize_vector(u) 97 | # rot_mat[1, :] = -u 98 | # rot_mat[2, :] = self.normalize_vector(self.direction) 99 | 100 | return rot_mat 101 | 102 | def get_translation_vector(self): 103 | rot_mat = self.get_rotation_matrix() 104 | trans = -np.dot(rot_mat.T, self.eye) 105 | return trans 106 | 107 | def get_intrinsic_matrix(self): 108 | int_mat = np.eye(3) 109 | 110 | int_mat[0, 0] = self.focal_x 111 | int_mat[1, 1] = self.focal_y 112 | int_mat[0, 1] = self.skew 113 | int_mat[0, 2] = self.principal_x 114 | int_mat[1, 2] = self.principal_y 115 | 116 | return int_mat 117 | 118 | def get_projection_matrix(self): 119 | ext_mat = self.get_extrinsic_matrix() 120 | int_mat = self.get_intrinsic_matrix() 121 | 122 | return np.matmul(int_mat, ext_mat) 123 | 124 | def get_extrinsic_matrix(self): 125 | rot_mat = self.get_rotation_matrix() 126 | int_mat = self.get_intrinsic_matrix() 127 | trans = self.get_translation_vector() 128 | 129 | extrinsic = np.eye(4) 130 | extrinsic[:3, :3] = rot_mat 131 | extrinsic[:3, 3] = trans 132 | 133 | return extrinsic[:3, :] 134 | 135 | def set_rotation_matrix(self, rot_mat): 136 | self.direction = rot_mat[2, :] 137 | self.up = -rot_mat[1, :] 138 | self.right = rot_mat[0, :] 139 | 140 | def set_intrinsic_matrix(self, int_mat): 141 | self.focal_x = int_mat[0, 0] 142 | self.focal_y = int_mat[1, 1] 143 | self.skew = int_mat[0, 1] 144 | self.principal_x = int_mat[0, 2] 145 | self.principal_y = int_mat[1, 2] 146 | 147 | def set_projection_matrix(self, proj_mat): 148 | res = cv2.decomposeProjectionMatrix(proj_mat) 149 | int_mat, rot_mat, camera_center_homo = res[0], res[1], res[2] 150 | camera_center = camera_center_homo[0:3] / camera_center_homo[3] 151 | camera_center = camera_center.reshape(-1) 152 | int_mat = int_mat / int_mat[2][2] 153 | 154 | self.set_intrinsic_matrix(int_mat) 155 | self.set_rotation_matrix(rot_mat) 156 | self.center = camera_center 157 | 158 | self.sanity_check() 159 | 160 | def get_gl_matrix(self): 161 | z_near = self.near 162 | z_far = self.far 163 | rot_mat = self.get_rotation_matrix() 164 | int_mat = self.get_intrinsic_matrix() 165 | trans = self.get_translation_vector() 166 | 167 | extrinsic = np.eye(4) 168 | extrinsic[:3, :3] = rot_mat 169 | extrinsic[:3, 3] = trans 170 | axis_adj = np.eye(4) 171 | axis_adj[2, 2] = -1 172 | axis_adj[1, 1] = -1 173 | model_view = np.matmul(axis_adj, extrinsic) 174 | 175 | projective = np.zeros([4, 4]) 176 | projective[:2, :2] = int_mat[:2, :2] 177 | projective[:2, 2:3] = -int_mat[:2, 2:3] 178 | projective[3, 2] = -1 179 | projective[2, 2] = (z_near + z_far) 180 | projective[2, 3] = (z_near * z_far) 181 | 182 | if self.ortho_ratio is None: 183 | ndc = ortho(0, self.width, 0, self.height, z_near, z_far) 184 | perspective = np.matmul(ndc, projective) 185 | else: 186 | perspective = ortho(-self.width * self.ortho_ratio / 2, self.width * self.ortho_ratio / 2, 187 | -self.height * self.ortho_ratio / 2, self.height * self.ortho_ratio / 2, 188 | z_near, z_far) 189 | 190 | return perspective, model_view 191 | 192 | 193 | def KRT_from_P(proj_mat, normalize_K=True): 194 | res = cv2.decomposeProjectionMatrix(proj_mat) 195 | K, Rot, camera_center_homog = res[0], res[1], res[2] 196 | camera_center = camera_center_homog[0:3] / camera_center_homog[3] 197 | trans = -Rot.dot(camera_center) 198 | if normalize_K: 199 | K = K / K[2][2] 200 | return K, Rot, trans 201 | 202 | 203 | def MVP_from_P(proj_mat, width, height, near=0.1, far=10000): 204 | ''' 205 | Convert OpenCV camera calibration matrix to OpenGL projection and model view matrix 206 | :param proj_mat: OpenCV camera projeciton matrix 207 | :param width: Image width 208 | :param height: Image height 209 | :param near: Z near value 210 | :param far: Z far value 211 | :return: OpenGL projection matrix and model view matrix 212 | ''' 213 | res = cv2.decomposeProjectionMatrix(proj_mat) 214 | K, Rot, camera_center_homog = res[0], res[1], res[2] 215 | camera_center = camera_center_homog[0:3] / camera_center_homog[3] 216 | trans = -Rot.dot(camera_center) 217 | K = K / K[2][2] 218 | 219 | extrinsic = np.eye(4) 220 | extrinsic[:3, :3] = Rot 221 | extrinsic[:3, 3:4] = trans 222 | axis_adj = np.eye(4) 223 | axis_adj[2, 2] = -1 224 | axis_adj[1, 1] = -1 225 | model_view = np.matmul(axis_adj, extrinsic) 226 | 227 | zFar = far 228 | zNear = near 229 | projective = np.zeros([4, 4]) 230 | projective[:2, :2] = K[:2, :2] 231 | projective[:2, 2:3] = -K[:2, 2:3] 232 | projective[3, 2] = -1 233 | projective[2, 2] = (zNear + zFar) 234 | projective[2, 3] = (zNear * zFar) 235 | 236 | ndc = ortho(0, width, 0, height, zNear, zFar) 237 | 238 | perspective = np.matmul(ndc, projective) 239 | 240 | return perspective, model_view 241 | -------------------------------------------------------------------------------- /lib/render/gl/__init__.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | from .framework import * 25 | from .render import * 26 | from .cam_render import * 27 | -------------------------------------------------------------------------------- /lib/render/gl/cam_render.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | from OpenGL.GLUT import * 25 | 26 | from .render import Render 27 | 28 | 29 | class CamRender(Render): 30 | def __init__(self, width=1600, height=1200, name='Cam Renderer', 31 | program_files=['simple.fs', 'simple.vs'], color_size=1, ms_rate=1): 32 | Render.__init__(self, width, height, name, program_files, color_size, ms_rate) 33 | self.camera = None 34 | 35 | glutDisplayFunc(self.display) 36 | glutKeyboardFunc(self.keyboard) 37 | 38 | def set_camera(self, camera): 39 | self.camera = camera 40 | self.projection_matrix, self.model_view_matrix = camera.get_gl_matrix() 41 | 42 | def set_matrices(self, projection, modelview): 43 | self.projection_matrix = projection 44 | self.model_view_matrix = modelview 45 | 46 | def keyboard(self, key, x, y): 47 | # up 48 | eps = 1 49 | # print(key) 50 | if key == b'w': 51 | self.camera.center += eps * self.camera.direction 52 | elif key == b's': 53 | self.camera.center -= eps * self.camera.direction 54 | if key == b'a': 55 | self.camera.center -= eps * self.camera.right 56 | elif key == b'd': 57 | self.camera.center += eps * self.camera.right 58 | if key == b' ': 59 | self.camera.center += eps * self.camera.up 60 | elif key == b'x': 61 | self.camera.center -= eps * self.camera.up 62 | elif key == b'i': 63 | self.camera.near += 0.1 * eps 64 | self.camera.far += 0.1 * eps 65 | elif key == b'o': 66 | self.camera.near -= 0.1 * eps 67 | self.camera.far -= 0.1 * eps 68 | 69 | self.projection_matrix, self.model_view_matrix = self.camera.get_gl_matrix() 70 | 71 | def show(self): 72 | glutMainLoop() 73 | -------------------------------------------------------------------------------- /lib/render/gl/color_render.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import numpy as np 25 | import random 26 | 27 | from .framework import * 28 | from .cam_render import CamRender 29 | 30 | 31 | class ColorRender(CamRender): 32 | def __init__(self, width=1600, height=1200, name='Color Renderer'): 33 | program_files = ['color.vs', 'color.fs'] 34 | CamRender.__init__(self, width, height, name, program_files=program_files) 35 | 36 | # WARNING: this differs from vertex_buffer and vertex_data in Render 37 | self.vert_buffer = {} 38 | self.vert_data = {} 39 | 40 | self.color_buffer = {} 41 | self.color_data = {} 42 | 43 | self.vertex_dim = {} 44 | self.n_vertices = {} 45 | 46 | def set_mesh(self, vertices, faces, color, faces_clr, mat_name='all'): 47 | self.vert_data[mat_name] = vertices[faces.reshape([-1])] 48 | self.n_vertices[mat_name] = self.vert_data[mat_name].shape[0] 49 | self.vertex_dim[mat_name] = self.vert_data[mat_name].shape[1] 50 | 51 | if mat_name not in self.vert_buffer.keys(): 52 | self.vert_buffer[mat_name] = glGenBuffers(1) 53 | glBindBuffer(GL_ARRAY_BUFFER, self.vert_buffer[mat_name]) 54 | glBufferData(GL_ARRAY_BUFFER, self.vert_data[mat_name], GL_STATIC_DRAW) 55 | 56 | self.color_data[mat_name] = color[faces_clr.reshape([-1])] 57 | if mat_name not in self.color_buffer.keys(): 58 | self.color_buffer[mat_name] = glGenBuffers(1) 59 | glBindBuffer(GL_ARRAY_BUFFER, self.color_buffer[mat_name]) 60 | glBufferData(GL_ARRAY_BUFFER, self.color_data[mat_name], GL_STATIC_DRAW) 61 | 62 | glBindBuffer(GL_ARRAY_BUFFER, 0) 63 | 64 | def cleanup(self): 65 | 66 | glBindBuffer(GL_ARRAY_BUFFER, 0) 67 | for key in self.vert_data: 68 | glDeleteBuffers(1, [self.vert_buffer[key]]) 69 | glDeleteBuffers(1, [self.color_buffer[key]]) 70 | 71 | self.vert_buffer = {} 72 | self.vert_data = {} 73 | 74 | self.color_buffer = {} 75 | self.color_data = {} 76 | 77 | self.render_texture_mat = {} 78 | 79 | self.vertex_dim = {} 80 | self.n_vertices = {} 81 | 82 | def draw(self): 83 | self.draw_init() 84 | 85 | glEnable(GL_MULTISAMPLE) 86 | 87 | glUseProgram(self.program) 88 | glUniformMatrix4fv(self.model_mat_unif, 1, GL_FALSE, self.model_view_matrix.transpose()) 89 | glUniformMatrix4fv(self.persp_mat_unif, 1, GL_FALSE, self.projection_matrix.transpose()) 90 | 91 | for mat in self.vert_buffer: 92 | # Handle vertex buffer 93 | glBindBuffer(GL_ARRAY_BUFFER, self.vert_buffer[mat]) 94 | glEnableVertexAttribArray(0) 95 | glVertexAttribPointer(0, self.vertex_dim[mat], GL_DOUBLE, GL_FALSE, 0, None) 96 | 97 | # Handle normal buffer 98 | glBindBuffer(GL_ARRAY_BUFFER, self.color_buffer[mat]) 99 | glEnableVertexAttribArray(1) 100 | glVertexAttribPointer(1, 3, GL_DOUBLE, GL_FALSE, 0, None) 101 | 102 | glDrawArrays(GL_TRIANGLES, 0, self.n_vertices[mat]) 103 | 104 | glDisableVertexAttribArray(1) 105 | glDisableVertexAttribArray(0) 106 | 107 | glBindBuffer(GL_ARRAY_BUFFER, 0) 108 | 109 | glUseProgram(0) 110 | 111 | glDisable(GL_MULTISAMPLE) 112 | 113 | self.draw_end() 114 | -------------------------------------------------------------------------------- /lib/render/gl/data/color.fs: -------------------------------------------------------------------------------- 1 | #version 330 core 2 | 3 | out vec4 FragColor; 4 | 5 | in vec3 Color; 6 | 7 | void main() 8 | { 9 | FragColor = vec4(Color,1.0); 10 | } 11 | -------------------------------------------------------------------------------- /lib/render/gl/data/color.vs: -------------------------------------------------------------------------------- 1 | #version 330 core 2 | 3 | layout (location = 0) in vec3 a_Position; 4 | layout (location = 1) in vec3 a_Color; 5 | 6 | out vec3 CamNormal; 7 | out vec3 CamPos; 8 | out vec3 Color; 9 | 10 | uniform mat4 ModelMat; 11 | uniform mat4 PerspMat; 12 | 13 | void main() 14 | { 15 | gl_Position = PerspMat * ModelMat * vec4(a_Position, 1.0); 16 | Color = a_Color; 17 | } -------------------------------------------------------------------------------- /lib/render/gl/data/geo.fs: -------------------------------------------------------------------------------- 1 | #version 330 core 2 | 3 | out vec4 FragColor; 4 | 5 | in vec3 CamNormal; 6 | in vec3 CamPos; 7 | 8 | void main() 9 | { 10 | vec3 light_direction = vec3(0, 0, 1); 11 | vec3 f_normal = normalize(CamNormal.xyz); 12 | vec4 specular_reflection = vec4(0.2) * pow(max(0.0, dot(reflect(-light_direction, f_normal), vec3(0, 0, -1))), 16.f); 13 | FragColor = vec4(dot(f_normal, light_direction)*vec3(1.0)+specular_reflection.xyz, 1.0); 14 | } 15 | -------------------------------------------------------------------------------- /lib/render/gl/data/geo.vs: -------------------------------------------------------------------------------- 1 | #version 330 core 2 | 3 | layout (location = 0) in vec3 a_Position; 4 | layout (location = 1) in vec3 a_Normal; 5 | 6 | out vec3 CamNormal; 7 | out vec3 CamPos; 8 | 9 | uniform mat4 ModelMat; 10 | uniform mat4 PerspMat; 11 | 12 | void main() 13 | { 14 | gl_Position = PerspMat * ModelMat * vec4(a_Position, 1.0); 15 | CamNormal = (ModelMat * vec4(a_Normal, 0.0)).xyz; 16 | CamPos = (ModelMat * vec4(a_Position, 1.0)).xyz; 17 | } -------------------------------------------------------------------------------- /lib/render/gl/data/normal.fs: -------------------------------------------------------------------------------- 1 | #version 330 2 | 3 | out vec4 FragColor; 4 | 5 | in vec3 CamNormal; 6 | 7 | void main() 8 | { 9 | vec3 cam_norm_normalized = normalize(CamNormal); 10 | vec3 rgb = (cam_norm_normalized + 1.0) / 2.0; 11 | FragColor = vec4(rgb, 1.0); 12 | } 13 | -------------------------------------------------------------------------------- /lib/render/gl/data/normal.vs: -------------------------------------------------------------------------------- 1 | #version 330 2 | 3 | layout (location = 0) in vec3 Position; 4 | layout (location = 1) in vec3 Normal; 5 | 6 | out vec3 CamNormal; 7 | 8 | uniform mat4 ModelMat; 9 | uniform mat4 PerspMat; 10 | 11 | void main() 12 | { 13 | gl_Position = PerspMat * ModelMat * vec4(Position, 1.0); 14 | CamNormal = (ModelMat * vec4(Normal, 0.0)).xyz; 15 | } 16 | -------------------------------------------------------------------------------- /lib/render/gl/data/quad.fs: -------------------------------------------------------------------------------- 1 | #version 330 core 2 | 3 | out vec4 FragColor; 4 | 5 | in vec2 TexCoord; 6 | 7 | uniform sampler2D screenTexture; 8 | 9 | void main() 10 | { 11 | FragColor = texture(screenTexture, TexCoord); 12 | } -------------------------------------------------------------------------------- /lib/render/gl/data/quad.vs: -------------------------------------------------------------------------------- 1 | #version 330 core 2 | 3 | layout (location = 0) in vec2 aPos; 4 | layout (location = 1) in vec2 aTexCoord; 5 | 6 | out vec2 TexCoord; 7 | 8 | void main() 9 | { 10 | gl_Position = vec4(aPos.x, aPos.y, 0.0, 1.0); 11 | TexCoord = aTexCoord; 12 | } -------------------------------------------------------------------------------- /lib/render/gl/framework.py: -------------------------------------------------------------------------------- 1 | # Mario Rosasco, 2016 2 | # adapted from framework.cpp, Copyright (C) 2010-2012 by Jason L. McKesson 3 | # This file is licensed under the MIT License. 4 | # 5 | # NB: Unlike in the framework.cpp organization, the main loop is contained 6 | # in the tutorial files, not in this framework file. Additionally, a copy of 7 | # this module file must exist in the same directory as the tutorial files 8 | # to be imported properly. 9 | 10 | 11 | import os 12 | 13 | from OpenGL.GL import * 14 | 15 | 16 | # Function that creates and compiles shaders according to the given type (a GL enum value) and 17 | # shader program (a file containing a GLSL program). 18 | def loadShader(shaderType, shaderFile): 19 | # check if file exists, get full path name 20 | strFilename = findFileOrThrow(shaderFile) 21 | shaderData = None 22 | with open(strFilename, 'r') as f: 23 | shaderData = f.read() 24 | 25 | shader = glCreateShader(shaderType) 26 | glShaderSource(shader, shaderData) # note that this is a simpler function call than in C 27 | 28 | # This shader compilation is more explicit than the one used in 29 | # framework.cpp, which relies on a glutil wrapper function. 30 | # This is made explicit here mainly to decrease dependence on pyOpenGL 31 | # utilities and wrappers, which docs caution may change in future versions. 32 | glCompileShader(shader) 33 | 34 | status = glGetShaderiv(shader, GL_COMPILE_STATUS) 35 | if status == GL_FALSE: 36 | # Note that getting the error log is much simpler in Python than in C/C++ 37 | # and does not require explicit handling of the string buffer 38 | strInfoLog = glGetShaderInfoLog(shader) 39 | strShaderType = "" 40 | if shaderType is GL_VERTEX_SHADER: 41 | strShaderType = "vertex" 42 | elif shaderType is GL_GEOMETRY_SHADER: 43 | strShaderType = "geometry" 44 | elif shaderType is GL_FRAGMENT_SHADER: 45 | strShaderType = "fragment" 46 | 47 | print("Compilation failure for " + strShaderType + " shader:\n" + str(strInfoLog)) 48 | 49 | return shader 50 | 51 | 52 | # Function that accepts a list of shaders, compiles them, and returns a handle to the compiled program 53 | def createProgram(shaderList): 54 | program = glCreateProgram() 55 | 56 | for shader in shaderList: 57 | glAttachShader(program, shader) 58 | 59 | glLinkProgram(program) 60 | 61 | status = glGetProgramiv(program, GL_LINK_STATUS) 62 | if status == GL_FALSE: 63 | # Note that getting the error log is much simpler in Python than in C/C++ 64 | # and does not require explicit handling of the string buffer 65 | strInfoLog = glGetProgramInfoLog(program) 66 | print("Linker failure: \n" + str(strInfoLog)) 67 | 68 | for shader in shaderList: 69 | glDetachShader(program, shader) 70 | 71 | return program 72 | 73 | 74 | # Helper function to locate and open the target file (passed in as a string). 75 | # Returns the full path to the file as a string. 76 | def findFileOrThrow(strBasename): 77 | # Keep constant names in C-style convention, for readability 78 | # when comparing to C(/C++) code. 79 | if os.path.isfile(strBasename): 80 | return strBasename 81 | 82 | LOCAL_FILE_DIR = "data" + os.sep 83 | GLOBAL_FILE_DIR = os.path.dirname(os.path.abspath(__file__)) + os.sep + "data" + os.sep 84 | 85 | strFilename = LOCAL_FILE_DIR + strBasename 86 | if os.path.isfile(strFilename): 87 | return strFilename 88 | 89 | strFilename = GLOBAL_FILE_DIR + strBasename 90 | if os.path.isfile(strFilename): 91 | return strFilename 92 | 93 | raise IOError('Could not find target file ' + strBasename) 94 | -------------------------------------------------------------------------------- /lib/render/gl/geo_render.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import numpy as np 25 | import random 26 | 27 | from .framework import * 28 | from .cam_render import CamRender 29 | 30 | 31 | class GeoRender(CamRender): 32 | def __init__(self, width=1600, height=1200, name='Geo Renderer'): 33 | program_files = ['geo.vs', 'geo.fs'] 34 | CamRender.__init__(self, width, height, name, program_files=program_files) 35 | 36 | # WARNING: this differs from vertex_buffer and vertex_data in Render 37 | self.vert_buffer = {} 38 | self.vert_data = {} 39 | 40 | self.norm_buffer = {} 41 | self.norm_data = {} 42 | 43 | self.vertex_dim = {} 44 | self.n_vertices = {} 45 | 46 | def set_mesh(self, vertices, faces, norms, faces_nml, mat_name='all'): 47 | self.vert_data[mat_name] = vertices[faces.reshape([-1])] 48 | self.n_vertices[mat_name] = self.vert_data[mat_name].shape[0] 49 | self.vertex_dim[mat_name] = self.vert_data[mat_name].shape[1] 50 | 51 | if mat_name not in self.vert_buffer.keys(): 52 | self.vert_buffer[mat_name] = glGenBuffers(1) 53 | glBindBuffer(GL_ARRAY_BUFFER, self.vert_buffer[mat_name]) 54 | glBufferData(GL_ARRAY_BUFFER, self.vert_data[mat_name], GL_STATIC_DRAW) 55 | 56 | self.norm_data[mat_name] = norms[faces_nml.reshape([-1])] 57 | if mat_name not in self.norm_buffer.keys(): 58 | self.norm_buffer[mat_name] = glGenBuffers(1) 59 | glBindBuffer(GL_ARRAY_BUFFER, self.norm_buffer[mat_name]) 60 | glBufferData(GL_ARRAY_BUFFER, self.norm_data[mat_name], GL_STATIC_DRAW) 61 | 62 | glBindBuffer(GL_ARRAY_BUFFER, 0) 63 | 64 | def cleanup(self): 65 | 66 | glBindBuffer(GL_ARRAY_BUFFER, 0) 67 | for key in self.vert_data: 68 | glDeleteBuffers(1, [self.vert_buffer[key]]) 69 | glDeleteBuffers(1, [self.norm_buffer[key]]) 70 | 71 | self.vert_buffer = {} 72 | self.vert_data = {} 73 | 74 | self.norm_buffer = {} 75 | self.norm_data = {} 76 | 77 | self.render_texture_mat = {} 78 | 79 | self.vertex_dim = {} 80 | self.n_vertices = {} 81 | 82 | def draw(self): 83 | self.draw_init() 84 | 85 | glEnable(GL_MULTISAMPLE) 86 | 87 | glUseProgram(self.program) 88 | glUniformMatrix4fv(self.model_mat_unif, 1, GL_FALSE, self.model_view_matrix.transpose()) 89 | glUniformMatrix4fv(self.persp_mat_unif, 1, GL_FALSE, self.projection_matrix.transpose()) 90 | 91 | for mat in self.vert_buffer: 92 | # Handle vertex buffer 93 | glBindBuffer(GL_ARRAY_BUFFER, self.vert_buffer[mat]) 94 | glEnableVertexAttribArray(0) 95 | glVertexAttribPointer(0, self.vertex_dim[mat], GL_DOUBLE, GL_FALSE, 0, None) 96 | 97 | # Handle normal buffer 98 | glBindBuffer(GL_ARRAY_BUFFER, self.norm_buffer[mat]) 99 | glEnableVertexAttribArray(1) 100 | glVertexAttribPointer(1, 3, GL_DOUBLE, GL_FALSE, 0, None) 101 | 102 | glDrawArrays(GL_TRIANGLES, 0, self.n_vertices[mat]) 103 | 104 | glDisableVertexAttribArray(1) 105 | glDisableVertexAttribArray(0) 106 | 107 | glBindBuffer(GL_ARRAY_BUFFER, 0) 108 | 109 | glUseProgram(0) 110 | 111 | glDisable(GL_MULTISAMPLE) 112 | 113 | self.draw_end() 114 | -------------------------------------------------------------------------------- /lib/render/gl/normal_render.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import numpy as np 25 | 26 | from .framework import * 27 | from .cam_render import CamRender 28 | 29 | 30 | class NormalRender(CamRender): 31 | def __init__(self, width=1600, height=1200, name='Normal Renderer'): 32 | CamRender.__init__(self, width, height, name, program_files=['normal.vs', 'normal.fs']) 33 | 34 | self.norm_buffer = glGenBuffers(1) 35 | 36 | self.norm_data = None 37 | 38 | def set_normal_mesh(self, vertices, faces, norms, face_normals): 39 | CamRender.set_mesh(self, vertices, faces) 40 | 41 | self.norm_data = norms[face_normals.reshape([-1])] 42 | 43 | glBindBuffer(GL_ARRAY_BUFFER, self.norm_buffer) 44 | glBufferData(GL_ARRAY_BUFFER, self.norm_data, GL_STATIC_DRAW) 45 | 46 | glBindBuffer(GL_ARRAY_BUFFER, 0) 47 | 48 | def draw(self): 49 | self.draw_init() 50 | 51 | glUseProgram(self.program) 52 | glUniformMatrix4fv(self.model_mat_unif, 1, GL_FALSE, self.model_view_matrix.transpose()) 53 | glUniformMatrix4fv(self.persp_mat_unif, 1, GL_FALSE, self.projection_matrix.transpose()) 54 | 55 | # Handle vertex buffer 56 | glBindBuffer(GL_ARRAY_BUFFER, self.vertex_buffer) 57 | 58 | glEnableVertexAttribArray(0) 59 | glVertexAttribPointer(0, self.vertex_dim, GL_DOUBLE, GL_FALSE, 0, None) 60 | 61 | # Handle normal buffer 62 | glBindBuffer(GL_ARRAY_BUFFER, self.norm_buffer) 63 | 64 | glEnableVertexAttribArray(1) 65 | glVertexAttribPointer(1, 3, GL_DOUBLE, GL_FALSE, 0, None) 66 | 67 | glDrawArrays(GL_TRIANGLES, 0, self.n_vertices) 68 | 69 | glDisableVertexAttribArray(1) 70 | glDisableVertexAttribArray(0) 71 | 72 | glBindBuffer(GL_ARRAY_BUFFER, 0) 73 | 74 | glUseProgram(0) 75 | 76 | self.draw_end() 77 | -------------------------------------------------------------------------------- /lib/render/gl/render.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import numpy as np 25 | from OpenGL.GLUT import * 26 | from .framework import * 27 | 28 | _glut_window = None 29 | 30 | class Render: 31 | def __init__(self, width=1600, height=1200, name='GL Renderer', 32 | program_files=['simple.fs', 'simple.vs'], color_size=1, ms_rate=1): 33 | self.width = width 34 | self.height = height 35 | self.name = name 36 | self.display_mode = GLUT_DOUBLE | GLUT_RGB | GLUT_DEPTH 37 | self.use_inverse_depth = False 38 | 39 | global _glut_window 40 | if _glut_window is None: 41 | glutInit() 42 | glutInitDisplayMode(self.display_mode) 43 | glutInitWindowSize(self.width, self.height) 44 | glutInitWindowPosition(0, 0) 45 | _glut_window = glutCreateWindow("My Render.") 46 | 47 | # glEnable(GL_DEPTH_CLAMP) 48 | glEnable(GL_DEPTH_TEST) 49 | 50 | glClampColor(GL_CLAMP_READ_COLOR, GL_FALSE) 51 | glClampColor(GL_CLAMP_FRAGMENT_COLOR, GL_FALSE) 52 | glClampColor(GL_CLAMP_VERTEX_COLOR, GL_FALSE) 53 | 54 | # init program 55 | shader_list = [] 56 | 57 | for program_file in program_files: 58 | _, ext = os.path.splitext(program_file) 59 | if ext == '.vs': 60 | shader_list.append(loadShader(GL_VERTEX_SHADER, program_file)) 61 | elif ext == '.fs': 62 | shader_list.append(loadShader(GL_FRAGMENT_SHADER, program_file)) 63 | elif ext == '.gs': 64 | shader_list.append(loadShader(GL_GEOMETRY_SHADER, program_file)) 65 | 66 | self.program = createProgram(shader_list) 67 | 68 | for shader in shader_list: 69 | glDeleteShader(shader) 70 | 71 | # Init uniform variables 72 | self.model_mat_unif = glGetUniformLocation(self.program, 'ModelMat') 73 | self.persp_mat_unif = glGetUniformLocation(self.program, 'PerspMat') 74 | 75 | self.vertex_buffer = glGenBuffers(1) 76 | 77 | # Init screen quad program and buffer 78 | self.quad_program, self.quad_buffer = self.init_quad_program() 79 | 80 | # Configure frame buffer 81 | self.frame_buffer = glGenFramebuffers(1) 82 | glBindFramebuffer(GL_FRAMEBUFFER, self.frame_buffer) 83 | 84 | self.intermediate_fbo = None 85 | if ms_rate > 1: 86 | # Configure texture buffer to render to 87 | self.color_buffer = [] 88 | for i in range(color_size): 89 | color_buffer = glGenTextures(1) 90 | multi_sample_rate = ms_rate 91 | glBindTexture(GL_TEXTURE_2D_MULTISAMPLE, color_buffer) 92 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE) 93 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE) 94 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR) 95 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR) 96 | glTexImage2DMultisample(GL_TEXTURE_2D_MULTISAMPLE, multi_sample_rate, GL_RGBA32F, self.width, self.height, GL_TRUE) 97 | glBindTexture(GL_TEXTURE_2D_MULTISAMPLE, 0) 98 | glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0 + i, GL_TEXTURE_2D_MULTISAMPLE, color_buffer, 0) 99 | self.color_buffer.append(color_buffer) 100 | 101 | self.render_buffer = glGenRenderbuffers(1) 102 | glBindRenderbuffer(GL_RENDERBUFFER, self.render_buffer) 103 | glRenderbufferStorageMultisample(GL_RENDERBUFFER, multi_sample_rate, GL_DEPTH24_STENCIL8, self.width, self.height) 104 | glBindRenderbuffer(GL_RENDERBUFFER, 0) 105 | glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_STENCIL_ATTACHMENT, GL_RENDERBUFFER, self.render_buffer) 106 | 107 | attachments = [] 108 | for i in range(color_size): 109 | attachments.append(GL_COLOR_ATTACHMENT0 + i) 110 | glDrawBuffers(color_size, attachments) 111 | glBindFramebuffer(GL_FRAMEBUFFER, 0) 112 | 113 | self.intermediate_fbo = glGenFramebuffers(1) 114 | glBindFramebuffer(GL_FRAMEBUFFER, self.intermediate_fbo) 115 | 116 | self.screen_texture = [] 117 | for i in range(color_size): 118 | screen_texture = glGenTextures(1) 119 | glBindTexture(GL_TEXTURE_2D, screen_texture) 120 | glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA32F, self.width, self.height, 0, GL_RGBA, GL_FLOAT, None) 121 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR) 122 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR) 123 | glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0 + i, GL_TEXTURE_2D, screen_texture, 0) 124 | self.screen_texture.append(screen_texture) 125 | 126 | glDrawBuffers(color_size, attachments) 127 | glBindFramebuffer(GL_FRAMEBUFFER, 0) 128 | else: 129 | self.color_buffer = [] 130 | for i in range(color_size): 131 | color_buffer = glGenTextures(1) 132 | glBindTexture(GL_TEXTURE_2D, color_buffer) 133 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE) 134 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE) 135 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST) 136 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST) 137 | glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA32F, self.width, self.height, 0, GL_RGBA, GL_FLOAT, None) 138 | glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0 + i, GL_TEXTURE_2D, color_buffer, 0) 139 | self.color_buffer.append(color_buffer) 140 | 141 | # Configure depth texture map to render to 142 | self.depth_buffer = glGenTextures(1) 143 | glBindTexture(GL_TEXTURE_2D, self.depth_buffer) 144 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT) 145 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT) 146 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST) 147 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST) 148 | glTexParameteri(GL_TEXTURE_2D, GL_DEPTH_TEXTURE_MODE, GL_INTENSITY) 149 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_COMPARE_MODE, GL_COMPARE_R_TO_TEXTURE) 150 | glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_COMPARE_FUNC, GL_LEQUAL) 151 | glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT, self.width, self.height, 0, GL_DEPTH_COMPONENT, GL_FLOAT, None) 152 | glFramebufferTexture2D(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_TEXTURE_2D, self.depth_buffer, 0) 153 | 154 | attachments = [] 155 | for i in range(color_size): 156 | attachments.append(GL_COLOR_ATTACHMENT0 + i) 157 | glDrawBuffers(color_size, attachments) 158 | self.screen_texture = self.color_buffer 159 | 160 | glBindFramebuffer(GL_FRAMEBUFFER, 0) 161 | 162 | 163 | # Configure texture buffer if needed 164 | self.render_texture = None 165 | 166 | # NOTE: original render_texture only support one input 167 | # this is tentative member of this issue 168 | self.render_texture_v2 = {} 169 | 170 | # Inner storage for buffer data 171 | self.vertex_data = None 172 | self.vertex_dim = None 173 | self.n_vertices = None 174 | 175 | self.model_view_matrix = None 176 | self.projection_matrix = None 177 | 178 | glutDisplayFunc(self.display) 179 | 180 | 181 | def init_quad_program(self): 182 | shader_list = [] 183 | 184 | shader_list.append(loadShader(GL_VERTEX_SHADER, "quad.vs")) 185 | shader_list.append(loadShader(GL_FRAGMENT_SHADER, "quad.fs")) 186 | 187 | the_program = createProgram(shader_list) 188 | 189 | for shader in shader_list: 190 | glDeleteShader(shader) 191 | 192 | # vertex attributes for a quad that fills the entire screen in Normalized Device Coordinates. 193 | # positions # texCoords 194 | quad_vertices = np.array( 195 | [-1.0, 1.0, 0.0, 1.0, 196 | -1.0, -1.0, 0.0, 0.0, 197 | 1.0, -1.0, 1.0, 0.0, 198 | 199 | -1.0, 1.0, 0.0, 1.0, 200 | 1.0, -1.0, 1.0, 0.0, 201 | 1.0, 1.0, 1.0, 1.0] 202 | ) 203 | 204 | quad_buffer = glGenBuffers(1) 205 | glBindBuffer(GL_ARRAY_BUFFER, quad_buffer) 206 | glBufferData(GL_ARRAY_BUFFER, quad_vertices, GL_STATIC_DRAW) 207 | 208 | glBindBuffer(GL_ARRAY_BUFFER, 0) 209 | 210 | return the_program, quad_buffer 211 | 212 | def set_mesh(self, vertices, faces): 213 | self.vertex_data = vertices[faces.reshape([-1])] 214 | self.vertex_dim = self.vertex_data.shape[1] 215 | self.n_vertices = self.vertex_data.shape[0] 216 | 217 | glBindBuffer(GL_ARRAY_BUFFER, self.vertex_buffer) 218 | glBufferData(GL_ARRAY_BUFFER, self.vertex_data, GL_STATIC_DRAW) 219 | 220 | glBindBuffer(GL_ARRAY_BUFFER, 0) 221 | 222 | def set_viewpoint(self, projection, model_view): 223 | self.projection_matrix = projection 224 | self.model_view_matrix = model_view 225 | 226 | def draw_init(self): 227 | glBindFramebuffer(GL_FRAMEBUFFER, self.frame_buffer) 228 | glEnable(GL_DEPTH_TEST) 229 | 230 | # glClearColor(0.0, 0.0, 0.0, 0.0) 231 | glClearColor(1.0, 1.0, 1.0, 0.0) #Black background 232 | 233 | if self.use_inverse_depth: 234 | glDepthFunc(GL_GREATER) 235 | glClearDepth(0.0) 236 | else: 237 | glDepthFunc(GL_LESS) 238 | glClearDepth(1.0) 239 | glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT) 240 | 241 | def draw_end(self): 242 | if self.intermediate_fbo is not None: 243 | for i in range(len(self.color_buffer)): 244 | glBindFramebuffer(GL_READ_FRAMEBUFFER, self.frame_buffer) 245 | glReadBuffer(GL_COLOR_ATTACHMENT0 + i) 246 | glBindFramebuffer(GL_DRAW_FRAMEBUFFER, self.intermediate_fbo) 247 | glDrawBuffer(GL_COLOR_ATTACHMENT0 + i) 248 | glBlitFramebuffer(0, 0, self.width, self.height, 0, 0, self.width, self.height, GL_COLOR_BUFFER_BIT, GL_NEAREST) 249 | 250 | glBindFramebuffer(GL_FRAMEBUFFER, 0) 251 | glDepthFunc(GL_LESS) 252 | glClearDepth(1.0) 253 | 254 | def draw(self): 255 | self.draw_init() 256 | 257 | glUseProgram(self.program) 258 | glUniformMatrix4fv(self.model_mat_unif, 1, GL_FALSE, self.model_view_matrix.transpose()) 259 | glUniformMatrix4fv(self.persp_mat_unif, 1, GL_FALSE, self.projection_matrix.transpose()) 260 | 261 | glBindBuffer(GL_ARRAY_BUFFER, self.vertex_buffer) 262 | 263 | glEnableVertexAttribArray(0) 264 | glVertexAttribPointer(0, self.vertex_dim, GL_DOUBLE, GL_FALSE, 0, None) 265 | 266 | glDrawArrays(GL_TRIANGLES, 0, self.n_vertices) 267 | 268 | glDisableVertexAttribArray(0) 269 | 270 | glBindBuffer(GL_ARRAY_BUFFER, 0) 271 | 272 | glUseProgram(0) 273 | 274 | self.draw_end() 275 | 276 | def get_color(self, color_id=0): 277 | glBindFramebuffer(GL_FRAMEBUFFER, self.intermediate_fbo if self.intermediate_fbo is not None else self.frame_buffer) 278 | glReadBuffer(GL_COLOR_ATTACHMENT0 + color_id) 279 | data = glReadPixels(0, 0, self.width, self.height, GL_RGBA, GL_FLOAT, outputType=None) 280 | glBindFramebuffer(GL_FRAMEBUFFER, 0) 281 | rgb = data.reshape(self.height, self.width, -1) 282 | rgb = np.flip(rgb, 0) 283 | return rgb 284 | 285 | def get_z_value(self): 286 | glBindFramebuffer(GL_FRAMEBUFFER, self.frame_buffer) 287 | data = glReadPixels(0, 0, self.width, self.height, GL_DEPTH_COMPONENT, GL_FLOAT, outputType=None) 288 | glBindFramebuffer(GL_FRAMEBUFFER, 0) 289 | z = data.reshape(self.height, self.width) 290 | z = np.flip(z, 0) 291 | return z 292 | 293 | def display(self): 294 | # First we draw a scene. 295 | # Notice the result is stored in the texture buffer. 296 | self.draw() 297 | 298 | # Then we return to the default frame buffer since we will display on the screen. 299 | glBindFramebuffer(GL_FRAMEBUFFER, 0) 300 | 301 | # Do the clean-up. 302 | # glClearColor(0.0, 0.0, 0.0, 0.0) #Black background 303 | glClearColor(1.0, 1.0, 1.0, 0.0) #Black background 304 | glClear(GL_COLOR_BUFFER_BIT) 305 | 306 | # We draw a rectangle which covers the whole screen. 307 | glUseProgram(self.quad_program) 308 | glBindBuffer(GL_ARRAY_BUFFER, self.quad_buffer) 309 | 310 | size_of_double = 8 311 | glEnableVertexAttribArray(0) 312 | glVertexAttribPointer(0, 2, GL_DOUBLE, GL_FALSE, 4 * size_of_double, None) 313 | glEnableVertexAttribArray(1) 314 | glVertexAttribPointer(1, 2, GL_DOUBLE, GL_FALSE, 4 * size_of_double, c_void_p(2 * size_of_double)) 315 | 316 | glDisable(GL_DEPTH_TEST) 317 | 318 | # The stored texture is then mapped to this rectangle. 319 | # properly assing color buffer texture 320 | glActiveTexture(GL_TEXTURE0) 321 | glBindTexture(GL_TEXTURE_2D, self.screen_texture[0]) 322 | glUniform1i(glGetUniformLocation(self.quad_program, 'screenTexture'), 0) 323 | 324 | glDrawArrays(GL_TRIANGLES, 0, 6) 325 | 326 | glDisableVertexAttribArray(1) 327 | glDisableVertexAttribArray(0) 328 | 329 | glEnable(GL_DEPTH_TEST) 330 | glBindBuffer(GL_ARRAY_BUFFER, 0) 331 | glUseProgram(0) 332 | 333 | glutSwapBuffers() 334 | glutPostRedisplay() 335 | 336 | def show(self): 337 | glutMainLoop() 338 | -------------------------------------------------------------------------------- /lib/render/glm.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import numpy as np 25 | 26 | 27 | def vec3(x, y, z): 28 | return np.array([x, y, z], dtype=np.float32) 29 | 30 | 31 | def radians(v): 32 | return np.radians(v) 33 | 34 | 35 | def identity(): 36 | return np.identity(4, dtype=np.float32) 37 | 38 | 39 | def empty(): 40 | return np.zeros([4, 4], dtype=np.float32) 41 | 42 | 43 | def magnitude(v): 44 | return np.linalg.norm(v) 45 | 46 | 47 | def normalize(v): 48 | m = magnitude(v) 49 | return v if m == 0 else v / m 50 | 51 | 52 | def dot(u, v): 53 | return np.sum(u * v) 54 | 55 | 56 | def cross(u, v): 57 | res = vec3(0, 0, 0) 58 | res[0] = u[1] * v[2] - u[2] * v[1] 59 | res[1] = u[2] * v[0] - u[0] * v[2] 60 | res[2] = u[0] * v[1] - u[1] * v[0] 61 | return res 62 | 63 | 64 | # below functions can be optimized 65 | 66 | def translate(m, v): 67 | res = np.copy(m) 68 | res[:, 3] = m[:, 0] * v[0] + m[:, 1] * v[1] + m[:, 2] * v[2] + m[:, 3] 69 | return res 70 | 71 | 72 | def rotate(m, angle, v): 73 | a = angle 74 | c = np.cos(a) 75 | s = np.sin(a) 76 | 77 | axis = normalize(v) 78 | temp = (1 - c) * axis 79 | 80 | rot = empty() 81 | rot[0][0] = c + temp[0] * axis[0] 82 | rot[0][1] = temp[0] * axis[1] + s * axis[2] 83 | rot[0][2] = temp[0] * axis[2] - s * axis[1] 84 | 85 | rot[1][0] = temp[1] * axis[0] - s * axis[2] 86 | rot[1][1] = c + temp[1] * axis[1] 87 | rot[1][2] = temp[1] * axis[2] + s * axis[0] 88 | 89 | rot[2][0] = temp[2] * axis[0] + s * axis[1] 90 | rot[2][1] = temp[2] * axis[1] - s * axis[0] 91 | rot[2][2] = c + temp[2] * axis[2] 92 | 93 | res = empty() 94 | res[:, 0] = m[:, 0] * rot[0][0] + m[:, 1] * rot[0][1] + m[:, 2] * rot[0][2] 95 | res[:, 1] = m[:, 0] * rot[1][0] + m[:, 1] * rot[1][1] + m[:, 2] * rot[1][2] 96 | res[:, 2] = m[:, 0] * rot[2][0] + m[:, 1] * rot[2][1] + m[:, 2] * rot[2][2] 97 | res[:, 3] = m[:, 3] 98 | return res 99 | 100 | 101 | def perspective(fovy, aspect, zNear, zFar): 102 | tanHalfFovy = np.tan(fovy / 2) 103 | 104 | res = empty() 105 | res[0][0] = 1 / (aspect * tanHalfFovy) 106 | res[1][1] = 1 / (tanHalfFovy) 107 | res[2][3] = -1 108 | res[2][2] = - (zFar + zNear) / (zFar - zNear) 109 | res[3][2] = -(2 * zFar * zNear) / (zFar - zNear) 110 | 111 | return res.T 112 | 113 | 114 | def ortho(left, right, bottom, top, zNear, zFar): 115 | # res = np.ones([4, 4], dtype=np.float32) 116 | res = identity() 117 | res[0][0] = 2 / (right - left) 118 | res[1][1] = 2 / (top - bottom) 119 | res[2][2] = - 2 / (zFar - zNear) 120 | res[3][0] = - (right + left) / (right - left) 121 | res[3][1] = - (top + bottom) / (top - bottom) 122 | res[3][2] = - (zFar + zNear) / (zFar - zNear) 123 | return res.T 124 | 125 | 126 | def lookat(eye, center, up): 127 | f = normalize(center - eye) 128 | s = normalize(cross(f, up)) 129 | u = cross(s, f) 130 | 131 | res = identity() 132 | res[0][0] = s[0] 133 | res[1][0] = s[1] 134 | res[2][0] = s[2] 135 | res[0][1] = u[0] 136 | res[1][1] = u[1] 137 | res[2][1] = u[2] 138 | res[0][2] = -f[0] 139 | res[1][2] = -f[1] 140 | res[2][2] = -f[2] 141 | res[3][0] = -dot(s, eye) 142 | res[3][1] = -dot(u, eye) 143 | res[3][2] = -dot(f, eye) 144 | return res.T 145 | 146 | 147 | def transform(d, m): 148 | return np.dot(m, d.T).T 149 | -------------------------------------------------------------------------------- /lib/render/mesh.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import numpy as np 25 | 26 | 27 | def save_obj_mesh(mesh_path, verts, faces): 28 | file = open(mesh_path, 'w') 29 | for v in verts: 30 | file.write('v %.4f %.4f %.4f\n' % (v[0], v[1], v[2])) 31 | for f in faces: 32 | f_plus = f + 1 33 | file.write('f %d %d %d\n' % (f_plus[0], f_plus[1], f_plus[2])) 34 | file.close() 35 | 36 | # https://github.com/ratcave/wavefront_reader 37 | def read_mtlfile(fname): 38 | materials = {} 39 | with open(fname) as f: 40 | lines = f.read().splitlines() 41 | 42 | for line in lines: 43 | if line: 44 | split_line = line.strip().split(' ', 1) 45 | if len(split_line) < 2: 46 | continue 47 | 48 | prefix, data = split_line[0], split_line[1] 49 | if 'newmtl' in prefix: 50 | material = {} 51 | materials[data] = material 52 | elif materials: 53 | if data: 54 | split_data = data.strip().split(' ') 55 | 56 | # assume texture maps are in the same level 57 | # WARNING: do not include space in your filename!! 58 | if 'map' in prefix: 59 | material[prefix] = split_data[-1].split('\\')[-1] 60 | elif len(split_data) > 1: 61 | material[prefix] = tuple(float(d) for d in split_data) 62 | else: 63 | try: 64 | material[prefix] = int(data) 65 | except ValueError: 66 | material[prefix] = float(data) 67 | 68 | return materials 69 | 70 | 71 | def load_obj_mesh_mtl(mesh_file): 72 | vertex_data = [] 73 | norm_data = [] 74 | uv_data = [] 75 | 76 | face_data = [] 77 | face_norm_data = [] 78 | face_uv_data = [] 79 | 80 | # face per material 81 | face_data_mat = {} 82 | face_norm_data_mat = {} 83 | face_uv_data_mat = {} 84 | 85 | # current material name 86 | mtl_data = None 87 | cur_mat = None 88 | 89 | if isinstance(mesh_file, str): 90 | f = open(mesh_file, "r") 91 | else: 92 | f = mesh_file 93 | for line in f: 94 | if isinstance(line, bytes): 95 | line = line.decode("utf-8") 96 | if line.startswith('#'): 97 | continue 98 | values = line.split() 99 | if not values: 100 | continue 101 | 102 | if values[0] == 'v': 103 | v = list(map(float, values[1:4])) 104 | vertex_data.append(v) 105 | elif values[0] == 'vn': 106 | vn = list(map(float, values[1:4])) 107 | norm_data.append(vn) 108 | elif values[0] == 'vt': 109 | vt = list(map(float, values[1:3])) 110 | uv_data.append(vt) 111 | elif values[0] == 'mtllib': 112 | mtl_data = read_mtlfile(mesh_file.replace(mesh_file.split('/')[-1],values[1])) 113 | elif values[0] == 'usemtl': 114 | cur_mat = values[1] 115 | elif values[0] == 'f': 116 | # local triangle data 117 | l_face_data = [] 118 | l_face_uv_data = [] 119 | l_face_norm_data = [] 120 | 121 | # quad mesh 122 | if len(values) > 4: 123 | f = list(map(lambda x: int(x.split('/')[0]) if int(x.split('/')[0]) < 0 else int(x.split('/')[0])-1, values[1:4])) 124 | l_face_data.append(f) 125 | f = list(map(lambda x: int(x.split('/')[0]) if int(x.split('/')[0]) < 0 else int(x.split('/')[0])-1, [values[3], values[4], values[1]])) 126 | l_face_data.append(f) 127 | # tri mesh 128 | else: 129 | f = list(map(lambda x: int(x.split('/')[0]) if int(x.split('/')[0]) < 0 else int(x.split('/')[0])-1, values[1:4])) 130 | l_face_data.append(f) 131 | # deal with texture 132 | if len(values[1].split('/')) >= 2: 133 | # quad mesh 134 | if len(values) > 4: 135 | f = list(map(lambda x: int(x.split('/')[1]) if int(x.split('/')[1]) < 0 else int(x.split('/')[1])-1, values[1:4])) 136 | l_face_uv_data.append(f) 137 | f = list(map(lambda x: int(x.split('/')[1]) if int(x.split('/')[1]) < 0 else int(x.split('/')[1])-1, [values[3], values[4], values[1]])) 138 | l_face_uv_data.append(f) 139 | # tri mesh 140 | elif len(values[1].split('/')[1]) != 0: 141 | f = list(map(lambda x: int(x.split('/')[1]) if int(x.split('/')[1]) < 0 else int(x.split('/')[1])-1, values[1:4])) 142 | l_face_uv_data.append(f) 143 | # deal with normal 144 | if len(values[1].split('/')) == 3: 145 | # quad mesh 146 | if len(values) > 4: 147 | f = list(map(lambda x: int(x.split('/')[2]) if int(x.split('/')[2]) < 0 else int(x.split('/')[2])-1, values[1:4])) 148 | l_face_norm_data.append(f) 149 | f = list(map(lambda x: int(x.split('/')[2]) if int(x.split('/')[2]) < 0 else int(x.split('/')[2])-1, [values[3], values[4], values[1]])) 150 | l_face_norm_data.append(f) 151 | # tri mesh 152 | elif len(values[1].split('/')[2]) != 0: 153 | f = list(map(lambda x: int(x.split('/')[2]) if int(x.split('/')[2]) < 0 else int(x.split('/')[2])-1, values[1:4])) 154 | l_face_norm_data.append(f) 155 | 156 | face_data += l_face_data 157 | face_uv_data += l_face_uv_data 158 | face_norm_data += l_face_norm_data 159 | 160 | if cur_mat is not None: 161 | if cur_mat not in face_data_mat.keys(): 162 | face_data_mat[cur_mat] = [] 163 | if cur_mat not in face_uv_data_mat.keys(): 164 | face_uv_data_mat[cur_mat] = [] 165 | if cur_mat not in face_norm_data_mat.keys(): 166 | face_norm_data_mat[cur_mat] = [] 167 | face_data_mat[cur_mat] += l_face_data 168 | face_uv_data_mat[cur_mat] += l_face_uv_data 169 | face_norm_data_mat[cur_mat] += l_face_norm_data 170 | 171 | vertices = np.array(vertex_data) 172 | faces = np.array(face_data) 173 | 174 | norms = np.array(norm_data) 175 | norms = normalize_v3(norms) 176 | face_normals = np.array(face_norm_data) 177 | 178 | uvs = np.array(uv_data) 179 | face_uvs = np.array(face_uv_data) 180 | 181 | out_tuple = (vertices, faces, norms, face_normals, uvs, face_uvs) 182 | 183 | if cur_mat is not None and mtl_data is not None: 184 | for key in face_data_mat: 185 | face_data_mat[key] = np.array(face_data_mat[key]) 186 | face_uv_data_mat[key] = np.array(face_uv_data_mat[key]) 187 | face_norm_data_mat[key] = np.array(face_norm_data_mat[key]) 188 | 189 | out_tuple += (face_data_mat, face_norm_data_mat, face_uv_data_mat, mtl_data) 190 | 191 | return out_tuple 192 | 193 | 194 | def load_obj_mesh(mesh_file, with_normal=False, with_texture=False): 195 | vertex_data = [] 196 | norm_data = [] 197 | uv_data = [] 198 | 199 | face_data = [] 200 | face_norm_data = [] 201 | face_uv_data = [] 202 | 203 | if isinstance(mesh_file, str): 204 | f = open(mesh_file, "r") 205 | else: 206 | f = mesh_file 207 | for line in f: 208 | if isinstance(line, bytes): 209 | line = line.decode("utf-8") 210 | if line.startswith('#'): 211 | continue 212 | values = line.split() 213 | if not values: 214 | continue 215 | 216 | if values[0] == 'v': 217 | v = list(map(float, values[1:4])) 218 | vertex_data.append(v) 219 | elif values[0] == 'vn': 220 | vn = list(map(float, values[1:4])) 221 | norm_data.append(vn) 222 | elif values[0] == 'vt': 223 | vt = list(map(float, values[1:3])) 224 | uv_data.append(vt) 225 | 226 | elif values[0] == 'f': 227 | # quad mesh 228 | if len(values) > 4: 229 | f = list(map(lambda x: int(x.split('/')[0]), values[1:4])) 230 | face_data.append(f) 231 | f = list(map(lambda x: int(x.split('/')[0]), [values[3], values[4], values[1]])) 232 | face_data.append(f) 233 | # tri mesh 234 | else: 235 | f = list(map(lambda x: int(x.split('/')[0]), values[1:4])) 236 | face_data.append(f) 237 | 238 | # deal with texture 239 | if len(values[1].split('/')) >= 2: 240 | # quad mesh 241 | if len(values) > 4: 242 | f = list(map(lambda x: int(x.split('/')[1]), values[1:4])) 243 | face_uv_data.append(f) 244 | f = list(map(lambda x: int(x.split('/')[1]), [values[3], values[4], values[1]])) 245 | face_uv_data.append(f) 246 | # tri mesh 247 | elif len(values[1].split('/')[1]) != 0: 248 | f = list(map(lambda x: int(x.split('/')[1]), values[1:4])) 249 | face_uv_data.append(f) 250 | # deal with normal 251 | if len(values[1].split('/')) == 3: 252 | # quad mesh 253 | if len(values) > 4: 254 | f = list(map(lambda x: int(x.split('/')[2]), values[1:4])) 255 | face_norm_data.append(f) 256 | f = list(map(lambda x: int(x.split('/')[2]), [values[3], values[4], values[1]])) 257 | face_norm_data.append(f) 258 | # tri mesh 259 | elif len(values[1].split('/')[2]) != 0: 260 | f = list(map(lambda x: int(x.split('/')[2]), values[1:4])) 261 | face_norm_data.append(f) 262 | 263 | vertices = np.array(vertex_data) 264 | faces = np.array(face_data) - 1 265 | 266 | if with_texture and with_normal: 267 | uvs = np.array(uv_data) 268 | face_uvs = np.array(face_uv_data) - 1 269 | norms = np.array(norm_data) 270 | if norms.shape[0] == 0: 271 | norms = compute_normal(vertices, faces) 272 | face_normals = faces 273 | else: 274 | norms = normalize_v3(norms) 275 | face_normals = np.array(face_norm_data) - 1 276 | return vertices, faces, norms, face_normals, uvs, face_uvs 277 | 278 | if with_texture: 279 | uvs = np.array(uv_data) 280 | face_uvs = np.array(face_uv_data) - 1 281 | return vertices, faces, uvs, face_uvs 282 | 283 | if with_normal: 284 | norms = np.array(norm_data) 285 | norms = normalize_v3(norms) 286 | face_normals = np.array(face_norm_data) - 1 287 | return vertices, faces, norms, face_normals 288 | 289 | return vertices, faces 290 | 291 | 292 | def normalize_v3(arr): 293 | ''' Normalize a numpy array of 3 component vectors shape=(n,3) ''' 294 | lens = np.sqrt(arr[:, 0] ** 2 + arr[:, 1] ** 2 + arr[:, 2] ** 2) 295 | eps = 0.00000001 296 | lens[lens < eps] = eps 297 | arr[:, 0] /= lens 298 | arr[:, 1] /= lens 299 | arr[:, 2] /= lens 300 | return arr 301 | 302 | 303 | def compute_normal(vertices, faces): 304 | # Create a zeroed array with the same type and shape as our vertices i.e., per vertex normal 305 | norm = np.zeros(vertices.shape, dtype=vertices.dtype) 306 | # Create an indexed view into the vertex array using the array of three indices for triangles 307 | tris = vertices[faces] 308 | # Calculate the normal for all the triangles, by taking the cross product of the vectors v1-v0, and v2-v0 in each triangle 309 | n = np.cross(tris[::, 1] - tris[::, 0], tris[::, 2] - tris[::, 0]) 310 | # n is now an array of normals per triangle. The length of each normal is dependent the vertices, 311 | # we need to normalize these, so that our next step weights each normal equally. 312 | normalize_v3(n) 313 | # now we have a normalized array of normals, one per triangle, i.e., per triangle normals. 314 | # But instead of one per triangle (i.e., flat shading), we add to each vertex in that triangle, 315 | # the triangles' normal. Multiple triangles would then contribute to every vertex, so we need to normalize again afterwards. 316 | # The cool part, we can actually add the normals through an indexed view of our (zeroed) per vertex normal array 317 | norm[faces[:, 0]] += n 318 | norm[faces[:, 1]] += n 319 | norm[faces[:, 2]] += n 320 | normalize_v3(norm) 321 | 322 | return norm 323 | 324 | # compute tangent and bitangent 325 | def compute_tangent(vertices, faces, normals, uvs, faceuvs): 326 | # NOTE: this could be numerically unstable around [0,0,1] 327 | # but other current solutions are pretty freaky somehow 328 | c1 = np.cross(normals, np.array([0,1,0.0])) 329 | tan = c1 330 | normalize_v3(tan) 331 | btan = np.cross(normals, tan) 332 | 333 | # NOTE: traditional version is below 334 | 335 | # pts_tris = vertices[faces] 336 | # uv_tris = uvs[faceuvs] 337 | 338 | # W = np.stack([pts_tris[::, 1] - pts_tris[::, 0], pts_tris[::, 2] - pts_tris[::, 0]],2) 339 | # UV = np.stack([uv_tris[::, 1] - uv_tris[::, 0], uv_tris[::, 2] - uv_tris[::, 0]], 1) 340 | 341 | # for i in range(W.shape[0]): 342 | # W[i,::] = W[i,::].dot(np.linalg.inv(UV[i,::])) 343 | 344 | # tan = np.zeros(vertices.shape, dtype=vertices.dtype) 345 | # tan[faces[:,0]] += W[:,:,0] 346 | # tan[faces[:,1]] += W[:,:,0] 347 | # tan[faces[:,2]] += W[:,:,0] 348 | 349 | # btan = np.zeros(vertices.shape, dtype=vertices.dtype) 350 | # btan[faces[:,0]] += W[:,:,1] 351 | # btan[faces[:,1]] += W[:,:,1] 352 | # btan[faces[:,2]] += W[:,:,1] 353 | 354 | # normalize_v3(tan) 355 | 356 | # ndott = np.sum(normals*tan, 1, keepdims=True) 357 | # tan = tan - ndott * normals 358 | 359 | # normalize_v3(btan) 360 | # normalize_v3(tan) 361 | 362 | # tan[np.sum(np.cross(normals, tan) * btan, 1) < 0,:] *= -1.0 363 | 364 | return tan, btan 365 | 366 | if __name__ == '__main__': 367 | pts, tri, nml, trin, uvs, triuv = load_obj_mesh('/home/ICT2000/ssaito/Documents/Body/tmp/Baseball_Pitching/0012.obj', True, True) 368 | compute_tangent(pts, tri, uvs, triuv) -------------------------------------------------------------------------------- /lib/sdf.py: -------------------------------------------------------------------------------- 1 | ''' 2 | MIT License 3 | 4 | Copyright (c) 2019 Shunsuke Saito, Zeng Huang, and Ryota Natsume 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | ''' 24 | import numpy as np 25 | 26 | 27 | def create_grid(resX, resY, resZ, b_min=np.array([-1, -1, -1]), b_max=np.array([1, 1, 1]), transform=None): 28 | ''' 29 | Create a dense grid of given resolution and bounding box 30 | :param resX: resolution along X axis 31 | :param resY: resolution along Y axis 32 | :param resZ: resolution along Z axis 33 | :param b_min: vec3 (x_min, y_min, z_min) bounding box corner 34 | :param b_max: vec3 (x_max, y_max, z_max) bounding box corner 35 | :return: [3, resX, resY, resZ] coordinates of the grid, and transform matrix from mesh index 36 | ''' 37 | coords = np.mgrid[:resX, :resY, :resZ] 38 | coords = coords.reshape(3, -1) 39 | coords_matrix = np.eye(4) 40 | length = b_max - b_min 41 | coords_matrix[0, 0] = length[0] / resX 42 | coords_matrix[1, 1] = length[1] / resY 43 | coords_matrix[2, 2] = length[2] / resZ 44 | coords_matrix[0:3, 3] = b_min 45 | coords = np.matmul(coords_matrix[:3, :3], coords) + coords_matrix[:3, 3:4] 46 | if transform is not None: 47 | coords = np.matmul(transform[:3, :3], coords) + transform[:3, 3:4] 48 | coords_matrix = np.matmul(transform, coords_matrix) 49 | coords = coords.reshape(3, resX, resY, resZ) 50 | return coords, coords_matrix 51 | 52 | 53 | def batch_eval(points, eval_func, num_samples=512 * 512 * 512): 54 | num_pts = points.shape[1] 55 | sdf = np.zeros(num_pts) 56 | 57 | num_batches = num_pts // num_samples 58 | for i in range(num_batches): 59 | sdf[i * num_samples:i * num_samples + num_samples] = eval_func( 60 | points[:, i * num_samples:i * num_samples + num_samples]) 61 | if num_pts % num_samples: 62 | sdf[num_batches * num_samples:] = eval_func(points[:, num_batches * num_samples:]) 63 | 64 | return sdf 65 | 66 | def batch_eval_tensor(points, eval_func, num_samples=512 * 512 * 512): 67 | num_pts = points.size(1) 68 | 69 | num_batches = num_pts // num_samples 70 | vals = [] 71 | for i in range(num_batches): 72 | vals.append(eval_func(points[:, i * num_samples:i * num_samples + num_samples])) 73 | if num_pts % num_samples: 74 | vals.append(eval_func(points[:, num_batches * num_samples:])) 75 | 76 | return np.concatenate(vals,0) 77 | 78 | def eval_grid(coords, eval_func, num_samples=512 * 512 * 512): 79 | resolution = coords.shape[1:4] 80 | coords = coords.reshape([3, -1]) 81 | sdf = batch_eval(coords, eval_func, num_samples=num_samples) 82 | return sdf.reshape(resolution) 83 | 84 | 85 | import time 86 | def eval_grid_octree(coords, eval_func, 87 | init_resolution=64, threshold=0.05, 88 | num_samples=512 * 512 * 512): 89 | resolution = coords.shape[1:4] 90 | 91 | sdf = np.zeros(resolution) 92 | 93 | notprocessed = np.zeros(resolution, dtype=np.bool) 94 | notprocessed[:-1,:-1,:-1] = True 95 | grid_mask = np.zeros(resolution, dtype=np.bool) 96 | 97 | reso = resolution[0] // init_resolution 98 | 99 | while reso > 0: 100 | # subdivide the grid 101 | grid_mask[0:resolution[0]:reso, 0:resolution[1]:reso, 0:resolution[2]:reso] = True 102 | # test samples in this iteration 103 | test_mask = np.logical_and(grid_mask, notprocessed) 104 | # print('step size:', reso, 'test sample size:', test_mask.sum()) 105 | points = coords[:, test_mask] 106 | 107 | sdf[test_mask] = batch_eval(points, eval_func, num_samples=num_samples) 108 | notprocessed[test_mask] = False 109 | 110 | # do interpolation 111 | if reso <= 1: 112 | break 113 | x_grid = np.arange(0, resolution[0], reso) 114 | y_grid = np.arange(0, resolution[1], reso) 115 | z_grid = np.arange(0, resolution[2], reso) 116 | 117 | v = sdf[tuple(np.meshgrid(x_grid, y_grid, z_grid, indexing='ij'))] 118 | 119 | v0 = v[:-1,:-1,:-1] 120 | v1 = v[:-1,:-1,1:] 121 | v2 = v[:-1,1:,:-1] 122 | v3 = v[:-1,1:,1:] 123 | v4 = v[1:,:-1,:-1] 124 | v5 = v[1:,:-1,1:] 125 | v6 = v[1:,1:,:-1] 126 | v7 = v[1:,1:,1:] 127 | 128 | x_grid = x_grid[:-1] + reso//2 129 | y_grid = y_grid[:-1] + reso//2 130 | z_grid = z_grid[:-1] + reso//2 131 | 132 | nonprocessed_grid = notprocessed[tuple(np.meshgrid(x_grid, y_grid, z_grid, indexing='ij'))] 133 | 134 | v = np.stack([v0,v1,v2,v3,v4,v5,v6,v7], 0) 135 | v_min = v.min(0) 136 | v_max = v.max(0) 137 | v = 0.5*(v_min+v_max) 138 | 139 | skip_grid = np.logical_and(((v_max - v_min) < threshold), nonprocessed_grid) 140 | 141 | n_x = resolution[0] // reso 142 | n_y = resolution[1] // reso 143 | n_z = resolution[2] // reso 144 | 145 | xs, ys, zs = np.where(skip_grid) 146 | for x, y, z in zip(xs*reso, ys*reso, zs*reso): 147 | sdf[x:(x+reso+1), y:(y+reso+1), z:(z+reso+1)] = v[x//reso,y//reso,z//reso] 148 | notprocessed[x:(x+reso+1), y:(y+reso+1), z:(z+reso+1)] = False 149 | reso //= 2 150 | 151 | return sdf.reshape(resolution) 152 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # This is a requirement.txt file 2 | # to install all these requirements: 3 | # `pip install -r requirements.txt` 4 | # Make sure you have pip installed! 5 | # To update just use `pip install -r requirements.txt --upgrade` 6 | torch 7 | torchvision 8 | Pillow # PIL 9 | scikit-image #skimage 10 | tqdm 11 | opencv-python # cv2 12 | trimesh 13 | PyOpenGL 14 | ffmpeg 15 | 16 | -------------------------------------------------------------------------------- /sample_images/test.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/facebookresearch/pifuhd/e47c4d918aaedd5f5608192b130bda150b1fb0ab/sample_images/test.png -------------------------------------------------------------------------------- /sample_images/test_keypoints.json: -------------------------------------------------------------------------------- 1 | {"version":1.3,"people":[{"person_id":[-1],"pose_keypoints_2d":[251.14,70.0905,0.920212,249.729,121.622,0.848532,205.176,121.648,0.783397,185.738,191.227,0.800347,216.332,199.615,0.829802,294.238,120.247,0.805623,306.9,189.876,0.865557,324.944,223.301,0.823568,251.178,263.668,0.671555,226.047,265.073,0.633188,227.432,370.844,0.798983,234.42,466.968,0.718138,281.782,262.271,0.647396,295.743,372.224,0.780934,316.553,472.555,0.721108,244.17,60.3191,0.876,260.855,60.3366,0.937398,230.252,60.3082,0.947452,271.992,60.2974,0.922649,354.182,494.776,0.658449,356.96,489.211,0.639241,306.767,485.045,0.680462,226.043,483.654,0.614895,216.363,480.89,0.687368,240.024,476.682,0.642223],"face_keypoints_2d":[],"hand_left_keypoints_2d":[],"hand_right_keypoints_2d":[],"pose_keypoints_3d":[],"face_keypoints_3d":[],"hand_left_keypoints_3d":[],"hand_right_keypoints_3d":[]},{"person_id":[-1],"pose_keypoints_2d":[395.944,283.101,0.888373,396.006,306.861,0.795637,372.222,306.829,0.772225,391.774,327.698,0.713164,433.532,327.688,0.73626,427.926,308.195,0.697673,443.259,347.19,0.214575,453.021,329.12,0.340148,398.738,363.94,0.583225,382.024,365.313,0.553282,380.634,383.443,0.619834,393.159,485.078,0.648966,418.185,363.89,0.574877,444.662,383.37,0.644425,462.8,473.927,0.765089,390.345,273.432,0.869964,405.621,273.465,0.888317,375.097,274.783,0.786201,418.156,277.609,0.699742,490.6,494.799,0.54446,494.758,487.844,0.580562,457.186,485.025,0.63643,395.953,496.228,0.254274,384.825,496.225,0.320764,394.565,492.017,0.421362],"face_keypoints_2d":[],"hand_left_keypoints_2d":[],"hand_right_keypoints_2d":[],"pose_keypoints_3d":[],"face_keypoints_3d":[],"hand_left_keypoints_3d":[],"hand_right_keypoints_3d":[]},{"person_id":[-1],"pose_keypoints_2d":[327.693,280.348,0.776371,324.912,298.466,0.448569,304.025,304.022,0.334807,0,0,0,0,0,0,348.576,295.664,0.665439,362.57,327.695,0.453743,352.798,356.962,0.209829,327.75,363.9,0.300198,312.373,365.326,0.227978,0,0,0,0,0,0,344.42,362.509,0.296135,369.462,376.432,0.0696449,0,0,0,319.326,273.359,0.772246,333.253,273.349,0.796905,308.194,276.19,0.243569,341.642,274.786,0.390684,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"face_keypoints_2d":[],"hand_left_keypoints_2d":[],"hand_right_keypoints_2d":[],"pose_keypoints_3d":[],"face_keypoints_3d":[],"hand_left_keypoints_3d":[],"hand_right_keypoints_3d":[]}]} -------------------------------------------------------------------------------- /scripts/demo.sh: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | python -m apps.simple_test 4 | # python apps/clean_mesh.py -f ./results/pifuhd_final/recon 5 | python -m apps.render_turntable -f ./results/pifuhd_final/recon -ww 512 -hh 512 -------------------------------------------------------------------------------- /scripts/download_trained_model.sh: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 2 | 3 | set -ex 4 | 5 | mkdir -p checkpoints 6 | cd checkpoints 7 | wget "https://dl.fbaipublicfiles.com/pifuhd/checkpoints/pifuhd.pt" pifuhd.pt 8 | cd .. 9 | --------------------------------------------------------------------------------