├── .gitignore ├── LICENSE ├── README.md ├── assets └── teaser.png ├── bake_texture_field.py ├── cameras.py ├── config.py ├── configs ├── discretization_agnostic │ ├── cat_cloud100k.yaml │ ├── cat_cloud10k.yaml │ ├── cat_dense.yaml │ ├── cat_iso.yaml │ ├── cat_qes.yaml │ ├── human_cloud400k.yaml │ ├── human_cloud40k.yaml │ ├── human_dense.yaml │ ├── human_iso.yaml │ └── human_qes.yaml ├── shape_descriptors │ ├── cat_efuncs.yaml │ ├── cat_hks.yaml │ ├── cat_shot.yaml │ ├── human_efuncs.yaml │ ├── human_hks.yaml │ └── human_shot.yaml ├── texture_reconstruction │ ├── intrinsic_cat.yaml │ ├── intrinsic_human.yaml │ ├── neutex_cat.yaml │ ├── neutex_human.yaml │ ├── tf_rff_cat.yaml │ └── tf_rff_human.yaml └── texture_transfer │ └── cat_orig.yaml ├── dataset.py ├── download_data.sh ├── eval.py ├── evaluation_metrics.py ├── layers.py ├── mesh.py ├── model.py ├── neutex ├── __init__.py ├── atlasnet_inverse.py ├── atlasnet_networks.py ├── network_utils.py ├── neutex.py ├── neutex_pretrain_mapping.ipynb ├── pretrained │ ├── cat_rescaled_rotated_pretrained_neutex_mapping.pt │ └── human_pretrained_neutex_mapping.pt └── texture_mlp.py ├── preprocess_dataset.py ├── preprocess_eigenfunctions.py ├── preprocessing_scripts ├── preprocess_cat_cloud100k.sh ├── preprocess_cat_cloud10k.sh ├── preprocess_cat_dataset.sh ├── preprocess_cat_dense.sh ├── preprocess_cat_iso.sh ├── preprocess_cat_qes.sh ├── preprocess_human_cloud400k.sh ├── preprocess_human_cloud40k.sh ├── preprocess_human_dataset.sh ├── preprocess_human_dense.sh ├── preprocess_human_iso.sh └── preprocess_human_qes.sh ├── ray_dataloader.py ├── render_video.py ├── render_view.py ├── renderer.py ├── texture_transfer ├── texture_transfer_cat_arap.ipynb ├── texture_transfer_cat_dense.ipynb ├── texture_transfer_cat_tosca2.ipynb └── texture_transfer_dog_tosca0.ipynb ├── train.py ├── trainer.py ├── training_scripts ├── train_cat_discretizations.sh ├── train_cat_shape_descriptors.sh ├── train_cat_tex_recon.sh ├── train_cat_texture_transfer.sh ├── train_human_discretizations.sh ├── train_human_shape_descriptors.sh └── train_human_tex_recon.sh └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | .ipynb_checkpoints 3 | .idea 4 | out 5 | data 6 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2022, Lukas Koestler, Daniel Grittner, Michael Moeller, Daniel Cremers, Zorah Laehner. 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | * Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

Intrinsic Neural Fields: Learning Functions on Manifolds

2 | 3 |

4 | Lukas Koestler1*   5 | Daniel Grittner1*   6 | Michael Moeller2   7 | Daniel Cremers1   8 | Zorah Lähner2   9 |

10 | 11 |

12 | *equal contribution    13 |

14 | 15 |

16 | 1Technical University of Munich   17 | 2University of Siegen
18 |

19 | 20 |

21 | European Conference on Computer Vision (ECCV) 2022, Tel Aviv, Israel 22 |

23 | 24 |

25 | arXiv | 26 | Video | 27 | Poster 28 |

29 | 30 |

 

31 | 32 | ![teaser](assets/teaser.png) 33 |

(a) Overview of our method. We use the eigenfunctions of the Laplace-Beltrami operator (LBO) at each point as a point embedding. This overcomes the spectral bias of the multilayer perceptron (MLP), and hence the combined intrinsic neural field can represent a high-frequency function on the surface. Notice that the point can be inside a triangle, and the function is clearly more detailed than the discretization (insets). (b) An intrinsic neural texture field trained on one shape (top) can be transferred to a new shape (bottom) without retraining. (c) Due to our intrinsic approach (LBO eigenfunctions) local geometry is maintained in close but separate parts, whereas an extrinsic approach (Random Fourier Features) shows bleeding artifacts when trained with sparse supervision.

34 | 35 |

 

36 | 37 | ## Abstract 38 | 39 |

Neural fields have gained significant attention in the computer vision community due to their excellent performance in novel view synthesis, geometry reconstruction, and generative modeling. Some of their advantages are a sound theoretic foundation and an easy implementation in current deep learning frameworks. While neural fields have been applied to signals on manifolds, e.g., for texture reconstruction, their representation has been limited to extrinsically embedding the shape into Euclidean space. The extrinsic embedding ignores known intrinsic manifold properties and is inflexible wrt. transfer of the learned function. To overcome these limitations, this work introduces intrinsic neural fields, a novel and versatile representation for neural fields on manifolds. Intrinsic neural fields combine the advantages of neural fields with the spectral properties of the Laplace-Beltrami operator. We show theoretically that intrinsic neural fields inherit many desirable properties of the extrinsic neural field framework but exhibit additional intrinsic qualities, like isometry invariance. In experiments, we show intrinsic neural fields can reconstruct high-fidelity textures from images with state-of-the-art quality and are robust to the discretization of the underlying manifold. We demonstrate the versatility of intrinsic neural fields by tackling various applications: texture transfer between deformed shapes & different shapes, texture reconstruction from real-world images with view dependence, and discretization-agnostic learning on meshes and point clouds.

40 | 41 | ![bunnygif](https://vision.in.tum.de/webshare/g/intrinsic-neural-fields/lindt_bunny.gif) 42 | 43 | ## Code and Data Release 44 | 45 | - [X] 📣 Release main code and data 46 | - [X] Release local triangulation code as a Python package for ray-pointcloud intersection. Has been merged into [nmwsharp/potpourri3d](https://github.com/nmwsharp/potpourri3d/commit/adb76f1566bba2ef2bc2d1a6e6c57346796c89f8). 47 | - [X] Publish BigBIRD configs, data, and scripts. Pushed into banch [bigbird](https://github.com/tum-vision/intrinsic-neural-fields/tree/bigbird) and will be merged into master after aditional checks. 48 | 49 | ## Related Work 50 | 51 | The following papers explore ideas similar to the ones presented within our work and will be referenced in an updated version of our arxiv paper. If there is another paper missing from the list, please reach out to us and we will add it. Papers are ordered by the initial publication date on arxiv. We believe that the variety of papers shows the potential of the underlying idea across a range of applications. 52 | 53 | - [Sign and Basis Invariant Networks for Spectral Graph Representation Learning](https://arxiv.org/abs/2202.13013) 54 | - [Generalised Implicit Neural Representations](https://arxiv.org/abs/2205.15674) 55 | - [Δ -PINNs: physics-informed neural networks on complex geometries](https://arxiv.org/abs/2209.03984) 56 | 57 | ## Data 58 | 59 | The data for the experiments can be downloaded by running the following command: 60 | 61 | ``` 62 | ./download_data.sh 63 | ``` 64 | 65 | ## Installation 66 | 67 | Run the following for creating a conda environment and installing the required dependencies: 68 | 69 | ``` 70 | conda create --name intrinsic-neural-fields python=3.6 -y 71 | conda activate intrinsic-neural-fields 72 | 73 | # Dependencies 74 | pip3 install --ignore-installed certifi "trimesh[all]" 75 | conda install pytorch torchvision cudatoolkit=11.3 -c pytorch -y 76 | pip3 install numpy==1.18.0 77 | conda install -c conda-forge igl pyembree tqdm torchinfo==1.5.4 imageio tensorboardx opencv pyyaml -y 78 | conda install scikit-image matplotlib -y 79 | pip3 install lpips crc32c robust_laplacian 80 | ``` 81 | 82 | ## Data Preprocessing 83 | 84 | 1) Preprocess eigenfunctions: 85 | 86 | ``` 87 | python preprocess_eigenfunctions.py ... # See preprocess_eigenfunctions.py for the arguments 88 | ``` 89 | 90 | 2) Preprocess the training splits (train, val, test): 91 | 92 | ``` 93 | python preprocess_dataset.py ... # See preprocess_dataset.py for the arguments 94 | ``` 95 | 96 | **Recommended:** For convenience, you can also use the prepared preprocessing scripts: 97 | 98 | ``` 99 | ./preprocessing_scripts/ 100 | ``` 101 | 102 | ## Training 103 | 104 | After the preprocessing, run the following for trainings 105 | 106 | ``` 107 | python train.py --allow_checkpoint_loading 108 | ``` 109 | 110 | After the training, you can run 111 | 112 | ``` 113 | python eval.py ... # See eval.py for the arguments 114 | ``` 115 | 116 | **Recommended:** For convenience, you can also use the prepared training scripts: 117 | 118 | ``` 119 | ./training_scripts/ 120 | ``` 121 | Please note that most scripts require the method as an input parameter. Please review the scripts for possible input parameters. 122 | 123 | ## Visualizations 124 | 125 | 1) Rendering a view from a given camera position 126 | 127 | ``` 128 | python render_view.py ... # See render_view.py for the arguments 129 | ``` 130 | 131 | 2) Bake the learned texture into a UV-map for exploration 132 | 133 | ``` 134 | python bake_texture_field.py ... # See bake_texture_field.py for the arguments 135 | ``` 136 | 137 | ## Expected Dataset structure for preprocessing 138 | 139 | Custom datasets can be used with the existing code if they have the following folder structure: 140 | 141 | dataset 142 | |_ train.lst (Defines which views are preprocessed for training) 143 | |_ test.lst (Defines which views are preprocessed for testing) 144 | |_ val.lst (Defines which views are preprocessed for validation) 145 | |_ (Represents a view) 146 | |_ depth 147 | |_ cameras.npz (camera parameters) 148 | |_ depth_0000.exr (depth map) 149 | |_ image 150 | |_ 000.png (GT view) 151 | |_ ... (the rest does not matter) 152 | |_ ... 153 | 154 | ## License 155 | 156 | The code is provided under a BSD 3-clause license. See the LICENSE file for details. Note also the different licenses for thirdparty submodules. 157 | -------------------------------------------------------------------------------- /assets/teaser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tum-vision/intrinsic-neural-fields/a4da0c832f7ae395606f72cffcd82dfef98b29e6/assets/teaser.png -------------------------------------------------------------------------------- /cameras.py: -------------------------------------------------------------------------------- 1 | """ 2 | Partly taken from 3 | https://github.com/tum-vision/tandem/blob/master/cva_mvsnet/models/datasets.py#L144 4 | """ 5 | import numpy as np 6 | import torch 7 | 8 | 9 | def _fx(cam: dict) -> float: 10 | return cam['K'][0, 0] 11 | 12 | 13 | def _cx(cam: dict) -> float: 14 | return cam['K'][0, 2] 15 | 16 | 17 | def _fy(cam: dict) -> float: 18 | return cam['K'][1, 1] 19 | 20 | 21 | def _cy(cam: dict) -> float: 22 | return cam['K'][1, 2] 23 | 24 | 25 | def _height(cam: dict) -> int: 26 | return cam['height'] 27 | 28 | 29 | def _width(cam: dict) -> int: 30 | return cam['width'] 31 | 32 | 33 | def _is_torch(cam: dict) -> bool: 34 | return torch.is_tensor(cam['K']) 35 | 36 | 37 | def cam_resize(cam: dict, 38 | height: int, 39 | width: int): 40 | """ 41 | Convert to new camera intrinsics for resize of image from original camera. 42 | :param cam: 43 | camera intrinsics 44 | :param height: 45 | height of resized frame 46 | :param width: 47 | width of resized frame 48 | :return: 49 | camera intrinsics for resized frame 50 | """ 51 | center_x = 0.5 * float(_width(cam) - 1) 52 | center_y = 0.5 * float(_height(cam) - 1) 53 | 54 | orig_cx_diff = _cx(cam) - center_x 55 | orig_cy_diff = _cy(cam) - center_y 56 | 57 | scaled_center_x = 0.5 * float(width - 1) 58 | scaled_center_y = 0.5 * float(height - 1) 59 | 60 | scale_x = float(width) / float(_width(cam)) 61 | scale_y = float(height) / float(_height(cam)) 62 | 63 | fx = scale_x * _fx(cam) 64 | fy = scale_y * _fy(cam) 65 | cx = scaled_center_x + scale_x * orig_cx_diff 66 | cy = scaled_center_y + scale_y * orig_cy_diff 67 | 68 | if _is_torch(cam): 69 | return { 70 | "K": torch.tensor([[fx, 0, cx, 0], 71 | [0, fy, cy, 0], 72 | [0, 0, 1, 0]], dtype=torch.float32), 73 | "height": height, 74 | "width": width, 75 | } 76 | else: 77 | return { 78 | "K": np.array([[fx, 0, cx, 0], 79 | [0, fy, cy, 0], 80 | [0, 0, 1, 0]]), 81 | "height": height, 82 | "width": width, 83 | } 84 | 85 | 86 | def cam_crop(cam: dict, 87 | height: int, 88 | width: int, 89 | col: int, 90 | row: int): 91 | fx = _fx(cam) 92 | fy = _fy(cam) 93 | cx = _cx(cam) - col 94 | cy = _cy(cam) - row 95 | 96 | if _is_torch(cam): 97 | return { 98 | "K": torch.tensor([[fx, 0, cx, 0], 99 | [0, fy, cy, 0], 100 | [0, 0, 1, 0]], dtype=torch.float32), 101 | "height": height, 102 | "width": width, 103 | } 104 | else: 105 | return { 106 | "K": np.array([[fx, 0, cx, 0], 107 | [0, fy, cy, 0], 108 | [0, 0, 1, 0]]), 109 | "height": height, 110 | "width": width, 111 | } 112 | 113 | 114 | def load_extr_and_intr_camera(camera_path): 115 | cameras = np.load(camera_path) 116 | camCv2world = torch.from_numpy(cameras["world_mat_0"]).to(dtype=torch.float32) 117 | K = torch.from_numpy(cameras["camera_mat_0"]).to(dtype=torch.float32) 118 | return camCv2world, K 119 | 120 | 121 | def _principal_point(K): 122 | # See https://github.com/alicevision/AliceVision/blob/d7a737f5d8b4ae32ca5f389c8266c49c4e733894/src/aliceVision/camera/Pinhole.hpp#L74 123 | return K[:2, 2] 124 | 125 | 126 | def _focal(K): 127 | # See https://github.com/alicevision/AliceVision/blob/d7a737f5d8b4ae32ca5f389c8266c49c4e733894/src/aliceVision/camera/Pinhole.hpp#L73 128 | return K[0,0] 129 | 130 | 131 | # Solve a single variable nonlinear equation 132 | # Find p' such that disto_func(p') = r2 approximately holds 133 | def _bisection_radius_solve(r2, disto_func): 134 | eps = 1e-8 135 | 136 | # Guess plausible upper and lower bound 137 | lb, ub = r2, r2 138 | while disto_func(lb) > r2: 139 | lb /= 1.05 140 | while disto_func(ub) < r2: 141 | ub *= 1.05 142 | 143 | # Bisection until accuracy is reached 144 | while eps < (ub - lb): 145 | m = (lb + ub) / 2 146 | if disto_func(m) > r2: 147 | ub = m 148 | else: 149 | lb = m 150 | 151 | return (lb + ub) / 2 152 | 153 | 154 | def _remove_distortion(p, disto_func): 155 | # See https://github.com/alicevision/AliceVision/blob/d7a737f5d8b4ae32ca5f389c8266c49c4e733894/src/aliceVision/camera/PinholeRadial.hpp#L167 156 | r2 = p[:,0]*p[:,0] + p[:,1]*p[:,1] 157 | for i in range(p.shape[0]): 158 | if r2 == 0: 159 | radius = 1 160 | else: 161 | radius = np.sqrt(_bisection_radius_solve(r2[i], disto_func) / r2) 162 | p[i] *= radius 163 | return p 164 | 165 | 166 | # Vectorized version of _bisection_radius_solve 167 | def _bisection_radius_solve_v2(r2, disto_func, radius_one_mask): 168 | eps = 1e-8 169 | 170 | f = lambda ps: disto_func(ps) - r2 171 | 172 | # Guess plausible upper and lower bound 173 | lb, ub = np.array(r2), np.array(r2) 174 | while True: 175 | cond = f(lb) > 0 176 | cond[radius_one_mask] = False 177 | if not np.any(cond): 178 | break 179 | lb[cond] /= 1.05 180 | 181 | while True: 182 | cond = f(ub) < 0 183 | cond[radius_one_mask] = False 184 | if not np.any(cond): 185 | break 186 | ub[cond] *= 1.05 187 | 188 | # Bisection until accuracy is reached for every entry 189 | while True: 190 | cond = eps < (ub - lb) 191 | cond[radius_one_mask] = False 192 | if not np.any(cond): 193 | break 194 | 195 | m = (lb + ub) / 2 196 | cond2 = f(m) > 0 197 | 198 | mask_ub = np.logical_and(cond, cond2) 199 | ub[mask_ub] = m[mask_ub] 200 | mask_lb = np.logical_and(cond, cond2 == False) 201 | lb[mask_lb] = m[mask_lb] 202 | 203 | return (lb + ub) / 2 204 | 205 | 206 | # Vectorized version of _remove_distortion 207 | def _remove_distortion_v2(p, disto_func): 208 | # See https://github.com/alicevision/AliceVision/blob/d7a737f5d8b4ae32ca5f389c8266c49c4e733894/src/aliceVision/camera/PinholeRadial.hpp#L167 209 | r2 = p[:,0]*p[:,0] + p[:,1]*p[:,1] 210 | radius_one_mask = r2 == 0 211 | 212 | radius = np.sqrt(_bisection_radius_solve_v2(r2, disto_func, radius_one_mask) / r2) 213 | radius[radius_one_mask] = 1 214 | 215 | return p * radius[..., None] 216 | 217 | 218 | def undistort_pixels_meshroom_radial_k3(p_2d, K, distortion): 219 | # The pixels are distorted. 220 | # Undistortion => cam2ima( remove_disto(ima2cam(p)) ) 221 | # See https://github.com/alicevision/AliceVision/blob/d7a737f5d8b4ae32ca5f389c8266c49c4e733894/src/aliceVision/camera/PinholeRadial.hpp#L179 222 | 223 | # See https://github.com/alicevision/AliceVision/blob/d7a737f5d8b4ae32ca5f389c8266c49c4e733894/src/aliceVision/camera/Pinhole.hpp#L84 224 | # cam2ima = focal() * p + principal_point() 225 | # ima2cam = ( p - principal_point() ) / focal() 226 | focal = _focal(K) 227 | principal_point = _principal_point(K) 228 | # Transform a point from the camera plane to the image plane 229 | cam2ima = lambda p: focal * p + principal_point 230 | # Transform a point from the image plane to the camera plane 231 | ima2cam = lambda p: (p - principal_point) / focal 232 | 233 | k1 = distortion[0] 234 | k2 = distortion[1] 235 | k3 = distortion[2] 236 | square = lambda x: x*x 237 | disto_func = lambda x: x * square(1 + x * (k1 + x * (k2 + x * k3))) # x == r2 238 | 239 | return cam2ima(_remove_distortion_v2(ima2cam(p_2d), disto_func)) 240 | 241 | 242 | # Supported distortion types 243 | class DistortionTypes: 244 | MESHROOM_RADIAL_K3 = "meshroom_radial_k3" 245 | -------------------------------------------------------------------------------- /config.py: -------------------------------------------------------------------------------- 1 | import os 2 | import torch 3 | import torch.nn.functional as F 4 | import yaml 5 | from shutil import copyfile 6 | 7 | from model import make_model 8 | from renderer import Renderer 9 | from mesh import load_first_k_eigenfunctions, load_mesh 10 | from ray_dataloader import create_ray_dataloader 11 | 12 | 13 | def _pretty_print_config(config, path): 14 | print("----------------------------------------------------------------") 15 | print(f"Loaded Config from {path}") 16 | print("================================================================") 17 | print(yaml.dump(config, default_flow_style=False)) 18 | print("================================================================\n") 19 | 20 | 21 | def _copy_config_file_into_out_dir(config, config_path): 22 | os.makedirs(config["training"]["out_dir"], exist_ok=True) 23 | copyfile(config_path, os.path.join(config["training"]["out_dir"], "config.yaml")) 24 | 25 | 26 | def load_config_file(path, allow_checkpoint_loading=False): 27 | with open(path, "r") as f: 28 | config = yaml.safe_load(f) 29 | 30 | out_dir = config["training"]["out_dir"] 31 | if os.path.exists(out_dir) and not allow_checkpoint_loading: 32 | raise RuntimeError(f"out_dir '{out_dir}' exists. Exit to not overwrite old results.") 33 | 34 | _pretty_print_config(config, path) 35 | _copy_config_file_into_out_dir(config, path) 36 | return config 37 | 38 | 39 | def load_config(path): 40 | with open(path, "r") as f: 41 | config = yaml.safe_load(f) 42 | return config 43 | 44 | 45 | def get_seed(config): 46 | return config.get("seed", 0) 47 | 48 | 49 | def get_log_dir(config): 50 | if not os.path.exists(config["training"]["out_dir"]): 51 | os.makedirs(config["training"]["out_dir"]) 52 | log_dir = os.path.join(config["training"]["out_dir"], "logs") 53 | return log_dir 54 | 55 | 56 | def get_data(config, device, num_workers_per_data_loader=6): 57 | mesh = load_mesh(config["data"]["mesh_path"]) 58 | data = { 59 | "train": create_ray_dataloader(config["data"]["preproc_data_path_train"], 60 | config["data"]["eigenfunctions_path"], 61 | config["model"].get("k"), 62 | config["model"].get("feature_strategy", "efuncs"), 63 | mesh, 64 | config["data"].get("rescale_strategy", "standard"), 65 | config["data"].get("embed_strategy"), 66 | config["data"].get("eigenvalues_path"), 67 | config["training"]["batch_size"], 68 | shuffle=True, 69 | drop_last=config["data"].get("train_drop_last", True), 70 | device=device), 71 | "val": create_ray_dataloader(config["data"]["preproc_data_path_eval"], 72 | config["data"]["eigenfunctions_path"], 73 | config["model"].get("k"), 74 | config["model"].get("feature_strategy", "efuncs"), 75 | mesh, 76 | config["data"].get("rescale_strategy", "standard"), 77 | config["data"].get("embed_strategy"), 78 | config["data"].get("eigenvalues_path"), 79 | config["training"]["batch_size"], 80 | shuffle=False, 81 | drop_last=False, 82 | device=device), 83 | } 84 | 85 | if hasattr(config["data"], "preproc_data_path_test"): 86 | data["test"] = create_ray_dataloader(config["data"]["preproc_data_path_test"], 87 | config["data"]["eigenfunctions_path"], 88 | config["model"].get("k"), 89 | config["model"].get("feature_strategy", "efuncs"), 90 | mesh, 91 | config["data"].get("rescale_strategy", "standard"), 92 | config["data"].get("embed_strategy"), 93 | config["data"].get("eigenvalues_path"), 94 | config["training"]["batch_size"], 95 | shuffle=False, 96 | drop_last=False, 97 | device=device) 98 | 99 | return data 100 | 101 | 102 | def get_model_and_optim(config, mesh, device): 103 | model = make_model(config["model"], mesh=mesh) 104 | # Note: Moving the model to GPU should always be done BEFORE constructing the optimizer. 105 | # See https://pytorch.org/docs/master/optim.html#torch.optim.Optimizer.zero_grad 106 | model = model.to(device) 107 | 108 | optim = torch.optim.Adam(model.parameters(), lr=config["training"]["lr"]) 109 | 110 | return model, optim 111 | 112 | 113 | def get_loss_fn(config): 114 | loss_type = config["training"]["loss_type"] 115 | if loss_type == "L2": 116 | return F.mse_loss 117 | if loss_type == "L1": 118 | return F.l1_loss 119 | if loss_type == "cauchy": 120 | return lambda rgb_pred, rgb_gt: ((20 / 255) * (20 / 255) * torch.log( 121 | 1 + (rgb_pred - rgb_gt) ** 2 / ((20 / 255) * (20 / 255)))).mean() 122 | raise RuntimeError(f"Unknown loss function: {loss_type}. Please use either 'L1', 'L2' or 'cauchy'") 123 | 124 | 125 | def get_renderer(config, model, mesh, device): 126 | feature_strategy = config["model"].get("feature_strategy", "efuncs") 127 | if feature_strategy == "efuncs": 128 | E = load_first_k_eigenfunctions(config["data"]["eigenfunctions_path"], 129 | config["model"]["k"], 130 | rescale_strategy=config["data"].get("rescale_strategy", "standard"), 131 | embed_strategy=config["data"].get("embed_strategy"), 132 | eigenvalues_path=config["data"].get("eigenvalues_path")) 133 | return Renderer(model, mesh, eigenfunctions=E, H=config["data"]["img_height"], 134 | W=config["data"]["img_width"], device=device) 135 | elif feature_strategy in ("ff", "rff", "xyz"): 136 | return Renderer(model, mesh, feature_strategy=feature_strategy, 137 | H=config["data"]["img_height"], W=config["data"]["img_width"], device=device) 138 | else: 139 | raise ValueError(f"Unknown feature strategy: {feature_strategy}") 140 | -------------------------------------------------------------------------------- /configs/discretization_agnostic/cat_cloud100k.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_cloud100000/train 4 | preproc_data_path_eval: data/preprocessed/cat_cloud100000/val 5 | preproc_data_path_test: data/preprocessed/cat_cloud100000/test 6 | eigenfunctions_path: data/preprocessed/cat_cloud100000/eigenfunctions_pc_vert_robust_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/discretizations/cat/cloud_100000.ply 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/discretization/cat_cloud100k 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/discretization_agnostic/cat_cloud10k.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_cloud10000/train 4 | preproc_data_path_eval: data/preprocessed/cat_cloud10000/val 5 | preproc_data_path_test: data/preprocessed/cat_cloud10000/test 6 | eigenfunctions_path: data/preprocessed/cat_cloud10000/eigenfunctions_pc_vert_robust_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/discretizations/cat/cloud_10000.ply 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/discretization/cat_cloud10k 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/discretization_agnostic/cat_dense.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_dense/train 4 | preproc_data_path_eval: data/preprocessed/cat_dense/val 5 | preproc_data_path_test: data/preprocessed/cat_dense/test 6 | eigenfunctions_path: data/preprocessed/cat_dense/eigenfunctions_robust_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/discretizations/cat/dens.ply 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/discretization/cat_dense 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/discretization_agnostic/cat_iso.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_isotropic/train 4 | preproc_data_path_eval: data/preprocessed/cat_isotropic/val 5 | preproc_data_path_test: data/preprocessed/cat_isotropic/test 6 | eigenfunctions_path: data/preprocessed/cat_isotropic/eigenfunctions_robust_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/discretizations/cat/iso.ply 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/discretizations/cat_iso 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/discretization_agnostic/cat_qes.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_qes/train 4 | preproc_data_path_eval: data/preprocessed/cat_qes/val 5 | preproc_data_path_test: data/preprocessed/cat_qes/test 6 | eigenfunctions_path: data/preprocessed/cat_qes/eigenfunctions_robust_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/discretizations/cat/qes.ply 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/discretizations/cat_qes 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/discretization_agnostic/human_cloud400k.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_cloud400000/train 4 | preproc_data_path_eval: data/preprocessed/human_cloud400000/val 5 | preproc_data_path_test: data/preprocessed/human_cloud400000/test 6 | eigenfunctions_path: data/preprocessed/human_cloud400000/eigenfunctions_pc_vert_robust_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/discretizations/human/cloud_400000.ply 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/human_dataset_v2_tiny/human_vis000 13 | - data/human_dataset_v2_tiny/human_vis001 14 | - data/human_dataset_v2_tiny/human_vis002 15 | - data/human_dataset_v2_tiny/human_vis003 16 | - data/human_dataset_v2_tiny/human_vis004 17 | eval_render_img_names: 18 | - human_vis000 19 | - human_vis001 20 | - human_vis002 21 | - human_vis003 22 | - human_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/discretizations/human_cloud400k 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/discretization_agnostic/human_cloud40k.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_cloud40000/train 4 | preproc_data_path_eval: data/preprocessed/human_cloud40000/val 5 | preproc_data_path_test: data/preprocessed/human_cloud40000/test 6 | eigenfunctions_path: data/preprocessed/human_cloud40000/eigenfunctions_pc_vert_robust_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/discretizations/human/cloud_40000.ply 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/human_dataset_v2_tiny/human_vis000 13 | - data/human_dataset_v2_tiny/human_vis001 14 | - data/human_dataset_v2_tiny/human_vis002 15 | - data/human_dataset_v2_tiny/human_vis003 16 | - data/human_dataset_v2_tiny/human_vis004 17 | eval_render_img_names: 18 | - human_vis000 19 | - human_vis001 20 | - human_vis002 21 | - human_vis003 22 | - human_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/discretizations/human_cloud40k 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/discretization_agnostic/human_dense.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_dense/train 4 | preproc_data_path_eval: data/preprocessed/human_dense/val 5 | preproc_data_path_test: data/preprocessed/human_dense/test 6 | eigenfunctions_path: data/preprocessed/human_dense/eigenfunctions_robust_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/discretizations/human/dens.ply 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/human_dataset_v2_tiny/human_vis000 13 | - data/human_dataset_v2_tiny/human_vis001 14 | - data/human_dataset_v2_tiny/human_vis002 15 | - data/human_dataset_v2_tiny/human_vis003 16 | - data/human_dataset_v2_tiny/human_vis004 17 | eval_render_img_names: 18 | - human_vis000 19 | - human_vis001 20 | - human_vis002 21 | - human_vis003 22 | - human_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/discretizations/human_dense 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/discretization_agnostic/human_iso.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_isotropic/train 4 | preproc_data_path_eval: data/preprocessed/human_isotropic/val 5 | preproc_data_path_test: data/preprocessed/human_isotropic/test 6 | eigenfunctions_path: data/preprocessed/human_isotropic/eigenfunctions_robust_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/discretizations/human/iso.ply 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/human_dataset_v2_tiny/human_vis000 13 | - data/human_dataset_v2_tiny/human_vis001 14 | - data/human_dataset_v2_tiny/human_vis002 15 | - data/human_dataset_v2_tiny/human_vis003 16 | - data/human_dataset_v2_tiny/human_vis004 17 | eval_render_img_names: 18 | - human_vis000 19 | - human_vis001 20 | - human_vis002 21 | - human_vis003 22 | - human_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/discretizations/human_iso 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/discretization_agnostic/human_qes.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_qes/train 4 | preproc_data_path_eval: data/preprocessed/human_qes/val 5 | preproc_data_path_test: data/preprocessed/human_qes/test 6 | eigenfunctions_path: data/preprocessed/human_qes/eigenfunctions_robust_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/discretizations/human/qes.ply 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/human_dataset_v2_tiny/human_vis000 13 | - data/human_dataset_v2_tiny/human_vis001 14 | - data/human_dataset_v2_tiny/human_vis002 15 | - data/human_dataset_v2_tiny/human_vis003 16 | - data/human_dataset_v2_tiny/human_vis004 17 | eval_render_img_names: 18 | - human_vis000 19 | - human_vis001 20 | - human_vis002 21 | - human_vis003 22 | - human_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/discretizations/human_qes 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 10 # 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/shape_descriptors/cat_efuncs.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/cat_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/cat_dataset_v2_tiny/test 6 | eigenfunctions_path: data/preprocessed/cat_efuncs/eigenfunctions_cotan_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/cat_rescaled_rotated/12221_Cat_v1_l3.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | k: 352 25 | num_layers: 6 26 | mlp_hidden_dim: 128 27 | skip_layer_idx: 3 28 | training: 29 | out_dir: out/shape_descriptors/cat_efuncs 30 | batch_size: 4096 31 | lr: 0.0001 32 | loss_type: L1 33 | render_every: 5 34 | print_every: 5 35 | epochs: 1000 36 | checkpoint_every: 5 37 | -------------------------------------------------------------------------------- /configs/shape_descriptors/cat_hks.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/cat_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/cat_dataset_v2_tiny/test 6 | eigenfunctions_path: data/shape_descriptors/cat_hks_352.npy 7 | mesh_path: data/cat_rescaled_rotated/12221_Cat_v1_l3.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | k: 352 25 | num_layers: 6 26 | mlp_hidden_dim: 128 27 | skip_layer_idx: 3 28 | training: 29 | out_dir: out/shape_descriptors/cat_hks 30 | batch_size: 4096 31 | lr: 0.0001 32 | loss_type: L1 33 | render_every: 5 34 | print_every: 5 35 | epochs: 1000 36 | checkpoint_every: 5 37 | -------------------------------------------------------------------------------- /configs/shape_descriptors/cat_shot.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/cat_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/cat_dataset_v2_tiny/test 6 | eigenfunctions_path: data/shape_descriptors/cat_shot.npy 7 | mesh_path: data/cat_rescaled_rotated/12221_Cat_v1_l3.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | k: 352 25 | num_layers: 6 26 | mlp_hidden_dim: 128 27 | skip_layer_idx: 3 28 | training: 29 | out_dir: out/shape_descriptors/cat_shot 30 | batch_size: 4096 31 | lr: 0.0001 32 | loss_type: L1 33 | render_every: 5 34 | print_every: 5 35 | epochs: 1000 36 | checkpoint_every: 5 37 | -------------------------------------------------------------------------------- /configs/shape_descriptors/human_efuncs.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/human_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/human_dataset_v2_tiny/test 6 | eigenfunctions_path: data/preprocessed/human_efuncs/eigenfunctions_cotan_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/human/RUST_3d_Low1.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/human_dataset_v2_tiny/human_vis000 13 | - data/human_dataset_v2_tiny/human_vis001 14 | - data/human_dataset_v2_tiny/human_vis002 15 | - data/human_dataset_v2_tiny/human_vis003 16 | - data/human_dataset_v2_tiny/human_vis004 17 | eval_render_img_names: 18 | - human_vis000 19 | - human_vis001 20 | - human_vis002 21 | - human_vis003 22 | - human_vis004 23 | model: 24 | k: 352 25 | num_layers: 6 26 | mlp_hidden_dim: 128 27 | skip_layer_idx: 3 28 | training: 29 | out_dir: out/shape_descriptors/human_efuncs 30 | batch_size: 4096 31 | lr: 0.0001 32 | loss_type: L1 33 | render_every: 5 34 | print_every: 5 35 | epochs: 1000 36 | checkpoint_every: 5 37 | -------------------------------------------------------------------------------- /configs/shape_descriptors/human_hks.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/human_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/human_dataset_v2_tiny/test 6 | eigenfunctions_path: data/shape_descriptors/human_hks_352.npy 7 | mesh_path: data/human/RUST_3d_Low1.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/human_dataset_v2_tiny/human_vis000 13 | - data/human_dataset_v2_tiny/human_vis001 14 | - data/human_dataset_v2_tiny/human_vis002 15 | - data/human_dataset_v2_tiny/human_vis003 16 | - data/human_dataset_v2_tiny/human_vis004 17 | eval_render_img_names: 18 | - human_vis000 19 | - human_vis001 20 | - human_vis002 21 | - human_vis003 22 | - human_vis004 23 | model: 24 | k: 352 25 | num_layers: 6 26 | mlp_hidden_dim: 128 27 | skip_layer_idx: 3 28 | training: 29 | out_dir: out/shape_descriptors/human_hks 30 | batch_size: 4096 31 | lr: 0.0001 32 | loss_type: L1 33 | render_every: 5 34 | print_every: 5 35 | epochs: 1000 36 | checkpoint_every: 5 37 | -------------------------------------------------------------------------------- /configs/shape_descriptors/human_shot.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/human_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/human_dataset_v2_tiny/test 6 | eigenfunctions_path: data/shape_descriptors/human_shot.npy 7 | mesh_path: data/human/RUST_3d_Low1.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/human_dataset_v2_tiny/human_vis000 13 | - data/human_dataset_v2_tiny/human_vis001 14 | - data/human_dataset_v2_tiny/human_vis002 15 | - data/human_dataset_v2_tiny/human_vis003 16 | - data/human_dataset_v2_tiny/human_vis004 17 | eval_render_img_names: 18 | - human_vis000 19 | - human_vis001 20 | - human_vis002 21 | - human_vis003 22 | - human_vis004 23 | model: 24 | k: 352 25 | num_layers: 6 26 | mlp_hidden_dim: 128 27 | skip_layer_idx: 3 28 | training: 29 | out_dir: out/shape_descriptors/human_shot 30 | batch_size: 4096 31 | lr: 0.0001 32 | loss_type: L1 33 | render_every: 5 34 | print_every: 5 35 | epochs: 1000 36 | checkpoint_every: 5 37 | -------------------------------------------------------------------------------- /configs/texture_reconstruction/intrinsic_cat.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/cat_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/cat_dataset_v2_tiny/test 6 | eigenfunctions_path: data/preprocessed/cat_efuncs/eigenfunctions_cotan_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/cat_rescaled_rotated/12221_Cat_v1_l3.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/texture_recon/intrinsic_cat 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 50 # 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/texture_reconstruction/intrinsic_human.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/human_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/human_dataset_v2_tiny/test 6 | eigenfunctions_path: data/preprocessed/human_efuncs/eigenfunctions_cotan_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/human/RUST_3d_Low1.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/human_dataset_v2_tiny/human_vis000 13 | - data/human_dataset_v2_tiny/human_vis001 14 | - data/human_dataset_v2_tiny/human_vis002 15 | - data/human_dataset_v2_tiny/human_vis003 16 | - data/human_dataset_v2_tiny/human_vis004 17 | eval_render_img_names: 18 | - human_vis000 19 | - human_vis001 20 | - human_vis002 21 | - human_vis003 22 | - human_vis004 23 | model: 24 | # Selected Efuncs: 0-255, 1793-2303, 3840-4095 25 | k: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095] 26 | num_layers: 6 27 | mlp_hidden_dim: 128 28 | skip_layer_idx: 3 29 | batchnorm: False 30 | training: 31 | out_dir: out/texture_recon/intrinsic_human 32 | batch_size: 4096 33 | lr: 0.0001 34 | loss_type: L1 35 | render_every: 5 36 | print_every: 1 37 | epochs: 1000 38 | checkpoint_every: 25 39 | -------------------------------------------------------------------------------- /configs/texture_reconstruction/neutex_cat.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/cat_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/cat_dataset_v2_tiny/test 6 | eigenfunctions_path: data/preprocessed/cat_efuncs/eigenfunctions_cotan_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/cat_rescaled_rotated/12221_Cat_v1_l3.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | type: neutex 25 | feature_strategy: xyz 26 | pretrained_path: neutex/pretrained/cat_rescaled_rotated_pretrained_neutex_mapping.pt 27 | # Linspace with embedding size for positional encoding => 170 * 2 * 3 + 3 = 1023 28 | k: 170 29 | use_logspace: False 30 | max_freq: 6 31 | training: 32 | out_dir: out/texture_recon/neutex_cat 33 | batch_size: 4096 34 | lr: 0.0001 35 | loss_type: L1 36 | render_every: 5 37 | print_every: 1 38 | epochs: 1000 39 | checkpoint_every: 5 40 | -------------------------------------------------------------------------------- /configs/texture_reconstruction/neutex_human.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/human_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/human_dataset_v2_tiny/test 6 | eigenfunctions_path: data/preprocessed/human_efuncs/eigenfunctions_cotan_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/human/RUST_3d_Low1.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/human_dataset_v2_tiny/human_vis000 13 | - data/human_dataset_v2_tiny/human_vis001 14 | - data/human_dataset_v2_tiny/human_vis002 15 | - data/human_dataset_v2_tiny/human_vis003 16 | - data/human_dataset_v2_tiny/human_vis004 17 | eval_render_img_names: 18 | - human_vis000 19 | - human_vis001 20 | - human_vis002 21 | - human_vis003 22 | - human_vis004 23 | model: 24 | type: neutex 25 | feature_strategy: xyz 26 | pretrained_path: neutex/pretrained/human_pretrained_neutex_mapping.pt 27 | # Linspace with embedding size for positional encoding => 170 * 2 * 3 + 3 = 1023 28 | k: 170 29 | use_logspace: False 30 | max_freq: 6 31 | training: 32 | out_dir: out/texture_recon/neutex_human 33 | batch_size: 4096 34 | lr: 0.0001 35 | loss_type: L1 36 | render_every: 5 37 | print_every: 1 38 | epochs: 1000 39 | checkpoint_every: 5 40 | -------------------------------------------------------------------------------- /configs/texture_reconstruction/tf_rff_cat.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/cat_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/cat_dataset_v2_tiny/test 6 | eigenfunctions_path: data/preprocessed/cat_efuncs/eigenfunctions_cotan_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/cat_rescaled_rotated/12221_Cat_v1_l3.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | # EXTRINSIC - RFF 25 | feature_strategy: rff 26 | # NOTE: in_dim = 510 * 2 (sin, cos) + 3 (include_input) = 1023 27 | embed_include_input: True 28 | k: 510 29 | embed_std: 8 30 | num_layers: 6 31 | mlp_hidden_dim: 128 32 | skip_layer_idx: 3 33 | batchnorm: False 34 | training: 35 | out_dir: out/texture_recon/tf_rff_cat 36 | batch_size: 4096 37 | lr: 0.0001 38 | loss_type: L1 39 | render_every: 5 40 | print_every: 1 41 | epochs: 1000 42 | checkpoint_every: 25 43 | -------------------------------------------------------------------------------- /configs/texture_reconstruction/tf_rff_human.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/human_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/human_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/human_dataset_v2_tiny/test 6 | eigenfunctions_path: data/preprocessed/human_efuncs/eigenfunctions_cotan_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/human/RUST_3d_Low1.obj 8 | use_tensor_dataset: False 9 | img_height: 512 10 | img_width: 512 11 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 12 | eval_render_input_paths: 13 | - data/human_dataset_v2_tiny/human_vis000 14 | - data/human_dataset_v2_tiny/human_vis001 15 | - data/human_dataset_v2_tiny/human_vis002 16 | - data/human_dataset_v2_tiny/human_vis003 17 | - data/human_dataset_v2_tiny/human_vis004 18 | eval_render_img_names: 19 | - human_vis000 20 | - human_vis001 21 | - human_vis002 22 | - human_vis003 23 | - human_vis004 24 | model: 25 | # EXTRINSIC - RFF 26 | feature_strategy: rff 27 | # NOTE: in_dim = 510 * 2 (sin, cos) + 3 (include_input) = 1023 28 | embed_include_input: True 29 | k: 510 30 | embed_std: 8 31 | 32 | num_layers: 6 33 | mlp_hidden_dim: 128 34 | skip_layer_idx: 3 35 | batchnorm: False 36 | training: 37 | out_dir: out/texture_recon/tf_rff_human 38 | batch_size: 4096 39 | lr: 0.0001 40 | loss_type: L1 41 | render_every: 5 42 | print_every: 1 43 | epochs: 1000 44 | checkpoint_every: 25 45 | -------------------------------------------------------------------------------- /configs/texture_transfer/cat_orig.yaml: -------------------------------------------------------------------------------- 1 | seed: 0 2 | data: 3 | preproc_data_path_train: data/preprocessed/cat_dataset_v2_tiny/train 4 | preproc_data_path_eval: data/preprocessed/cat_dataset_v2_tiny/val 5 | preproc_data_path_test: data/preprocessed/cat_dataset_v2_tiny/test 6 | eigenfunctions_path: data/texture_transfer/cat_orig_eigenfunctions_cotan_kmax4096_skip_first_efuncs.npy 7 | mesh_path: data/cat_rescaled_rotated/12221_Cat_v1_l3.obj 8 | img_height: 512 9 | img_width: 512 10 | # Images which are completely rendered after the evaluation step (not part of the training dataset) 11 | eval_render_input_paths: 12 | - data/cat_dataset_v2_tiny/cat_vis000 13 | - data/cat_dataset_v2_tiny/cat_vis001 14 | - data/cat_dataset_v2_tiny/cat_vis002 15 | - data/cat_dataset_v2_tiny/cat_vis003 16 | - data/cat_dataset_v2_tiny/cat_vis004 17 | eval_render_img_names: 18 | - cat_vis000 19 | - cat_vis001 20 | - cat_vis002 21 | - cat_vis003 22 | - cat_vis004 23 | model: 24 | k: 512 25 | num_layers: 6 26 | mlp_hidden_dim: 128 27 | skip_layer_idx: 3 28 | batchnorm: False 29 | training: 30 | out_dir: out/texture_transfer_source/orig_cat 31 | batch_size: 4096 32 | lr: 0.0001 33 | loss_type: L1 34 | render_every: 5 35 | print_every: 5 36 | epochs: 500 37 | checkpoint_every: 25 38 | -------------------------------------------------------------------------------- /dataset.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | import os 4 | import imageio 5 | import json 6 | 7 | from cameras import DistortionTypes 8 | from mesh import get_k_eigenfunc_vec_vals, load_first_k_eigenfunctions 9 | from utils import load_obj_mask_as_tensor, load_cameras 10 | 11 | 12 | def load_preprocessed_data(preproc_data_path): 13 | data = {} 14 | 15 | vertex_idxs_of_hit_faces = np.load(os.path.join(preproc_data_path, "vids_of_hit_faces.npy")) 16 | data["vertex_idxs_of_hit_faces"] = torch.from_numpy(vertex_idxs_of_hit_faces).to(dtype=torch.int64) 17 | 18 | barycentric_coords = np.load(os.path.join(preproc_data_path, "barycentric_coords.npy")) 19 | data["barycentric_coords"] = torch.from_numpy(barycentric_coords).to(dtype=torch.float32) 20 | 21 | expected_rgbs = np.load(os.path.join(preproc_data_path, "expected_rgbs.npy")) 22 | data["expected_rgbs"] = torch.from_numpy(expected_rgbs).to(dtype=torch.float32) 23 | 24 | unit_ray_dirs_path = os.path.join(preproc_data_path, "unit_ray_dirs.npy") 25 | face_idxs_path = os.path.join(preproc_data_path, "face_idxs.npy") 26 | if os.path.exists(unit_ray_dirs_path) and os.path.exists(face_idxs_path): 27 | unit_ray_dirs = np.load(unit_ray_dirs_path) 28 | data["unit_ray_dirs"] = torch.from_numpy(unit_ray_dirs).to(dtype=torch.float32) 29 | 30 | face_idxs = np.load(face_idxs_path) 31 | data["face_idxs"] = torch.from_numpy(face_idxs).to(dtype=torch.int64) 32 | 33 | return data 34 | 35 | 36 | class MeshViewsPreprocessedDataset(torch.utils.data.Dataset): 37 | def __init__(self, 38 | preproc_data_path, 39 | eigenfunctions_path, 40 | k, 41 | feature_strategy="efuncs", 42 | mesh=None, 43 | rescale_strategy="standard", 44 | eigenvalues_path=None, 45 | embed_strategy=None, 46 | transforms=None): 47 | assert os.path.exists(preproc_data_path) 48 | self.feature_strategy = feature_strategy 49 | if self.feature_strategy == "efuncs": 50 | self.k = k 51 | self.E = load_first_k_eigenfunctions(eigenfunctions_path, 52 | self.k, 53 | rescale_strategy=rescale_strategy, 54 | embed_strategy=embed_strategy, 55 | eigenvalues_path=eigenvalues_path) 56 | elif self.feature_strategy in ("ff", "rff", "xyz"): 57 | assert mesh is not None 58 | self.vertices = torch.from_numpy(mesh.vertices).to(dtype=torch.float32) 59 | else: 60 | raise ValueError(f"Unknown input feature strategy: {self.feature_strategy}") 61 | 62 | data = load_preprocessed_data(preproc_data_path) 63 | self.vertex_idxs_of_hit_faces = data["vertex_idxs_of_hit_faces"] 64 | self.barycentric_coords = data["barycentric_coords"] 65 | self.expected_rgbs = data["expected_rgbs"] 66 | 67 | self.unit_ray_dirs = data.get("unit_ray_dirs") 68 | self.face_idxs = data.get("face_idxs") 69 | 70 | self.transforms = transforms 71 | 72 | def get_eigenfunctions(self): 73 | return self.E 74 | 75 | def __len__(self): 76 | return len(self.expected_rgbs) 77 | 78 | def __getitem__(self, idx): 79 | item = {} 80 | 81 | assert idx < len(self.expected_rgbs) 82 | vertex_idxs_of_hit_faces = self.vertex_idxs_of_hit_faces[idx] # 3 83 | barycentric_coords = self.barycentric_coords[idx] # 3 84 | 85 | if self.feature_strategy == "efuncs": 86 | eigenfuncs = get_k_eigenfunc_vec_vals(self.E, vertex_idxs_of_hit_faces.unsqueeze(0), barycentric_coords.unsqueeze(0)) 87 | assert eigenfuncs.dtype == torch.float32 88 | item["eigenfunctions"] = eigenfuncs.squeeze(0) 89 | elif self.feature_strategy in ("ff", "rff", "xyz"): 90 | item["xyz"] = self.vertices[vertex_idxs_of_hit_faces].T @ barycentric_coords 91 | else: 92 | raise ValueError(f"Unknown input feature strategy: {self.feature_strategy}") 93 | 94 | expected_rgbs = self.expected_rgbs[idx] 95 | assert expected_rgbs.dtype == torch.float32 96 | item["expected_rgbs"] = expected_rgbs 97 | 98 | if self.unit_ray_dirs is not None: 99 | assert self.face_idxs is not None 100 | item["unit_ray_dirs"] = self.unit_ray_dirs[idx] 101 | item["hit_face_idxs"] = self.face_idxs[idx] 102 | 103 | if self.transforms is not None: 104 | return self.transforms(item) 105 | 106 | return item 107 | 108 | 109 | class MeshViewsDataset(torch.utils.data.Dataset): 110 | def __init__(self, dataset_path, split, H=512, W=512, background="white"): 111 | self.dataset_path = dataset_path 112 | self.H = H 113 | self.W = W 114 | self.background = background 115 | with open(os.path.join(self.dataset_path, f"{split}.lst"), "r") as file_handle: 116 | self.mesh_views_list = [line[:-1] if line.endswith('\n') else line for line in file_handle.readlines()] 117 | 118 | def __len__(self): 119 | return len(self.mesh_views_list) 120 | 121 | def __getitem__(self, idx): 122 | assert idx < len(self.mesh_views_list) 123 | 124 | mesh_view_path = os.path.join(self.dataset_path, self.mesh_views_list[idx]) 125 | 126 | # Load cameras 127 | camCv2world, K = load_cameras(mesh_view_path) 128 | 129 | # Load object mask 130 | obj_mask = load_obj_mask_as_tensor(mesh_view_path) 131 | bg_mask_1d = (obj_mask == False).reshape(-1) 132 | obj_mask_1d = obj_mask.reshape(-1) 133 | 134 | # Load view image 135 | img = imageio.imread(os.path.join(mesh_view_path, "image", "000.png")) 136 | img = torch.from_numpy(img).to(dtype=torch.float32) 137 | img /= 255. 138 | img = img.reshape(-1, 3) 139 | # Ensure that background is correct and everything besides the object is set to the background color. 140 | if self.background == "white": 141 | img[bg_mask_1d] = 1.0 142 | else: 143 | assert False, "Currently only white background is supported" 144 | img = img.reshape(self.H, self.W, 3) 145 | 146 | return { 147 | "camCv2world": camCv2world, 148 | "K": K, 149 | "img": img, 150 | "obj_mask_1d": obj_mask_1d 151 | } 152 | 153 | 154 | #=== Meshroom Radial K3 155 | 156 | 157 | def load_meshroom_metadata(dataset_path, split): 158 | with open(os.path.join(dataset_path, f"{split}_data.json"), "r") as file_handle: 159 | metadata = json.load(file_handle) 160 | return metadata 161 | 162 | 163 | class MeshroomRadialK3Dataset(torch.utils.data.Dataset): 164 | def __init__(self, dataset_path, split, *, H, W): 165 | self.dataset_path = dataset_path 166 | self.H = H 167 | self.W = W 168 | self.metadata = load_meshroom_metadata(dataset_path, split) 169 | 170 | self.K = torch.from_numpy(np.array(self.metadata["K"]).astype(np.float32)) 171 | self.distortion_params = list(map(float, self.metadata["distortion_params"])) 172 | 173 | def __len__(self): 174 | return len(self.metadata["views"]) 175 | 176 | def __getitem__(self, idx): 177 | assert idx < len(self.metadata["views"]) 178 | 179 | cur_view = self.metadata["views"][idx] 180 | 181 | # Load view image 182 | img = imageio.imread(os.path.join(self.dataset_path, cur_view["view_file"])) / 255. 183 | img = torch.from_numpy(img).to(dtype=torch.float32) 184 | 185 | # Get masks 186 | obj_mask = np.load(os.path.join(self.dataset_path, cur_view["obj_mask_file"])) 187 | bg_mask = obj_mask == False 188 | 189 | # Mask out background of the image 190 | img[bg_mask] = 1. 191 | 192 | cam2world = torch.from_numpy(np.array(cur_view["cam2world"]).astype(np.float32)) 193 | cam2world = cam2world[:3] # 3x4 194 | 195 | return { 196 | "camCv2world": cam2world, 197 | "K": self.K, 198 | "distortion_params": self.distortion_params, 199 | "distortion_type": DistortionTypes.MESHROOM_RADIAL_K3, 200 | "img": img, 201 | "obj_mask_1d": obj_mask.reshape(-1) 202 | } 203 | -------------------------------------------------------------------------------- /download_data.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | mkdir data 4 | 5 | dataPoints=("cat_rescaled_rotated" "cat_dataset_v2_tiny" "cat_tri" "discretizations" "human" "human_dataset_v2_tiny" "human_tri" "shape_descriptors" "texture_transfer") 6 | 7 | for dataPoint in ${dataPoints[@]}; do 8 | wget "https://vision.in.tum.de/webshare/g/intrinsic-neural-fields/data/${dataPoint}.zip" -P data 9 | unzip "data/${dataPoint}.zip" -d data 10 | rm -rf "data/${dataPoint}.zip" 11 | done 12 | -------------------------------------------------------------------------------- /eval.py: -------------------------------------------------------------------------------- 1 | import os 2 | import argparse 3 | import numpy as np 4 | import torch 5 | import matplotlib.pyplot as plt 6 | import pickle 7 | from tqdm import tqdm 8 | import random 9 | # https://github.com/richzhang/PerceptualSimilarity#a-basic-usage 10 | import lpips 11 | 12 | from utils import load_trained_model 13 | from dataset import MeshViewsDataset, MeshroomRadialK3Dataset 14 | from config import load_config, get_seed 15 | from mesh import load_mesh, get_ray_mesh_intersector, load_first_k_eigenfunctions 16 | from evaluation_metrics import psnr, dssim 17 | from bake_texture_field import bake_texture 18 | from renderer import Renderer 19 | 20 | 21 | def parse_args(): 22 | parser = argparse.ArgumentParser() 23 | parser.add_argument("output_path", type=str) 24 | parser.add_argument("config_path", type=str) 25 | parser.add_argument("dataset_path", type=str) 26 | parser.add_argument("split", type=str) 27 | parser.add_argument("--uv_mesh_path", type=str, default=None) 28 | parser.add_argument("--background", nargs='?', type=str, default="white") 29 | args = parser.parse_args() 30 | return args 31 | 32 | 33 | def main(): 34 | args = parse_args() 35 | 36 | if args.uv_mesh_path is not None: 37 | print("Baking texture into UV-map...") 38 | bake_texture(args.output_path, args.uv_mesh_path, args.config_path) 39 | print("Done.") 40 | 41 | config = load_config(args.config_path) 42 | 43 | if torch.cuda.is_available(): 44 | device = "cuda" 45 | torch.backends.cudnn.benchmark = True 46 | else: 47 | device = "cpu" 48 | 49 | seed = get_seed(config) 50 | random.seed(seed) 51 | np.random.seed(seed) 52 | torch.manual_seed(seed) 53 | 54 | # Load dataset 55 | H, W = config["data"]["img_height"], config["data"]["img_width"] 56 | 57 | dataset_type = config["data"].get("type") 58 | if dataset_type is None: 59 | dataset = MeshViewsDataset(args.dataset_path, 60 | args.split, 61 | H=H, 62 | W=W, 63 | background=args.background) 64 | elif dataset_type == "meshroom_radial_k3": 65 | dataset = MeshroomRadialK3Dataset(args.dataset_path, 66 | args.split, 67 | H=H, 68 | W=W) 69 | else: 70 | raise NotImplementedError(f"Unknown dataset type: {dataset_type}") 71 | 72 | dataloader = torch.utils.data.DataLoader(dataset, 73 | batch_size=None, 74 | shuffle=False, 75 | drop_last=False) 76 | 77 | # Load mesh 78 | mesh = load_mesh(config["data"]["mesh_path"]) 79 | 80 | # Load features 81 | feature_strategy = config["model"].get("feature_strategy", "efuncs") 82 | 83 | if feature_strategy == "efuncs": 84 | # Load eigenfunctions 85 | features = load_first_k_eigenfunctions(config["data"]["eigenfunctions_path"], 86 | config["model"].get("k"), 87 | rescale_strategy=config["data"].get("rescale_strategy", 88 | "standard"), 89 | embed_strategy=config["data"].get("embed_strategy"), 90 | eigenvalues_path=config["data"].get("eigenvalues_path")) 91 | elif feature_strategy in ("xyz", "ff", "rff"): 92 | features = torch.from_numpy(mesh.vertices).to(dtype=torch.float32) 93 | else: 94 | raise ValueError(f"Unknown feature strategy: {feature_strategy}") 95 | 96 | # Ray-mesh intersector 97 | ray_mesh_intersector = get_ray_mesh_intersector(mesh) 98 | 99 | # Load trained model 100 | weights_path = os.path.join(config["training"]["out_dir"], "model.pt") 101 | 102 | model = load_trained_model(config["model"], 103 | weights_path, 104 | device, 105 | mesh=mesh) 106 | model = model.eval() 107 | 108 | # Process each view 109 | eval_metrics_results = {} 110 | 111 | os.makedirs(args.output_path, exist_ok=True) 112 | 113 | lpips_fn = lpips.LPIPS(net='alex') 114 | 115 | total_psnr = 0 116 | total_dssim = 0 117 | total_lpips = 0 118 | total = 0 119 | 120 | if feature_strategy == "efuncs": 121 | renderer = Renderer(model, mesh, eigenfunctions=features, 122 | feature_strategy=feature_strategy, H=H, W=W, device=device) 123 | else: 124 | assert feature_strategy in ("xyz", "ff", "rff") 125 | renderer = Renderer(model, mesh, feature_strategy=feature_strategy, H=H, W=W, device=device) 126 | 127 | with torch.no_grad(): 128 | for i, batch in enumerate(tqdm(dataloader)): 129 | view_id = f"{i:03d}" 130 | 131 | camCv2world = batch["camCv2world"] 132 | K = batch["K"] 133 | real_img = batch["img"] 134 | obj_mask_1d = batch["obj_mask_1d"] 135 | distortion_params = batch.get("distortion_params") 136 | distortion_type = batch.get("distortion_type") 137 | 138 | fake_img, hit_ray_idxs = renderer.render(camCv2world, 139 | K, 140 | eval_render=True, 141 | distortion_coeffs=distortion_params, 142 | distortion_type=distortion_type) 143 | 144 | # Use hit ray mask & (logical) obj_mask instead of object mask due to some minor errors within the 145 | # ray-mesh intersection. 146 | # The hit_ray_mask tells us which pixels hit the mesh. 147 | hit_ray_mask = torch.zeros(H * W, dtype=torch.bool) 148 | hit_ray_mask[hit_ray_idxs] = True 149 | obj_mask_1d = torch.logical_and(hit_ray_mask, obj_mask_1d) 150 | 151 | # Store an unmasked version of the rendering 152 | fake_img_raw = fake_img.clone().numpy() 153 | 154 | # Mask out background of both images 155 | bg_mask_1d = obj_mask_1d == False 156 | 157 | fake_img = fake_img.reshape(-1, 3) 158 | fake_img[bg_mask_1d] = 1. 159 | fake_img = fake_img.reshape(H, W, 3) 160 | 161 | real_img = real_img.reshape(-1, 3) 162 | real_img[bg_mask_1d] = 1. 163 | real_img = real_img.reshape(H, W, 3) 164 | 165 | lpips_input_real = real_img.permute(2, 0, 1).unsqueeze(0) 166 | lpips_input_fake = fake_img.permute(2, 0, 1).unsqueeze(0) 167 | 168 | fake_img = fake_img.numpy() 169 | real_img = real_img.numpy() 170 | 171 | # Evaluation Metrics 172 | metrics = { 173 | "psnr": psnr(fake_img, real_img, obj_mask_1d), 174 | "dssim_rescaled": dssim(fake_img, real_img) * 100, 175 | "lpips_rescaled": lpips_fn(lpips_input_fake, lpips_input_real).item() * 100, 176 | } 177 | 178 | total_psnr += metrics["psnr"] 179 | total_dssim += metrics["dssim_rescaled"] 180 | total_lpips += metrics["lpips_rescaled"] 181 | total += 1 182 | 183 | # Store rendered view and evaluation metrics 184 | eval_metrics_results[view_id] = metrics 185 | plt.imsave(os.path.join(args.output_path, f"{view_id}_fake_raw.png"), fake_img_raw) 186 | plt.imsave(os.path.join(args.output_path, f"{view_id}_fake.png"), fake_img) 187 | plt.imsave(os.path.join(args.output_path, f"{view_id}_real.png"), real_img) 188 | 189 | # Store the metrics results. 190 | with open(os.path.join(args.output_path, "evaluation_metrics.pkl"), "wb") as f: 191 | pickle.dump(eval_metrics_results, f) 192 | 193 | print(f"PSNR: {total_psnr / total}, DSSIM: {total_dssim / total}, LPIPS: {total_lpips / total}") 194 | 195 | 196 | if __name__ == "__main__": 197 | main() 198 | -------------------------------------------------------------------------------- /evaluation_metrics.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from skimage.metrics import structural_similarity 3 | 4 | 5 | def psnr(fake_img, real_img, obj_mask_1d=None): 6 | """ 7 | Peak Signal-To-Noise-Ratio 8 | 9 | Images have dimension HxWx3 and are in [0,1] 10 | 11 | Here: MAX := 1.0 12 | PSNR = 10 * log10(MAX^2 / MSE) = 20 * log10(MAX / sqrt(MSE)) 13 | """ 14 | assert fake_img.shape == real_img.shape 15 | if obj_mask_1d is not None: 16 | fake_img = fake_img.reshape(-1, 3)[obj_mask_1d] 17 | real_img = real_img.reshape(-1, 3)[obj_mask_1d] 18 | mse = np.mean((fake_img - real_img)**2) 19 | if mse == 0: 20 | return float('inf') 21 | PIXEL_MAX = 1 22 | return 20 * np.log10(PIXEL_MAX / np.sqrt(mse)) 23 | 24 | 25 | def epoch_psnr(epoch_mse): 26 | return -10 * np.log10(epoch_mse) 27 | 28 | 29 | def dssim(fake_image, real_image): 30 | """ 31 | Structural Dissimilarity based on Structural Similarity Index Metric (SSIM) 32 | """ 33 | assert fake_image.shape == real_image.shape and fake_image.shape[2] == 3 34 | return (1 - structural_similarity(fake_image, real_image, multichannel=True)) / 2 35 | -------------------------------------------------------------------------------- /layers.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | 5 | 6 | class FourierFeatEnc(nn.Module): 7 | """ 8 | Inspired by 9 | https://github.com/facebookresearch/pytorch3d/blob/fc4dd80208bbcf6f834e7e1594db0e106961fd60/pytorch3d/renderer/implicit/harmonic_embedding.py#L10 10 | """ 11 | def __init__(self, k, include_input=True, use_logspace=False, max_freq=None): 12 | super(FourierFeatEnc, self).__init__() 13 | if use_logspace: 14 | freq_bands = 2 ** torch.arange(0, k) * torch.pi 15 | else: 16 | assert max_freq is not None 17 | freq_bands = 2 ** torch.linspace(0, max_freq, steps=k+1)[:-1] * torch.pi 18 | self.register_buffer("freq_bands", freq_bands, persistent=False) 19 | self.include_input = include_input 20 | 21 | def forward(self, x): 22 | embed = (x[..., None] * self.freq_bands).view(*x.size()[:-1], -1) 23 | if self.include_input: 24 | return torch.cat((embed.cos(), embed.sin(), x), dim=-1) 25 | return torch.cat((embed.cos(), embed.sin()), dim=-1) 26 | 27 | 28 | class RandomFourierFeatEnc(nn.Module): 29 | def __init__(self, k, std=1., in_dim=3, dtype=torch.float32, include_input=True): 30 | super(RandomFourierFeatEnc, self).__init__() 31 | B = torch.randn((in_dim, k), dtype=dtype) * std 32 | self.register_buffer("B", B, persistent=True) 33 | self.include_input = include_input 34 | 35 | def forward(self, x): 36 | embed = (2 * torch.pi * x) @ self.B 37 | if self.include_input: 38 | return torch.cat((embed.cos(), embed.sin(), x), dim=-1) 39 | return torch.cat((embed.cos(), embed.sin()), dim=-1) 40 | 41 | 42 | class Sine(nn.Module): 43 | def __init__(self): 44 | super(Sine, self).__init__() 45 | 46 | def forward(self, x): 47 | return torch.sin(x) 48 | 49 | 50 | class LinearWithConcatAndActivation(nn.Module): 51 | def __init__(self, x_in_dim, y_in_dim, out_dim, batchnorm=False, activation=nn.ReLU): 52 | super(LinearWithConcatAndActivation, self).__init__() 53 | self.Lx = nn.Linear(x_in_dim, out_dim) 54 | self.Ly = nn.Linear(y_in_dim, out_dim) 55 | self.actn = activation() 56 | self.batchnorm = None 57 | if batchnorm: 58 | self.batchnorm = nn.BatchNorm1d(out_dim) 59 | 60 | def forward(self, x, y): 61 | out = self.actn(self.Lx(x) + self.Ly(y)) 62 | return out if self.batchnorm is None else self.batchnorm(out) 63 | 64 | 65 | class MLP(nn.Module): 66 | def __init__(self, 67 | in_channels, 68 | hidden_channels, 69 | out_channels, 70 | num_layers, 71 | use_bn=False, 72 | use_ln=False, 73 | dropout=0.5, 74 | activation='relu', 75 | residual=False): 76 | super(MLP, self).__init__() 77 | self.lins = nn.ModuleList() 78 | if use_bn: self.bns = nn.ModuleList() 79 | if use_ln: self.lns = nn.ModuleList() 80 | 81 | if num_layers == 1: 82 | # linear mapping 83 | self.lins.append(nn.Linear(in_channels, out_channels)) 84 | else: 85 | self.lins.append(nn.Linear(in_channels, hidden_channels)) 86 | if use_bn: self.bns.append(nn.BatchNorm1d(hidden_channels)) 87 | if use_ln: self.lns.append(nn.LayerNorm(hidden_channels)) 88 | for layer in range(num_layers-2): 89 | self.lins.append(nn.Linear(hidden_channels, hidden_channels)) 90 | if use_bn: self.bns.append(nn.BatchNorm1d(hidden_channels)) 91 | if use_ln: self.lns.append(nn.LayerNorm(hidden_channels)) 92 | self.lins.append(nn.Linear(hidden_channels, out_channels)) 93 | if activation == 'relu': 94 | self.activation = nn.ReLU() 95 | elif activation == 'gelu': 96 | self.activation = nn.GELU() 97 | elif activation == 'tanh': 98 | self.activation = nn.Tanh() 99 | else: 100 | raise ValueError('Invalid activation') 101 | self.use_bn = use_bn 102 | self.use_ln = use_ln 103 | self.dropout = dropout 104 | self.residual = residual 105 | 106 | def forward(self, x): 107 | x_prev = x 108 | for i, lin in enumerate(self.lins[:-1]): 109 | x = lin(x) 110 | x = self.activation(x) 111 | if self.use_bn: 112 | if x.ndim == 2: 113 | x = self.bns[i](x) 114 | elif x.ndim == 3: 115 | x = self.bns[i](x.transpose(2,1)).transpose(2,1) 116 | else: 117 | raise ValueError('invalid dimension of x') 118 | if self.use_ln: x = self.lns[i](x) 119 | if self.residual and x_prev.shape == x.shape: x = x + x_prev 120 | x = F.dropout(x, p=self.dropout, training=self.training) 121 | x_prev = x 122 | x = self.lins[-1](x) 123 | if self.residual and x_prev.shape == x.shape: 124 | x = x + x_prev 125 | return x 126 | -------------------------------------------------------------------------------- /neutex/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tum-vision/intrinsic-neural-fields/a4da0c832f7ae395606f72cffcd82dfef98b29e6/neutex/__init__.py -------------------------------------------------------------------------------- /neutex/atlasnet_inverse.py: -------------------------------------------------------------------------------- 1 | """ 2 | Adapted from 3 | https://github.com/fbxiang/NeuTex/blob/master/models/atlasnet/inverse.py 4 | """ 5 | import torch 6 | from torch import nn 7 | import torch.nn.functional as F 8 | from .network_utils import init_weights 9 | 10 | 11 | class MappingManifold(nn.Module): 12 | # def __init__(self, code_size, input_dim, output_dim, hidden_size=128, num_layers=2): 13 | def __init__(self, input_dim, output_dim, hidden_size=128, num_layers=2): 14 | """ 15 | template_size: input size 16 | """ 17 | super().__init__() 18 | # self.code_size = code_size 19 | self.input_dim = input_dim 20 | self.output_dim = output_dim 21 | self.hidden_neurons = hidden_size 22 | self.num_layers = num_layers 23 | 24 | # self.linear1 = nn.Linear(self.input_dim, self.code_size) 25 | # self.linear2 = nn.Linear(self.code_size, self.hidden_neurons) 26 | 27 | # We only keep linear1 since we dont have a latent vector anymore 28 | self.linear1 = nn.Linear(self.input_dim, self.hidden_neurons) 29 | 30 | init_weights(self.linear1) 31 | # init_weights(self.linear2) 32 | 33 | self.linear_list = nn.ModuleList( 34 | [ 35 | nn.Linear(self.hidden_neurons, self.hidden_neurons) 36 | for _ in range(self.num_layers) 37 | ] 38 | ) 39 | 40 | for l in self.linear_list: 41 | init_weights(l) 42 | 43 | self.last_linear = nn.Linear(self.hidden_neurons, self.output_dim) 44 | init_weights(self.last_linear) 45 | 46 | self.activation = F.relu 47 | 48 | # def forward(self, x, latent): 49 | def forward(self, x): 50 | # x = self.linear1(x) + latent[:, None] 51 | # x = self.activation(x) 52 | # x = self.activation(self.linear2(x)) 53 | x = self.activation(self.linear1(x)) 54 | for i in range(self.num_layers): 55 | x = self.activation(self.linear_list[i](x)) 56 | return self.last_linear(x) 57 | 58 | 59 | # Transforms 3D points on the manifold into uv-map points 60 | class InverseAtlasnet(nn.Module): 61 | # def __init__(self, num_primitives, code_size, primitive_type="sphere"): 62 | def __init__(self, num_primitives, primitive_type="sphere"): 63 | super().__init__() 64 | 65 | if primitive_type == 'square': 66 | self.output_dim = 2 67 | else: 68 | self.output_dim = 3 69 | 70 | self.encoders = nn.ModuleList( 71 | # Adding an additional dimension to the output_dim for calculating weights 72 | [MappingManifold(3, self.output_dim + 1) for _ in range(0, num_primitives)] 73 | # [MappingManifold(code_size, 3, self.output_dim + 1) for _ in range(0, num_primitives)] 74 | ) 75 | 76 | # def forward(self, latent_vector, points): 77 | def forward(self, points): 78 | """ 79 | Args: 80 | points: :math:`(N,*,3)` 81 | """ 82 | input_shape = points.shape 83 | 84 | points = points.view(points.shape[0], -1, 3) 85 | 86 | output = [ 87 | encoder(points) for encoder in self.encoders 88 | # encoder(points, latent_vector) for encoder in self.encoders 89 | ] # (N, *, 3)[primitives] 90 | output = torch.stack(output, dim=-2) # (N, *, primitives, 3) 91 | output = output.view(input_shape[:-1] + output.shape[-2:]) 92 | 93 | if self.output_dim == 2: 94 | uv = torch.tanh(output[..., :-1]) 95 | else: 96 | uv = F.normalize(output[..., :-1], dim=-1) 97 | 98 | # Building weights for combining the different primitives 99 | weights_logits = output[..., -1] 100 | weights = torch.softmax(weights_logits, dim=-1) 101 | return uv, weights, weights_logits 102 | -------------------------------------------------------------------------------- /neutex/atlasnet_networks.py: -------------------------------------------------------------------------------- 1 | """ 2 | Adapted from 3 | https://github.com/fbxiang/NeuTex/blob/master/models/atlasnet/networks.py 4 | """ 5 | import torch 6 | from torch import nn 7 | import torch.nn.functional as F 8 | import numpy as np 9 | from .network_utils import init_weights 10 | 11 | 12 | class Mapping2Dto3D(nn.Module): 13 | """ 14 | Modified AtlasNet core function 15 | """ 16 | 17 | def __init__( 18 | self, 19 | # code_size, 20 | input_point_dim, 21 | hidden_size=512, 22 | num_layers=2, 23 | activation="relu", 24 | ): 25 | """ 26 | template_size: input size 27 | """ 28 | super().__init__() 29 | assert activation in ["relu", "softplus"] 30 | 31 | # self.code_size = code_size 32 | self.input_size = input_point_dim 33 | self.dim_output = 3 34 | self.hidden_neurons = hidden_size 35 | self.num_layers = num_layers 36 | 37 | self.linear1 = nn.Linear(self.input_size, self.hidden_neurons) 38 | # self.linear1 = nn.Linear(self.input_size, self.code_size) 39 | # self.linear2 = nn.Linear(self.code_size, self.hidden_neurons) 40 | 41 | init_weights(self.linear1) 42 | # init_weights(self.linear2) 43 | 44 | self.linear_list = nn.ModuleList( 45 | [ 46 | nn.Linear(self.hidden_neurons, self.hidden_neurons) 47 | for _ in range(self.num_layers) 48 | ] 49 | ) 50 | 51 | for l in self.linear_list: 52 | init_weights(l) 53 | 54 | self.last_linear = nn.Linear(self.hidden_neurons, self.dim_output) 55 | init_weights(self.last_linear) 56 | 57 | if activation == "relu": 58 | self.activation = F.relu 59 | elif activation == "softplus": 60 | self.activation = F.softplus 61 | 62 | # self.diff_geom = DiffGeomProps() 63 | 64 | # def _forward(self, x, latent): 65 | def _forward(self, x): 66 | # x = self.linear1(x) + latent 67 | # x = self.activation(x) 68 | # x = self.activation(self.linear2(x)) 69 | x = self.activation(self.linear1(x)) 70 | for i in range(self.num_layers): 71 | x = self.activation(self.linear_list[i](x)) 72 | return self.last_linear(x) 73 | 74 | # def forward(self, x, latent): 75 | def forward(self, x): 76 | assert x.shape[-1] == self.input_size 77 | # self.uv_ = x 78 | # self.xyz_ = self._forward(x, latent) 79 | self.xyz_ = self._forward(x) 80 | return self.xyz_ 81 | 82 | """ 83 | def compute_normal(self, x, latent, eps=0.01): 84 | assert x.shape[-1] == self.input_size 85 | with torch.no_grad(): 86 | angles = torch.rand(x.shape[:-1], dtype=x.dtype).to(x.device) * np.pi * 2 87 | _x = torch.cos(angles) 88 | _y = torch.sin(angles) 89 | dir1 = torch.stack([_x, _y], dim=-1) 90 | dir2 = torch.stack([-_y, _x], dim=-1) 91 | 92 | if self.input_size == 2: 93 | x1 = x + dir1 * eps 94 | x2 = x + dir2 * eps 95 | 96 | elif self.input_size == 3: 97 | with torch.no_grad(): 98 | v1 = torch.zeros_like(x, dtype=x.dtype).to(x.device) 99 | v2 = torch.zeros_like(x, dtype=x.dtype).to(x.device) 100 | v1[..., 2] = 1 101 | v2[..., 0] = 1 102 | 103 | c1 = x.cross(v1) 104 | c2 = x.cross(v2) 105 | mask = (c1 * c1).sum(-1, keepdim=True) > (c2 * c2).sum(-1, keepdim=True) 106 | mask = mask.float() 107 | t1 = c1 * mask + c2 * (1 - mask) 108 | t1 = F.normalize(t1, dim=-1) 109 | t2 = x.cross(t1) 110 | d1 = t1 * dir1[..., [0]] + t2 * dir1[..., [1]] 111 | d2 = t1 * dir2[..., [0]] + t2 * dir2[..., [1]] 112 | 113 | x1 = F.normalize(x + d1 * eps, dim=-1) 114 | x2 = F.normalize(x + d2 * eps, dim=-1) 115 | 116 | p = self._forward(x, latent) 117 | p1 = self._forward(x1, latent) 118 | p2 = self._forward(x2, latent) 119 | normal = F.normalize((p1 - p).cross(p2 - p), dim=-1) 120 | 121 | return p, normal 122 | """ 123 | 124 | 125 | class SquareTemplate: 126 | def __init__(self): 127 | self.regular_num_points = 0 128 | 129 | def get_random_points(self, npoints, device): 130 | with torch.no_grad(): 131 | rand_grid = (torch.rand((npoints, 2)) * 2 - 1).to(device).float() 132 | return rand_grid 133 | 134 | def get_regular_points(self, npoints=2500, device=None): 135 | ''' 136 | Get regular points on a Square 137 | ''' 138 | assert int(npoints ** 0.5) ** 2 == npoints 139 | assert device is not None, "device needs to be provided for get_regular_points" 140 | 141 | side_length = int(npoints ** 0.5) 142 | 143 | uv = np.stack( 144 | np.meshgrid(*([np.linspace(-1, 1, side_length)] * 2), indexing="ij"), 145 | axis=-1, 146 | ).reshape((-1, 2)) 147 | 148 | points = torch.FloatTensor(uv).to(device) 149 | return points.requires_grad_() 150 | 151 | 152 | class SphereTemplate: 153 | def get_random_points(self, npoints, device): 154 | ''' 155 | Get random points on a Sphere 156 | Return Tensor of Size [x, 3, x ... x] 157 | ''' 158 | with torch.no_grad(): 159 | points = torch.randn((npoints, 3)).to(device).float() * 2 - 1 160 | points = F.normalize(points, dim=-1) 161 | return points 162 | 163 | def get_regular_points(self, npoints, device): 164 | ''' 165 | Get regular points on a Sphere 166 | Return Tensor of Size [x, 3] 167 | ''' 168 | import trimesh 169 | mesh = trimesh.creation.icosphere(6) 170 | return torch.tensor(mesh.vertices).to(device).float() 171 | 172 | 173 | # Transforms uv-map points into 3D points on the manifold 174 | class Atlasnet(nn.Module): 175 | def __init__( 176 | self, 177 | num_points_per_primitive, 178 | num_primitives, 179 | activation, 180 | primitive_type="square", 181 | ): 182 | """ 183 | Core Atlasnet module : decoder to meshes and pointclouds. 184 | This network takes an embedding in the form of a latent vector and returns a pointcloud or a mesh 185 | Author : Thibault Groueix 01.11.2019 186 | :param opt: 187 | """ 188 | super().__init__() 189 | 190 | if primitive_type == "square": 191 | self.template_class = SquareTemplate 192 | self.input_point_dim = 2 193 | elif primitive_type == "sphere": 194 | self.template_class = SphereTemplate 195 | self.input_point_dim = 3 196 | else: 197 | raise Exception("Unknown primitive type {}".format(primitive_type)) 198 | 199 | self.num_points_per_primitive = num_points_per_primitive 200 | self.num_primitives = num_primitives 201 | 202 | # Initialize templates 203 | self.template = [self.template_class() for i in range(0, num_primitives)] 204 | 205 | # Intialize deformation networks 206 | self.decoder = nn.ModuleList( 207 | [ 208 | Mapping2Dto3D(self.input_point_dim, activation=activation) 209 | for _ in range(0, num_primitives) 210 | ] 211 | ) 212 | 213 | with torch.no_grad(): 214 | self.label = torch.zeros(num_points_per_primitive * num_primitives).long() 215 | for i in range(num_primitives): 216 | self.label[ 217 | num_points_per_primitive * i : num_points_per_primitive * (i + 1) 218 | ] = i 219 | 220 | def get_label(self, device): 221 | self.label = self.label.to(device) 222 | return self.label 223 | 224 | # def forward(self, latent_vector, regular_point_count=None): 225 | def sample_uvs_and_transform_to_3d(self, regular_point_count=None, device="cuda"): 226 | """ 227 | Deform points from self.template using the embedding latent_vector 228 | :param latent_vector: an opt.bottleneck size vector encoding a 3D shape or an image. size : batch, bottleneck 229 | :return: A deformed pointcloud os size : batch, nb_prim, num_point, 3 230 | """ 231 | if regular_point_count is None: 232 | input_points = [ 233 | self.template[i].get_random_points( 234 | self.num_points_per_primitive, device, 235 | ) # num_points_per_primitive x in_dim 236 | for i in range(self.num_primitives) 237 | ] 238 | else: 239 | input_points = [ 240 | self.template[i].get_regular_points( 241 | regular_point_count, device 242 | ) 243 | for i in range(self.num_primitives) 244 | ] 245 | 246 | points2d = [ 247 | self.decoder[i]( 248 | input_points[i].unsqueeze(0) # 1 x num_points_per_primitive x in_dim 249 | ).unsqueeze(1) # 1 x num_points_per_primitive x 3 250 | # self.decoder[i]( 251 | # input_points[i].unsqueeze(0), latent_vector.unsqueeze(1) 252 | # ).unsqueeze(1) 253 | for i in range(0, self.num_primitives) 254 | ] 255 | 256 | output_points = torch.cat(points2d, dim=0) # nb_prim x num_point x 3 257 | input_points = torch.stack(input_points, dim=0) # nb_prim x num_points x in_dim 258 | 259 | return input_points, output_points.contiguous() 260 | 261 | # Mapping from UV points to 3D 262 | # def map_uv_to_3d(self, latent_vector, uvs): 263 | def forward(self, uvs): 264 | assert uvs.shape[-1] == self.input_point_dim 265 | assert uvs.shape[-2] == self.num_primitives 266 | input_shape = uvs.shape 267 | output = torch.stack( 268 | [ 269 | self.decoder[i]( 270 | uvs[..., i, :].view(input_shape[0], -1, self.input_point_dim), 271 | # latent_vector.unsqueeze(1), 272 | ) 273 | for i in range(self.num_primitives) 274 | ], 275 | dim=-2, 276 | ) 277 | return output.view(input_shape[:-1] + (3,)) 278 | 279 | """ 280 | def map_and_normal(self, latent_vector, uvs, eps=0.01): 281 | assert uvs.shape[-1] == self.input_point_dim 282 | assert uvs.shape[-2] == self.num_primitives 283 | input_shape = uvs.shape 284 | outputs = [] 285 | normals = [] 286 | for i in range(self.num_primitives): 287 | output, normal = self.decoder[i].compute_normal( 288 | uvs[..., i, :].view(input_shape[0], -1, self.input_point_dim), 289 | latent_vector.unsqueeze(1), 290 | eps=eps, 291 | ) 292 | outputs.append(output) 293 | normals.append(normal) 294 | 295 | outputs = torch.stack(outputs, dim=-2).view(input_shape[:-1] + (3,)) 296 | normals = torch.stack(normals, dim=-2).view(input_shape[:-1] + (3,)) 297 | return outputs, normals 298 | """ 299 | -------------------------------------------------------------------------------- /neutex/network_utils.py: -------------------------------------------------------------------------------- 1 | """ 2 | Adapted from 3 | https://github.com/fbxiang/NeuTex/blob/master/models/networks.py 4 | """ 5 | import torch 6 | import torch.nn as nn 7 | from torch.nn import init 8 | import functools 9 | from torch.optim import lr_scheduler 10 | import numpy as np 11 | 12 | 13 | def get_nonlinearity_layer(activation_type='PReLU'): 14 | if activation_type == 'ReLU': 15 | nonlinearity_layer = nn.ReLU(True) 16 | elif activation_type == 'SELU': 17 | nonlinearity_layer = nn.SELU(True) 18 | elif activation_type == 'LeakyReLU': 19 | nonlinearity_layer = nn.LeakyReLU(0.1, True) 20 | elif activation_type == 'PReLU': 21 | nonlinearity_layer = nn.PReLU() 22 | else: 23 | raise NotImplementedError('activation layer [{}] is not found'.format(activation_type)) 24 | return nonlinearity_layer 25 | 26 | 27 | def get_norm_layer(norm_type='instance'): 28 | if norm_type == 'batch': 29 | norm_layer = functools.partial(nn.BatchNorm2d, affine=True) 30 | elif norm_type == 'instance': 31 | norm_layer = functools.partial(nn.InstanceNorm2d, affine=False, track_running_stats=False) 32 | # norm_layer = functools.partial(nn.InstanceNorm2d, affine=True, track_running_stats=True) 33 | elif norm_type == 'group': 34 | norm_layer = functools.partial(nn.GroupNorm, num_groups=16, affine=True) 35 | elif norm_type == 'layer': 36 | norm_layer = nn.LayerNorm 37 | elif norm_type == 'none': 38 | norm_layer = None 39 | else: 40 | raise NotImplementedError('normalization layer [%s] is not found' % norm_type) 41 | return norm_layer 42 | 43 | 44 | def get_scheduler(optimizer, opt): 45 | if opt.lr_policy == 'lambda': 46 | 47 | def lambda_rule(it): 48 | lr_l = 1.0 - max(0, it - opt.niter) / float(opt.niter_decay + 1) 49 | return lr_l 50 | 51 | scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda_rule) 52 | elif opt.lr_policy == 'step': 53 | scheduler = lr_scheduler.StepLR(optimizer, step_size=opt.lr_decay_iters, gamma=0.1) 54 | elif opt.lr_policy == 'plateau': 55 | scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, 56 | mode='min', 57 | factor=0.2, 58 | threshold=0.01, 59 | patience=5) 60 | else: 61 | return NotImplementedError('learning rate policy [%s] is not implemented', opt.lr_policy) 62 | return scheduler 63 | 64 | 65 | def get_xavier_multiplier(m, gain): 66 | if isinstance(m, nn.Conv1d): 67 | ksize = m.kernel_size[0] 68 | n1 = m.in_channels 69 | n2 = m.out_channels 70 | 71 | std = gain * np.sqrt(2.0 / ((n1 + n2) * ksize)) 72 | elif isinstance(m, nn.ConvTranspose1d): 73 | ksize = m.kernel_size[0] // m.stride[0] 74 | n1 = m.in_channels 75 | n2 = m.out_channels 76 | 77 | std = gain * np.sqrt(2.0 / ((n1 + n2) * ksize)) 78 | elif isinstance(m, nn.Conv2d): 79 | ksize = m.kernel_size[0] * m.kernel_size[1] 80 | n1 = m.in_channels 81 | n2 = m.out_channels 82 | 83 | std = gain * np.sqrt(2.0 / ((n1 + n2) * ksize)) 84 | elif isinstance(m, nn.ConvTranspose2d): 85 | ksize = m.kernel_size[0] * m.kernel_size[1] // m.stride[0] // m.stride[1] 86 | n1 = m.in_channels 87 | n2 = m.out_channels 88 | 89 | std = gain * np.sqrt(2.0 / ((n1 + n2) * ksize)) 90 | elif isinstance(m, nn.Conv3d): 91 | ksize = m.kernel_size[0] * m.kernel_size[1] * m.kernel_size[2] 92 | n1 = m.in_channels 93 | n2 = m.out_channels 94 | 95 | std = gain * np.sqrt(2.0 / ((n1 + n2) * ksize)) 96 | elif isinstance(m, nn.ConvTranspose3d): 97 | ksize = m.kernel_size[0] * m.kernel_size[1] * m.kernel_size[2] // m.stride[0] // m.stride[ 98 | 1] // m.stride[2] 99 | n1 = m.in_channels 100 | n2 = m.out_channels 101 | 102 | std = gain * np.sqrt(2.0 / ((n1 + n2) * ksize)) 103 | elif isinstance(m, nn.Linear): 104 | n1 = m.in_features 105 | n2 = m.out_features 106 | 107 | std = gain * np.sqrt(2.0 / (n1 + n2)) 108 | else: 109 | return None 110 | 111 | return std 112 | 113 | 114 | def xavier_uniform_(m, gain): 115 | std = get_xavier_multiplier(m, gain) 116 | m.weight.data.uniform_(-std * np.sqrt(3.0), std * np.sqrt(3.0)) 117 | 118 | 119 | def init_weights(net, init_type='xavier_uniform', gain=1): 120 | def init_func(m): 121 | classname = m.__class__.__name__ 122 | if hasattr(m, 'weight') and (classname.find('Conv') != -1 or classname.find('Linear') != -1): 123 | if init_type == 'xavier_uniform': 124 | xavier_uniform_(m, gain) 125 | elif init_type == 'normal': 126 | init.normal_(m.weight.data, 0.0, gain) 127 | elif init_type == 'xavier': 128 | init.xavier_normal_(m.weight.data, gain=gain) 129 | elif init_type == 'kaiming': 130 | init.kaiming_normal_(m.weight.data, a=0, mode='fan_in') 131 | elif init_type == 'orthogonal': 132 | init.orthogonal_(m.weight.data, gain=gain) 133 | else: 134 | raise NotImplementedError('initialization method [{}] is not implemented'.format(init_type)) 135 | if hasattr(m, 'bias') and m.bias is not None: 136 | init.constant_(m.bias.data, 0.0) 137 | elif classname.find('BatchNorm2d') != -1: 138 | init.normal_(m.weight.data, 1.0, gain) 139 | init.constant_(m.bias.data, 0.0) 140 | 141 | # if classname == 'ConvTranspose2d': 142 | # m.weight.data[:, :, 0::2, 1::2] = m.weight.data[:, :, 0::2, 0::2] 143 | # m.weight.data[:, :, 1::2, 0::2] = m.weight.data[:, :, 0::2, 0::2] 144 | # m.weight.data[:, :, 1::2, 1::2] = m.weight.data[:, :, 0::2, 0::2] 145 | # elif classname == 'ConvTranspose3d': 146 | # m.weight.data[:, :, 0::2, 0::2, 1::2] = m.weight.data[:, :, 0::2, 0::2, 0::2] 147 | # m.weight.data[:, :, 0::2, 1::2, 0::2] = m.weight.data[:, :, 0::2, 0::2, 0::2] 148 | # m.weight.data[:, :, 0::2, 1::2, 1::2] = m.weight.data[:, :, 0::2, 0::2, 0::2] 149 | # m.weight.data[:, :, 1::2, 0::2, 0::2] = m.weight.data[:, :, 0::2, 0::2, 0::2] 150 | # m.weight.data[:, :, 1::2, 0::2, 1::2] = m.weight.data[:, :, 0::2, 0::2, 0::2] 151 | # m.weight.data[:, :, 1::2, 1::2, 0::2] = m.weight.data[:, :, 0::2, 0::2, 0::2] 152 | # m.weight.data[:, :, 1::2, 1::2, 1::2] = m.weight.data[:, :, 0::2, 0::2, 0::2] 153 | 154 | net.apply(init_func) 155 | 156 | 157 | def init_seq(s, init_type='xavier_uniform'): 158 | '''initialize sequential model''' 159 | for a, b in zip(s[:-1], s[1:]): 160 | if isinstance(b, nn.ReLU): 161 | init_weights(a, init_type, nn.init.calculate_gain('relu')) 162 | elif isinstance(b, nn.LeakyReLU): 163 | init_weights(a, init_type, nn.init.calculate_gain('leaky_relu', b.negative_slope)) 164 | else: 165 | init_weights(a, init_type) 166 | init_weights(s[-1]) 167 | -------------------------------------------------------------------------------- /neutex/pretrained/cat_rescaled_rotated_pretrained_neutex_mapping.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tum-vision/intrinsic-neural-fields/a4da0c832f7ae395606f72cffcd82dfef98b29e6/neutex/pretrained/cat_rescaled_rotated_pretrained_neutex_mapping.pt -------------------------------------------------------------------------------- /neutex/pretrained/human_pretrained_neutex_mapping.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tum-vision/intrinsic-neural-fields/a4da0c832f7ae395606f72cffcd82dfef98b29e6/neutex/pretrained/human_pretrained_neutex_mapping.pt -------------------------------------------------------------------------------- /preprocess_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import torch 4 | import argparse 5 | from tqdm import tqdm 6 | import imageio 7 | import json 8 | 9 | from cameras import DistortionTypes 10 | from mesh import MeshViewPreProcessor 11 | from utils import load_obj_mask_as_tensor, load_depth_as_numpy, load_cameras 12 | from dataset import load_meshroom_metadata 13 | 14 | 15 | def parse_args(): 16 | parser = argparse.ArgumentParser(description="Preprocess the dataset") 17 | parser.add_argument("out_dir", type=str, help="Path to the directory where the preprocessed data should be stored") 18 | parser.add_argument("path_to_mesh", type=str, help="Path to the mesh file") 19 | parser.add_argument("dataset_path", type=str, help="Path to the dataset which is preprocessed") 20 | parser.add_argument("split", type=str, help="Dataset split") 21 | parser.add_argument("--dataset_type", type=str, help="Dataset Type", default=None) 22 | parser.add_argument("--check_depth", action="store_true", help="Will check the depth from ray-mesh intersection with the depth map of the dataset.") 23 | args = parser.parse_args() 24 | return args 25 | 26 | 27 | def preprocess_views(mesh_view_pre_proc, mesh_views_list, dataset_path, check_depth=False): 28 | for mesh_view in tqdm(mesh_views_list): 29 | mesh_view_path = os.path.join(dataset_path, mesh_view) 30 | 31 | camCv2world, K = load_cameras(mesh_view_path) 32 | 33 | # Load depth map for building a mask 34 | obj_mask = load_obj_mask_as_tensor(mesh_view_path) 35 | 36 | # Load image 37 | img = imageio.imread(os.path.join(mesh_view_path, "image", "000.png")) 38 | img = torch.from_numpy(img).to(dtype=torch.float32) 39 | img /= 255. 40 | 41 | # Preprocess and cache the current view 42 | if check_depth: 43 | depth = load_depth_as_numpy(mesh_view_path) 44 | assert np.all(np.isfinite(depth)) 45 | assert np.all(depth > 0) 46 | assert np.all(depth <= 1e10) 47 | mesh_view_pre_proc.cache_single_view(camCv2world, K, obj_mask, img, depth_check=depth) 48 | else: 49 | mesh_view_pre_proc.cache_single_view(camCv2world, K, obj_mask, img) 50 | 51 | mesh_view_pre_proc.write_to_disk() 52 | 53 | 54 | def preprocess_meshroom_data(mesh_view_pre_proc, metadata, dataset_path): 55 | # Get H and W 56 | img = imageio.imread(os.path.join(dataset_path, metadata["views"][0]["view_file"])) 57 | H, W = img.shape[:2] 58 | 59 | K = torch.from_numpy(np.array(metadata["K"]).astype(np.float32)) 60 | distortion_params = list(map(float, metadata["distortion_params"])) 61 | distortion_type = "meshroom_radial_k3" 62 | 63 | for view_data in tqdm(metadata["views"]): 64 | # Load view image 65 | img = imageio.imread(os.path.join(dataset_path, view_data["view_file"])) / 255. 66 | img = torch.from_numpy(img).to(torch.float32) 67 | 68 | # Get background mask 69 | obj_mask = np.load(os.path.join(dataset_path, view_data["obj_mask_file"])) 70 | bg_mask = obj_mask == False 71 | 72 | # Ensure that we have a white background 73 | img[bg_mask] = 1 74 | 75 | # Get cam2world 76 | cam2world = np.array(view_data["cam2world"]).astype(np.float32) 77 | cam2world = torch.from_numpy(cam2world) 78 | cam2world = cam2world[:3] 79 | 80 | # Preprocess and cache the current view 81 | mesh_view_pre_proc.cache_single_view(cam2world, 82 | K, 83 | obj_mask, 84 | img, 85 | distortion_coeffs=distortion_params, 86 | distortion_type=DistortionTypes.MESHROOM_RADIAL_K3) 87 | 88 | mesh_view_pre_proc.write_to_disk() 89 | 90 | 91 | def preprocess_dataset(split, dataset_path, path_to_mesh, out_dir, dataset_type, check_depth): 92 | split_out_dir = os.path.join(out_dir, split) 93 | 94 | if os.path.exists(split_out_dir): 95 | raise RuntimeError(f"Error: You are trying to overwrite the following directory: {split_out_dir}") 96 | os.makedirs(split_out_dir, exist_ok=True) 97 | 98 | mesh_view_pre_proc = MeshViewPreProcessor(path_to_mesh, split_out_dir) 99 | 100 | if dataset_type is None: 101 | with open(os.path.join(dataset_path, f"{split}.lst"), "r") as file_handle: 102 | mesh_views_list = [line[:-1] if line.endswith('\n') else line for line in file_handle.readlines()] 103 | preprocess_views(mesh_view_pre_proc, mesh_views_list, dataset_path, check_depth=check_depth) 104 | elif dataset_type == "meshroom_radial_k3": 105 | metadata = load_meshroom_metadata(dataset_path, split) 106 | preprocess_meshroom_data(mesh_view_pre_proc, metadata, dataset_path) 107 | else: 108 | raise NotImplementedError(f"Unknown dataset type: {type}") 109 | 110 | 111 | def main(): 112 | args = parse_args() 113 | print("Preprocessing dataset...") 114 | preprocess_dataset(args.split, args.dataset_path, args.path_to_mesh, args.out_dir, args.dataset_type, args.check_depth) 115 | 116 | 117 | if __name__ == "__main__": 118 | main() 119 | -------------------------------------------------------------------------------- /preprocess_eigenfunctions.py: -------------------------------------------------------------------------------- 1 | import os 2 | import argparse 3 | import numpy as np 4 | 5 | from mesh import EigenfuncsProcessor 6 | 7 | 8 | def parse_args(): 9 | parser = argparse.ArgumentParser(description="Preprocess the eigenfunctions of a mesh") 10 | parser.add_argument("out_dir", type=str, help="Path to the directory where the preprocessed eigenfunctions should be stored") 11 | parser.add_argument("path_to_mesh", type=str, help="Path to the mesh file") 12 | parser.add_argument("k_max", type=int, help="Maximum k for the eigenfunctions") 13 | parser.add_argument("--laplacian_type", default="cotan", type=str, help="Laplacian type: cotan, robust, pc_vert_robust") 14 | parser.add_argument('--keep_first_efunc', default=False, action="store_true") 15 | args = parser.parse_args() 16 | return args 17 | 18 | 19 | def get_geometry_type(args): 20 | return "pointcloud" if args.laplacian_type == "pc_vert_robust" else "mesh" 21 | 22 | 23 | def main(): 24 | args = parse_args() 25 | print(f"Computing the first {args.k_max} eigenfunctions of a {get_geometry_type(args)}...") 26 | 27 | # Compute the eigenfunctions 28 | eigenfuncs_proc = EigenfuncsProcessor(args.path_to_mesh, args.k_max, args.laplacian_type, 29 | skip_first_efunc=(not args.keep_first_efunc)) 30 | 31 | os.makedirs(args.out_dir, exist_ok=True) 32 | 33 | skip_suffix = "all_efuncs" if args.keep_first_efunc else "skip_first_efuncs" 34 | # Save eigenfunctions on disk 35 | np.save(os.path.join(args.out_dir, f"eigenfunctions_{args.laplacian_type}_kmax{args.k_max}_{skip_suffix}.npy"), 36 | eigenfuncs_proc.get_eigenfunctions()) 37 | 38 | # Save eigenvalues on disk 39 | np.save(os.path.join(args.out_dir, f"eigenvalues_{args.laplacian_type}_kmax{args.k_max}_{skip_suffix}.npy"), 40 | eigenfuncs_proc.get_eigenvalues()) 41 | 42 | # Symlink of mesh 43 | mesh_dst_path = os.path.join(args.out_dir, os.path.basename(args.path_to_mesh)) 44 | if not os.path.exists(mesh_dst_path): 45 | os.symlink(src=args.path_to_mesh, dst=mesh_dst_path) 46 | 47 | print("Done.") 48 | 49 | 50 | if __name__ == "__main__": 51 | main() 52 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_cat_cloud100k.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/cat_cloud100000 5 | OUT_DATASET_DIR=data/preprocessed/cat_cloud100000 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/discretizations/cat/cloud_100000.ply $NUM_EIGENFUNCTIONS --laplacian_type pc_vert_robust 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/cloud_100000.ply $DATA_DIR/cat_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/cloud_100000.ply $DATA_DIR/cat_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/cloud_100000.ply $DATA_DIR/cat_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_cat_cloud10k.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/cat_cloud10000 5 | OUT_DATASET_DIR=data/preprocessed/cat_cloud10000 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/discretizations/cat/cloud_10000.ply $NUM_EIGENFUNCTIONS --laplacian_type pc_vert_robust 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/cloud_10000.ply $DATA_DIR/cat_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/cloud_10000.ply $DATA_DIR/cat_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/cloud_10000.ply $DATA_DIR/cat_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_cat_dataset.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/cat_efuncs 5 | OUT_DATASET_DIR=data/preprocessed/cat_dataset_v2_tiny 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/cat_rescaled_rotated/12221_Cat_v1_l3.obj $NUM_EIGENFUNCTIONS 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/cat_rescaled_rotated/12221_Cat_v1_l3.obj $DATA_DIR/cat_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/cat_rescaled_rotated/12221_Cat_v1_l3.obj $DATA_DIR/cat_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/cat_rescaled_rotated/12221_Cat_v1_l3.obj $DATA_DIR/cat_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_cat_dense.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/cat_dense 5 | OUT_DATASET_DIR=data/preprocessed/cat_dense 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/discretizations/cat/cloud_100000.ply $NUM_EIGENFUNCTIONS --laplacian_type robust 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/cloud_100000.ply $DATA_DIR/cat_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/cloud_100000.ply $DATA_DIR/cat_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/cloud_100000.ply $DATA_DIR/cat_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_cat_iso.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/cat_isotropic 5 | OUT_DATASET_DIR=data/preprocessed/cat_isotropic 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/discretizations/cat/cloud_100000.ply $NUM_EIGENFUNCTIONS --laplacian_type robust 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/iso.ply $DATA_DIR/cat_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/iso.ply $DATA_DIR/cat_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/iso.ply $DATA_DIR/cat_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_cat_qes.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/cat_qes 5 | OUT_DATASET_DIR=data/preprocessed/cat_qes 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/discretizations/cat/cloud_100000.ply $NUM_EIGENFUNCTIONS --laplacian_type robust 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/qes.ply $DATA_DIR/cat_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/qes.ply $DATA_DIR/cat_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/cat/qes.ply $DATA_DIR/cat_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_human_cloud400k.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/human_cloud400000 5 | OUT_DATASET_DIR=data/preprocessed/human_cloud400000 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/discretizations/human/cloud_400000.ply $NUM_EIGENFUNCTIONS --laplacian_type pc_vert_robust 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/cloud_400000.ply $DATA_DIR/human_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/cloud_400000.ply $DATA_DIR/human_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/cloud_400000.ply $DATA_DIR/human_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_human_cloud40k.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/human_cloud40000 5 | OUT_DATASET_DIR=data/preprocessed/human_cloud40000 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/discretizations/human/cloud_40000.ply $NUM_EIGENFUNCTIONS --laplacian_type pc_vert_robust 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/cloud_40000.ply $DATA_DIR/human_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/cloud_40000.ply $DATA_DIR/human_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/cloud_40000.ply $DATA_DIR/human_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_human_dataset.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/human_efuncs 5 | OUT_DATASET_DIR=data/preprocessed/human_dataset_v2_tiny 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/human/RUST_3d_Low1.obj $NUM_EIGENFUNCTIONS 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/human/RUST_3d_Low1.obj $DATA_DIR/human_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/human/RUST_3d_Low1.obj $DATA_DIR/human_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/human/RUST_3d_Low1.obj $DATA_DIR/human_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_human_dense.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/human_dense 5 | OUT_DATASET_DIR=data/preprocessed/human_dense 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunction 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/discretizations/human/dens.ply $NUM_EIGENFUNCTIONS --laplacian_type robust 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/dens.ply $DATA_DIR/human_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/dens.ply $DATA_DIR/human_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/dens.ply $DATA_DIR/human_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_human_iso.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/human_isotropic 5 | OUT_DATASET_DIR=data/preprocessed/human_isotropic 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/discretizations/human/iso.ply $NUM_EIGENFUNCTIONS --laplacian_type robust 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/iso.ply $DATA_DIR/human_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/iso.ply $DATA_DIR/human_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/iso.ply $DATA_DIR/human_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /preprocessing_scripts/preprocess_human_qes.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | DATA_DIR=data 4 | OUT_EFUNCS_DIR=data/preprocessed/human_qes 5 | OUT_DATASET_DIR=data/preprocessed/human_qes 6 | 7 | NUM_EIGENFUNCTIONS=4096 8 | 9 | # Preprocess eigenfunctions 10 | python preprocess_eigenfunctions.py $OUT_EFUNCS_DIR $DATA_DIR/discretizations/human/qes.ply $NUM_EIGENFUNCTIONS --laplacian_type robust 11 | 12 | # Preprocess views 13 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/qes.ply $DATA_DIR/human_dataset_v2_tiny train 14 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/qes.ply $DATA_DIR/human_dataset_v2_tiny val 15 | python preprocess_dataset.py $OUT_DATASET_DIR $DATA_DIR/discretizations/human/qes.ply $DATA_DIR/human_dataset_v2_tiny test 16 | -------------------------------------------------------------------------------- /ray_dataloader.py: -------------------------------------------------------------------------------- 1 | import torch 2 | 3 | from mesh import get_k_eigenfunc_vec_vals, load_first_k_eigenfunctions 4 | from dataset import load_preprocessed_data 5 | 6 | 7 | def create_ray_dataloader(preproc_data_path, 8 | eigenfunctions_path, 9 | k, 10 | feature_strategy, 11 | mesh, 12 | rescale_strategy, 13 | eigenvalues_path, 14 | embed_strategy, 15 | batch_size, 16 | shuffle, 17 | drop_last, 18 | device="cuda"): 19 | # LOAD DATA 20 | 21 | # Get input features 22 | if feature_strategy == "efuncs": 23 | features = load_first_k_eigenfunctions(eigenfunctions_path, 24 | k, 25 | rescale_strategy=rescale_strategy, 26 | embed_strategy=embed_strategy, 27 | eigenvalues_path=eigenvalues_path) 28 | elif feature_strategy in ("ff", "rff", "xyz"): 29 | assert mesh is not None 30 | features = torch.from_numpy(mesh.vertices).to(dtype=torch.float32) 31 | else: 32 | raise ValueError(f"Unknown input feature strategy: {feature_strategy}") 33 | 34 | # Get ray-mesh intersection information 35 | data = load_preprocessed_data(preproc_data_path) 36 | vertex_idxs_of_hit_faces = data["vertex_idxs_of_hit_faces"] 37 | barycentric_coords = data["barycentric_coords"] 38 | expected_rgbs = data["expected_rgbs"] 39 | 40 | # Get view dependence information 41 | unit_ray_dirs = data.get("unit_ray_dirs") 42 | face_idxs = data.get("face_idxs") 43 | 44 | return RayDataLoader(features, 45 | feature_strategy, 46 | vertex_idxs_of_hit_faces, 47 | barycentric_coords, 48 | expected_rgbs, 49 | unit_ray_dirs, 50 | face_idxs, 51 | batch_size, 52 | shuffle, 53 | drop_last, 54 | device=device) 55 | 56 | 57 | class RayDataLoader: 58 | def __init__(self, 59 | features, 60 | feature_strategy, 61 | vertex_idxs_of_hit_faces, 62 | barycentric_coords, 63 | expected_rgbs, 64 | unit_ray_dirs, 65 | face_idxs, 66 | batch_size, 67 | shuffle, 68 | drop_last, 69 | device="cuda"): 70 | self.device = device 71 | 72 | # INITIALIZE DATA AND MOVE TO DEVICE 73 | self.features = features.to(self.device) 74 | self.feature_strategy = feature_strategy 75 | self.vertex_idxs_of_hit_faces = vertex_idxs_of_hit_faces.to(self.device) 76 | self.barycentric_coords = barycentric_coords.to(self.device) 77 | self.expected_rgbs = expected_rgbs.to(self.device) 78 | self.unit_ray_dirs = unit_ray_dirs 79 | self.face_idxs = face_idxs 80 | if self.unit_ray_dirs is not None: 81 | assert self.face_idxs is not None 82 | self.unit_ray_dirs = self.unit_ray_dirs.to(self.device) 83 | self.face_idxs = self.face_idxs.to(self.device) 84 | 85 | # DATALOADING SPECIFICS 86 | 87 | self.shuffle = shuffle 88 | self.drop_last = drop_last 89 | 90 | self.B = batch_size 91 | self.N = self.vertex_idxs_of_hit_faces.shape[0] 92 | if self.drop_last: 93 | self.num_batches = self.N // self.B 94 | else: 95 | self.num_batches = (self.N + self.B - 1) // self.B 96 | 97 | self.i = 0 98 | self.idxs = torch.arange(self.N, device=self.device) 99 | 100 | def __len__(self): 101 | return self.num_batches 102 | 103 | def __iter__(self): 104 | if self.shuffle: 105 | self.idxs = torch.randperm(self.N, device=self.device) 106 | self.i = 0 107 | return self 108 | 109 | def _get_next_batch_idxs(self): 110 | low = self.i * self.B 111 | high = min((self.i + 1) * self.B, self.N) 112 | self.i += 1 113 | return self.idxs[low:high] 114 | 115 | def __next__(self): 116 | if self.i >= self.num_batches: 117 | raise StopIteration 118 | 119 | batch_idxs = self._get_next_batch_idxs() 120 | 121 | batch = { 122 | "expected_rgbs": self.expected_rgbs[batch_idxs], 123 | } 124 | 125 | vertex_idxs_of_hit_faces_local = self.vertex_idxs_of_hit_faces[batch_idxs] # B x 3 126 | barycentric_coords = self.barycentric_coords[batch_idxs] # B x 3 127 | 128 | if self.feature_strategy == "efuncs": 129 | efuncs_local = get_k_eigenfunc_vec_vals(self.features, 130 | vertex_idxs_of_hit_faces_local, 131 | barycentric_coords) 132 | assert efuncs_local.dtype == torch.float32 133 | batch["eigenfunctions"] = efuncs_local 134 | elif self.feature_strategy in ("ff", "rff", "xyz"): 135 | features_local = self.features[vertex_idxs_of_hit_faces_local] # B x 3 x 3 136 | batch["xyz"] = torch.bmm(barycentric_coords.unsqueeze(1), features_local).squeeze(1) # B x 3 137 | else: 138 | raise ValueError(f"Unknown input feature strategy: {self.feature_strategy}") 139 | 140 | if self.unit_ray_dirs is not None: 141 | assert self.face_idxs is not None 142 | batch["unit_ray_dirs"] = self.unit_ray_dirs[batch_idxs] 143 | batch["hit_face_idxs"] = self.face_idxs[batch_idxs] 144 | 145 | return batch 146 | 147 | 148 | # TESTING 149 | if __name__ == "__main__": 150 | device = "cuda" 151 | 152 | vertex_idxs_of_hit_faces = torch.tensor([[0, 1, 2], 153 | [1, 2, 3], 154 | [7, 8, 9], 155 | [5, 6, 7], 156 | [3, 4, 5] 157 | ]) 158 | barycentric = torch.tensor([[1, 0, 0], 159 | [1, 0, 0], 160 | [1, 0, 0], 161 | [1, 0, 0], 162 | [1, 0, 0]], dtype=torch.float32) 163 | expected_rgbs = torch.ones((5, 3), dtype=torch.float32) 164 | batch_size = 2 165 | 166 | # Test Intrinsic Loading 167 | k = 5 168 | efuncs = torch.rand((10, k)) 169 | intrinsic_dataloader = RayDataLoader(efuncs, 170 | "efuncs", 171 | vertex_idxs_of_hit_faces, 172 | barycentric, 173 | expected_rgbs, 174 | None, 175 | None, 176 | batch_size, 177 | False, 178 | True, 179 | device=device) 180 | 181 | total_elements = 0 182 | for batch in intrinsic_dataloader: 183 | assert (batch_size, k) == batch["eigenfunctions"].shape 184 | total_elements += batch["eigenfunctions"].shape[0] 185 | assert total_elements == ((barycentric.shape[0] // batch_size) * batch_size) 186 | 187 | # Test Extrinsic Loading 188 | vertices = torch.rand((10, 3)) 189 | extrinsic_dataloader = RayDataLoader(vertices, 190 | "xyz", 191 | vertex_idxs_of_hit_faces, 192 | barycentric, 193 | expected_rgbs, 194 | None, 195 | None, 196 | batch_size, 197 | False, 198 | True, 199 | device=device) 200 | 201 | total_elements = 0 202 | for batch in extrinsic_dataloader: 203 | assert (batch_size, 3) == batch["xyz"].shape 204 | total_elements += batch["xyz"].shape[0] 205 | assert total_elements == ((barycentric.shape[0] // batch_size) * batch_size) 206 | -------------------------------------------------------------------------------- /render_video.py: -------------------------------------------------------------------------------- 1 | def isnotebook(): 2 | try: 3 | shell = get_ipython().__class__.__name__ 4 | if shell == 'ZMQInteractiveShell': 5 | return True # Jupyter notebook or qtconsole 6 | elif shell == 'TerminalInteractiveShell': 7 | return False # Terminal running IPython 8 | else: 9 | return False # Other type (?) 10 | except NameError: 11 | return False # Probably standard Python interpreter 12 | 13 | import os, sys 14 | if isnotebook() and sys.path[-1] != "../intrinsic-neural-fields": 15 | sys.path.append("../intrinsic-neural-fields") 16 | from os.path import join, exists, basename, dirname 17 | import argparse, shlex 18 | import matplotlib.pyplot as plt 19 | from fractions import Fraction 20 | 21 | import numpy as np 22 | import torch 23 | from cameras import cam_resize, load_extr_and_intr_camera, cam_crop 24 | from config import load_config 25 | from renderer import make_renderer_with_trained_model 26 | 27 | if isnotebook(): 28 | print("LOADING tqdm.notebook") 29 | from tqdm.notebook import tqdm 30 | else: 31 | print("LOADING tqdm for a python script") 32 | from tqdm import tqdm 33 | 34 | 35 | resolutions = { 36 | "2160p": (3840, 2160), 37 | "1080p": (1920, 1080), 38 | "720p": (1280, 720), 39 | } 40 | 41 | def parse_args(s=None): 42 | parser = argparse.ArgumentParser() 43 | parser.add_argument("--out_dir", type=str) 44 | parser.add_argument("--config_path", type=str) 45 | parser.add_argument("--cameras_path", type=str) 46 | parser.add_argument("--height", type=int) 47 | parser.add_argument("--width", type=int) 48 | parser.add_argument("--resolution", choices=tuple(resolutions.keys())) 49 | parser.add_argument("--turns", type=float, default=2) 50 | parser.add_argument("--duration", type=float, default=15) 51 | parser.add_argument("--fps", type=int, default=60) 52 | if s is None: 53 | # Called as script 54 | args = parser.parse_args() 55 | else: 56 | # Used in notebook 57 | args = parser.parse_args(shlex.split(s)) 58 | return args 59 | 60 | 61 | if __name__ == "__main__": 62 | args = parse_args() 63 | config = load_config(args.config_path) 64 | 65 | 66 | ### Make Renderer ### 67 | renderer = make_renderer_with_trained_model(config) 68 | def render_view(camCv2world_in, cam_intrinsic, pose_obj=np.eye(4)): 69 | pose_obj = torch.from_numpy(pose_obj).to(dtype=camCv2world_in.dtype, device=camCv2world_in.device) 70 | camCv2world = torch.eye(4, dtype=camCv2world_in.dtype, device=camCv2world_in.device) 71 | camCv2world[:3, :4] = camCv2world_in 72 | pose_render = pose_obj @ camCv2world 73 | renderer.set_height(cam_intrinsic["height"]) 74 | renderer.set_width(cam_intrinsic["width"]) 75 | return renderer.render(pose_render[:3], cam_intrinsic["K"]) 76 | 77 | 78 | ### Change Intrinsics ### 79 | camCv2world, K = load_extr_and_intr_camera(args.cameras_path) 80 | cam_orig = { 81 | "K": K, 82 | "height": config["data"]["img_height"], 83 | "width": config["data"]["img_width"], 84 | } 85 | 86 | res = resolutions[args.resolution] 87 | factor_width = Fraction(res[0], cam_orig['width']) 88 | factor_height = Fraction(res[1], cam_orig['height']) 89 | factor = min(factor_height, factor_width) 90 | 91 | new_width = cam_orig['width']*factor 92 | new_height = cam_orig['height']*factor 93 | 94 | assert int(new_height) == new_height 95 | assert int(new_width) == new_width 96 | 97 | new_width, new_height = int(new_width), int(new_height) 98 | assert (res[0] - new_width) % 2 == 0 99 | assert (res[1] - new_height) % 2 == 0 100 | pad_width = (res[0] - new_width) //2 101 | pad_height = (res[1] - new_height) //2 102 | 103 | cam_resized = cam_resize(cam_orig, height=new_height, width=new_width) 104 | cam_render = cam_crop(cam_resized, height=res[1], width=res[0], col=-pad_width, row=-pad_height) 105 | 106 | # Assert that this is a "synthetic optimal" cam 107 | assert cam_render['K'][0,0] == cam_render['K'][1,1] 108 | assert cam_render['width']*0.5-0.5 == cam_render['K'][0,2] 109 | assert cam_render['height']*0.5-0.5 == cam_render['K'][1,2] 110 | 111 | ### Loop over rotations and render ### 112 | folder_name = basename(dirname(args.config_path)) 113 | image_folder = join(args.out_dir, "images", folder_name+"_"+args.resolution) 114 | 115 | os.makedirs(image_folder, exist_ok=True) 116 | 117 | num_images = args.fps*args.duration 118 | assert num_images == int(num_images) 119 | num_images = int(num_images) 120 | angles = np.linspace(0, args.turns*360, num_images) 121 | 122 | for idx, a in enumerate(tqdm(angles)): 123 | st, ct = np.sin(np.deg2rad(a)), np.cos(np.deg2rad(a)) 124 | rot = np.array([ 125 | [ct, -st, 0], 126 | [st, ct, 0], 127 | [0, 0, 1]]) 128 | 129 | pose_obj = np.eye(4) 130 | pose_obj[:3, :3] = rot 131 | 132 | view = render_view(camCv2world, cam_render, pose_obj=pose_obj) 133 | 134 | plt.imsave(join(image_folder, f"{idx:04d}.jpg"), view) 135 | 136 | 137 | ### Make video with ffmpeg ### 138 | video_name = join(args.out_dir, folder_name+"_"+args.resolution+".mp4") 139 | os.system(f"/usr/bin/ffmpeg -y -framerate {args.fps} -pattern_type glob -i '{image_folder}/*.jpg' -c:v libx264 -crf 17 -pix_fmt yuv420p {video_name}") 140 | -------------------------------------------------------------------------------- /render_view.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import cv2 3 | 4 | from cameras import cam_resize, load_extr_and_intr_camera 5 | from config import load_config 6 | from renderer import make_renderer_with_trained_model 7 | 8 | 9 | def parse_args(): 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument("--output_path", type=str) 12 | parser.add_argument("--config_path", type=str) 13 | parser.add_argument("--cameras_path", type=str) 14 | parser.add_argument("--height", nargs="?", type=int, default=None) 15 | parser.add_argument("--width", nargs="?", type=int, default=None) 16 | args = parser.parse_args() 17 | return args 18 | 19 | 20 | def main(): 21 | args = parse_args() 22 | 23 | config = load_config(args.config_path) 24 | 25 | renderer = make_renderer_with_trained_model(config) 26 | def render_view(camCv2world, cam_intrinsic): 27 | renderer.set_height(cam_intrinsic["height"]) 28 | renderer.set_width(cam_intrinsic["width"]) 29 | return renderer.render(camCv2world, cam_intrinsic["K"]) 30 | 31 | camCv2world, K = load_extr_and_intr_camera(args.cameras_path) 32 | cam_intrinsic = { 33 | "K": K, 34 | "height": config["data"]["img_height"], 35 | "width": config["data"]["img_width"], 36 | } 37 | 38 | height = args.height if args.height is not None else config["data"]["img_height"] 39 | width = args.width if args.width is not None else config["data"]["img_width"] 40 | view = render_view(camCv2world, cam_resize(cam_intrinsic, height, width)) 41 | 42 | cv2.imwrite(args.output_path, view[..., ::-1]) 43 | 44 | 45 | if __name__ == "__main__": 46 | main() 47 | -------------------------------------------------------------------------------- /renderer.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import torch 4 | 5 | from mesh import load_first_k_eigenfunctions, ray_tracing, get_ray_mesh_intersector, ray_tracing_xyz, load_mesh 6 | from utils import to_device, load_trained_model, batchify_dict_data 7 | 8 | 9 | def make_renderer_with_trained_model(config, device="cuda"): 10 | # Load mesh 11 | mesh = load_mesh(config["data"]["mesh_path"]) 12 | 13 | # Load features 14 | feature_strategy = config["model"].get("feature_strategy", "efuncs") 15 | 16 | if feature_strategy == "efuncs": 17 | # Load eigenfunctions 18 | efuncs = load_first_k_eigenfunctions(config["data"]["eigenfunctions_path"], 19 | config["model"].get("k"), 20 | rescale_strategy=config["data"].get("rescale_strategy", 21 | "standard"), 22 | embed_strategy=config["data"].get("embed_strategy"), 23 | eigenvalues_path=config["data"].get("eigenvalues_path")) 24 | else: 25 | efuncs = None 26 | 27 | # Load trained model 28 | weights_path = os.path.join(config["training"]["out_dir"], "model.pt") 29 | model = load_trained_model(config["model"], weights_path, device, mesh=mesh) 30 | 31 | return Renderer(model, mesh, eigenfunctions=efuncs, feature_strategy=feature_strategy, 32 | device=device, H=config["data"]["img_height"], W=config["data"]["img_width"]) 33 | 34 | 35 | class Renderer: 36 | def __init__(self, model, mesh, eigenfunctions=None, feature_strategy="efuncs", background="white", device="cpu", *, H, W): 37 | self.model = model 38 | self.mesh = mesh 39 | self.ray_mesh_intersector = get_ray_mesh_intersector(self.mesh) 40 | self.feature_strategy = feature_strategy 41 | 42 | if self.feature_strategy == "efuncs": 43 | self.features = eigenfunctions 44 | elif self.feature_strategy in ("ff", "rff", "xyz"): 45 | self.features = torch.from_numpy(mesh.vertices).to(dtype=torch.float32) 46 | else: 47 | raise ValueError(f"Unknown feature strategy: {self.feature_strategy}") 48 | 49 | self.H = H 50 | self.W = W 51 | self.background = background 52 | self.device = device 53 | 54 | def set_height(self, height): 55 | self.H = height 56 | 57 | def set_width(self, width): 58 | self.W = width 59 | 60 | def apply_mesh_transform(self, transform): 61 | self.mesh.apply_transform(transform) 62 | self.ray_mesh_intersector = get_ray_mesh_intersector(self.mesh) 63 | 64 | @torch.no_grad() 65 | def render(self, camCv2world, K, obj_mask_1d=None, eval_render=False, distortion_coeffs=None, distortion_type=None): 66 | assert obj_mask_1d is None or obj_mask_1d.size()[0] == self.H*self.W 67 | 68 | self.model.eval() 69 | 70 | if self.feature_strategy == "efuncs": 71 | eigenfunction_vector_values, hit_ray_idxs, unit_ray_dirs, face_idxs = ray_tracing(self.ray_mesh_intersector, 72 | self.mesh, 73 | self.features, 74 | camCv2world, 75 | K, 76 | obj_mask_1d=obj_mask_1d, 77 | H=self.H, 78 | W=self.W, 79 | batched=True, 80 | distortion_coeffs=distortion_coeffs, 81 | distortion_type=distortion_type) 82 | assert eigenfunction_vector_values.dtype == torch.float32 83 | data = { 84 | "eigenfunctions": eigenfunction_vector_values, 85 | "unit_ray_dirs": unit_ray_dirs, 86 | "hit_face_idxs": face_idxs, 87 | } 88 | num_rays = eigenfunction_vector_values.shape[0] 89 | elif self.feature_strategy in ("ff", "rff", "xyz"): 90 | hit_points_xyz, hit_ray_idxs, unit_ray_dirs, face_idxs = ray_tracing_xyz(self.ray_mesh_intersector, 91 | self.mesh, 92 | self.features, 93 | camCv2world, 94 | K, 95 | obj_mask_1d=obj_mask_1d, 96 | H=self.H, 97 | W=self.W, 98 | batched=True, 99 | distortion_coeffs=distortion_coeffs, 100 | distortion_type=distortion_type) 101 | data = { 102 | "xyz": hit_points_xyz, 103 | "unit_ray_dirs": unit_ray_dirs, 104 | "hit_face_idxs": face_idxs 105 | } 106 | num_rays = hit_points_xyz.shape[0] 107 | else: 108 | raise ValueError(f"Unknown feature strategy: {self.feature_strategy}") 109 | 110 | assert num_rays > 0 111 | 112 | # Inference in batches to support rendering large views 113 | total_pred_rgbs = [] 114 | batch_size = 1 << 15 115 | for batch in batchify_dict_data(data, num_rays, batch_size): 116 | batch = to_device(batch, device=self.device) 117 | pred_rgbs = self.model(batch).cpu() 118 | total_pred_rgbs.append(pred_rgbs) 119 | pred_rgbs = torch.concat(total_pred_rgbs, dim=0) 120 | 121 | # We now need to bring the predicted RGB colors into the correct ordering again 122 | # since the ray-mesh intersection does not preserve ordering 123 | assert obj_mask_1d is None or obj_mask_1d.dtype == torch.bool 124 | N = self.H * self.W if obj_mask_1d is None else obj_mask_1d.sum() 125 | if self.background == "white": 126 | img = torch.ones((N, 3), device="cpu", dtype=torch.float32) 127 | else: 128 | assert self.background == "black" 129 | img = torch.zeros((N, 3), device="cpu", dtype=torch.float32) 130 | img[hit_ray_idxs] = pred_rgbs 131 | 132 | # If we only kept the object, then img does not have the correct resolution yet. 133 | # Therefore, we must upscale it one more time taking the object mask into account. 134 | if obj_mask_1d is not None: 135 | M = self.H * self.W 136 | if self.background == "white": 137 | img_unmasked = torch.ones((M, 3), device="cpu", dtype=torch.float32) 138 | else: 139 | assert self.background == "black" 140 | img_unmasked = torch.zeros((M, 3), device="cpu", dtype=torch.float32) 141 | img_unmasked[obj_mask_1d] = img 142 | img = img_unmasked 143 | 144 | if eval_render: 145 | return img.reshape(self.H, self.W, 3), hit_ray_idxs 146 | return img.reshape(self.H, self.W, 3).numpy() 147 | -------------------------------------------------------------------------------- /train.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | import argparse 4 | import random 5 | import os 6 | 7 | from config import load_config_file, get_data, get_model_and_optim, get_loss_fn, get_renderer, get_seed 8 | from trainer import Trainer 9 | from mesh import load_mesh 10 | from utils import model_summary 11 | 12 | 13 | def parse_args(): 14 | parser = argparse.ArgumentParser() 15 | parser.add_argument("config_path", type=str) 16 | parser.add_argument('--allow_checkpoint_loading', default=False, action="store_true") 17 | parser.add_argument('--data_parallel', default=False, action="store_true") 18 | args = parser.parse_args() 19 | return args 20 | 21 | 22 | def main(): 23 | args = parse_args() 24 | 25 | config = load_config_file(args.config_path, args.allow_checkpoint_loading) 26 | 27 | seed = get_seed(config) 28 | random.seed(seed) 29 | np.random.seed(seed) 30 | torch.manual_seed(seed) 31 | 32 | if torch.cuda.is_available(): 33 | device = "cuda" 34 | torch.backends.cudnn.benchmark = True 35 | else: 36 | device = "cpu" 37 | 38 | mesh = load_mesh(config["data"]["mesh_path"]) 39 | data = get_data(config, device) 40 | 41 | model, optim = get_model_and_optim(config, mesh, device) 42 | 43 | # Print model summary 44 | model_summary(model, data) 45 | 46 | if args.data_parallel: 47 | device_ids = [int(x) for x in os.environ.get("CUDA_VISIBLE_DEVICES", "").split(",")] 48 | model = torch.nn.DataParallel(model, device_ids=device_ids) 49 | 50 | loss_fn = get_loss_fn(config) 51 | renderer = get_renderer(config, model, mesh, device) 52 | 53 | # Seed again because different model architectures change seed. Make train samples consistent. 54 | # https://discuss.pytorch.org/t/shuffle-issue-in-dataloader-how-to-get-the-same-data-shuffle-results-with-fixed-seed-but-different-network/45357/9 55 | random.seed(seed) 56 | np.random.seed(seed) 57 | torch.manual_seed(seed) 58 | 59 | trainer = Trainer(model, optim, loss_fn, renderer, data, mesh, config, device) 60 | trainer.train() 61 | 62 | 63 | if __name__ == "__main__": 64 | main() 65 | -------------------------------------------------------------------------------- /training_scripts/train_cat_discretizations.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Select one of the following methods: "cloud10k", "cloud100k", "dense", "iso", "qes" 4 | method="$1" 5 | 6 | if [ "$method" = "cloud10k" ]; then 7 | echo "Selected discretization: cloud10k" 8 | CONFIG_PATH=configs/discretization_agnostic/cat_cloud10k.yaml 9 | EVAL_OUT_DIR=out/discretization/cat_cloud10k/test_eval 10 | elif [ "$method" = "cloud100k" ]; then 11 | echo "Selected discretization: cloud100k" 12 | CONFIG_PATH=configs/discretization_agnostic/cat_cloud100k.yaml 13 | EVAL_OUT_DIR=out/discretization/cat_cloud100k/test_eval 14 | elif [ "$method" = "dense" ]; then 15 | echo "Selected discretization: dense" 16 | CONFIG_PATH=configs/discretization_agnostic/cat_dense.yaml 17 | EVAL_OUT_DIR=out/discretization/cat_dense/test_eval 18 | elif [ "$method" = "iso" ]; then 19 | echo "Selected discretization: iso" 20 | CONFIG_PATH=configs/discretization_agnostic/cat_iso.yaml 21 | EVAL_OUT_DIR=out/discretization/cat_iso/test_eval 22 | elif [ "$method" = "qes" ]; then 23 | echo "Selected discretization: qes" 24 | CONFIG_PATH=configs/discretization_agnostic/cat_qes.yaml 25 | EVAL_OUT_DIR=out/discretization/cat_qes/test_eval 26 | else 27 | echo "Unknown method: $method. Must be one of the following: cloud10k, cloud100k, dense, iso, qes" 28 | exit 1 29 | fi 30 | 31 | # Perform training 32 | python train.py $CONFIG_PATH --allow_checkpoint_loading 33 | 34 | # Evaluation on test split 35 | python eval.py $EVAL_OUT_DIR $CONFIG_PATH data/cat_dataset_v2_tiny test 36 | -------------------------------------------------------------------------------- /training_scripts/train_cat_shape_descriptors.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Select one of the following shape descriptors: "efuncs", "hks", "shot" 4 | shapeDescriptor="$1" 5 | 6 | if [ "$method" = "efuncs" ]; then 7 | echo "Selected shape descriptor: Eigenfunctions" 8 | CONFIG_PATH=configs/shape_descriptors/cat_efuncs.yaml 9 | EVAL_OUT_DIR=out/shape_descriptors/cat_efuncs/test_eval 10 | elif [ "$method" = "hks" ]; then 11 | echo "Selected shape descriptor: HKS" 12 | CONFIG_PATH=configs/shape_descriptors/cat_hks.yaml 13 | EVAL_OUT_DIR=out/shape_descriptors/cat_efuncs/test_eval 14 | elif [ "$method" = "shot" ]; then 15 | echo "Selected shape descriptor: SHOT" 16 | CONFIG_PATH=configs/shape_descriptors/cat_shot.yaml 17 | EVAL_OUT_DIR=out/shape_descriptors/cat_efuncs/test_eval 18 | else 19 | echo "Unknown shape descriptor: $method. Must be one of the following: efuncs, hks, shot" 20 | exit 1 21 | fi 22 | 23 | # Perform training 24 | python train.py $CONFIG_PATH --allow_checkpoint_loading 25 | 26 | # Evaluation on test split and bake the texture 27 | python eval.py $EVAL_OUT_DIR $CONFIG_PATH data/cat_dataset_v2_tiny test --uv_mesh_path data/cat_tri/12221_Cat_v1_l3.obj 28 | -------------------------------------------------------------------------------- /training_scripts/train_cat_tex_recon.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Select one of the following methods: "tf+rff", "neutex", "intrinsic" 4 | method="$1" 5 | 6 | if [ "$method" = "intrinsic" ]; then 7 | echo "Selected method: Intrinsic" 8 | CONFIG_PATH=configs/texture_reconstruction/intrinsic_cat.yaml 9 | EVAL_OUT_DIR=out/texture_recon/intrinsic_cat/test_eval 10 | elif [ "$method" = "tf+rff" ]; then 11 | echo "Selected method: TF + RFF" 12 | CONFIG_PATH=configs/texture_reconstruction/tf_rff_cat.yaml 13 | EVAL_OUT_DIR=out/texture_recon/tf_rff_cat/test_eval 14 | elif [ "$method" = "neutex" ]; then 15 | echo "Selected method: NeuTex" 16 | CONFIG_PATH=configs/texture_reconstruction/neutex_cat.yaml 17 | EVAL_OUT_DIR=out/texture_recon/neutex_cat/test_eval 18 | else 19 | echo "Unknown method: $method. Must be one of the following: tf+rff, neutex, intrinsic" 20 | exit 1 21 | fi 22 | 23 | # Perform training 24 | python train.py $CONFIG_PATH --allow_checkpoint_loading 25 | 26 | # Evaluation on test split and bake the texture 27 | python eval.py $EVAL_OUT_DIR $CONFIG_PATH data/cat_dataset_v2_tiny test --uv_mesh_path data/cat_tri/12221_Cat_v1_l3.obj 28 | -------------------------------------------------------------------------------- /training_scripts/train_cat_texture_transfer.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | CONFIG_PATH=configs/texture_transfer/cat_orig.yaml 4 | EVAL_OUT_DIR=out/texture_transfer_source/orig_cat/test_eval 5 | 6 | # Perform training 7 | python train.py $CONFIG_PATH --allow_checkpoint_loading 8 | 9 | # Evaluation on test split 10 | python eval.py $EVAL_OUT_DIR $CONFIG_PATH data/cat_dataset_v2_tiny test 11 | -------------------------------------------------------------------------------- /training_scripts/train_human_discretizations.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Select one of the following methods: "cloud40k", "cloud400k", "dense", "iso", "qes" 4 | method="$1" 5 | 6 | if [ "$method" = "cloud40k" ]; then 7 | echo "Selected discretization: cloud40k" 8 | CONFIG_PATH=configs/discretization_agnostic/human_cloud40k.yaml 9 | EVAL_OUT_DIR=out/discretization/human_cloud40k/test_eval 10 | elif [ "$method" = "cloud400k" ]; then 11 | echo "Selected discretization: cloud400k" 12 | CONFIG_PATH=configs/discretization_agnostic/human_cloud400k.yaml 13 | EVAL_OUT_DIR=out/discretization/human_cloud400k/test_eval 14 | elif [ "$method" = "dense" ]; then 15 | echo "Selected discretization: dense" 16 | CONFIG_PATH=configs/discretization_agnostic/human_dense.yaml 17 | EVAL_OUT_DIR=out/discretization/human_dense/test_eval 18 | elif [ "$method" = "iso" ]; then 19 | echo "Selected discretization: iso" 20 | CONFIG_PATH=configs/discretization_agnostic/human_iso.yaml 21 | EVAL_OUT_DIR=out/discretization/human_iso/test_eval 22 | elif [ "$method" = "qes" ]; then 23 | echo "Selected discretization: qes" 24 | CONFIG_PATH=configs/discretization_agnostic/human_qes.yaml 25 | EVAL_OUT_DIR=out/discretization/human_qes/test_eval 26 | else 27 | echo "Unknown method: $method. Must be one of the following: cloud40k, cloud400k, dense, iso, qes" 28 | exit 1 29 | fi 30 | 31 | # Perform training 32 | python train.py $CONFIG_PATH --allow_checkpoint_loading 33 | 34 | # Evaluation on test split 35 | python eval.py $EVAL_OUT_DIR $CONFIG_PATH data/human_dataset_v2_tiny test 36 | -------------------------------------------------------------------------------- /training_scripts/train_human_shape_descriptors.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Select one of the following shape descriptors: "efuncs", "hks", "shot" 4 | shapeDescriptor="$1" 5 | 6 | if [ "$method" = "efuncs" ]; then 7 | echo "Selected shape descriptor: Eigenfunctions" 8 | CONFIG_PATH=configs/shape_descriptors/human_efuncs.yaml 9 | EVAL_OUT_DIR=out/shape_descriptors/human_efuncs/test_eval 10 | elif [ "$method" = "hks" ]; then 11 | echo "Selected shape descriptor: HKS" 12 | CONFIG_PATH=configs/shape_descriptors/human_hks.yaml 13 | EVAL_OUT_DIR=out/shape_descriptors/human_efuncs/test_eval 14 | elif [ "$method" = "shot" ]; then 15 | echo "Selected shape descriptor: SHOT" 16 | CONFIG_PATH=configs/shape_descriptors/human_shot.yaml 17 | EVAL_OUT_DIR=out/shape_descriptors/human_efuncs/test_eval 18 | else 19 | echo "Unknown shape descriptor: $method. Must be one of the following: efuncs, hks, shot" 20 | exit 1 21 | fi 22 | 23 | # Perform training 24 | python train.py $CONFIG_PATH --allow_checkpoint_loading 25 | 26 | # Evaluation on test split and bake the texture 27 | python eval.py $EVAL_OUT_DIR $CONFIG_PATH data/human_dataset_v2_tiny test --uv_mesh_path data/human_tri/RUST_3d_Low1.obj 28 | -------------------------------------------------------------------------------- /training_scripts/train_human_tex_recon.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Select one of the following methods: "tf+rff", "neutex", "intrinsic" 4 | method="$1" 5 | 6 | if [ "$method" = "intrinsic" ]; then 7 | echo "Selected method: Intrinsic" 8 | CONFIG_PATH=configs/texture_reconstruction/intrinsic_human.yaml 9 | EVAL_OUT_DIR=out/texture_recon/intrinsic_human/test_eval 10 | elif [ "$method" = "tf+rff" ]; then 11 | echo "Selected method: TF + RFF" 12 | CONFIG_PATH=configs/texture_reconstruction/tf_rff_human.yaml 13 | EVAL_OUT_DIR=out/texture_recon/tf_rff_human/test_eval 14 | elif [ "$method" = "neutex" ]; then 15 | echo "Selected method: NeuTex" 16 | CONFIG_PATH=configs/texture_reconstruction/neutex_human.yaml 17 | EVAL_OUT_DIR=out/texture_recon/neutex_human/test_eval 18 | else 19 | echo "Unknown method: $method. Must be one of the following: tf+rff, neutex, intrinsic" 20 | exit 1 21 | fi 22 | 23 | # Perform training 24 | python train.py $CONFIG_PATH --allow_checkpoint_loading 25 | 26 | # Evaluation on test split and bake the texture 27 | python eval.py $EVAL_OUT_DIR $CONFIG_PATH data/human_dataset_v2_tiny test --uv_mesh_path data/human_tri/RUST_3d_Low1.obj 28 | -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | import sys 4 | from torchinfo import summary 5 | import os 6 | import imageio 7 | 8 | from model import make_model 9 | 10 | # Make sure loading .exr works for imageio 11 | try: 12 | imageio.plugins.freeimage.download() 13 | except FileExistsError: 14 | # Ignore 15 | pass 16 | 17 | 18 | def tensor_mem_size_in_bytes(x): 19 | return sys.getsizeof(x.storage()) 20 | 21 | 22 | def load_trained_model(model_config, weights_path, device, mesh=None): 23 | model = make_model(model_config, mesh=mesh) 24 | data = torch.load(weights_path) 25 | if "model_state_dict" in data: 26 | model.load_state_dict(data["model_state_dict"]) 27 | else: 28 | model.load_state_dict(data) 29 | return model.to(device) 30 | 31 | 32 | def load_cameras(view_path): 33 | cameras = np.load(os.path.join(view_path, "depth", "cameras.npz")) 34 | camCv2world = torch.from_numpy(cameras["world_mat_0"]).to(dtype=torch.float32) 35 | K = torch.from_numpy(cameras["camera_mat_0"]).to(dtype=torch.float32) 36 | return camCv2world, K 37 | 38 | 39 | def model_summary(model, data): 40 | data_batch = next(iter(data["train"])) 41 | summary(model, input_data=[data_batch]) 42 | 43 | 44 | def load_obj_mask_as_tensor(view_path): 45 | if view_path.endswith(".npy"): 46 | return np.load(view_path) 47 | 48 | depth_path = os.path.join(view_path, "depth", "depth_0000.exr") 49 | if os.path.exists(depth_path): 50 | depth_map = imageio.imread(depth_path)[..., 0] 51 | 52 | mask_value = 1.e+10 53 | obj_mask = depth_map != mask_value 54 | else: 55 | mask_path = os.path.join(view_path, "depth", "mask.png") 56 | assert os.path.exists(mask_path), "Must have depth or mask" 57 | mask = imageio.imread(mask_path) 58 | obj_mask = mask != 0 # 0 is invalid 59 | 60 | obj_mask = torch.from_numpy(obj_mask) 61 | return obj_mask 62 | 63 | 64 | def load_depth_as_numpy(view_path): 65 | depth_path = os.path.join(view_path, "depth", "depth_0000.exr") 66 | assert os.path.exists(depth_path) 67 | depth_map = imageio.imread(depth_path)[..., 0] 68 | 69 | return depth_map 70 | 71 | 72 | def batchify_dict_data(data_dict, input_total_size, batch_size): 73 | idxs = np.arange(0, input_total_size) 74 | batch_idxs = np.split(idxs, np.arange(batch_size, input_total_size, batch_size), axis=0) 75 | 76 | batches = [] 77 | for cur_idxs in batch_idxs: 78 | data = {} 79 | for key in data_dict.keys(): 80 | data[key] = data_dict[key][cur_idxs] 81 | batches.append(data) 82 | 83 | return batches 84 | 85 | 86 | ########################################################################################## 87 | # The following is taken from: 88 | # https://github.com/tum-vision/tandem/blob/master/cva_mvsnet/utils.py 89 | ########################################################################################## 90 | 91 | 92 | # convert a function into recursive style to handle nested dict/list/tuple variables 93 | def make_recursive_func(func): 94 | def wrapper(vars, **kwargs): 95 | if isinstance(vars, list): 96 | return [wrapper(x, **kwargs) for x in vars] 97 | elif isinstance(vars, tuple): 98 | return tuple([wrapper(x, **kwargs) for x in vars]) 99 | elif isinstance(vars, dict): 100 | return {k: wrapper(v, **kwargs) for k, v in vars.items()} 101 | else: 102 | return func(vars, **kwargs) 103 | 104 | return wrapper 105 | 106 | 107 | @make_recursive_func 108 | def tensor2float(vars): 109 | if isinstance(vars, float): 110 | return vars 111 | elif isinstance(vars, torch.Tensor): 112 | return vars.data.item() 113 | else: 114 | raise NotImplementedError("invalid input type {} for tensor2float".format(type(vars))) 115 | 116 | 117 | @make_recursive_func 118 | def tensor2numpy(vars): 119 | if isinstance(vars, np.ndarray): 120 | return vars 121 | elif isinstance(vars, torch.Tensor): 122 | return vars.detach().cpu().numpy().copy() 123 | else: 124 | raise NotImplementedError("invalid input type {} for tensor2numpy".format(type(vars))) 125 | 126 | 127 | @make_recursive_func 128 | def tocuda(vars): 129 | if isinstance(vars, torch.Tensor): 130 | return vars.to(torch.device("cuda")) 131 | elif isinstance(vars, str): 132 | return vars 133 | else: 134 | raise NotImplementedError("invalid input type {} for tensor2numpy".format(type(vars))) 135 | 136 | 137 | @make_recursive_func 138 | def to_device(x, *, device): 139 | if torch.is_tensor(x): 140 | return x.to(device) 141 | elif isinstance(x, str): 142 | return x 143 | else: 144 | raise NotImplementedError(f"Invalid type for to_device: {type(x)}") 145 | 146 | ########################################################################################## 147 | --------------------------------------------------------------------------------