├── .gitignore ├── .vscode └── settings.json ├── LICENSE ├── README.md ├── camcenters.npy ├── ccv_batchscript ├── cloud_size_test └── results.txt ├── clouds ├── camcenters.npy ├── pointcloud_down16.npy ├── pointcloud_down2.npy ├── pointcloud_down32.npy ├── pointcloud_down4.npy └── pointcloud_down8.npy ├── config_cit.txt ├── config_cit_hi_res.txt ├── config_deepvoxels_greek.txt ├── config_fern.txt ├── config_lego.txt ├── download_example_data.sh ├── download_example_weights.sh ├── env2.yml ├── environment.yml ├── extract_mesh.ipynb ├── fast_results ├── imgs │ ├── FastNeRF_sparse_18x.png │ ├── FastNeRF_sparse_2x.png │ ├── FastNeRF_sparse_3x.png │ ├── FastNeRF_sparse_6x.png │ ├── FastNeRF_sparse_9x.png │ ├── GT0.png │ └── NeRF_render.png ├── results.txt ├── results2.txt └── sampling_rate_accuracy.png ├── fern_example2 ├── args.txt ├── config.txt ├── model_200000.npy └── model_fine_200000.npy ├── fern_pc_results ├── results.txt ├── results2.txt ├── samps_psnr.png └── time_psnr.png ├── hierarchical_performance.py ├── imgs └── pipeline.jpg ├── installation.md ├── load_blender.py ├── load_deepvoxels.py ├── load_llff.py ├── paper_configs ├── blender_config.txt ├── deepvoxels_config.txt └── llff_config.txt ├── point_cloud.py ├── point_cloud_results.py ├── render_demo.ipynb ├── render_demo.py ├── render_demo_fast.py ├── render_stats ├── run_env.sh ├── run_nerf.py ├── run_nerf_fast.py ├── run_nerf_helpers.py ├── run_nerf_helpers_fast.py ├── sandbox.py ├── sum.out ├── tiny_nerf.ipynb └── tiny_nerf_data.npz.1 /.gitignore: -------------------------------------------------------------------------------- 1 | **/.ipynb_checkpoints 2 | **/__pycache__ 3 | *.png 4 | *.mp4 5 | *.npy 6 | *.npz 7 | *.dae 8 | data/* 9 | logs/* -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "python.pythonPath": "/Users/trevorhouchens/opt/anaconda3/envs/nerf/bin/python" 3 | } -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 bmild 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ### Builds on the original Neural Radiance Fields (NeRF) paper. 2 | ### Documentation for the original paper is included below 3 | 4 | # NeRF: Neural Radiance Fields 5 | ### [Project Page](http://tancik.com/nerf) | [Video](https://youtu.be/JuH79E8rdKc) | [Paper](https://arxiv.org/abs/2003.08934) | [Data](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) 6 | [![Open Tiny-NeRF in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/bmild/nerf/blob/master/tiny_nerf.ipynb)
7 | Tensorflow implementation of optimizing a neural representation for a single scene and rendering new views.

8 | [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis](http://tancik.com/nerf) 9 | [Ben Mildenhall](https://people.eecs.berkeley.edu/~bmild/)\*1, 10 | [Pratul P. Srinivasan](https://people.eecs.berkeley.edu/~pratul/)\*1, 11 | [Matthew Tancik](http://tancik.com/)\*1, 12 | [Jonathan T. Barron](http://jonbarron.info/)2, 13 | [Ravi Ramamoorthi](http://cseweb.ucsd.edu/~ravir/)3, 14 | [Ren Ng](https://www2.eecs.berkeley.edu/Faculty/Homepages/yirenng.html)1
15 | 1UC Berkeley, 2Google Research, 3UC San Diego 16 | \*denotes equal contribution 17 | in ECCV 2020 (Oral Presentation, Best Paper Honorable Mention) 18 | 19 | 20 | 21 | ## TL;DR quickstart 22 | 23 | To setup a conda environment, download example training data, begin the training process, and launch Tensorboard: 24 | ``` 25 | conda env create -f environment.yml 26 | conda activate nerf 27 | bash download_example_data.sh 28 | python run_nerf.py --config config_fern.txt 29 | tensorboard --logdir=logs/summaries --port=6006 30 | ``` 31 | If everything works without errors, you can now go to `localhost:6006` in your browser and watch the "Fern" scene train. 32 | 33 | ## Setup 34 | 35 | Python 3 dependencies: 36 | 37 | * Tensorflow 1.15 38 | * matplotlib 39 | * numpy 40 | * imageio 41 | * configargparse 42 | 43 | The LLFF data loader requires ImageMagick. 44 | 45 | We provide a conda environment setup file including all of the above dependencies. Create the conda environment `nerf` by running: 46 | ``` 47 | conda env create -f environment.yml 48 | ``` 49 | 50 | You will also need the [LLFF code](http://github.com/fyusion/llff) (and COLMAP) set up to compute poses if you want to run on your own real data. 51 | 52 | ## What is a NeRF? 53 | 54 | A neural radiance field is a simple fully connected network (weights are ~5MB) trained to reproduce input views of a single scene using a rendering loss. The network directly maps from spatial location and viewing direction (5D input) to color and opacity (4D output), acting as the "volume" so we can use volume rendering to differentiably render new views. 55 | 56 | Optimizing a NeRF takes between a few hours and a day or two (depending on resolution) and only requires a single GPU. Rendering an image from an optimized NeRF takes somewhere between less than a second and ~30 seconds, again depending on resolution. 57 | 58 | 59 | ## Running code 60 | 61 | Here we show how to run our code on two example scenes. You can download the rest of the synthetic and real data used in the paper [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1). 62 | 63 | ### Optimizing a NeRF 64 | 65 | Run 66 | ``` 67 | bash download_example_data.sh 68 | ``` 69 | to get the our synthetic Lego dataset and the LLFF Fern dataset. 70 | 71 | To optimize a low-res Fern NeRF: 72 | ``` 73 | python run_nerf.py --config config_fern.txt 74 | ``` 75 | After 200k iterations (about 15 hours), you should get a video like this at `logs/fern_test/fern_test_spiral_200000_rgb.mp4`: 76 | 77 | ![ferngif](https://people.eecs.berkeley.edu/~bmild/nerf/fern_200k_256w.gif) 78 | 79 | To optimize a low-res Lego NeRF: 80 | ``` 81 | python run_nerf.py --config config_lego.txt 82 | ``` 83 | After 200k iterations, you should get a video like this: 84 | 85 | ![legogif](https://people.eecs.berkeley.edu/~bmild/nerf/lego_200k_256w.gif) 86 | 87 | ### Rendering a NeRF 88 | 89 | Run 90 | ``` 91 | bash download_example_weights.sh 92 | ``` 93 | to get a pretrained high-res NeRF for the Fern dataset. Now you can use [`render_demo.ipynb`](https://github.com/bmild/nerf/blob/master/render_demo.ipynb) to render new views. 94 | 95 | ### Replicating the paper results 96 | 97 | The example config files run at lower resolutions than the quantitative/qualitative results in the paper and video. To replicate the results from the paper, start with the config files in [`paper_configs/`](https://github.com/bmild/nerf/tree/master/paper_configs). Our synthetic Blender data and LLFF scenes are hosted [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and the DeepVoxels data is hosted by Vincent Sitzmann [here](https://drive.google.com/open?id=1lUvJWB6oFtT8EQ_NzBrXnmi25BufxRfl). 98 | 99 | ### Extracting geometry from a NeRF 100 | 101 | Check out [`extract_mesh.ipynb`](https://github.com/bmild/nerf/blob/master/extract_mesh.ipynb) for an example of running marching cubes to extract a triangle mesh from a trained NeRF network. You'll need the install the [PyMCubes](https://github.com/pmneila/PyMCubes) package for marching cubes plus the [trimesh](https://github.com/mikedh/trimesh) and [pyrender](https://github.com/mmatl/pyrender) packages if you want to render the mesh inside the notebook: 102 | ``` 103 | pip install trimesh pyrender PyMCubes 104 | ``` 105 | 106 | ## Generating poses for your own scenes 107 | 108 | ### Don't have poses? 109 | 110 | We recommend using the `imgs2poses.py` script from the [LLFF code](https://github.com/fyusion/llff). Then you can pass the base scene directory into our code using `--datadir ` along with `-dataset_type llff`. You can take a look at the `config_fern.txt` config file for example settings to use for a forward facing scene. For a spherically captured 360 scene, we recomment adding the `--no_ndc --spherify --lindisp` flags. 111 | 112 | ### Already have poses! 113 | 114 | In `run_nerf.py` and all other code, we use the same pose coordinate system as in OpenGL: the local camera coordinate system of an image is defined in a way that the X axis points to the right, the Y axis upwards, and the Z axis backwards as seen from the image. 115 | 116 | Poses are stored as 3x4 numpy arrays that represent camera-to-world transformation matrices. The other data you will need is simple pinhole camera intrinsics (`hwf = [height, width, focal length]`) and near/far scene bounds. Take a look at [our data loading code](https://github.com/bmild/nerf/blob/master/run_nerf.py#L406) to see more. 117 | 118 | ## Citation 119 | 120 | ``` 121 | @inproceedings{mildenhall2020nerf, 122 | title={NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis}, 123 | author={Ben Mildenhall and Pratul P. Srinivasan and Matthew Tancik and Jonathan T. Barron and Ravi Ramamoorthi and Ren Ng}, 124 | year={2020}, 125 | booktitle={ECCV}, 126 | } 127 | ``` 128 | -------------------------------------------------------------------------------- /camcenters.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/camcenters.npy -------------------------------------------------------------------------------- /ccv_batchscript: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Request a GPU partition node and access to 1 GPU 4 | #SBATCH -p gpu --gres=gpu:1 5 | 6 | # Request 1 CPU core 7 | #SBATCH -n 1 8 | 9 | #SBATCH --mem=8G 10 | #SBATCH -t 16:15:00 11 | #SBATCH -o sum2.out 12 | 13 | module load python/3.7.4 14 | module load anaconda/ 15 | 16 | source activate nerf 17 | 18 | python run_nerf.py --config config_cit_hi_res.txt > logs/cit_res_output.txt -------------------------------------------------------------------------------- /cloud_size_test/results.txt: -------------------------------------------------------------------------------- 1 | {"cloud_size": [12954816, 3238704, 809676, 202419, 50604, 12651], "mse": [ -------------------------------------------------------------------------------- /clouds/camcenters.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/clouds/camcenters.npy -------------------------------------------------------------------------------- /clouds/pointcloud_down16.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/clouds/pointcloud_down16.npy -------------------------------------------------------------------------------- /clouds/pointcloud_down2.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/clouds/pointcloud_down2.npy -------------------------------------------------------------------------------- /clouds/pointcloud_down32.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/clouds/pointcloud_down32.npy -------------------------------------------------------------------------------- /clouds/pointcloud_down4.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/clouds/pointcloud_down4.npy -------------------------------------------------------------------------------- /clouds/pointcloud_down8.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/clouds/pointcloud_down8.npy -------------------------------------------------------------------------------- /config_cit.txt: -------------------------------------------------------------------------------- 1 | expname = cit_test 2 | basedir = ./logs 3 | datadir = ./data/nerf_llff_data/cit 4 | dataset_type = llff 5 | 6 | factor = 8 7 | llffhold = 8 8 | 9 | N_rand = 1024 10 | N_samples = 64 11 | N_importance = 64 12 | 13 | use_viewdirs = True 14 | raw_noise_std = 1e0 15 | 16 | -------------------------------------------------------------------------------- /config_cit_hi_res.txt: -------------------------------------------------------------------------------- 1 | expname = cit_hi_res 2 | basedir = ./FastNeRF/logs 3 | datadir = ./FastNeRF/data/nerf_llff_data/cit 4 | dataset_type = llff 5 | 6 | factor = 2 7 | llffhold = 8 8 | 9 | N_rand = 1024 10 | N_samples = 64 11 | N_importance = 64 12 | 13 | use_viewdirs = True 14 | raw_noise_std = 1e0 15 | 16 | -------------------------------------------------------------------------------- /config_deepvoxels_greek.txt: -------------------------------------------------------------------------------- 1 | expname = dvox_greek_test 2 | basedir = ./logs 3 | # datadir = /your/path/to/deepvoxels/data 4 | dataset_type = deepvoxels 5 | shape = greek 6 | 7 | no_batching = True 8 | 9 | N_samples = 64 10 | N_importance = 64 11 | 12 | use_viewdirs = True 13 | 14 | white_bkgd = True 15 | 16 | N_rand = 1024 -------------------------------------------------------------------------------- /config_fern.txt: -------------------------------------------------------------------------------- 1 | expname = fern_test 2 | basedir = ./logs 3 | datadir = ./data/nerf_llff_data/fern 4 | dataset_type = llff 5 | 6 | factor = 8 7 | llffhold = 8 8 | 9 | N_rand = 1024 10 | N_samples = 64 11 | N_importance = 64 12 | 13 | use_viewdirs = True 14 | raw_noise_std = 1e0 15 | 16 | -------------------------------------------------------------------------------- /config_lego.txt: -------------------------------------------------------------------------------- 1 | expname = lego_test 2 | basedir = ./logs 3 | datadir = ./data/nerf_synthetic/lego 4 | dataset_type = blender 5 | 6 | half_res = True 7 | no_batching = True 8 | 9 | N_samples = 64 10 | N_importance = 64 11 | 12 | use_viewdirs = True 13 | 14 | white_bkgd = True 15 | 16 | N_rand = 1024 -------------------------------------------------------------------------------- /download_example_data.sh: -------------------------------------------------------------------------------- 1 | wget https://people.eecs.berkeley.edu/~bmild/nerf/tiny_nerf_data.npz --no-check-certificate 2 | mkdir -p data 3 | cd data 4 | wget https://people.eecs.berkeley.edu/~bmild/nerf/nerf_example_data.zip --no-check-certificate 5 | unzip nerf_example_data.zip 6 | cd .. 7 | -------------------------------------------------------------------------------- /download_example_weights.sh: -------------------------------------------------------------------------------- 1 | mkdir -p logs 2 | cd logs 3 | wget https://people.eecs.berkeley.edu/~bmild/nerf/fern_example_weights.zip 4 | unzip fern_example_weights.zip 5 | wget https://people.eecs.berkeley.edu/~bmild/nerf/lego_example_weights.zip 6 | unzip lego_example_weights.zip -------------------------------------------------------------------------------- /env2.yml: -------------------------------------------------------------------------------- 1 | # To run: conda env create -f environment.yml 2 | name: nerf 3 | channels: 4 | - conda-forge 5 | dependencies: 6 | - python=3.7 7 | - pip 8 | - tensorflow 9 | - numpy 10 | - matplotlib 11 | - imageio 12 | - imageio-ffmpeg 13 | - configargparse 14 | - imagemagick 15 | -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | # To run: conda env create -f environment.yml 2 | name: nerf 3 | channels: 4 | - conda-forge 5 | dependencies: 6 | - python=3.7 7 | - pip 8 | - cudatoolkit=10.0 9 | - tensorflow-gpu==1.15 10 | - numpy 11 | - matplotlib 12 | - imageio 13 | - imageio-ffmpeg 14 | - configargparse 15 | - imagemagick 16 | -------------------------------------------------------------------------------- /fast_results/imgs/FastNeRF_sparse_18x.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fast_results/imgs/FastNeRF_sparse_18x.png -------------------------------------------------------------------------------- /fast_results/imgs/FastNeRF_sparse_2x.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fast_results/imgs/FastNeRF_sparse_2x.png -------------------------------------------------------------------------------- /fast_results/imgs/FastNeRF_sparse_3x.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fast_results/imgs/FastNeRF_sparse_3x.png -------------------------------------------------------------------------------- /fast_results/imgs/FastNeRF_sparse_6x.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fast_results/imgs/FastNeRF_sparse_6x.png -------------------------------------------------------------------------------- /fast_results/imgs/FastNeRF_sparse_9x.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fast_results/imgs/FastNeRF_sparse_9x.png -------------------------------------------------------------------------------- /fast_results/imgs/GT0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fast_results/imgs/GT0.png -------------------------------------------------------------------------------- /fast_results/imgs/NeRF_render.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fast_results/imgs/NeRF_render.png -------------------------------------------------------------------------------- /fast_results/results.txt: -------------------------------------------------------------------------------- 1 | {"down_x": [1, 4, 9, 36, 81, 324], "psnr": [], "mse": [], "time": []} -------------------------------------------------------------------------------- /fast_results/results2.txt: -------------------------------------------------------------------------------- 1 | {"down_x": [1, 4, 9, 36, 81, 324], "psnr": [23.33826446533203, 22.831159591674805, 22.835399627685547, 22.83519172668457, 22.83094596862793, 22.830163955688477], "mse": [0.004636320751160383, 0.005210555624216795, 0.005205471534281969, 0.005205720663070679, 0.005210810340940952, 0.005211748648434877], "time": [16.27228832244873, 14.011771440505981, 13.990169763565063, 13.985945224761963, 13.978806018829346, 13.980828523635864]} -------------------------------------------------------------------------------- /fast_results/sampling_rate_accuracy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fast_results/sampling_rate_accuracy.png -------------------------------------------------------------------------------- /fern_example2/args.txt: -------------------------------------------------------------------------------- 1 | ISdepth = 8 2 | ISwidth = 256 3 | N_importance = 128 4 | N_rand = 4096 5 | N_samples = 64 6 | basedir = ../logs_fwd_noise 7 | bilinear_rays = False 8 | chunk = 32768 9 | config = config_fwd_f4_b4k.txt 10 | datadir = ../data/real_iconic/data2_fernvlsb/ 11 | dataset_type = llff 12 | expname = fern_std1e0 13 | exptrans = True 14 | factor = 4 15 | ft_path = None 16 | full_res = False 17 | i_embed = 0 18 | lindisp = False 19 | llffhold = 8 20 | lrate = 0.0005 21 | lrate_decay = 250 22 | mid_split = 0 23 | multires = 10 24 | multires_alt = 0 25 | multires_views = 4 26 | netdepth = 8 27 | netfine = True 28 | netwidth = 256 29 | no_batching = False 30 | no_ndc = False 31 | no_reload = False 32 | numholdout = 100 33 | numtest = 100 34 | perturb = 1.0 35 | raw_noise_std = 1.0 36 | render_only = False 37 | rgb_requery = False 38 | shape = greek 39 | sphere_flip = False 40 | spherify = False 41 | use_viewdirs = True 42 | white_bkgd = False 43 | -------------------------------------------------------------------------------- /fern_example2/config.txt: -------------------------------------------------------------------------------- 1 | datadir = ./data/nerf_llff_data/fern 2 | dataset_type = llff 3 | factor = 4 4 | 5 | no_batching = False 6 | i_embed = 0 7 | N_samples = 64 8 | N_importance = 128 9 | use_viewdirs = True 10 | lrate_decay = 250 11 | 12 | llffhold = 8 13 | 14 | N_rand = 4096 -------------------------------------------------------------------------------- /fern_example2/model_200000.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fern_example2/model_200000.npy -------------------------------------------------------------------------------- /fern_example2/model_fine_200000.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fern_example2/model_fine_200000.npy -------------------------------------------------------------------------------- /fern_pc_results/results.txt: -------------------------------------------------------------------------------- 1 | {"down_x": [1, 4, 9, 36, 81, 324], "psnr": [23.33826446533203, 22.831159591674805, 22.835399627685547, 22.83519172668457, 22.83094596862793, 22.830163955688477], "mse": [0.004636320751160383, 0.005210555624216795, 0.005205471534281969, 0.005205720663070679, 0.005210810340940952, 0.005211748648434877], "time": [16.27228832244873, 14.011771440505981, 13.990169763565063, 13.985945224761963, 13.978806018829346, 13.980828523635864]} -------------------------------------------------------------------------------- /fern_pc_results/results2.txt: -------------------------------------------------------------------------------- 1 | {"down_x": [1, 4, 9, 36, 81, 324], "psnr": [23.33826446533203, 22.831159591674805, 22.835399627685547, 22.83519172668457, 22.83094596862793, 22.830163955688477], "mse": [0.004636320751160383, 0.005210555624216795, 0.005205471534281969, 0.005205720663070679, 0.005210810340940952, 0.005211748648434877], "time": [16.27228832244873, 14.011771440505981, 13.990169763565063, 13.985945224761963, 13.978806018829346, 13.980828523635864]} -------------------------------------------------------------------------------- /fern_pc_results/samps_psnr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fern_pc_results/samps_psnr.png -------------------------------------------------------------------------------- /fern_pc_results/time_psnr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/fern_pc_results/time_psnr.png -------------------------------------------------------------------------------- /hierarchical_performance.py: -------------------------------------------------------------------------------- 1 | import os, sys 2 | # os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true' 3 | os.environ['CUDA_VISIBLE_DEVICES'] = '1' 4 | import tensorflow as tf 5 | tf.compat.v1.enable_eager_execution() 6 | 7 | import numpy as np 8 | import imageio 9 | import json 10 | import random 11 | import time 12 | import pprint 13 | 14 | import matplotlib.pyplot as plt 15 | 16 | import run_nerf_fast 17 | import run_nerf_helpers_fast 18 | import run_nerf 19 | 20 | from load_llff import load_llff_data 21 | from load_deepvoxels import load_dv_data 22 | from load_blender import load_blender_data 23 | 24 | # profiling 25 | import cProfile 26 | import pstats 27 | from pstats import SortKey 28 | 29 | # CELL 1 30 | 31 | basedir = './logs' 32 | expname = 'fern_example' 33 | # expname = 'fern_test' 34 | 35 | config = os.path.join(basedir, expname, 'config.txt') 36 | print('Args:') 37 | print(open(config, 'r').read()) 38 | parser = run_nerf_fast.config_parser() 39 | 40 | weights_name = 'model_200000.npy' 41 | # weights_name = 'model_000700.npy' 42 | args = parser.parse_args('--config {} --ft_path {}'.format(config, os.path.join(basedir, expname, weights_name))) 43 | print('loaded args') 44 | 45 | images, poses, bds, render_poses, i_test = load_llff_data(args.datadir, args.factor, 46 | recenter=True, bd_factor=.75, 47 | spherify=args.spherify) 48 | H, W, focal = poses[0,:3,-1].astype(np.float32) 49 | poses = poses[:, :3, :4] 50 | print(f"NUM IMAGES: {poses.shape[0]}") 51 | 52 | H = int(H) 53 | W = int(W) 54 | hwf = [H, W, focal] 55 | 56 | images = images.astype(np.float32) 57 | poses = poses.astype(np.float32) 58 | 59 | if args.no_ndc: 60 | near = tf.reduce_min(bds) * .9 61 | far = tf.reduce_max(bds) * 1. 62 | else: 63 | near = 0. 64 | far = 1. 65 | 66 | # get the test and train sets 67 | if not isinstance(i_test, list): 68 | i_test = [i_test] 69 | 70 | if args.llffhold > 0: 71 | print('Auto LLFF holdout,', args.llffhold) 72 | i_test = np.arange(images.shape[0])[::args.llffhold] 73 | 74 | i_train = np.array([i for i in np.arange(int(images.shape[0])) if 75 | (i not in i_test)]) 76 | 77 | down = 2 78 | 79 | gt = images[i_test[0]] 80 | gt_down = np.zeros((H//down, W//down, 3)) 81 | for i in range(gt_down.shape[0]): 82 | for j in range(gt_down.shape[1]): 83 | patch = gt[down*i:down*(i+1), (down*j):down*(j+1)] 84 | patch = np.reshape(patch, (-1,3)) 85 | gt_down[i,j] = np.mean(patch, axis=0) 86 | 87 | # Create nerf model 88 | 89 | def new_render(img_dir, fast=False,r2=128,d=3): 90 | _, render_kwargs_test, start, grad_vars, models = run_nerf_fast.create_nerf(args) 91 | 92 | bds_dict = { 93 | 'near' : tf.cast(near, tf.float32), 94 | 'far' : tf.cast(far, tf.float32), 95 | } 96 | render_kwargs_test.update(bds_dict) 97 | 98 | print('Render kwargs:') 99 | pprint.pprint(render_kwargs_test) 100 | 101 | 102 | 103 | render_kwargs_fast = {k : render_kwargs_test[k] for k in render_kwargs_test} 104 | render_kwargs_fast['N_importance'] = r2 105 | 106 | # c2w = np.eye(4)[:3,:4].astype(np.float32) # identity pose matrix 107 | c2w = poses[0] 108 | start_time = time.time() 109 | if fast: 110 | test = run_nerf_fast.render(H//down, W//down, focal/down, c2w=c2w, d_factor=d, **render_kwargs_fast) 111 | else: 112 | test = run_nerf.render(H//down, W//down, focal/down, c2w=c2w, pc=False, **render_kwargs_fast) 113 | end_time = time.time() 114 | net = end_time - start_time 115 | 116 | img = np.clip(test[0],0,1) 117 | if not fast: 118 | plt.imsave(os.path.join(img_dir, f"NeRF_render.png"), images[i_test[1]]) 119 | else: 120 | plt.imsave(os.path.join(img_dir, f"FastNeRF_sparse_{d}x.png"), images[i_test[1]]) 121 | 122 | mse = run_nerf_helpers_fast.img2mse(tf.cast(gt_down, tf.float32), img) 123 | psnr = run_nerf_helpers_fast.mse2psnr(mse) 124 | mse = float(mse) 125 | psnr = float(psnr) 126 | 127 | return net, mse, psnr 128 | 129 | 130 | res_dir = "./fast_results" 131 | img_dir = os.path.join(res_dir, "imgs") 132 | 133 | plt.imsave(os.path.join(img_dir, f"GT0.png"), gt_down) 134 | 135 | down_x = [1,2,3,6,9,18] 136 | psnr = [] 137 | mse = [] 138 | ts = [] 139 | for x in down_x: 140 | print(f"Running with {x}x reduced sampling") 141 | if x == 1: 142 | t, m, p = new_render(img_dir, fast=False,r2=128,d=3) 143 | else: 144 | t, m, p = new_render(img_dir, fast=True, r2=192, d=x) 145 | psnr.append(p) 146 | mse.append(m) 147 | ts.append(t) 148 | res = {} 149 | res['down_x'] = [x**2 for x in down_x] 150 | res['psnr'] = psnr 151 | res['mse'] = mse 152 | res['time'] = ts 153 | print(res) 154 | 155 | with open(os.path.join(res_dir, 'results.txt'), 'w') as outfile: 156 | json.dump(res,outfile) 157 | 158 | 159 | fig, ax = plt.subplots(1,1) 160 | fig.suptitle('Accuracy vs Sampling Rate') 161 | ax.set_xlabel('Sampling Rate (1/x sampled)') 162 | ax.set_ylabel('PSNR') 163 | plt.xscale('log') 164 | ax.plot(res['down_x'][1:],res['psnr'][1:], label="Fast NeRF") 165 | ax.scatter([res['down_x'][0]], res['psnr'][0], c="red", label="NeRF") 166 | ax.legend() 167 | plt.savefig(os.path.join(res_dir, 'sampling_rate_accuracy.png')) 168 | 169 | fig, ax = plt.subplots(1,1) 170 | fig.suptitle('Accuracy vs Time') 171 | ax.set_xlabel('Running Time (seconds)') 172 | ax.set_ylabel('PSNR') 173 | plt.xscale('log') 174 | ax.plot(res['time'][1:],res['psnr'][1:], label="Fast NeRF") 175 | ax.scatter([res['time'][0]], res['psnr'][0], c="red", label="NeRF") 176 | ax.legend() 177 | plt.savefig(os.path.join(res_dir, 'time_accuracy.png')) 178 | 179 | 180 | 181 | -------------------------------------------------------------------------------- /imgs/pipeline.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/imgs/pipeline.jpg -------------------------------------------------------------------------------- /installation.md: -------------------------------------------------------------------------------- 1 | #Getting Set Up 2 | 3 | Setting up the conda environment and downloading data and weights for the fern scene 4 | ``` 5 | conda env create -f environment.yml 6 | conda activate nerf 7 | bash download_example_data.sh 8 | ``` 9 | Double check that within the 'logs' directory there is a folder titled 'fern_example' 10 | Within 'fern_example' should be 'config.txt', 'args.txt', and two .npy files of model weights 11 | 12 | #Rendering! 13 | 14 | You can try rendering a low resolution image using render_demo.py 15 | 16 | To render with normal nerf: 17 | ``` 18 | python render_demo.py --vanilla 19 | ``` 20 | 21 | To render using point cloud depth estimation: 22 | ``` 23 | python render_demo.py --cloud 24 | ``` 25 | 26 | To render using sparse sampling + interpolation: 27 | ``` 28 | python render_demo.py --interpolate 29 | ``` 30 | 31 | The using --cloud may lead to slow rendering. 32 | You may also pass in an integer using the --down flag. Both dimensions of the new views are divided by this number. The default is 12 and it is not recommended to go lower that 4 if you aren't using a GPU. 33 | NOTE: If you are passing --interpolate, then --down must be a product of elements of the set {2,2,3,3,7} (e.g. 2,3,4,6,9) ¯\_(ツ)_/¯ it's a feature! 34 | 35 | To visualize the depth bounds and the point cloud: 36 | ``` 37 | python render_demo.py --cloud_viz 38 | ``` 39 | First you will see near bounds, then far bounds, then the new camera center, it's view, and the full point cloud. 40 | 41 | #Where the important things are 42 | run_nerf.py, run_nerf_helpers.py - Most network functionality for vanilla NeRF as well as the point cloud based approach 43 | run_nerf_fast.py, run_nerf_helpers_fast.py - Most network functionality the sparse sampling model 44 | point_cloud.py - Methods to generate and write point clouds, as well as to read point clouds and project them to NDC space 45 | 46 | A variety of other scripts are used to produce our figures 47 | 48 | 49 | 50 | 51 | -------------------------------------------------------------------------------- /load_blender.py: -------------------------------------------------------------------------------- 1 | import os 2 | import tensorflow as tf 3 | import numpy as np 4 | import imageio 5 | import json 6 | 7 | 8 | 9 | 10 | trans_t = lambda t : tf.convert_to_tensor([ 11 | [1,0,0,0], 12 | [0,1,0,0], 13 | [0,0,1,t], 14 | [0,0,0,1], 15 | ], dtype=tf.float32) 16 | 17 | rot_phi = lambda phi : tf.convert_to_tensor([ 18 | [1,0,0,0], 19 | [0,tf.cos(phi),-tf.sin(phi),0], 20 | [0,tf.sin(phi), tf.cos(phi),0], 21 | [0,0,0,1], 22 | ], dtype=tf.float32) 23 | 24 | rot_theta = lambda th : tf.convert_to_tensor([ 25 | [tf.cos(th),0,-tf.sin(th),0], 26 | [0,1,0,0], 27 | [tf.sin(th),0, tf.cos(th),0], 28 | [0,0,0,1], 29 | ], dtype=tf.float32) 30 | 31 | 32 | def pose_spherical(theta, phi, radius): 33 | c2w = trans_t(radius) 34 | c2w = rot_phi(phi/180.*np.pi) @ c2w 35 | c2w = rot_theta(theta/180.*np.pi) @ c2w 36 | c2w = np.array([[-1,0,0,0],[0,0,1,0],[0,1,0,0],[0,0,0,1]]) @ c2w 37 | return c2w 38 | 39 | 40 | 41 | def load_blender_data(basedir, half_res=False, testskip=1): 42 | splits = ['train', 'val', 'test'] 43 | metas = {} 44 | for s in splits: 45 | with open(os.path.join(basedir, 'transforms_{}.json'.format(s)), 'r') as fp: 46 | metas[s] = json.load(fp) 47 | 48 | all_imgs = [] 49 | all_poses = [] 50 | counts = [0] 51 | for s in splits: 52 | meta = metas[s] 53 | imgs = [] 54 | poses = [] 55 | if s=='train' or testskip==0: 56 | skip = 1 57 | else: 58 | skip = testskip 59 | 60 | for frame in meta['frames'][::skip]: 61 | fname = os.path.join(basedir, frame['file_path'] + '.png') 62 | imgs.append(imageio.imread(fname)) 63 | poses.append(np.array(frame['transform_matrix'])) 64 | imgs = (np.array(imgs) / 255.).astype(np.float32) # keep all 4 channels (RGBA) 65 | poses = np.array(poses).astype(np.float32) 66 | counts.append(counts[-1] + imgs.shape[0]) 67 | all_imgs.append(imgs) 68 | all_poses.append(poses) 69 | 70 | i_split = [np.arange(counts[i], counts[i+1]) for i in range(3)] 71 | 72 | imgs = np.concatenate(all_imgs, 0) 73 | poses = np.concatenate(all_poses, 0) 74 | 75 | H, W = imgs[0].shape[:2] 76 | camera_angle_x = float(meta['camera_angle_x']) 77 | focal = .5 * W / np.tan(.5 * camera_angle_x) 78 | 79 | render_poses = tf.stack([pose_spherical(angle, -30.0, 4.0) for angle in np.linspace(-180,180,40+1)[:-1]],0) 80 | 81 | if half_res: 82 | imgs = tf.image.resize_area(imgs, [400, 400]).numpy() 83 | H = H//2 84 | W = W//2 85 | focal = focal/2. 86 | 87 | return imgs, poses, render_poses, [H, W, focal], i_split 88 | 89 | 90 | -------------------------------------------------------------------------------- /load_deepvoxels.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import imageio 4 | 5 | 6 | def load_dv_data(scene='cube', basedir='/data/deepvoxels', testskip=8): 7 | 8 | 9 | def parse_intrinsics(filepath, trgt_sidelength, invert_y=False): 10 | # Get camera intrinsics 11 | with open(filepath, 'r') as file: 12 | f, cx, cy = list(map(float, file.readline().split()))[:3] 13 | grid_barycenter = np.array(list(map(float, file.readline().split()))) 14 | near_plane = float(file.readline()) 15 | scale = float(file.readline()) 16 | height, width = map(float, file.readline().split()) 17 | 18 | try: 19 | world2cam_poses = int(file.readline()) 20 | except ValueError: 21 | world2cam_poses = None 22 | 23 | if world2cam_poses is None: 24 | world2cam_poses = False 25 | 26 | world2cam_poses = bool(world2cam_poses) 27 | 28 | print(cx,cy,f,height,width) 29 | 30 | cx = cx / width * trgt_sidelength 31 | cy = cy / height * trgt_sidelength 32 | f = trgt_sidelength / height * f 33 | 34 | fx = f 35 | if invert_y: 36 | fy = -f 37 | else: 38 | fy = f 39 | 40 | # Build the intrinsic matrices 41 | full_intrinsic = np.array([[fx, 0., cx, 0.], 42 | [0., fy, cy, 0], 43 | [0., 0, 1, 0], 44 | [0, 0, 0, 1]]) 45 | 46 | return full_intrinsic, grid_barycenter, scale, near_plane, world2cam_poses 47 | 48 | 49 | def load_pose(filename): 50 | assert os.path.isfile(filename) 51 | nums = open(filename).read().split() 52 | return np.array([float(x) for x in nums]).reshape([4,4]).astype(np.float32) 53 | 54 | 55 | H = 512 56 | W = 512 57 | deepvoxels_base = '{}/train/{}/'.format(basedir, scene) 58 | 59 | full_intrinsic, grid_barycenter, scale, near_plane, world2cam_poses = parse_intrinsics(os.path.join(deepvoxels_base, 'intrinsics.txt'), H) 60 | print(full_intrinsic, grid_barycenter, scale, near_plane, world2cam_poses) 61 | focal = full_intrinsic[0,0] 62 | print(H, W, focal) 63 | 64 | 65 | def dir2poses(posedir): 66 | poses = np.stack([load_pose(os.path.join(posedir, f)) for f in sorted(os.listdir(posedir)) if f.endswith('txt')], 0) 67 | transf = np.array([ 68 | [1,0,0,0], 69 | [0,-1,0,0], 70 | [0,0,-1,0], 71 | [0,0,0,1.], 72 | ]) 73 | poses = poses @ transf 74 | poses = poses[:,:3,:4].astype(np.float32) 75 | return poses 76 | 77 | posedir = os.path.join(deepvoxels_base, 'pose') 78 | poses = dir2poses(posedir) 79 | testposes = dir2poses('{}/test/{}/pose'.format(basedir, scene)) 80 | testposes = testposes[::testskip] 81 | valposes = dir2poses('{}/validation/{}/pose'.format(basedir, scene)) 82 | valposes = valposes[::testskip] 83 | 84 | imgfiles = [f for f in sorted(os.listdir(os.path.join(deepvoxels_base, 'rgb'))) if f.endswith('png')] 85 | imgs = np.stack([imageio.imread(os.path.join(deepvoxels_base, 'rgb', f))/255. for f in imgfiles], 0).astype(np.float32) 86 | 87 | 88 | testimgd = '{}/test/{}/rgb'.format(basedir, scene) 89 | imgfiles = [f for f in sorted(os.listdir(testimgd)) if f.endswith('png')] 90 | testimgs = np.stack([imageio.imread(os.path.join(testimgd, f))/255. for f in imgfiles[::testskip]], 0).astype(np.float32) 91 | 92 | valimgd = '{}/validation/{}/rgb'.format(basedir, scene) 93 | imgfiles = [f for f in sorted(os.listdir(valimgd)) if f.endswith('png')] 94 | valimgs = np.stack([imageio.imread(os.path.join(valimgd, f))/255. for f in imgfiles[::testskip]], 0).astype(np.float32) 95 | 96 | all_imgs = [imgs, valimgs, testimgs] 97 | counts = [0] + [x.shape[0] for x in all_imgs] 98 | counts = np.cumsum(counts) 99 | i_split = [np.arange(counts[i], counts[i+1]) for i in range(3)] 100 | 101 | imgs = np.concatenate(all_imgs, 0) 102 | poses = np.concatenate([poses, valposes, testposes], 0) 103 | 104 | render_poses = testposes 105 | 106 | print(poses.shape, imgs.shape) 107 | 108 | return imgs, poses, render_poses, [H,W,focal], i_split 109 | 110 | 111 | -------------------------------------------------------------------------------- /load_llff.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import os, imageio 3 | 4 | 5 | ########## Slightly modified version of LLFF data loading code 6 | ########## see https://github.com/Fyusion/LLFF for original 7 | 8 | def _minify(basedir, factors=[], resolutions=[]): 9 | needtoload = False 10 | for r in factors: 11 | imgdir = os.path.join(basedir, 'images_{}'.format(r)) 12 | if not os.path.exists(imgdir): 13 | needtoload = True 14 | for r in resolutions: 15 | imgdir = os.path.join(basedir, 'images_{}x{}'.format(r[1], r[0])) 16 | if not os.path.exists(imgdir): 17 | needtoload = True 18 | if not needtoload: 19 | return 20 | 21 | from shutil import copy 22 | from subprocess import check_output 23 | 24 | imgdir = os.path.join(basedir, 'images') 25 | imgs = [os.path.join(imgdir, f) for f in sorted(os.listdir(imgdir))] 26 | imgs = [f for f in imgs if any([f.endswith(ex) for ex in ['JPG', 'jpg', 'png', 'jpeg', 'PNG']])] 27 | imgdir_orig = imgdir 28 | 29 | wd = os.getcwd() 30 | 31 | for r in factors + resolutions: 32 | if isinstance(r, int): 33 | name = 'images_{}'.format(r) 34 | resizearg = '{}%'.format(100./r) 35 | else: 36 | name = 'images_{}x{}'.format(r[1], r[0]) 37 | resizearg = '{}x{}'.format(r[1], r[0]) 38 | imgdir = os.path.join(basedir, name) 39 | if os.path.exists(imgdir): 40 | continue 41 | 42 | print('Minifying', r, basedir) 43 | 44 | os.makedirs(imgdir) 45 | check_output('cp {}/* {}'.format(imgdir_orig, imgdir), shell=True) 46 | 47 | ext = imgs[0].split('.')[-1] 48 | args = ' '.join(['mogrify', '-resize', resizearg, '-format', 'png', '*.{}'.format(ext)]) 49 | print(args) 50 | os.chdir(imgdir) 51 | check_output(args, shell=True) 52 | os.chdir(wd) 53 | 54 | if ext != 'png': 55 | check_output('rm {}/*.{}'.format(imgdir, ext), shell=True) 56 | print('Removed duplicates') 57 | print('Done') 58 | 59 | 60 | 61 | 62 | def _load_data(basedir, factor=None, width=None, height=None, load_imgs=True): 63 | 64 | poses_arr = np.load(os.path.join(basedir, 'poses_bounds.npy')) 65 | poses = poses_arr[:, :-2].reshape([-1, 3, 5]).transpose([1,2,0]) 66 | bds = poses_arr[:, -2:].transpose([1,0]) 67 | 68 | img0 = [os.path.join(basedir, 'images', f) for f in sorted(os.listdir(os.path.join(basedir, 'images'))) \ 69 | if f.endswith('JPG') or f.endswith('jpg') or f.endswith('png')][0] 70 | sh = imageio.imread(img0).shape 71 | 72 | sfx = '' 73 | 74 | if factor is not None: 75 | sfx = '_{}'.format(factor) 76 | _minify(basedir, factors=[factor]) 77 | factor = factor 78 | elif height is not None: 79 | factor = sh[0] / float(height) 80 | width = int(sh[1] / factor) 81 | _minify(basedir, resolutions=[[height, width]]) 82 | sfx = '_{}x{}'.format(width, height) 83 | elif width is not None: 84 | factor = sh[1] / float(width) 85 | height = int(sh[0] / factor) 86 | _minify(basedir, resolutions=[[height, width]]) 87 | sfx = '_{}x{}'.format(width, height) 88 | else: 89 | factor = 1 90 | 91 | imgdir = os.path.join(basedir, 'images' + sfx) 92 | if not os.path.exists(imgdir): 93 | print( imgdir, 'does not exist, returning' ) 94 | return 95 | 96 | imgfiles = [os.path.join(imgdir, f) for f in sorted(os.listdir(imgdir)) if f.endswith('JPG') or f.endswith('jpg') or f.endswith('png')] 97 | if poses.shape[-1] != len(imgfiles): 98 | print( 'Mismatch between imgs {} and poses {} !!!!'.format(len(imgfiles), poses.shape[-1]) ) 99 | return 100 | 101 | sh = imageio.imread(imgfiles[0]).shape 102 | poses[:2, 4, :] = np.array(sh[:2]).reshape([2, 1]) 103 | poses[2, 4, :] = poses[2, 4, :] * 1./factor 104 | 105 | if not load_imgs: 106 | return poses, bds 107 | 108 | def imread(f): 109 | if f.endswith('png'): 110 | return imageio.imread(f, ignoregamma=True) 111 | else: 112 | return imageio.imread(f) 113 | 114 | imgs = imgs = [imread(f)[...,:3]/255. for f in imgfiles] 115 | imgs = np.stack(imgs, -1) 116 | 117 | print('Loaded image data', imgs.shape, poses[:,-1,0]) 118 | return poses, bds, imgs 119 | 120 | 121 | 122 | 123 | 124 | 125 | def normalize(x): 126 | return x / np.linalg.norm(x) 127 | 128 | def viewmatrix(z, up, pos): 129 | vec2 = normalize(z) 130 | vec1_avg = up 131 | vec0 = normalize(np.cross(vec1_avg, vec2)) 132 | vec1 = normalize(np.cross(vec2, vec0)) 133 | m = np.stack([vec0, vec1, vec2, pos], 1) 134 | return m 135 | 136 | def ptstocam(pts, c2w): 137 | tt = np.matmul(c2w[:3,:3].T, (pts-c2w[:3,3])[...,np.newaxis])[...,0] 138 | return tt 139 | 140 | def poses_avg(poses): 141 | 142 | hwf = poses[0, :3, -1:] 143 | 144 | center = poses[:, :3, 3].mean(0) 145 | vec2 = normalize(poses[:, :3, 2].sum(0)) 146 | up = poses[:, :3, 1].sum(0) 147 | c2w = np.concatenate([viewmatrix(vec2, up, center), hwf], 1) 148 | 149 | return c2w 150 | 151 | 152 | 153 | def render_path_spiral(c2w, up, rads, focal, zdelta, zrate, rots, N): 154 | render_poses = [] 155 | rads = np.array(list(rads) + [1.]) 156 | hwf = c2w[:,4:5] 157 | 158 | for theta in np.linspace(0., 2. * np.pi * rots, N+1)[:-1]: 159 | c = np.dot(c2w[:3,:4], np.array([np.cos(theta), -np.sin(theta), -np.sin(theta*zrate), 1.]) * rads) 160 | z = normalize(c - np.dot(c2w[:3,:4], np.array([0,0,-focal, 1.]))) 161 | render_poses.append(np.concatenate([viewmatrix(z, up, c), hwf], 1)) 162 | return render_poses 163 | 164 | 165 | 166 | def recenter_poses(poses): 167 | 168 | poses_ = poses+0 169 | bottom = np.reshape([0,0,0,1.], [1,4]) 170 | c2w = poses_avg(poses) 171 | c2w = np.concatenate([c2w[:3,:4], bottom], -2) 172 | bottom = np.tile(np.reshape(bottom, [1,1,4]), [poses.shape[0],1,1]) 173 | poses = np.concatenate([poses[:,:3,:4], bottom], -2) 174 | 175 | poses = np.linalg.inv(c2w) @ poses 176 | poses_[:,:3,:4] = poses[:,:3,:4] 177 | poses = poses_ 178 | return poses 179 | 180 | 181 | ##################### 182 | 183 | 184 | def spherify_poses(poses, bds): 185 | 186 | p34_to_44 = lambda p : np.concatenate([p, np.tile(np.reshape(np.eye(4)[-1,:], [1,1,4]), [p.shape[0], 1,1])], 1) 187 | 188 | rays_d = poses[:,:3,2:3] 189 | rays_o = poses[:,:3,3:4] 190 | 191 | def min_line_dist(rays_o, rays_d): 192 | A_i = np.eye(3) - rays_d * np.transpose(rays_d, [0,2,1]) 193 | b_i = -A_i @ rays_o 194 | pt_mindist = np.squeeze(-np.linalg.inv((np.transpose(A_i, [0,2,1]) @ A_i).mean(0)) @ (b_i).mean(0)) 195 | return pt_mindist 196 | 197 | pt_mindist = min_line_dist(rays_o, rays_d) 198 | 199 | center = pt_mindist 200 | up = (poses[:,:3,3] - center).mean(0) 201 | 202 | vec0 = normalize(up) 203 | vec1 = normalize(np.cross([.1,.2,.3], vec0)) 204 | vec2 = normalize(np.cross(vec0, vec1)) 205 | pos = center 206 | c2w = np.stack([vec1, vec2, vec0, pos], 1) 207 | 208 | poses_reset = np.linalg.inv(p34_to_44(c2w[None])) @ p34_to_44(poses[:,:3,:4]) 209 | 210 | rad = np.sqrt(np.mean(np.sum(np.square(poses_reset[:,:3,3]), -1))) 211 | 212 | sc = 1./rad 213 | poses_reset[:,:3,3] *= sc 214 | bds *= sc 215 | rad *= sc 216 | 217 | centroid = np.mean(poses_reset[:,:3,3], 0) 218 | zh = centroid[2] 219 | radcircle = np.sqrt(rad**2-zh**2) 220 | new_poses = [] 221 | 222 | for th in np.linspace(0.,2.*np.pi, 120): 223 | 224 | camorigin = np.array([radcircle * np.cos(th), radcircle * np.sin(th), zh]) 225 | up = np.array([0,0,-1.]) 226 | 227 | vec2 = normalize(camorigin) 228 | vec0 = normalize(np.cross(vec2, up)) 229 | vec1 = normalize(np.cross(vec2, vec0)) 230 | pos = camorigin 231 | p = np.stack([vec0, vec1, vec2, pos], 1) 232 | 233 | new_poses.append(p) 234 | 235 | new_poses = np.stack(new_poses, 0) 236 | 237 | new_poses = np.concatenate([new_poses, np.broadcast_to(poses[0,:3,-1:], new_poses[:,:3,-1:].shape)], -1) 238 | poses_reset = np.concatenate([poses_reset[:,:3,:4], np.broadcast_to(poses[0,:3,-1:], poses_reset[:,:3,-1:].shape)], -1) 239 | 240 | return poses_reset, new_poses, bds 241 | 242 | 243 | def load_llff_data(basedir, factor=8, recenter=True, bd_factor=.75, spherify=False, path_zflat=False): 244 | 245 | 246 | poses, bds, imgs = _load_data(basedir, factor=factor) # factor=8 downsamples original imgs by 8x 247 | print('Loaded', basedir, bds.min(), bds.max()) 248 | 249 | # Correct rotation matrix ordering and move variable dim to axis 0 250 | poses = np.concatenate([poses[:, 1:2, :], -poses[:, 0:1, :], poses[:, 2:, :]], 1) 251 | poses = np.moveaxis(poses, -1, 0).astype(np.float32) 252 | imgs = np.moveaxis(imgs, -1, 0).astype(np.float32) 253 | images = imgs 254 | bds = np.moveaxis(bds, -1, 0).astype(np.float32) 255 | 256 | # Rescale if bd_factor is provided 257 | sc = 1. if bd_factor is None else 1./(bds.min() * bd_factor) 258 | poses[:,:3,3] *= sc 259 | bds *= sc 260 | 261 | if recenter: 262 | poses = recenter_poses(poses) 263 | 264 | if spherify: 265 | poses, render_poses, bds = spherify_poses(poses, bds) 266 | 267 | else: 268 | 269 | c2w = poses_avg(poses) 270 | print('recentered', c2w.shape) 271 | print(c2w[:3,:4]) 272 | 273 | ## Get spiral 274 | # Get average pose 275 | up = normalize(poses[:, :3, 1].sum(0)) 276 | 277 | # Find a reasonable "focus depth" for this dataset 278 | close_depth, inf_depth = bds.min()*.9, bds.max()*5. 279 | dt = .75 280 | mean_dz = 1./(((1.-dt)/close_depth + dt/inf_depth)) 281 | focal = mean_dz 282 | 283 | # Get radii for spiral path 284 | shrink_factor = .8 285 | zdelta = close_depth * .2 286 | tt = poses[:,:3,3] # ptstocam(poses[:3,3,:].T, c2w).T 287 | rads = np.percentile(np.abs(tt), 90, 0) 288 | c2w_path = c2w 289 | N_views = 120 290 | N_rots = 2 291 | if path_zflat: 292 | # zloc = np.percentile(tt, 10, 0)[2] 293 | zloc = -close_depth * .1 294 | c2w_path[:3,3] = c2w_path[:3,3] + zloc * c2w_path[:3,2] 295 | rads[2] = 0. 296 | N_rots = 1 297 | N_views/=2 298 | 299 | # Generate poses for spiral path 300 | render_poses = render_path_spiral(c2w_path, up, rads, focal, zdelta, zrate=.5, rots=N_rots, N=N_views) 301 | 302 | 303 | render_poses = np.array(render_poses).astype(np.float32) 304 | 305 | c2w = poses_avg(poses) 306 | print('Data:') 307 | print(poses.shape, images.shape, bds.shape) 308 | 309 | dists = np.sum(np.square(c2w[:3,3] - poses[:,:3,3]), -1) 310 | i_test = np.argmin(dists) 311 | print('HOLDOUT view is', i_test) 312 | 313 | images = images.astype(np.float32) 314 | poses = poses.astype(np.float32) 315 | 316 | return images, poses, bds, render_poses, i_test 317 | 318 | 319 | 320 | -------------------------------------------------------------------------------- /paper_configs/blender_config.txt: -------------------------------------------------------------------------------- 1 | # This replicates the paper result for "Lego" 2 | # when trained to 500k iters. Settings are the 3 | # same for all other synthetic blender scenes. 4 | 5 | expname = blender_paper_lego 6 | basedir = ./logs 7 | datadir = ./data/nerf_synthetic/lego 8 | dataset_type = blender 9 | 10 | no_batching = True 11 | 12 | use_viewdirs = True 13 | white_bkgd = True 14 | lrate_decay = 500 15 | 16 | N_samples = 64 17 | N_importance = 128 18 | N_rand = 1024 19 | 20 | -------------------------------------------------------------------------------- /paper_configs/deepvoxels_config.txt: -------------------------------------------------------------------------------- 1 | # This replicates the paper result for "Cube" 2 | # when trained to 200k iters. Settings are the 3 | # same for all other DeepVoxels scenes. 4 | 5 | # You'll have to point the "datadir" argument 6 | # to wherever you've placed the downloaded 7 | # DeepVoxels dataset (base directory with the 8 | # "train", "test", "validation" folders). 9 | 10 | expname = dvox_paper_cube 11 | shape = cube 12 | basedir = ./logs 13 | datadir = ./data/deepvoxels 14 | dataset_type = deepvoxels 15 | 16 | no_batching = True 17 | 18 | use_viewdirs = True 19 | white_bkgd = True 20 | lrate_decay = 200 21 | 22 | N_samples = 64 23 | N_importance = 128 24 | N_rand = 4096 -------------------------------------------------------------------------------- /paper_configs/llff_config.txt: -------------------------------------------------------------------------------- 1 | # This replicates the paper result for "Fern" 2 | # when trained to 200k iters. Settings are the 3 | # same for all other LLFF-style real scenes. 4 | 5 | expname = llff_paper_fern 6 | basedir = ./logs 7 | datadir = ./data/nerf_llff_data/fern 8 | dataset_type = llff 9 | factor = 4 10 | llffhold = 8 11 | 12 | no_batching = False 13 | 14 | use_viewdirs = True 15 | lrate_decay = 250 16 | raw_noise_std = 1.0 17 | 18 | N_samples = 64 19 | N_importance = 128 20 | N_rand = 4096 -------------------------------------------------------------------------------- /point_cloud.py: -------------------------------------------------------------------------------- 1 | import os, sys 2 | # os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true' 3 | os.environ['CUDA_VISIBLE_DEVICES'] = '1' 4 | import tensorflow as tf 5 | tf.compat.v1.enable_eager_execution() 6 | 7 | import numpy as np 8 | import imageio 9 | import json 10 | import random 11 | import time 12 | import pprint 13 | 14 | import matplotlib.pyplot as plt 15 | 16 | import run_nerf 17 | from run_nerf_helpers_fast import * 18 | 19 | from load_llff import load_llff_data 20 | from load_deepvoxels import load_dv_data 21 | from load_blender import load_blender_data 22 | 23 | from mpl_toolkits.mplot3d import Axes3D 24 | import matplotlib.pyplot as plt 25 | 26 | 27 | def load_nerf(args): 28 | 29 | _, render_kwargs_test, start, grad_vars, models = run_nerf.create_nerf(args) 30 | 31 | # The point cloud functionality should only be used for ndc image sets 32 | near = 0. 33 | far = 1. 34 | 35 | bds_dict = { 36 | 'near' : tf.cast(near, tf.float32), 37 | 'far' : tf.cast(far, tf.float32), 38 | } 39 | render_kwargs_test.update(bds_dict) 40 | 41 | render_kwargs_fast = {k : render_kwargs_test[k] for k in render_kwargs_test} 42 | render_kwargs_fast['N_importance'] = 128 43 | return render_kwargs_fast 44 | 45 | def depth_bounds(hwf, c2w, cloudsize=16, bounds=False, show_view=False): 46 | epsilon = 0.10 47 | 48 | near = 1. 49 | H, W, focal = hwf 50 | world_points = np.load(f"./clouds/pointcloud_down{cloudsize}.npy") 51 | 52 | # find directional rays from camera center towards world points 53 | points_o = np.broadcast_to(c2w[:3, -1], np.shape(world_points)) 54 | camera_points = np.sum(world_points[..., np.newaxis, :] * np.linalg.inv(c2w[:3, :3]), -1) 55 | camera_points = camera_points - points_o 56 | cam_normed = camera_points / (-camera_points[:,2:]) 57 | 58 | # get rays from directions 59 | _,cam_rays = get_rays_np(H, W, focal, c2w, override_dirs=cam_normed) 60 | 61 | # get ndc points and set z equal to disparity 62 | depth = (world_points / cam_rays)[:,2] 63 | _, ndc_cloud = ndc_rays(H, W, focal, near, points_o, cam_rays) 64 | ndc_cloud = ndc_cloud.numpy() 65 | disp = 1. / depth 66 | ndc_cloud[:,2] = disp 67 | 68 | # find range of view in ndc coords 69 | gen_os, generated_rays = get_rays_np(H, W, focal, c2w) 70 | # print(generated_rays) 71 | _, generated_ndc = ndc_rays(H, W, focal, near, gen_os, generated_rays) 72 | generated_ndc = generated_ndc.numpy() 73 | # print(generated_ndc) 74 | 75 | low_x = generated_ndc[0,0,0] 76 | high_x = generated_ndc[0,-1,0] 77 | low_y = generated_ndc[0,0,1] 78 | high_y = generated_ndc[-1,0,1] 79 | 80 | # cloud_low_x = np.min(ndc_cloud[...,0]) 81 | # cloud_high_x = np.max(ndc_cloud[...,0]) 82 | # cloud_low_y = np.min(ndc_cloud[...,1]) 83 | # cloud_high_y = np.max(ndc_cloud[...,1]) 84 | 85 | # print("generated") 86 | # print(f'x- {low_x},{high_x}') 87 | # print(f'y- {low_y}{high_y}') 88 | # print("cloud") 89 | # print(f'x- {cloud_low_x}{cloud_high_x}') 90 | # print(f'y- {cloud_low_y}{cloud_high_y}') 91 | 92 | # find pixel size in ndc space 93 | near_bound = np.ones((H, W)) *-1. 94 | far_bound = np.ones((H, W)) * 2. 95 | pix_height = (high_y - low_y) / (H) 96 | pix_width = (high_x - low_x) / (W) 97 | # print(f'h: {pix_height}') 98 | # print(f'w: {pix_width}') 99 | 100 | # rays are center of pixels, adjust bounds accordingly 101 | low_x = low_x - (pix_width * 0.5) 102 | high_x = high_x + (pix_width * 0.5) 103 | low_y = low_y - (pix_height * 0.5) 104 | high_y = high_y + (pix_height * 0.5) 105 | 106 | # check every point in the cloud to update the bounds 107 | for a in range(ndc_cloud.shape[0]): 108 | p = ndc_cloud[a] 109 | # find image coords 110 | i = int((p[0] - low_x) // pix_width) 111 | j = int((p[1] - low_y) // pix_height) 112 | # check if image coords are valid 113 | if i < W and i >= 0 and j < H and j >= 0: 114 | # compare to near bound 115 | if p[2] > near_bound[j,i]: 116 | near_bound[j,i] = p[2] 117 | # compare to far_bound 118 | if p[2] < far_bound[j,i]: 119 | far_bound[j,i] = p[2] 120 | 121 | # max bounds if they haven't been updated 122 | near_bound[near_bound < 0.] = 1. 123 | far_bound[far_bound > 1.0] = 0. 124 | # invert since sampling is done 0-1 125 | near_bound = 1. - near_bound 126 | far_bound = 1. - far_bound 127 | 128 | near_bound = np.clip(near_bound - epsilon, 0., 1.) 129 | far_bound = np.clip(far_bound + epsilon, 0., 1.) 130 | 131 | # max and min local region 132 | translated_near = [] 133 | translated_far = [] 134 | for i in range(-2,3): 135 | for j in range(-2,3): 136 | # i = i-1 137 | # j = j-1 138 | near_base = np.zeros((H,W)) 139 | far_base = np.ones((H,W)) 140 | if i < 0: 141 | if j < 0: 142 | near_base[:-1,:-1] = near_bound[1:,1:] 143 | far_base[:-1,:-1] = far_bound[1:,1:] 144 | if j == 0: 145 | near_base[:-1,:] = near_bound[1:,:] 146 | far_base[:-1,:] = far_bound[1:,:] 147 | if j > 0: 148 | near_base[:-1,1:] = near_bound[1:,:-1] 149 | far_base[:-1,1:] = far_bound[1:,:-1] 150 | if i > 0: 151 | if j < 0: 152 | near_base[1:,:-1] = near_bound[:-1,1:] 153 | far_base[1:,:-1] = far_bound[:-1,1:] 154 | if j == 0: 155 | near_base[1:,:] = near_bound[:-1,:] 156 | far_base[1:,:] = far_bound[:-1,:] 157 | if j > 0: 158 | near_base[1:,1:] = near_bound[:-1,:-1] 159 | far_base[1:,1:] = far_bound[:-1,:-1] 160 | if i == 0: 161 | if j < 0: 162 | near_base[:,:-1] = near_bound[:,1:] 163 | far_base[:,:-1] = far_bound[:,1:] 164 | if j == 0: 165 | near_base[:,:] = near_bound[:,:] 166 | far_base[:,:] = far_bound[:,:] 167 | if j > 0: 168 | near_base[:,1:] = near_bound[:,:-1] 169 | far_base[:,1:] = far_bound[:,:-1] 170 | translated_near.append(near_base) 171 | translated_far.append(far_base) 172 | near_stack = np.stack(translated_near, axis=-1) 173 | far_stack = np.stack(translated_far, axis=-1) 174 | near_bound = np.amin(near_stack, axis=-1) 175 | far_bound = np.amax(far_stack, axis=-1) 176 | 177 | 178 | # Show near and far bounds 179 | if bounds: 180 | plt.imshow(near_bound, vmin=0, vmax=1) 181 | plt.show() 182 | 183 | plt.imshow(far_bound, vmin=0, vmax=1) 184 | plt.show() 185 | 186 | # NOTE: Use to show world points relative to camera view 187 | if show_view: 188 | fig = plt.figure() 189 | ax = fig.add_subplot(111, projection='3d') 190 | plot_scene(ax, world_points, [c2w], [hwf]) 191 | plt.show() 192 | return (near_bound, far_bound) 193 | 194 | 195 | def make_point_cloud(hwf, poses, i_train, args, render_kwargs, down=32): 196 | ''' 197 | Makes 3D point cloud using estimated depth data from images 198 | 199 | ''' 200 | near = 1. 201 | H, W, focal = hwf 202 | # use only the training images 203 | all_points = [] 204 | centers = [] 205 | for i in range(len(i_train)): 206 | print(f'Working on image #{i+1}') 207 | # c2w = poses[i_train[i]] 208 | c2w = poses[i_train[i]] 209 | # print(f"Using {c2w}") 210 | # print(c2w) 211 | centers.append(c2w[np.newaxis,:3, -1]) 212 | rays_o, rays_d = get_rays(H//down, W//down, focal/down, c2w) 213 | 214 | res = run_nerf.render(H//down, W//down, focal/down, c2w=c2w, **render_kwargs) 215 | t_prime = res[3]['depth_map'] 216 | # plt.imshow(t_prime) 217 | # plt.show() 218 | 219 | # convert to numpy 220 | rays_o, rays_d, t_prime = rays_o.numpy(), rays_d.numpy(), t_prime.numpy() 221 | oz, dz = rays_o[...,2], rays_d[...,2] 222 | 223 | # accounting for shifted origin before ndc conversion 224 | tn = -(near + oz) / dz 225 | # plt.imshow(tn) 226 | # plt.show() 227 | # print(tn) 228 | on = rays_o + tn[..., np.newaxis] * rays_d 229 | # print("RAYO") 230 | # print(rays_o) 231 | # print("RAY_D") 232 | # print(rays_d) 233 | # print("ON") 234 | # print(on) 235 | oz = on[...,2] 236 | # plt.imshow(oz) 237 | # plt.show() 238 | # solve for t given t prime using equation 15 from the paper 239 | # t_prime should be the ndc ray depth, while t is the real world ray depth 240 | 241 | t = (-1. * t_prime) + 1. 242 | # plt.imshow(t) 243 | # plt.show() 244 | t = 1. / t 245 | # plt.imshow(t) 246 | # plt.show() 247 | t = oz * t 248 | # plt.imshow(t) 249 | # plt.show() 250 | t = t - oz 251 | # plt.imshow(t) 252 | # plt.show() 253 | t = t / dz 254 | # plt.imshow(t) 255 | # plt.show() 256 | 257 | # get point cloud 258 | points_3d = on + t[..., np.newaxis] * rays_d 259 | 260 | points_3d = points_3d.reshape((-1,3)) 261 | 262 | all_points.append(points_3d) 263 | 264 | all_points = np.concatenate(all_points, axis=0) 265 | centers = np.concatenate(centers, axis=0) 266 | np.save(f"./clouds/pointcloud_down{down}.npy", all_points) 267 | np.save("./clouds/camcenters.npy", centers) 268 | 269 | 270 | # # plot 271 | # fig = plt.figure() 272 | # ax = fig.add_subplot(111, projection='3d') 273 | # ax.scatter(points_3d[..., 0], points_3d[..., 1], points_3d[..., 2]) 274 | 275 | # plt.show() 276 | 277 | 278 | 279 | 280 | 281 | 282 | if __name__ == "__main__": 283 | basedir = './logs' 284 | # NOTE: select what to make the point cloud for 285 | expname = 'fern_example' 286 | config = os.path.join(basedir, expname, 'config.txt') 287 | parser = run_nerf.config_parser() 288 | 289 | weights_name = 'model_200000.npy' 290 | args = parser.parse_args('--config {} --ft_path {}'.format(config, os.path.join(basedir, expname, weights_name))) 291 | 292 | images, poses, bds, render_poses, i_test = load_llff_data(args.datadir, args.factor, 293 | recenter=True, bd_factor=.75, 294 | spherify=args.spherify) 295 | H, W, focal = poses[0,:3,-1].astype(np.float32) 296 | poses = poses[:, :3, :4] 297 | 298 | H = int(H) 299 | W = int(W) 300 | hwf = [H, W, focal] 301 | 302 | images = images.astype(np.float32) 303 | poses = poses.astype(np.float32) 304 | 305 | # get the test and train sets 306 | if not isinstance(i_test, list): 307 | i_test = [i_test] 308 | 309 | if args.llffhold > 0: 310 | print('Auto LLFF holdout,', args.llffhold) 311 | i_test = np.arange(images.shape[0])[::args.llffhold] 312 | 313 | i_train = np.array([i for i in np.arange(int(images.shape[0])) if 314 | (i not in i_test)]) 315 | 316 | render_kwargs = load_nerf(args) 317 | 318 | # print("Num images") 319 | # print(i_train.shape) 320 | # make_point_cloud(hwf, poses, i_train, args, render_kwargs, down=1) 321 | for x in [2,4,8,16,32]: 322 | print(f'Generating point cloud that is {x}x downsampled') 323 | make_point_cloud(hwf, poses, i_train, args, render_kwargs, down=x) 324 | # down = 4 325 | # hwf = [H//64, W//64, focal//64] 326 | # depth_bounds(hwf, poses[i_test[0]]) 327 | 328 | -------------------------------------------------------------------------------- /point_cloud_results.py: -------------------------------------------------------------------------------- 1 | import os, sys 2 | # os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true' 3 | os.environ['CUDA_VISIBLE_DEVICES'] = '1' 4 | import tensorflow as tf 5 | tf.compat.v1.enable_eager_execution() 6 | 7 | import numpy as np 8 | import imageio 9 | import json 10 | import random 11 | import time 12 | import pprint 13 | import argparse 14 | 15 | import matplotlib.pyplot as plt 16 | 17 | import run_nerf 18 | 19 | from load_llff import load_llff_data 20 | from load_deepvoxels import load_dv_data 21 | from load_blender import load_blender_data 22 | 23 | # profiling 24 | import cProfile 25 | import pstats 26 | from pstats import SortKey 27 | 28 | # CELL 1 29 | def get_data(): 30 | basedir = './logs' 31 | expname = 'fern_example' 32 | 33 | config = os.path.join(basedir, expname, 'config.txt') 34 | print('Args:') 35 | print(open(config, 'r').read()) 36 | parser = run_nerf.config_parser() 37 | 38 | weights_name = 'model_200000.npy' 39 | args = parser.parse_args('--config {} --ft_path {}'.format(config, os.path.join(basedir, expname, weights_name))) 40 | print('loaded args') 41 | 42 | images, poses, bds, render_poses, i_test = load_llff_data(args.datadir, args.factor, 43 | recenter=True, bd_factor=.75, 44 | spherify=args.spherify) 45 | H, W, focal = poses[0,:3,-1].astype(np.float32) 46 | poses = poses[:, :3, :4] 47 | H = int(H) 48 | W = int(W) 49 | hwf = [H, W, focal] 50 | 51 | images = images.astype(np.float32) 52 | poses = poses.astype(np.float32) 53 | 54 | near = 0. 55 | far = 1. 56 | 57 | if not isinstance(i_test, list): 58 | i_test = [i_test] 59 | 60 | if args.llffhold > 0: 61 | print('Auto LLFF holdout,', args.llffhold) 62 | i_test = np.arange(images.shape[0])[::args.llffhold] 63 | 64 | _, render_kwargs, start, grad_vars, models = run_nerf.create_nerf(args) 65 | 66 | bds_dict = { 67 | 'near' : tf.cast(near, tf.float32), 68 | 'far' : tf.cast(far, tf.float32), 69 | } 70 | render_kwargs.update(bds_dict) 71 | 72 | print('Render kwargs:') 73 | pprint.pprint(render_kwargs) 74 | 75 | results = {} 76 | results['pc'] = {} 77 | results['no_pc'] = {} 78 | 79 | # NOTE: Where to output results! 80 | result_directory = "./fern_pc_results" 81 | img_dir = os.path.join(result_directory, "imgs") 82 | 83 | down = 1 84 | 85 | plt.imsave(os.path.join(img_dir, f"GT{i_test[0]}.png"), images[i_test[0]]) 86 | plt.imsave(os.path.join(img_dir, f"GT{i_test[1]}.png"), images[i_test[1]]) 87 | 88 | for num_samps in [4,8,16,32,64]: 89 | print(f'Running {num_samps} sample test') 90 | for pc in [True, False]: 91 | print(f'{"not " if not pc else ""}using pc') 92 | results['pc' if pc else 'no_pc'][num_samps] = {} 93 | render_kwargs['N_samples'] = num_samps 94 | render_kwargs['N_importance'] = 2*num_samps 95 | 96 | total_time = 0 97 | total_mse = 0 98 | total_psnr = 0 99 | for i in [i_test[0], i_test[1]]: 100 | gt = images[i] 101 | start_time = time.time() 102 | ret_vals = run_nerf.render(H//down, W//down, focal/down, c2w=poses[i], pc=pc, cloudsize=16, **render_kwargs) 103 | end_time = time.time() 104 | 105 | # add to cum time 106 | total_time += (end_time - start_time) 107 | 108 | # add to accuracy 109 | img = np.clip(ret_vals[0],0,1) 110 | # TODO: make sure this is commented out for real results (just used to test that it runs) 111 | # mse = run_nerf.img2mse(np.zeros((H//down, W//down,3), dtype=np.float32), img) 112 | mse = run_nerf.img2mse(gt, img) 113 | psnr = run_nerf.mse2psnr(mse) 114 | total_mse += float(mse) 115 | total_psnr += float(psnr) 116 | 117 | plt.imsave(os.path.join(img_dir, f'IMG{i}_{"pc" if pc else "no_pc"}_{num_samps}samples.png'), img) 118 | 119 | total_time /= 2. 120 | total_mse /= 2. 121 | total_psnr /= 2. 122 | results['pc' if pc else 'no_pc'][num_samps]['time'] = total_time 123 | results['pc' if pc else 'no_pc'][num_samps]['mse'] = total_mse 124 | results['pc' if pc else 'no_pc'][num_samps]['psnr'] = total_psnr 125 | 126 | with open(os.path.join(result_directory, 'results.txt'), 'w') as outfile: 127 | json.dump(results,outfile) 128 | 129 | def plot_data(): 130 | ''' 131 | Plot the data from results.txt in the given results directory 132 | ''' 133 | 134 | res_dir = "./fern_pc_results" 135 | 136 | with open(os.path.join(res_dir, 'results.txt')) as json_file: 137 | res = json.load(json_file) 138 | 139 | pc_time = [] 140 | nopc_time = [] 141 | pc_psnr = [] 142 | nopc_psnr = [] 143 | all_samps = [4,8,16,32,64] 144 | for num_samps in [4,8,16,32,64]: 145 | pc_time.append(res['pc'][str(num_samps)]['time']) 146 | nopc_time.append(res['no_pc'][str(num_samps)]['time']) 147 | pc_psnr.append(res['pc'][str(num_samps)]['psnr']) 148 | nopc_psnr.append(res['no_pc'][str(num_samps)]['psnr']) 149 | 150 | fig, ax = plt.subplots(1,1) 151 | fig.suptitle('PSNR vs Num Samples') 152 | ax.set_xlabel('Samples per Ray') 153 | ax.set_ylabel('PSNR') 154 | plt.xscale('log') 155 | ax.plot(all_samps,nopc_psnr, label="NeRF") 156 | ax.plot(all_samps,pc_psnr, label="NeRF with Point Cloud") 157 | ax.legend() 158 | plt.savefig(os.path.join(res_dir, 'samps_psnr.png')) 159 | 160 | fig, ax = plt.subplots(1,1) 161 | fig.suptitle('PSNR vs Running Time') 162 | ax.set_xlabel('Time (seconds)') 163 | ax.set_ylabel('PSNR') 164 | plt.xscale('log') 165 | ax.plot(nopc_time,nopc_psnr, label="NeRF") 166 | ax.plot(pc_time, pc_psnr, label="NeRF with Point Cloud") 167 | ax.legend() 168 | plt.savefig(os.path.join(res_dir, 'time_psnr.png')) 169 | 170 | 171 | 172 | 173 | def cloud_size_vs_performance(): 174 | basedir = './logs' 175 | expname = 'fern_example' 176 | 177 | config = os.path.join(basedir, expname, 'config.txt') 178 | print('Args:') 179 | print(open(config, 'r').read()) 180 | parser = run_nerf.config_parser() 181 | 182 | weights_name = 'model_200000.npy' 183 | args = parser.parse_args('--config {} --ft_path {}'.format(config, os.path.join(basedir, expname, weights_name))) 184 | print('loaded args') 185 | 186 | images, poses, bds, render_poses, i_test = load_llff_data(args.datadir, args.factor, 187 | recenter=True, bd_factor=.75, 188 | spherify=args.spherify) 189 | H, W, focal = poses[0,:3,-1].astype(np.float32) 190 | poses = poses[:, :3, :4] 191 | H = int(H) 192 | W = int(W) 193 | hwf = [H, W, focal] 194 | 195 | images = images.astype(np.float32) 196 | poses = poses.astype(np.float32) 197 | 198 | near = 0. 199 | far = 1. 200 | 201 | if not isinstance(i_test, list): 202 | i_test = [i_test] 203 | 204 | if args.llffhold > 0: 205 | print('Auto LLFF holdout,', args.llffhold) 206 | i_test = np.arange(images.shape[0])[::args.llffhold] 207 | 208 | _, render_kwargs, start, grad_vars, models = run_nerf.create_nerf(args) 209 | to_use = i_test[0] 210 | 211 | bds_dict = { 212 | 'near' : tf.cast(near, tf.float32), 213 | 'far' : tf.cast(far, tf.float32), 214 | } 215 | render_kwargs.update(bds_dict) 216 | 217 | print('Render kwargs:') 218 | pprint.pprint(render_kwargs) 219 | 220 | res_dir = "./cloud_size_test" 221 | 222 | res = {} 223 | res['cloud_size'] = [] 224 | res['mse'] = [] 225 | res['psnr'] = [] 226 | res['time'] = [] 227 | 228 | for i in [1,2,4,8,16,32]: 229 | print(f'Running with cloud downsampled {i}x') 230 | start_time = time.time() 231 | ret_vals = run_nerf.render(H, W, focal, c2w=poses[to_use], pc=True, cloudsize=i, **render_kwargs) 232 | end_time = time.time() 233 | img = np.clip(ret_vals[0],0,1) 234 | mse = run_nerf.img2mse(images[to_use], img) 235 | psnr = run_nerf.mse2psnr(mse) 236 | res['cloud_size'].append((17 * H * W) // (i * i)) 237 | res['mse'].append(float(mse)) 238 | res['psnr'].append(float(psnr)) 239 | res['time'].append(end_time - start_time) 240 | 241 | # a = [1,2,4,8,16,32] 242 | # b = [1/x for x in a] 243 | 244 | # make plots 245 | # cs vs psnr 246 | fig, ax = plt.subplots(1,1) 247 | fig.suptitle('PSNR vs Point Cloud Size') 248 | ax.set_xlabel('Cloud Size') 249 | ax.set_ylabel('PSNR') 250 | plt.xscale('log') 251 | ax.plot(res['cloud_size'],res['psnr']) 252 | plt.savefig(os.path.join(res_dir, 'cs_psnr.png')) 253 | 254 | fig, ax = plt.subplots(1,1) 255 | fig.suptitle('PSNR vs Running Time') 256 | ax.set_xlabel('Time') 257 | ax.set_ylabel('PSNR') 258 | plt.xscale('log') 259 | ax.plot(res['time'],res['psnr']) 260 | plt.savefig(os.path.join(res_dir, 'time_psnr.png')) 261 | 262 | fig, ax = plt.subplots(1,1) 263 | fig.suptitle('Running Time vs Cloud Size') 264 | ax.set_xlabel('Cloud Size') 265 | ax.set_ylabel('Running Time') 266 | plt.xscale('log') 267 | plt.yscale('log') 268 | ax.plot(res['cloud_size'],res['time']) 269 | plt.savefig(os.path.join(res_dir, 'cs_time.png')) 270 | 271 | with open(os.path.join(res_dir, 'results.txt'), 'w') as outfile: 272 | json.dump(res,outfile) 273 | 274 | 275 | if __name__ == "__main__": 276 | parser = argparse.ArgumentParser(description='Test the point cloud method vs vanilla NeRF') 277 | parser.add_argument('--generate', action='store_true', help='generate data') 278 | parser.add_argument('--cloudsize', action='store_true', help='compare gains from larger cloud size') 279 | parser.add_argument('--plot', action='store_true', help='plot existing data') 280 | args = parser.parse_args() 281 | 282 | if args.generate: 283 | get_data() 284 | 285 | if args.cloudsize: 286 | cloud_size_vs_performance() 287 | 288 | if args.plot: 289 | plot_data() -------------------------------------------------------------------------------- /render_demo.py: -------------------------------------------------------------------------------- 1 | import os, sys 2 | # os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true' 3 | os.environ['CUDA_VISIBLE_DEVICES'] = '1' 4 | import tensorflow as tf 5 | tf.compat.v1.enable_eager_execution() 6 | 7 | import numpy as np 8 | import imageio 9 | import json 10 | import random 11 | import time 12 | import pprint 13 | import argparse 14 | 15 | import matplotlib.pyplot as plt 16 | 17 | import run_nerf 18 | import run_nerf_fast 19 | import point_cloud 20 | 21 | from load_llff import load_llff_data 22 | from load_deepvoxels import load_dv_data 23 | from load_blender import load_blender_data 24 | 25 | # profiling 26 | import cProfile 27 | import pstats 28 | from pstats import SortKey 29 | 30 | # CELL 1 31 | 32 | basedir = './logs' 33 | expname = 'fern_example' 34 | # expname = 'fern_test' 35 | 36 | config = os.path.join(basedir, expname, 'config.txt') 37 | print('Args:') 38 | print(open(config, 'r').read()) 39 | parser = run_nerf.config_parser() 40 | 41 | weights_name = 'model_200000.npy' 42 | # weights_name = 'model_000700.npy' 43 | args = parser.parse_args('--config {} --ft_path {}'.format(config, os.path.join(basedir, expname, weights_name))) 44 | print('loaded args') 45 | 46 | images, poses, bds, render_poses, i_test = load_llff_data(args.datadir, args.factor, 47 | recenter=True, bd_factor=.75, 48 | spherify=args.spherify) 49 | H, W, focal = poses[0,:3,-1].astype(np.float32) 50 | poses = poses[:, :3, :4] 51 | H = int(H) 52 | W = int(W) 53 | hwf = [H, W, focal] 54 | 55 | images = images.astype(np.float32) 56 | poses = poses.astype(np.float32) 57 | 58 | if args.no_ndc: 59 | near = tf.reduce_min(bds) * .9 60 | far = tf.reduce_max(bds) * 1. 61 | else: 62 | near = 0. 63 | far = 1. 64 | 65 | 66 | # CELL 2 67 | 68 | # Create nerf model 69 | 70 | def new_render(pc=False,sparse=False, down=16): 71 | _, render_kwargs_test, start, grad_vars, models = run_nerf.create_nerf(args) 72 | 73 | bds_dict = { 74 | 'near' : tf.cast(near, tf.float32), 75 | 'far' : tf.cast(far, tf.float32), 76 | } 77 | render_kwargs_test.update(bds_dict) 78 | 79 | print('Render kwargs:') 80 | pprint.pprint(render_kwargs_test) 81 | 82 | 83 | render_kwargs_fast = {k : render_kwargs_test[k] for k in render_kwargs_test} 84 | render_kwargs_fast['N_importance'] = 128 85 | 86 | # c2w = np.eye(4)[:3,:4].astype(np.float32) # identity pose matrix 87 | if not sparse: 88 | test = run_nerf.render(H//down, W//down, focal/down, c2w=poses[0], pc=pc, cloudsize=16, **render_kwargs_fast) 89 | else: 90 | test = run_nerf_fast.render(H//down, W//down, focal/down, c2w=poses[0], **render_kwargs_fast) 91 | img = np.clip(test[0],0,1) 92 | disp = test[1] 93 | disp = (disp - np.min(disp)) / (np.max(disp) - np.min(disp)) 94 | return img,disp 95 | 96 | 97 | # # profile rendering 98 | # cProfile.run('img,disp = new_render()', 'render_stats') 99 | 100 | # # show results 101 | # plt.imshow(img) 102 | # plt.show() 103 | 104 | # # profiling results 105 | # p = pstats.Stats('render_stats') 106 | # p.sort_stats(SortKey.CUMULATIVE).print_stats(20) 107 | 108 | if __name__ == "__main__": 109 | parser2 = argparse.ArgumentParser(description='Show FastNeRF rendering in action') 110 | parser2.add_argument('--vanilla', action='store_true', help='renders using vanilla nerf') 111 | parser2.add_argument('--cloud', action='store_true', help='renders by limiting the depth range using a point cloud') 112 | parser2.add_argument('--interpolate', action='store_true', help='renders by sparsely (1/9) sampling the first pass and interpolating pdfs') 113 | parser2.add_argument('--cloud_viz', action='store_true', help='Visualizes the point cloud and shows depth lower and upper bounds') 114 | parser2.add_argument('--down', type=int, default=12, help='Downsampling factor for the novel view resolution. 16 recommended unless on GPU') 115 | args2 = parser2.parse_args() 116 | 117 | down = 16 118 | if args2.down is not None: 119 | down = args2.down 120 | 121 | if args2.vanilla: 122 | img, disp = new_render(down=down) 123 | plt.imshow(img) 124 | plt.show() 125 | if args2.cloud: 126 | img, disp = new_render(pc=True, down=down) 127 | plt.imshow(img) 128 | plt.show() 129 | if args2.interpolate: 130 | img, disp = new_render(sparse=True, down=down) 131 | plt.imshow(img) 132 | plt.show() 133 | 134 | if args2.cloud_viz: 135 | point_cloud.depth_bounds([H//down,W//down,focal/down], poses[0], cloudsize=16, bounds=True, show_view=True) 136 | -------------------------------------------------------------------------------- /render_demo_fast.py: -------------------------------------------------------------------------------- 1 | import os, sys 2 | # os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true' 3 | os.environ['CUDA_VISIBLE_DEVICES'] = '1' 4 | import tensorflow as tf 5 | tf.compat.v1.enable_eager_execution() 6 | 7 | import numpy as np 8 | import imageio 9 | import json 10 | import random 11 | import time 12 | import pprint 13 | 14 | import matplotlib.pyplot as plt 15 | 16 | import run_nerf_fast 17 | 18 | from load_llff import load_llff_data 19 | from load_deepvoxels import load_dv_data 20 | from load_blender import load_blender_data 21 | 22 | # profiling 23 | import cProfile 24 | import pstats 25 | from pstats import SortKey 26 | 27 | # CELL 1 28 | 29 | basedir = './logs' 30 | expname = 'fern_example' 31 | # expname = 'fern_test' 32 | 33 | config = os.path.join(basedir, expname, 'config.txt') 34 | print('Args:') 35 | print(open(config, 'r').read()) 36 | parser = run_nerf_fast.config_parser() 37 | 38 | weights_name = 'model_200000.npy' 39 | # weights_name = 'model_000700.npy' 40 | args = parser.parse_args('--config {} --ft_path {}'.format(config, os.path.join(basedir, expname, weights_name))) 41 | print('loaded args') 42 | 43 | images, poses, bds, render_poses, i_test = load_llff_data(args.datadir, args.factor, 44 | recenter=True, bd_factor=.75, 45 | spherify=args.spherify) 46 | H, W, focal = poses[0,:3,-1].astype(np.float32) 47 | poses = poses[:, :3, :4] 48 | print(f"NUM IMAGES: {poses.shape[0]}") 49 | 50 | H = int(H) 51 | W = int(W) 52 | hwf = [H, W, focal] 53 | 54 | images = images.astype(np.float32) 55 | poses = poses.astype(np.float32) 56 | 57 | if args.no_ndc: 58 | near = tf.reduce_min(bds) * .9 59 | far = tf.reduce_max(bds) * 1. 60 | else: 61 | near = 0. 62 | far = 1. 63 | 64 | # get the test and train sets 65 | if not isinstance(i_test, list): 66 | i_test = [i_test] 67 | 68 | if args.llffhold > 0: 69 | print('Auto LLFF holdout,', args.llffhold) 70 | i_test = np.arange(images.shape[0])[::args.llffhold] 71 | 72 | i_train = np.array([i for i in np.arange(int(images.shape[0])) if 73 | (i not in i_test)]) 74 | 75 | 76 | # CELL 2 77 | 78 | # Create nerf model 79 | 80 | def new_render(): 81 | _, render_kwargs_test, start, grad_vars, models = run_nerf_fast.create_nerf(args) 82 | 83 | bds_dict = { 84 | 'near' : tf.cast(near, tf.float32), 85 | 'far' : tf.cast(far, tf.float32), 86 | } 87 | render_kwargs_test.update(bds_dict) 88 | 89 | print('Render kwargs:') 90 | pprint.pprint(render_kwargs_test) 91 | 92 | 93 | down = 4 94 | render_kwargs_fast = {k : render_kwargs_test[k] for k in render_kwargs_test} 95 | render_kwargs_fast['N_importance'] = 128 96 | 97 | # c2w = np.eye(4)[:3,:4].astype(np.float32) # identity pose matrix 98 | c2w = poses[0] 99 | test = run_nerf_fast.render(H//down, W//down, focal/down, c2w=c2w, **render_kwargs_fast) 100 | print(test) 101 | img = np.clip(test[0],0,1) 102 | disp = test[1] 103 | disp = (disp - np.min(disp)) / (np.max(disp) - np.min(disp)) 104 | acc_alpha = test[2] 105 | return img,disp,acc_alpha 106 | 107 | 108 | # profile rendering 109 | cProfile.run('img,disp, acc_alpha = new_render()', 'render_stats') 110 | # img, disp,acc_alpha = new_render() 111 | 112 | # show results 113 | plt.imshow(img) 114 | plt.show() 115 | 116 | # profiling results 117 | p = pstats.Stats('render_stats') 118 | p.sort_stats(SortKey.CUMULATIVE).print_stats(20) 119 | 120 | # CELL 3 121 | 122 | # down = 8 # trade off resolution+aliasing for render speed to make this video faster 123 | # frames = [] 124 | # for i, c2w in enumerate(render_poses): 125 | # if i%8==0: print(i) 126 | # test = run_nerf_fast.render(H//down, W//down, focal/down, c2w=c2w[:3,:4], **render_kwargs_fast) 127 | # frames.append((255*np.clip(test[0],0,1)).astype(np.uint8)) 128 | 129 | # print('done, saving') 130 | # f = 'logs/fern_example/video.mp4' 131 | # imageio.mimwrite(f, frames, fps=30, quality=8) 132 | 133 | # from IPython.display import Video 134 | # Video(f, height=320) 135 | 136 | # Cell 4 137 | 138 | # from ipywidgets import interactive, widgets 139 | # import matplotlib.pyplot as plt 140 | # import numpy as np 141 | 142 | 143 | # def f(x, y, z): 144 | 145 | # c2w = tf.convert_to_tensor([ 146 | # [1,0,0,x], 147 | # [0,1,0,y], 148 | # [0,0,1,z], 149 | # [0,0,0,1], 150 | # ], dtype=tf.float32) 151 | 152 | # test = run_nerf_fast.render(H//down, W//down, focal/down, c2w=c2w, **render_kwargs_fast) 153 | # img = np.clip(test[0],0,1) 154 | 155 | # plt.figure(2, figsize=(20,6)) 156 | # plt.imshow(img) 157 | # plt.show() 158 | 159 | 160 | # sldr = lambda : widgets.FloatSlider( 161 | # value=0., 162 | # min=-1., 163 | # max=1., 164 | # step=.01, 165 | # ) 166 | 167 | # names = ['x', 'y', 'z'] 168 | 169 | # interactive_plot = interactive(f, **{n : sldr() for n in names}) 170 | # interactive_plot -------------------------------------------------------------------------------- /render_stats: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/render_stats -------------------------------------------------------------------------------- /run_env.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | module load python/3.7.4 3 | module load anaconda/ 4 | 5 | source activate nerf -------------------------------------------------------------------------------- /run_nerf.py: -------------------------------------------------------------------------------- 1 | import os 2 | os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true' 3 | 4 | import sys 5 | import tensorflow as tf 6 | import numpy as np 7 | import imageio 8 | import json 9 | import random 10 | import time 11 | from run_nerf_helpers import * 12 | from point_cloud import * 13 | from load_llff import load_llff_data 14 | from load_deepvoxels import load_dv_data 15 | from load_blender import load_blender_data 16 | 17 | config = tf.ConfigProto() 18 | config.gpu_options.allow_growth = True 19 | sess = tf.Session(config=config) 20 | 21 | 22 | tf.compat.v1.enable_eager_execution() 23 | 24 | 25 | def batchify(fn, chunk): 26 | """Constructs a version of 'fn' that applies to smaller batches.""" 27 | if chunk is None: 28 | return fn 29 | 30 | def ret(inputs): 31 | return tf.concat([fn(inputs[i:i+chunk]) for i in range(0, inputs.shape[0], chunk)], 0) 32 | return ret 33 | 34 | 35 | def run_network(inputs, viewdirs, fn, embed_fn, embeddirs_fn, netchunk=1024*64): 36 | """Prepares inputs and applies network 'fn'.""" 37 | 38 | inputs_flat = tf.reshape(inputs, [-1, inputs.shape[-1]]) 39 | 40 | embedded = embed_fn(inputs_flat) 41 | if viewdirs is not None: 42 | input_dirs = tf.broadcast_to(viewdirs[:, None], inputs.shape) 43 | input_dirs_flat = tf.reshape(input_dirs, [-1, input_dirs.shape[-1]]) 44 | embedded_dirs = embeddirs_fn(input_dirs_flat) 45 | embedded = tf.concat([embedded, embedded_dirs], -1) 46 | 47 | outputs_flat = batchify(fn, netchunk)(embedded) 48 | outputs = tf.reshape(outputs_flat, list( 49 | inputs.shape[:-1]) + [outputs_flat.shape[-1]]) 50 | return outputs 51 | 52 | 53 | def render_rays(ray_batch, 54 | network_fn, 55 | network_query_fn, 56 | N_samples, 57 | retraw=False, 58 | lindisp=False, 59 | pc=False, 60 | perturb=0., 61 | N_importance=0, 62 | network_fine=None, 63 | white_bkgd=False, 64 | raw_noise_std=0., 65 | verbose=False): 66 | """Volumetric rendering. 67 | 68 | Args: 69 | ray_batch: array of shape [batch_size, ...]. All information necessary 70 | for sampling along a ray, including: ray origin, ray direction, min 71 | dist, max dist, and unit-magnitude viewing direction. 72 | network_fn: function. Model for predicting RGB and density at each point 73 | in space. 74 | network_query_fn: function used for passing queries to network_fn. 75 | N_samples: int. Number of different times to sample along each ray. 76 | retraw: bool. If True, include model's raw, unprocessed predictions. 77 | lindisp: bool. If True, sample linearly in inverse depth rather than in depth. 78 | perturb: float, 0 or 1. If non-zero, each ray is sampled at stratified 79 | random points in time. 80 | N_importance: int. Number of additional times to sample along each ray. 81 | These samples are only passed to network_fine. 82 | network_fine: "fine" network with same spec as network_fn. 83 | white_bkgd: bool. If True, assume a white background. 84 | raw_noise_std: ... 85 | verbose: bool. If True, print more debugging info. 86 | 87 | Returns: 88 | rgb_map: [num_rays, 3]. Estimated RGB color of a ray. Comes from fine model. 89 | disp_map: [num_rays]. Disparity map. 1 / depth. 90 | acc_map: [num_rays]. Accumulated opacity along each ray. Comes from fine model. 91 | raw: [num_rays, num_samples, 4]. Raw predictions from model. 92 | rgb0: See rgb_map. Output for coarse model. 93 | disp0: See disp_map. Output for coarse model. 94 | acc0: See acc_map. Output for coarse model. 95 | z_std: [num_rays]. Standard deviation of distances along ray for each 96 | sample. 97 | """ 98 | 99 | def raw2outputs(raw, z_vals, rays_d): 100 | """Transforms model's predictions to semantically meaningful values. 101 | 102 | Args: 103 | raw: [num_rays, num_samples along ray, 4]. Prediction from model. 104 | z_vals: [num_rays, num_samples along ray]. Integration time. 105 | rays_d: [num_rays, 3]. Direction of each ray. 106 | 107 | Returns: 108 | rgb_map: [num_rays, 3]. Estimated RGB color of a ray. 109 | disp_map: [num_rays]. Disparity map. Inverse of depth map. 110 | acc_map: [num_rays]. Sum of weights along each ray. 111 | weights: [num_rays, num_samples]. Weights assigned to each sampled color. 112 | depth_map: [num_rays]. Estimated distance to object. 113 | """ 114 | # Function for computing density from model prediction. This value is 115 | # strictly between [0, 1]. 116 | def raw2alpha(raw, dists, act_fn=tf.nn.relu): return 1.0 - \ 117 | tf.exp(-act_fn(raw) * dists) 118 | 119 | # Compute 'distance' (in time) between each integration time along a ray. 120 | dists = z_vals[..., 1:] - z_vals[..., :-1] 121 | 122 | # The 'distance' from the last integration time is infinity. 123 | dists = tf.concat( 124 | [dists, tf.broadcast_to([1e10], dists[..., :1].shape)], 125 | axis=-1) # [N_rays, N_samples] 126 | 127 | # Multiply each distance by the norm of its corresponding direction ray 128 | # to convert to real world distance (accounts for non-unit directions). 129 | dists = dists * tf.linalg.norm(rays_d[..., None, :], axis=-1) 130 | 131 | # Extract RGB of each sample position along each ray. 132 | rgb = tf.math.sigmoid(raw[..., :3]) # [N_rays, N_samples, 3] 133 | 134 | # Add noise to model's predictions for density. Can be used to 135 | # regularize network during training (prevents floater artifacts). 136 | noise = 0. 137 | if raw_noise_std > 0.: 138 | noise = tf.random.normal(raw[..., 3].shape) * raw_noise_std 139 | 140 | # Predict density of each sample along each ray. Higher values imply 141 | # higher likelihood of being absorbed at this point. 142 | alpha = raw2alpha(raw[..., 3] + noise, dists) # [N_rays, N_samples] 143 | 144 | # Compute weight for RGB of each sample along each ray. A cumprod() is 145 | # used to express the idea of the ray not having reflected up to this 146 | # sample yet. 147 | # [N_rays, N_samples] 148 | weights = alpha * \ 149 | tf.math.cumprod(1.-alpha + 1e-10, axis=-1, exclusive=True) 150 | 151 | # Computed weighted color of each sample along each ray. 152 | rgb_map = tf.reduce_sum( 153 | weights[..., None] * rgb, axis=-2) # [N_rays, 3] 154 | 155 | # Estimated depth map is expected distance. 156 | depth_map = tf.reduce_sum(weights * z_vals, axis=-1) 157 | 158 | # Disparity map is inverse depth. 159 | disp_map = 1./tf.maximum(1e-10, depth_map / 160 | tf.reduce_sum(weights, axis=-1)) 161 | 162 | # Sum of weights along each ray. This value is in [0, 1] up to numerical error. 163 | acc_map = tf.reduce_sum(weights, -1) 164 | 165 | # To composite onto a white background, use the accumulated alpha map. 166 | if white_bkgd: 167 | rgb_map = rgb_map + (1.-acc_map[..., None]) 168 | 169 | return rgb_map, disp_map, acc_map, weights, depth_map 170 | 171 | ############################### 172 | # batch size 173 | N_rays = ray_batch.shape[0] 174 | 175 | # Extract ray origin, direction. 176 | rays_o, rays_d = ray_batch[:, 0:3], ray_batch[:, 3:6] # [N_rays, 3] each 177 | 178 | # Extract unit-normalized viewing direction. 179 | viewdirs = ray_batch[:, -3:] if ray_batch.shape[-1] > 8 else None 180 | 181 | # Extract lower, upper bound for ray distance. 182 | bounds = tf.reshape(ray_batch[..., 6:8], [-1, 1, 2]) 183 | near, far = bounds[..., 0], bounds[..., 1] # [-1,1] 184 | 185 | # Decide where to sample along each ray. Under the logic, all rays will be sampled at 186 | # the same times. 187 | t_vals = tf.linspace(0., 1., N_samples) 188 | if not lindisp: 189 | # Space integration times linearly between 'near' and 'far'. Same 190 | # integration points will be used for all rays. 191 | z_vals = near * (1.-t_vals) + far * (t_vals) 192 | else: 193 | # Sample linearly in inverse depth (disparity). 194 | z_vals = 1./(1./near * (1.-t_vals) + 1./far * (t_vals)) 195 | z_vals = tf.broadcast_to(z_vals, [N_rays, N_samples]) 196 | # if pc: 197 | # z_vals = z_vals[:,1:-1] 198 | # first = tf.zeros((N_rays,1)) 199 | # last = tf.ones((N_rays,1)) 200 | # z_vals = tf.concat([first,z_vals,last], -1) 201 | 202 | # Perturb sampling time along each ray. 203 | if perturb > 0.: 204 | # get intervals between samples 205 | mids = .5 * (z_vals[..., 1:] + z_vals[..., :-1]) 206 | upper = tf.concat([mids, z_vals[..., -1:]], -1) 207 | lower = tf.concat([z_vals[..., :1], mids], -1) 208 | # stratified samples in those intervals 209 | t_rand = tf.random.uniform(z_vals.shape) 210 | z_vals = lower + (upper - lower) * t_rand 211 | 212 | # Points in space to evaluate model at. 213 | pts = rays_o[..., None, :] + rays_d[..., None, :] * \ 214 | z_vals[..., :, None] # [N_rays, N_samples, 3] 215 | 216 | # Evaluate model at each point. 217 | raw = network_query_fn(pts, viewdirs, network_fn) # [N_rays, N_samples, 4] 218 | rgb_map, disp_map, acc_map, weights, depth_map = raw2outputs( 219 | raw, z_vals, rays_d) 220 | 221 | if N_importance > 0: 222 | rgb_map_0, disp_map_0, acc_map_0 = rgb_map, disp_map, acc_map 223 | 224 | # Obtain additional integration times to evaluate based on the weights 225 | # assigned to colors in the coarse model. 226 | z_vals_mid = .5 * (z_vals[..., 1:] + z_vals[..., :-1]) 227 | z_samples = sample_pdf( 228 | z_vals_mid, weights[..., 1:-1], N_importance, det=(perturb == 0.)) 229 | z_samples = tf.stop_gradient(z_samples) 230 | 231 | # Obtain all points to evaluate color, density at. 232 | z_vals = tf.sort(tf.concat([z_vals, z_samples], -1), -1) 233 | pts = rays_o[..., None, :] + rays_d[..., None, :] * \ 234 | z_vals[..., :, None] # [N_rays, N_samples + N_importance, 3] 235 | 236 | # Make predictions with network_fine. 237 | run_fn = network_fn if network_fine is None else network_fine 238 | raw = network_query_fn(pts, viewdirs, run_fn) 239 | rgb_map, disp_map, acc_map, weights, depth_map = raw2outputs( 240 | raw, z_vals, rays_d) 241 | 242 | ret = {'rgb_map': rgb_map, 'disp_map': disp_map, 'acc_map': acc_map, 'depth_map': depth_map} 243 | if retraw: 244 | ret['raw'] = raw 245 | if N_importance > 0: 246 | ret['rgb0'] = rgb_map_0 247 | ret['disp0'] = disp_map_0 248 | ret['acc0'] = acc_map_0 249 | ret['z_std'] = tf.math.reduce_std(z_samples, -1) # [N_rays] 250 | 251 | for k in ret: 252 | tf.debugging.check_numerics(ret[k], 'output {}'.format(k)) 253 | 254 | return ret 255 | 256 | 257 | def batchify_rays(rays_flat, chunk=1024*32,pc=False, **kwargs): 258 | """Render rays in smaller minibatches to avoid OOM.""" 259 | all_ret = {} 260 | for i in range(0, rays_flat.shape[0], chunk): 261 | ret = render_rays(rays_flat[i:i+chunk], pc=pc, **kwargs) 262 | for k in ret: 263 | if k not in all_ret: 264 | all_ret[k] = [] 265 | all_ret[k].append(ret[k]) 266 | 267 | all_ret = {k: tf.concat(all_ret[k], 0) for k in all_ret} 268 | return all_ret 269 | 270 | 271 | def render(H, W, focal, 272 | chunk=1024*32, rays=None, c2w=None, ndc=True, 273 | near=0., far=1., pc=False, cloudsize=1, 274 | use_viewdirs=False, c2w_staticcam=None, 275 | **kwargs): 276 | """Render rays 277 | 278 | Args: 279 | H: int. Height of image in pixels. 280 | W: int. Width of image in pixels. 281 | focal: float. Focal length of pinhole camera. 282 | chunk: int. Maximum number of rays to process simultaneously. Used to 283 | control maximum memory usage. Does not affect final results. 284 | rays: array of shape [2, batch_size, 3]. Ray origin and direction for 285 | each example in batch. 286 | c2w: array of shape [3, 4]. Camera-to-world transformation matrix. 287 | ndc: bool. If True, represent ray origin, direction in NDC coordinates. 288 | near: float or array of shape [batch_size]. Nearest distance for a ray. 289 | far: float or array of shape [batch_size]. Farthest distance for a ray. 290 | use_viewdirs: bool. If True, use viewing direction of a point in space in model. 291 | c2w_staticcam: array of shape [3, 4]. If not None, use this transformation matrix for 292 | camera while using other c2w argument for viewing directions. 293 | 294 | Returns: 295 | rgb_map: [batch_size, 3]. Predicted RGB values for rays. 296 | disp_map: [batch_size]. Disparity map. Inverse of depth. 297 | acc_map: [batch_size]. Accumulated opacity (alpha) along a ray. 298 | extras: dict with everything returned by render_rays(). 299 | """ 300 | 301 | if c2w is not None: 302 | # special case to render full image 303 | rays_o, rays_d = get_rays(H, W, focal, c2w) 304 | else: 305 | # use provided ray batch 306 | rays_o, rays_d = rays 307 | 308 | if use_viewdirs: 309 | # provide ray directions as input 310 | viewdirs = rays_d 311 | if c2w_staticcam is not None: 312 | # special case to visualize effect of viewdirs 313 | rays_o, rays_d = get_rays(H, W, focal, c2w_staticcam) 314 | 315 | # Make all directions unit magnitude. 316 | # shape: [batch_size, 3] 317 | viewdirs = viewdirs / tf.linalg.norm(viewdirs, axis=-1, keepdims=True) 318 | viewdirs = tf.cast(tf.reshape(viewdirs, [-1, 3]), dtype=tf.float32) 319 | 320 | sh = rays_d.shape # [..., 3] 321 | if ndc: 322 | # for forward facing scenes 323 | rays_o, rays_d = ndc_rays( 324 | H, W, focal, tf.cast(1., tf.float32), rays_o, rays_d) 325 | 326 | # Create ray batch 327 | rays_o = tf.cast(tf.reshape(rays_o, [-1, 3]), dtype=tf.float32) 328 | rays_d = tf.cast(tf.reshape(rays_d, [-1, 3]), dtype=tf.float32) 329 | near, far = near * \ 330 | tf.ones_like(rays_d[..., :1]), far * tf.ones_like(rays_d[..., :1]) 331 | if pc: 332 | near, far = depth_bounds([H, W, focal], c2w, cloudsize=cloudsize) 333 | near = tf.cast(tf.reshape(near, (-1,1)), tf.float32) 334 | far = tf.cast(tf.reshape(far, (-1,1)), tf.float32) 335 | 336 | # (ray origin, ray direction, min dist, max dist) for each ray 337 | rays = tf.concat([rays_o, rays_d, near, far], axis=-1) 338 | if use_viewdirs: 339 | # (ray origin, ray direction, min dist, max dist, normalized viewing direction) 340 | rays = tf.concat([rays, viewdirs], axis=-1) 341 | 342 | # Render and reshape 343 | all_ret = batchify_rays(rays, chunk, pc=pc, **kwargs) 344 | for k in all_ret: 345 | k_sh = list(sh[:-1]) + list(all_ret[k].shape[1:]) 346 | all_ret[k] = tf.reshape(all_ret[k], k_sh) 347 | 348 | k_extract = ['rgb_map', 'disp_map', 'acc_map'] 349 | ret_list = [all_ret[k] for k in k_extract] 350 | ret_dict = {k: all_ret[k] for k in all_ret if k not in k_extract} 351 | return ret_list + [ret_dict] 352 | 353 | 354 | def render_path(render_poses, hwf, chunk, render_kwargs, gt_imgs=None, savedir=None, render_factor=0): 355 | 356 | H, W, focal = hwf 357 | 358 | if render_factor != 0: 359 | # Render downsampled for speed 360 | H = H//render_factor 361 | W = W//render_factor 362 | focal = focal/render_factor 363 | 364 | rgbs = [] 365 | disps = [] 366 | 367 | t = time.time() 368 | for i, c2w in enumerate(render_poses): 369 | print(i, time.time() - t) 370 | t = time.time() 371 | rgb, disp, acc, _ = render( 372 | H, W, focal, chunk=chunk, c2w=c2w[:3, :4], **render_kwargs) 373 | rgbs.append(rgb.numpy()) 374 | disps.append(disp.numpy()) 375 | if i == 0: 376 | print(rgb.shape, disp.shape) 377 | 378 | if gt_imgs is not None and render_factor == 0: 379 | p = -10. * np.log10(np.mean(np.square(rgb - gt_imgs[i]))) 380 | print(p) 381 | 382 | if savedir is not None: 383 | rgb8 = to8b(rgbs[-1]) 384 | filename = os.path.join(savedir, '{:03d}.png'.format(i)) 385 | imageio.imwrite(filename, rgb8) 386 | 387 | rgbs = np.stack(rgbs, 0) 388 | disps = np.stack(disps, 0) 389 | 390 | return rgbs, disps 391 | 392 | 393 | def create_nerf(args): 394 | """Instantiate NeRF's MLP model.""" 395 | 396 | embed_fn, input_ch = get_embedder(args.multires, args.i_embed) 397 | 398 | input_ch_views = 0 399 | embeddirs_fn = None 400 | if args.use_viewdirs: 401 | embeddirs_fn, input_ch_views = get_embedder( 402 | args.multires_views, args.i_embed) 403 | output_ch = 4 404 | skips = [4] 405 | model = init_nerf_model( 406 | D=args.netdepth, W=args.netwidth, 407 | input_ch=input_ch, output_ch=output_ch, skips=skips, 408 | input_ch_views=input_ch_views, use_viewdirs=args.use_viewdirs) 409 | grad_vars = model.trainable_variables 410 | models = {'model': model} 411 | 412 | model_fine = None 413 | if args.N_importance > 0: 414 | model_fine = init_nerf_model( 415 | D=args.netdepth_fine, W=args.netwidth_fine, 416 | input_ch=input_ch, output_ch=output_ch, skips=skips, 417 | input_ch_views=input_ch_views, use_viewdirs=args.use_viewdirs) 418 | grad_vars += model_fine.trainable_variables 419 | models['model_fine'] = model_fine 420 | 421 | def network_query_fn(inputs, viewdirs, network_fn): return run_network( 422 | inputs, viewdirs, network_fn, 423 | embed_fn=embed_fn, 424 | embeddirs_fn=embeddirs_fn, 425 | netchunk=args.netchunk) 426 | 427 | render_kwargs_train = { 428 | 'network_query_fn': network_query_fn, 429 | 'perturb': args.perturb, 430 | 'N_importance': args.N_importance, 431 | 'network_fine': model_fine, 432 | 'N_samples': args.N_samples, 433 | 'network_fn': model, 434 | 'use_viewdirs': args.use_viewdirs, 435 | 'white_bkgd': args.white_bkgd, 436 | 'raw_noise_std': args.raw_noise_std, 437 | } 438 | 439 | # NDC only good for LLFF-style forward facing data 440 | if args.dataset_type != 'llff' or args.no_ndc: 441 | print('Not ndc!') 442 | render_kwargs_train['ndc'] = False 443 | render_kwargs_train['lindisp'] = args.lindisp 444 | 445 | render_kwargs_test = { 446 | k: render_kwargs_train[k] for k in render_kwargs_train} 447 | render_kwargs_test['perturb'] = False 448 | render_kwargs_test['raw_noise_std'] = 0. 449 | 450 | start = 0 451 | basedir = args.basedir 452 | expname = args.expname 453 | 454 | if args.ft_path is not None and args.ft_path != 'None': 455 | ckpts = [args.ft_path] 456 | else: 457 | ckpts = [os.path.join(basedir, expname, f) for f in sorted(os.listdir(os.path.join(basedir, expname))) if 458 | ('model_' in f and 'fine' not in f and 'optimizer' not in f)] 459 | print('Found ckpts', ckpts) 460 | if len(ckpts) > 0 and not args.no_reload: 461 | ft_weights = ckpts[-1] 462 | print('Reloading from', ft_weights) 463 | model.set_weights(np.load(ft_weights, allow_pickle=True)) 464 | start = int(ft_weights[-10:-4]) + 1 465 | print('Resetting step to', start) 466 | 467 | if model_fine is not None: 468 | ft_weights_fine = '{}_fine_{}'.format( 469 | ft_weights[:-11], ft_weights[-10:]) 470 | print('Reloading fine from', ft_weights_fine) 471 | model_fine.set_weights(np.load(ft_weights_fine, allow_pickle=True)) 472 | 473 | return render_kwargs_train, render_kwargs_test, start, grad_vars, models 474 | 475 | 476 | def config_parser(): 477 | 478 | import configargparse 479 | parser = configargparse.ArgumentParser() 480 | parser.add_argument('--config', is_config_file=True, 481 | help='config file path') 482 | parser.add_argument("--expname", type=str, help='experiment name') 483 | parser.add_argument("--basedir", type=str, default='./logs/', 484 | help='where to store ckpts and logs') 485 | parser.add_argument("--datadir", type=str, 486 | default='./data/llff/fern', help='input data directory') 487 | 488 | # training options 489 | parser.add_argument("--netdepth", type=int, default=8, 490 | help='layers in network') 491 | parser.add_argument("--netwidth", type=int, default=256, 492 | help='channels per layer') 493 | parser.add_argument("--netdepth_fine", type=int, 494 | default=8, help='layers in fine network') 495 | parser.add_argument("--netwidth_fine", type=int, default=256, 496 | help='channels per layer in fine network') 497 | parser.add_argument("--N_rand", type=int, default=32*32*4, 498 | help='batch size (number of random rays per gradient step)') 499 | parser.add_argument("--lrate", type=float, 500 | default=5e-4, help='learning rate') 501 | parser.add_argument("--lrate_decay", type=int, default=250, 502 | help='exponential learning rate decay (in 1000s)') 503 | parser.add_argument("--chunk", type=int, default=1024*32, 504 | help='number of rays processed in parallel, decrease if running out of memory') 505 | parser.add_argument("--netchunk", type=int, default=1024*64, 506 | help='number of pts sent through network in parallel, decrease if running out of memory') 507 | parser.add_argument("--no_batching", action='store_true', 508 | help='only take random rays from 1 image at a time') 509 | parser.add_argument("--no_reload", action='store_true', 510 | help='do not reload weights from saved ckpt') 511 | parser.add_argument("--ft_path", type=str, default=None, 512 | help='specific weights npy file to reload for coarse network') 513 | parser.add_argument("--random_seed", type=int, default=None, 514 | help='fix random seed for repeatability') 515 | 516 | # pre-crop options 517 | parser.add_argument("--precrop_iters", type=int, default=0, 518 | help='number of steps to train on central crops') 519 | parser.add_argument("--precrop_frac", type=float, 520 | default=.5, help='fraction of img taken for central crops') 521 | 522 | # rendering options 523 | parser.add_argument("--N_samples", type=int, default=64, 524 | help='number of coarse samples per ray') 525 | parser.add_argument("--N_importance", type=int, default=0, 526 | help='number of additional fine samples per ray') 527 | parser.add_argument("--perturb", type=float, default=1., 528 | help='set to 0. for no jitter, 1. for jitter') 529 | parser.add_argument("--use_viewdirs", action='store_true', 530 | help='use full 5D input instead of 3D') 531 | parser.add_argument("--i_embed", type=int, default=0, 532 | help='set 0 for default positional encoding, -1 for none') 533 | parser.add_argument("--multires", type=int, default=10, 534 | help='log2 of max freq for positional encoding (3D location)') 535 | parser.add_argument("--multires_views", type=int, default=4, 536 | help='log2 of max freq for positional encoding (2D direction)') 537 | parser.add_argument("--raw_noise_std", type=float, default=0., 538 | help='std dev of noise added to regularize sigma_a output, 1e0 recommended') 539 | 540 | parser.add_argument("--render_only", action='store_true', 541 | help='do not optimize, reload weights and render out render_poses path') 542 | parser.add_argument("--render_test", action='store_true', 543 | help='render the test set instead of render_poses path') 544 | parser.add_argument("--render_factor", type=int, default=0, 545 | help='downsampling factor to speed up rendering, set 4 or 8 for fast preview') 546 | 547 | # dataset options 548 | parser.add_argument("--dataset_type", type=str, default='llff', 549 | help='options: llff / blender / deepvoxels') 550 | parser.add_argument("--testskip", type=int, default=8, 551 | help='will load 1/N images from test/val sets, useful for large datasets like deepvoxels') 552 | 553 | # deepvoxels flags 554 | parser.add_argument("--shape", type=str, default='greek', 555 | help='options : armchair / cube / greek / vase') 556 | 557 | # blender flags 558 | parser.add_argument("--white_bkgd", action='store_true', 559 | help='set to render synthetic data on a white bkgd (always use for dvoxels)') 560 | parser.add_argument("--half_res", action='store_true', 561 | help='load blender synthetic data at 400x400 instead of 800x800') 562 | 563 | # llff flags 564 | parser.add_argument("--factor", type=int, default=8, 565 | help='downsample factor for LLFF images') 566 | parser.add_argument("--no_ndc", action='store_true', 567 | help='do not use normalized device coordinates (set for non-forward facing scenes)') 568 | parser.add_argument("--lindisp", action='store_true', 569 | help='sampling linearly in disparity rather than depth') 570 | parser.add_argument("--spherify", action='store_true', 571 | help='set for spherical 360 scenes') 572 | parser.add_argument("--llffhold", type=int, default=8, 573 | help='will take every 1/N images as LLFF test set, paper uses 8') 574 | 575 | # logging/saving options 576 | parser.add_argument("--i_print", type=int, default=100, 577 | help='frequency of console printout and metric loggin') 578 | parser.add_argument("--i_img", type=int, default=500, 579 | help='frequency of tensorboard image logging') 580 | parser.add_argument("--i_weights", type=int, default=10000, 581 | help='frequency of weight ckpt saving') 582 | parser.add_argument("--i_testset", type=int, default=50000, 583 | help='frequency of testset saving') 584 | parser.add_argument("--i_video", type=int, default=50000, 585 | help='frequency of render_poses video saving') 586 | 587 | return parser 588 | 589 | 590 | def train(): 591 | 592 | parser = config_parser() 593 | args = parser.parse_args() 594 | 595 | if args.random_seed is not None: 596 | print('Fixing random seed', args.random_seed) 597 | np.random.seed(args.random_seed) 598 | tf.compat.v1.set_random_seed(args.random_seed) 599 | 600 | # Load data 601 | 602 | if args.dataset_type == 'llff': 603 | images, poses, bds, render_poses, i_test = load_llff_data(args.datadir, args.factor, 604 | recenter=True, bd_factor=.75, 605 | spherify=args.spherify) 606 | hwf = poses[0, :3, -1] 607 | poses = poses[:, :3, :4] 608 | print('Loaded llff', images.shape, 609 | render_poses.shape, hwf, args.datadir) 610 | if not isinstance(i_test, list): 611 | i_test = [i_test] 612 | 613 | if args.llffhold > 0: 614 | print('Auto LLFF holdout,', args.llffhold) 615 | i_test = np.arange(images.shape[0])[::args.llffhold] 616 | 617 | i_val = i_test 618 | i_train = np.array([i for i in np.arange(int(images.shape[0])) if 619 | (i not in i_test and i not in i_val)]) 620 | 621 | print('DEFINING BOUNDS') 622 | if args.no_ndc: 623 | near = tf.reduce_min(bds) * .9 624 | far = tf.reduce_max(bds) * 1. 625 | else: 626 | near = 0. 627 | far = 1. 628 | print('NEAR FAR', near, far) 629 | 630 | elif args.dataset_type == 'blender': 631 | images, poses, render_poses, hwf, i_split = load_blender_data( 632 | args.datadir, args.half_res, args.testskip) 633 | print('Loaded blender', images.shape, 634 | render_poses.shape, hwf, args.datadir) 635 | i_train, i_val, i_test = i_split 636 | 637 | near = 2. 638 | far = 6. 639 | 640 | if args.white_bkgd: 641 | images = images[..., :3]*images[..., -1:] + (1.-images[..., -1:]) 642 | else: 643 | images = images[..., :3] 644 | 645 | elif args.dataset_type == 'deepvoxels': 646 | 647 | images, poses, render_poses, hwf, i_split = load_dv_data(scene=args.shape, 648 | basedir=args.datadir, 649 | testskip=args.testskip) 650 | 651 | print('Loaded deepvoxels', images.shape, 652 | render_poses.shape, hwf, args.datadir) 653 | i_train, i_val, i_test = i_split 654 | 655 | hemi_R = np.mean(np.linalg.norm(poses[:, :3, -1], axis=-1)) 656 | near = hemi_R-1. 657 | far = hemi_R+1. 658 | 659 | else: 660 | print('Unknown dataset type', args.dataset_type, 'exiting') 661 | return 662 | 663 | # Cast intrinsics to right types 664 | H, W, focal = hwf 665 | H, W = int(H), int(W) 666 | hwf = [H, W, focal] 667 | 668 | if args.render_test: 669 | render_poses = np.array(poses[i_test]) 670 | 671 | # Create log dir and copy the config file 672 | basedir = args.basedir 673 | expname = args.expname 674 | os.makedirs(os.path.join(basedir, expname), exist_ok=True) 675 | f = os.path.join(basedir, expname, 'args.txt') 676 | with open(f, 'w') as file: 677 | for arg in sorted(vars(args)): 678 | attr = getattr(args, arg) 679 | file.write('{} = {}\n'.format(arg, attr)) 680 | if args.config is not None: 681 | f = os.path.join(basedir, expname, 'config.txt') 682 | with open(f, 'w') as file: 683 | file.write(open(args.config, 'r').read()) 684 | 685 | # Create nerf model 686 | render_kwargs_train, render_kwargs_test, start, grad_vars, models = create_nerf( 687 | args) 688 | 689 | bds_dict = { 690 | 'near': tf.cast(near, tf.float32), 691 | 'far': tf.cast(far, tf.float32), 692 | } 693 | render_kwargs_train.update(bds_dict) 694 | render_kwargs_test.update(bds_dict) 695 | 696 | # Short circuit if only rendering out from trained model 697 | if args.render_only: 698 | print('RENDER ONLY') 699 | if args.render_test: 700 | # render_test switches to test poses 701 | images = images[i_test] 702 | else: 703 | # Default is smoother render_poses path 704 | images = None 705 | 706 | testsavedir = os.path.join(basedir, expname, 'renderonly_{}_{:06d}'.format( 707 | 'test' if args.render_test else 'path', start)) 708 | os.makedirs(testsavedir, exist_ok=True) 709 | print('test poses shape', render_poses.shape) 710 | 711 | rgbs, _ = render_path(render_poses, hwf, args.chunk, render_kwargs_test, 712 | gt_imgs=images, savedir=testsavedir, render_factor=args.render_factor) 713 | print('Done rendering', testsavedir) 714 | imageio.mimwrite(os.path.join(testsavedir, 'video.mp4'), 715 | to8b(rgbs), fps=30, quality=8) 716 | 717 | return 718 | 719 | # Create optimizer 720 | lrate = args.lrate 721 | if args.lrate_decay > 0: 722 | lrate = tf.keras.optimizers.schedules.ExponentialDecay(lrate, 723 | decay_steps=args.lrate_decay * 1000, decay_rate=0.1) 724 | optimizer = tf.keras.optimizers.Adam(lrate) 725 | models['optimizer'] = optimizer 726 | 727 | global_step = tf.compat.v1.train.get_or_create_global_step() 728 | global_step.assign(start) 729 | 730 | # Prepare raybatch tensor if batching random rays 731 | N_rand = args.N_rand 732 | use_batching = not args.no_batching 733 | if use_batching: 734 | # For random ray batching. 735 | # 736 | # Constructs an array 'rays_rgb' of shape [N*H*W, 3, 3] where axis=1 is 737 | # interpreted as, 738 | # axis=0: ray origin in world space 739 | # axis=1: ray direction in world space 740 | # axis=2: observed RGB color of pixel 741 | print('get rays') 742 | # get_rays_np() returns rays_origin=[H, W, 3], rays_direction=[H, W, 3] 743 | # for each pixel in the image. This stack() adds a new dimension. 744 | rays = [get_rays_np(H, W, focal, p) for p in poses[:, :3, :4]] 745 | rays = np.stack(rays, axis=0) # [N, ro+rd, H, W, 3] 746 | print('done, concats') 747 | # [N, ro+rd+rgb, H, W, 3] 748 | rays_rgb = np.concatenate([rays, images[:, None, ...]], 1) 749 | # [N, H, W, ro+rd+rgb, 3] 750 | rays_rgb = np.transpose(rays_rgb, [0, 2, 3, 1, 4]) 751 | rays_rgb = np.stack([rays_rgb[i] 752 | for i in i_train], axis=0) # train images only 753 | # [(N-1)*H*W, ro+rd+rgb, 3] 754 | rays_rgb = np.reshape(rays_rgb, [-1, 3, 3]) 755 | rays_rgb = rays_rgb.astype(np.float32) 756 | print('shuffle rays') 757 | np.random.shuffle(rays_rgb) 758 | print('done') 759 | i_batch = 0 760 | 761 | N_iters = 1000000 762 | print('Begin') 763 | print('TRAIN views are', i_train) 764 | print('TEST views are', i_test) 765 | print('VAL views are', i_val) 766 | 767 | # Summary writers 768 | writer = tf.contrib.summary.create_file_writer( 769 | os.path.join(basedir, 'summaries', expname)) 770 | writer.set_as_default() 771 | 772 | time0 = time.time() 773 | for i in range(start, N_iters): 774 | 775 | # Sample random ray batch 776 | 777 | if use_batching: 778 | # Random over all images 779 | batch = rays_rgb[i_batch:i_batch+N_rand] # [B, 2+1, 3*?] 780 | batch = tf.transpose(batch, [1, 0, 2]) 781 | 782 | # batch_rays[i, n, xyz] = ray origin or direction, example_id, 3D position 783 | # target_s[n, rgb] = example_id, observed color. 784 | batch_rays, target_s = batch[:2], batch[2] 785 | 786 | i_batch += N_rand 787 | if i_batch >= rays_rgb.shape[0]: 788 | np.random.shuffle(rays_rgb) 789 | i_batch = 0 790 | 791 | else: 792 | # Random from one image 793 | img_i = np.random.choice(i_train) 794 | target = images[img_i] 795 | pose = poses[img_i, :3, :4] 796 | 797 | if N_rand is not None: 798 | rays_o, rays_d = get_rays(H, W, focal, pose) 799 | if i < args.precrop_iters: 800 | dH = int(H//2 * args.precrop_frac) 801 | dW = int(W//2 * args.precrop_frac) 802 | coords = tf.stack(tf.meshgrid( 803 | tf.range(H//2 - dH, H//2 + dH), 804 | tf.range(W//2 - dW, W//2 + dW), 805 | indexing='ij'), -1) 806 | if i < 10: 807 | print('precrop', dH, dW, coords[0,0], coords[-1,-1]) 808 | else: 809 | coords = tf.stack(tf.meshgrid( 810 | tf.range(H), tf.range(W), indexing='ij'), -1) 811 | coords = tf.reshape(coords, [-1, 2]) 812 | select_inds = np.random.choice( 813 | coords.shape[0], size=[N_rand], replace=False) 814 | select_inds = tf.gather_nd(coords, select_inds[:, tf.newaxis]) 815 | rays_o = tf.gather_nd(rays_o, select_inds) 816 | rays_d = tf.gather_nd(rays_d, select_inds) 817 | batch_rays = tf.stack([rays_o, rays_d], 0) 818 | target_s = tf.gather_nd(target, select_inds) 819 | 820 | ##### Core optimization loop ##### 821 | 822 | with tf.GradientTape() as tape: 823 | 824 | # Make predictions for color, disparity, accumulated opacity. 825 | rgb, disp, acc, extras = render( 826 | H, W, focal, chunk=args.chunk, rays=batch_rays, 827 | verbose=i < 10, retraw=True, **render_kwargs_train) 828 | 829 | # Compute MSE loss between predicted and true RGB. 830 | img_loss = img2mse(rgb, target_s) 831 | trans = extras['raw'][..., -1] 832 | loss = img_loss 833 | psnr = mse2psnr(img_loss) 834 | 835 | # Add MSE loss for coarse-grained model 836 | if 'rgb0' in extras: 837 | img_loss0 = img2mse(extras['rgb0'], target_s) 838 | loss += img_loss0 839 | psnr0 = mse2psnr(img_loss0) 840 | 841 | gradients = tape.gradient(loss, grad_vars) 842 | optimizer.apply_gradients(zip(gradients, grad_vars)) 843 | 844 | dt = time.time()-time0 845 | 846 | ##### end ##### 847 | 848 | # Rest is logging 849 | 850 | def save_weights(net, prefix, i): 851 | path = os.path.join( 852 | basedir, expname, '{}_{:06d}.npy'.format(prefix, i)) 853 | np.save(path, net.get_weights()) 854 | print('saved weights at', path) 855 | 856 | if i % args.i_weights == 0: 857 | for k in models: 858 | save_weights(models[k], k, i) 859 | 860 | if i % args.i_video == 0 and i > 0: 861 | 862 | rgbs, disps = render_path( 863 | render_poses, hwf, args.chunk, render_kwargs_test) 864 | print('Done, saving', rgbs.shape, disps.shape) 865 | moviebase = os.path.join( 866 | basedir, expname, '{}_spiral_{:06d}_'.format(expname, i)) 867 | imageio.mimwrite(moviebase + 'rgb.mp4', 868 | to8b(rgbs), fps=30, quality=8) 869 | imageio.mimwrite(moviebase + 'disp.mp4', 870 | to8b(disps / np.max(disps)), fps=30, quality=8) 871 | 872 | if args.use_viewdirs: 873 | render_kwargs_test['c2w_staticcam'] = render_poses[0][:3, :4] 874 | rgbs_still, _ = render_path( 875 | render_poses, hwf, args.chunk, render_kwargs_test) 876 | render_kwargs_test['c2w_staticcam'] = None 877 | imageio.mimwrite(moviebase + 'rgb_still.mp4', 878 | to8b(rgbs_still), fps=30, quality=8) 879 | 880 | if i % args.i_testset == 0 and i > 0: 881 | testsavedir = os.path.join( 882 | basedir, expname, 'testset_{:06d}'.format(i)) 883 | os.makedirs(testsavedir, exist_ok=True) 884 | print('test poses shape', poses[i_test].shape) 885 | render_path(poses[i_test], hwf, args.chunk, render_kwargs_test, 886 | gt_imgs=images[i_test], savedir=testsavedir) 887 | print('Saved test set') 888 | 889 | if i % args.i_print == 0 or i < 10: 890 | 891 | print(expname, i, psnr.numpy(), loss.numpy(), global_step.numpy()) 892 | print('iter time {:.05f}'.format(dt)) 893 | with tf.contrib.summary.record_summaries_every_n_global_steps(args.i_print): 894 | tf.contrib.summary.scalar('loss', loss) 895 | tf.contrib.summary.scalar('psnr', psnr) 896 | tf.contrib.summary.histogram('tran', trans) 897 | if args.N_importance > 0: 898 | tf.contrib.summary.scalar('psnr0', psnr0) 899 | 900 | if i % args.i_img == 0: 901 | 902 | # Log a rendered validation view to Tensorboard 903 | img_i = np.random.choice(i_val) 904 | target = images[img_i] 905 | pose = poses[img_i, :3, :4] 906 | 907 | rgb, disp, acc, extras = render(H, W, focal, chunk=args.chunk, c2w=pose, 908 | **render_kwargs_test) 909 | 910 | psnr = mse2psnr(img2mse(rgb, target)) 911 | 912 | # Save out the validation image for Tensorboard-free monitoring 913 | testimgdir = os.path.join(basedir, expname, 'tboard_val_imgs') 914 | if i==0: 915 | os.makedirs(testimgdir, exist_ok=True) 916 | imageio.imwrite(os.path.join(testimgdir, '{:06d}.png'.format(i)), to8b(rgb)) 917 | 918 | with tf.contrib.summary.record_summaries_every_n_global_steps(args.i_img): 919 | 920 | tf.contrib.summary.image('rgb', to8b(rgb)[tf.newaxis]) 921 | tf.contrib.summary.image( 922 | 'disp', disp[tf.newaxis, ..., tf.newaxis]) 923 | tf.contrib.summary.image( 924 | 'acc', acc[tf.newaxis, ..., tf.newaxis]) 925 | 926 | tf.contrib.summary.scalar('psnr_holdout', psnr) 927 | tf.contrib.summary.image('rgb_holdout', target[tf.newaxis]) 928 | 929 | if args.N_importance > 0: 930 | 931 | with tf.contrib.summary.record_summaries_every_n_global_steps(args.i_img): 932 | tf.contrib.summary.image( 933 | 'rgb0', to8b(extras['rgb0'])[tf.newaxis]) 934 | tf.contrib.summary.image( 935 | 'disp0', extras['disp0'][tf.newaxis, ..., tf.newaxis]) 936 | tf.contrib.summary.image( 937 | 'z_std', extras['z_std'][tf.newaxis, ..., tf.newaxis]) 938 | 939 | global_step.assign_add(1) 940 | 941 | 942 | if __name__ == '__main__': 943 | train() 944 | 945 | -------------------------------------------------------------------------------- /run_nerf_fast.py: -------------------------------------------------------------------------------- 1 | import os 2 | os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true' 3 | 4 | import sys 5 | import tensorflow as tf 6 | import numpy as np 7 | import imageio 8 | import json 9 | import random 10 | import time 11 | from run_nerf_helpers_fast import * 12 | from point_cloud import * 13 | from load_llff import load_llff_data 14 | from load_deepvoxels import load_dv_data 15 | from load_blender import load_blender_data 16 | 17 | config = tf.ConfigProto() 18 | config.gpu_options.allow_growth = True 19 | sess = tf.Session(config=config) 20 | 21 | 22 | tf.compat.v1.enable_eager_execution() 23 | 24 | 25 | def batchify(fn, chunk): 26 | """Constructs a version of 'fn' that applies to smaller batches.""" 27 | if chunk is None: 28 | return fn 29 | 30 | def ret(inputs): 31 | return tf.concat([fn(inputs[i:i+chunk]) for i in range(0, inputs.shape[0], chunk)], 0) 32 | return ret 33 | 34 | 35 | def run_network(inputs, viewdirs, fn, embed_fn, embeddirs_fn, netchunk=1024*64): 36 | """Prepares inputs and applies network 'fn'.""" 37 | 38 | inputs_flat = tf.reshape(inputs, [-1, inputs.shape[-1]]) 39 | 40 | embedded = embed_fn(inputs_flat) 41 | if viewdirs is not None: 42 | input_dirs = tf.broadcast_to(viewdirs[:, None], inputs.shape) 43 | input_dirs_flat = tf.reshape(input_dirs, [-1, input_dirs.shape[-1]]) 44 | embedded_dirs = embeddirs_fn(input_dirs_flat) 45 | embedded = tf.concat([embedded, embedded_dirs], -1) 46 | 47 | outputs_flat = batchify(fn, netchunk)(embedded) 48 | outputs = tf.reshape(outputs_flat, list( 49 | inputs.shape[:-1]) + [outputs_flat.shape[-1]]) 50 | return outputs 51 | 52 | def raw2outputs(raw, z_vals, rays_d, raw_noise_std=0, white_bkgd=None): 53 | """Transforms model's predictions to semantically meaningful values. 54 | 55 | Args: 56 | raw: [num_rays, num_samples along ray, 4]. Prediction from model. 57 | z_vals: [num_rays, num_samples along ray]. Integration time. 58 | rays_d: [num_rays, 3]. Direction of each ray. 59 | 60 | Returns: 61 | rgb_map: [num_rays, 3]. Estimated RGB color of a ray. 62 | disp_map: [num_rays]. Disparity map. Inverse of depth map. 63 | acc_map: [num_rays]. Sum of weights along each ray. 64 | weights: [num_rays, num_samples]. Weights assigned to each sampled color. 65 | depth_map: [num_rays]. Estimated distance to object. 66 | """ 67 | # Function for computing density from model prediction. This value is 68 | # strictly between [0, 1]. 69 | def raw2alpha(raw, dists, act_fn=tf.nn.relu): return 1.0 - \ 70 | tf.exp(-act_fn(raw) * dists) 71 | 72 | # Compute 'distance' (in time) between each integration time along a ray. 73 | dists = z_vals[..., 1:] - z_vals[..., :-1] 74 | # The 'distance' from the last integration time is infinity. 75 | dists = tf.concat( 76 | [dists, tf.broadcast_to([1e10], dists[..., :1].shape)], 77 | axis=-1) # [N_rays, N_samples] 78 | 79 | # Multiply each distance by the norm of its corresponding direction ray 80 | # to convert to real world distance (accounts for non-unit directions). 81 | dists = dists * tf.linalg.norm(rays_d[..., None, :], axis=-1) 82 | 83 | # Extract RGB of each sample position along each ray. 84 | rgb = tf.math.sigmoid(raw[..., :3]) # [N_rays, N_samples, 3] 85 | 86 | # Add noise to model's predictions for density. Can be used to 87 | # regularize network during training (prevents floater artifacts). 88 | noise = 0. 89 | if raw_noise_std > 0.: 90 | noise = tf.random.normal(raw[..., 3].shape) * raw_noise_std 91 | 92 | # Predict density of each sample along each ray. Higher values imply 93 | # higher likelihood of being absorbed at this point. 94 | alpha = raw2alpha(raw[..., 3] + noise, dists) # [N_rays, N_samples] 95 | 96 | # Compute weight for RGB of each sample along each ray. A cumprod() is 97 | # used to express the idea of the ray not having reflected up to this 98 | # sample yet. 99 | # [N_rays, N_samples] 100 | weights = alpha * \ 101 | tf.math.cumprod(1.-alpha + 1e-10, axis=-1, exclusive=True) 102 | 103 | # Computed weighted color of each sample along each ray. 104 | rgb_map = tf.reduce_sum( 105 | weights[..., None] * rgb, axis=-2) # [N_rays, 3] 106 | 107 | # Estimated depth map is expected distance. 108 | depth_map = tf.reduce_sum(weights * z_vals, axis=-1) 109 | 110 | # Disparity map is inverse depth. 111 | disp_map = 1./tf.maximum(1e-10, depth_map / 112 | tf.reduce_sum(weights, axis=-1)) 113 | 114 | # Sum of weights along each ray. This value is in [0, 1] up to numerical error. 115 | acc_map = tf.reduce_sum(weights, -1) 116 | 117 | # To composite onto a white background, use the accumulated alpha map. 118 | if white_bkgd: 119 | rgb_map = rgb_map + (1.-acc_map[..., None]) 120 | 121 | return rgb_map, disp_map, acc_map, weights, depth_map 122 | 123 | 124 | def render_rays(ray_batch, 125 | network_fn, 126 | network_query_fn, 127 | N_samples, 128 | retraw=False, 129 | lindisp=False, 130 | perturb=0., 131 | N_importance=0, 132 | network_fine=None, 133 | white_bkgd=False, 134 | raw_noise_std=0., 135 | verbose=False): 136 | """Volumetric rendering. 137 | 138 | Args: 139 | ray_batch: array of shape [batch_size, ...]. All information necessary 140 | for sampling along a ray, including: ray origin, ray direction, min 141 | dist, max dist, and unit-magnitude viewing direction. 142 | network_fn: function. Model for predicting RGB and density at each point 143 | in space. 144 | network_query_fn: function used for passing queries to network_fn. 145 | N_samples: int. Number of different times to sample along each ray. 146 | retraw: bool. If True, include model's raw, unprocessed predictions. 147 | lindisp: bool. If True, sample linearly in inverse depth rather than in depth. 148 | perturb: float, 0 or 1. If non-zero, each ray is sampled at stratified 149 | random points in time. 150 | N_importance: int. Number of additional times to sample along each ray. 151 | These samples are only passed to network_fine. 152 | network_fine: "fine" network with same spec as network_fn. 153 | white_bkgd: bool. If True, assume a white background. 154 | raw_noise_std: ... 155 | verbose: bool. If True, print more debugging info. 156 | 157 | Returns: 158 | rgb_map: [num_rays, 3]. Estimated RGB color of a ray. Comes from fine model. 159 | disp_map: [num_rays]. Disparity map. 1 / depth. 160 | acc_map: [num_rays]. Accumulated opacity along each ray. Comes from fine model. 161 | raw: [num_rays, num_samples, 4]. Raw predictions from model. 162 | rgb0: See rgb_map. Output for coarse model. 163 | disp0: See disp_map. Output for coarse model. 164 | acc0: See acc_map. Output for coarse model. 165 | z_std: [num_rays]. Standard deviation of distances along ray for each 166 | sample. 167 | """ 168 | 169 | ############################### 170 | # batch size 171 | N_rays = ray_batch.shape[0] 172 | 173 | # Extract ray origin, direction. 174 | rays_o, rays_d = ray_batch[:, 0:3], ray_batch[:, 3:6] # [N_rays, 3] each 175 | 176 | # Extract unit-normalized viewing direction. 177 | viewdirs = ray_batch[:, -3:] if ray_batch.shape[-1] > 8 else None 178 | 179 | # Extract lower, upper bound for ray distance. 180 | bounds = tf.reshape(ray_batch[..., 6:8], [-1, 1, 2]) 181 | near, far = bounds[..., 0], bounds[..., 1] # [-1,1] 182 | 183 | # Decide where to sample along each ray. Under the logic, all rays will be sampled at 184 | # the same times. 185 | t_vals = tf.linspace(0., 1., N_samples) 186 | if not lindisp: 187 | # Space integration times linearly between 'near' and 'far'. Same 188 | # integration points will be used for all rays. 189 | z_vals = near * (1.-t_vals) + far * (t_vals) 190 | else: 191 | # Sample linearly in inverse depth (disparity). 192 | z_vals = 1./(1./near * (1.-t_vals) + 1./far * (t_vals)) 193 | z_vals = tf.broadcast_to(z_vals, [N_rays, N_samples]) 194 | 195 | # Perturb sampling time along each ray. 196 | if perturb > 0.: 197 | # get intervals between samples 198 | mids = .5 * (z_vals[..., 1:] + z_vals[..., :-1]) 199 | upper = tf.concat([mids, z_vals[..., -1:]], -1) 200 | lower = tf.concat([z_vals[..., :1], mids], -1) 201 | # stratified samples in those intervals 202 | t_rand = tf.random.uniform(z_vals.shape) 203 | z_vals = lower + (upper - lower) * t_rand 204 | 205 | # Points in space to evaluate model at. 206 | pts = rays_o[..., None, :] + rays_d[..., None, :] * \ 207 | z_vals[..., :, None] # [N_rays, N_samples, 3] 208 | 209 | # Evaluate model at each point. 210 | raw = network_query_fn(pts, viewdirs, network_fn) # [N_rays, N_samples, 4] 211 | rgb_map, disp_map, acc_map, weights, depth_map = raw2outputs( 212 | raw, z_vals, rays_d, raw_noise_std, white_bkgd) 213 | 214 | if N_importance > 0: 215 | rgb_map_0, disp_map_0, acc_map_0 = rgb_map, disp_map, acc_map 216 | 217 | # Obtain additional integration times to evaluate based on the weights 218 | # assigned to colors in the coarse model. 219 | z_vals_mid = .5 * (z_vals[..., 1:] + z_vals[..., :-1]) 220 | z_samples = sample_pdf( 221 | z_vals_mid, weights[..., 1:-1], N_importance, det=(perturb == 0.)) 222 | z_samples = tf.stop_gradient(z_samples) 223 | 224 | # Obtain all points to evaluate color, density at. 225 | z_vals = tf.sort(tf.concat([z_vals, z_samples], -1), -1) 226 | pts = rays_o[..., None, :] + rays_d[..., None, :] * \ 227 | z_vals[..., :, None] # [N_rays, N_samples + N_importance, 3] 228 | 229 | # Make predictions with network_fine. 230 | run_fn = network_fn if network_fine is None else network_fine 231 | raw = network_query_fn(pts, viewdirs, run_fn) 232 | rgb_map, disp_map, acc_map, weights, depth_map = raw2outputs( 233 | raw, z_vals, rays_d, raw_noise_std, white_bkgd) 234 | 235 | z_mids = .5 * (z_vals[..., 1:] + z_vals[..., :-1]) 236 | weights = weights[...,1:-1] 237 | 238 | ret = {'rgb_map': rgb_map, 'disp_map': disp_map, 'acc_map': acc_map, 'weights': weights, 'z_mids': z_mids} 239 | if retraw: 240 | ret['raw'] = raw 241 | if N_importance > 0: 242 | ret['rgb0'] = rgb_map_0 243 | ret['disp0'] = disp_map_0 244 | ret['acc0'] = acc_map_0 245 | ret['z_std'] = tf.math.reduce_std(z_samples, -1) # [N_rays] 246 | 247 | for k in ret: 248 | tf.debugging.check_numerics(ret[k], 'output {}'.format(k)) 249 | 250 | return ret 251 | 252 | # TODO: flatten the hxw dimensions of the inputs so its consistent with the other method 253 | def render_rays_with_pdf(z_vals_mid, weights, ray_batch, render_info, 254 | network_fn, 255 | network_query_fn, 256 | N_samples, 257 | retraw=False, 258 | lindisp=False, 259 | perturb=0., 260 | N_importance=0, 261 | network_fine=None, 262 | white_bkgd=False, 263 | raw_noise_std=0., 264 | verbose=False, 265 | chunk=1024*32): 266 | ''' 267 | Render a ray given a known pdf 268 | ''' 269 | NUM_SAMPLES = N_importance 270 | 271 | H, W, y_offset, x_offset, d_factor = render_info 272 | 273 | # this doesn't work 274 | #interpolate estimated z mids and weights 275 | # interpolated_z_mids = mat_bilinear_interpolation(z_vals_mid, H, W, x_offset, y_offset, d_factor) 276 | # interpolated_weights = mat_bilinear_interpolation(weights, H, W, x_offset, y_offset, d_factor) 277 | 278 | # batch size 279 | N_rays = ray_batch.shape[0] 280 | 281 | # Extract ray origin, direction. 282 | rays_o, rays_d = ray_batch[..., 0:3], ray_batch[..., 3:6] # [h, w, 3] each 283 | 284 | # Extract unit-normalized viewing direction. 285 | viewdirs = ray_batch[..., -3:] if ray_batch.shape[-1] > 8 else None 286 | 287 | # Extract lower, upper bound for ray distance. 288 | bounds = tf.reshape(ray_batch[..., 6:8], [-1, 1, 2]) 289 | near, far = bounds[..., 0], bounds[..., 1] # [-1,1] 290 | 291 | # Obtain additional integration times to evaluate based on the weights 292 | # assigned to colors in the coarse model. 293 | # z_samples = sample_pdf( 294 | # interpolated_z_mids, interpolated_weights, NUM_SAMPLES, det=(perturb == 0.)) 295 | # z_samples = tf.stop_gradient(z_samples) 296 | 297 | # Interpolation sampling 298 | z_samples = weighted_sampling_interpolation(z_vals_mid, weights, H, W, x_offset, y_offset, d_factor, NUM_SAMPLES, det=(perturb == 0.)) 299 | # Obtain all points to evaluate color, density at. 300 | # z_vals = tf.sort(tf.concat([z_vals, z_samples], -1), -1) 301 | pts = rays_o[..., None, :] + rays_d[..., None, :] * \ 302 | z_samples[..., :, None] # [N_rays, N_samples + N_importance, 3] 303 | 304 | # Make predictions with network_fine. 305 | og_shape = pts.shape[:2] 306 | pts = tf.reshape(pts, (list([-1]) + list(pts.shape[2:]))) 307 | viewdirs = tf.reshape(viewdirs, (list([-1]) + list(viewdirs.shape[2:]))) 308 | # also reshape for teh raw2outputs conversion 309 | z_samples = tf.reshape(z_samples, (list([-1]) + list(z_samples.shape[2:]))) 310 | rays_d = tf.reshape(rays_d, (list([-1]) + list(rays_d.shape[2:]))) 311 | 312 | 313 | run_fn = network_fn if network_fine is None else network_fine 314 | all_ret = {'rgb_map':[], 315 | 'disp_map':[], 316 | 'acc_map':[],} 317 | if retraw: 318 | all_ret['raw'] = [] 319 | 320 | # for batch in range(0, pts.shape[0], chunk): 321 | raw = network_query_fn(pts, viewdirs, run_fn) 322 | rgb_map, disp_map, acc_map, weights, depth_map = raw2outputs( 323 | raw, z_samples, rays_d, raw_noise_std, white_bkgd) 324 | ret = {'rgb_map': rgb_map, 'disp_map': disp_map, 'acc_map': acc_map} 325 | if retraw: 326 | ret['raw'] = raw 327 | for k in ret: 328 | all_ret[k].append(ret[k]) 329 | 330 | for k in all_ret: 331 | tf.debugging.check_numerics(all_ret[k], 'output {}'.format(k)) 332 | 333 | all_ret = {k: tf.concat(all_ret[k], 0) for k in all_ret} 334 | 335 | return all_ret 336 | 337 | 338 | def batchify_rays(rays_flat, chunk=1024*32, **kwargs): 339 | """Render rays in smaller minibatches to avoid OOM.""" 340 | all_ret = {} 341 | for i in range(0, rays_flat.shape[0], chunk): 342 | ret = render_rays(rays_flat[i:i+chunk], **kwargs) 343 | for k in ret: 344 | if k not in all_ret: 345 | all_ret[k] = [] 346 | all_ret[k].append(ret[k]) 347 | 348 | all_ret = {k: tf.concat(all_ret[k], 0) for k in all_ret} 349 | return all_ret 350 | 351 | 352 | def render(H, W, focal, 353 | chunk=1024*32, rays=None, c2w=None, ndc=True, 354 | near=0., far=1., d_factor=3, 355 | use_viewdirs=False, c2w_staticcam=None, 356 | **kwargs): 357 | """Render rays 358 | 359 | Args: 360 | H: int. Height of image in pixels. 361 | W: int. Width of image in pixels. 362 | focal: float. Focal length of pinhole camera. 363 | chunk: int. Maximum number of rays to process simultaneously. Used to 364 | control maximum memory usage. Does not affect final results. 365 | rays: array of shape [2, batch_size, 3]. Ray origin and direction for 366 | each example in batch. 367 | c2w: array of shape [3, 4]. Camera-to-world transformation matrix. 368 | ndc: bool. If True, represent ray origin, direction in NDC coordinates. 369 | near: float or array of shape [batch_size]. Nearest distance for a ray. 370 | far: float or array of shape [batch_size]. Farthest distance for a ray. 371 | use_viewdirs: bool. If True, use viewing direction of a point in space in model. 372 | c2w_staticcam: array of shape [3, 4]. If not None, use this transformation matrix for 373 | camera while using other c2w argument for viewing directions. 374 | 375 | Returns: 376 | rgb_map: [batch_size, 3]. Predicted RGB values for rays. 377 | disp_map: [batch_size]. Disparity map. Inverse of depth. 378 | acc_map: [batch_size]. Accumulated opacity (alpha) along a ray. 379 | extras: dict with everything returned by render_rays(). 380 | """ 381 | 382 | if c2w is not None: 383 | # special case to render full image 384 | rays_o, rays_d = get_rays(H, W, focal, c2w) 385 | else: 386 | # use provided ray batch 387 | rays_o, rays_d = rays 388 | 389 | # using gather or slicing 390 | gather = False 391 | 392 | if gather: 393 | # temporarily flatten 394 | rays_o = tf.reshape(rays_o, [-1, 3]) 395 | rays_d = tf.reshape(rays_d, [-1, 3]) 396 | 397 | 398 | # downsampling test 399 | # d_factor = 3 400 | # xs = np.array([range(0,W,d_factor),]*(H // d_factor)) 401 | # ys = np.array([range(0,H,d_factor),]*(W // d_factor)).transpose() 402 | # ys = ys * W 403 | # inds = xs + ys 404 | # inds = inds.flatten() 405 | # inds = tf.convert_to_tensor(inds, dtype=tf.int32) 406 | # rays_o = tf.gather(rays_o, inds, axis=0) 407 | # rays_d = tf.gather(rays_d, inds, axis=0) 408 | # near, far = near * tf.ones_like(rays_d[..., :1]), far * tf.ones_like(rays_d[..., :1]) 409 | 410 | 411 | # downsampling with all start shifts 412 | # Shape: group# x group height x group width x 3 413 | 414 | small_h = H//d_factor + (H % d_factor > 0) 415 | small_w = W//d_factor + (W % d_factor > 0) 416 | 417 | ray_groups_o = np.zeros((d_factor**2, small_h, small_w, 3)) 418 | ray_groups_d = np.zeros((d_factor**2, small_h, small_w, 3)) 419 | 420 | # Shape: group# x group height x group width x 1 421 | near_groups = near * np.ones((d_factor**2, small_h, small_w, 1)) 422 | far_groups = far * np.ones((d_factor**2, small_h, small_w, 1)) 423 | 424 | # TODO: account for case where H and W aren't multiples of d_factor 425 | for i in range(d_factor): 426 | for j in range(d_factor): 427 | if gather: 428 | xs = range(j, W, d_factor) 429 | ys = range(i, H, d_factor) 430 | width = len(xs) 431 | height = len(ys) 432 | xs = np.array([xs,]*height) 433 | ys = np.array([ys,]*width).transpose() 434 | ys = ys * W 435 | inds = xs + ys 436 | inds = inds.flatten() 437 | inds = tf.convert_to_tensor(inds, dtype=tf.int32) 438 | # get rays and reshape 439 | rays_g_o = tf.gather(rays_o, inds, axis=0) 440 | rays_g_d = tf.gather(rays_d, inds, axis=0) 441 | rays_g_o = tf.reshape(rays_g_o, (height, width, 3)) 442 | rays_g_d = tf.reshape(rays_g_d, (height, width, 3)) 443 | # put rays into larger tensor 444 | ray_groups_o[i*d_factor + j, :height, :width, :] = rays_g_o 445 | ray_groups_d[i*d_factor + j, :height, :width, :] = rays_g_d 446 | else: 447 | rays_g_o = rays_o[i::d_factor, j::d_factor] 448 | rays_g_d = rays_d[i::d_factor, j::d_factor] 449 | # put rays into larger tensor 450 | # indexing accounts for the height/width being either H//2 or H//2 + 1 451 | # depending on the start index 452 | h_size = small_h - (H%d_factor < (i+1) and (not H%d_factor == 0)) 453 | w_size = small_w - (W%d_factor < (j+1) and (not W%d_factor == 0)) 454 | ray_groups_o[i*d_factor + j, :h_size, :w_size, :] = rays_g_o 455 | ray_groups_d[i*d_factor + j, :h_size, :w_size, :] = rays_g_d 456 | 457 | ray_groups_o = tf.convert_to_tensor(ray_groups_o, dtype=tf.float32) 458 | ray_groups_d = tf.convert_to_tensor(ray_groups_d, dtype=tf.float32) 459 | near_groups = tf.convert_to_tensor(near_groups, dtype=tf.float32) 460 | far_groups = tf.convert_to_tensor(far_groups, dtype=tf.float32) 461 | 462 | # return to original ratio (downsampled shape) 463 | # rays_o = tf.reshape(rays_o, (H//d_factor, W//d_factor, 3)) 464 | # rays_d = tf.reshape(rays_d, (H//d_factor, W//d_factor, 3)) 465 | 466 | if use_viewdirs: 467 | viewdir_groups = [] 468 | 469 | for i in range(d_factor**2): 470 | # provide ray directions as input 471 | viewdirs = ray_groups_d[i] 472 | 473 | # NOTE: c2w_staticcam is not supported in our faster version 474 | # -Trevor 475 | # if c2w_staticcam is not None: 476 | # # special case to visualize effect of viewdirs 477 | # rays_o, rays_d = get_rays(H, W, focal, c2w_staticcam) 478 | 479 | # Make all directions unit magnitude. 480 | # shape: [batch_size, 3] 481 | viewdirs = viewdirs / tf.linalg.norm(viewdirs, axis=-1, keepdims=True) 482 | # viewdirs = tf.cast(tf.reshape(viewdirs, [-1, 3]), dtype=tf.float32) 483 | viewdirs = tf.cast(viewdirs, dtype=tf.float32) 484 | viewdir_groups.append(viewdirs) 485 | viewdir_groups = tf.stack(viewdir_groups, axis=0) 486 | 487 | sh = ray_groups_d.shape # [..., 3] 488 | if ndc: 489 | o_groups = [] 490 | d_groups = [] 491 | for i in range(d_factor**2): 492 | # for forward facing scenes 493 | ray_oi, ray_di = ndc_rays( 494 | H, W, focal, tf.cast(1., tf.float32), ray_groups_o[i], ray_groups_d[i]) 495 | o_groups.append(ray_oi) 496 | d_groups.append(ray_di) 497 | ray_groups_o = tf.stack(o_groups, axis=0) 498 | ray_groups_d = tf.stack(d_groups, axis=0) 499 | 500 | # This removes the view structure (HxW) 501 | # Create ray batch 502 | # rays_o = tf.cast(tf.reshape(rays_o, [-1, 3]), dtype=tf.float32) 503 | # rays_d = tf.cast(tf.reshape(rays_d, [-1, 3]), dtype=tf.float32) 504 | # near, far = near * \ 505 | # tf.ones_like(rays_d[..., :1]), far * tf.ones_like(rays_d[..., :1]) 506 | 507 | ray_groups_o = tf.cast(ray_groups_o, dtype=tf.float32) 508 | ray_groups_d = tf.cast(ray_groups_d, dtype=tf.float32) 509 | near_groups = tf.cast(near_groups, dtype=tf.float32) 510 | far_groups = tf.cast(far_groups, dtype=tf.float32) 511 | 512 | # (ray origin, ray direction, min dist, max dist) for each ray 513 | rays = tf.concat([ray_groups_o, ray_groups_d, near_groups, far_groups], axis=-1) 514 | if use_viewdirs: 515 | # (ray origin, ray direction, min dist, max dist, normalized viewing direction) 516 | rays = tf.concat([rays, viewdir_groups], axis=-1) 517 | data_size = rays.shape[-1] 518 | 519 | # Render and reshape 520 | all_dicts = [] 521 | merged_ret = {} 522 | for x in range(d_factor**2): 523 | if x == 0: 524 | all_ret = batchify_rays(tf.reshape(rays[0], [-1, data_size]), chunk, **kwargs) 525 | else: 526 | # keep the hxw structure for interpolation 527 | x_offset = x%d_factor 528 | y_offset = x//d_factor 529 | render_info = [H, W, y_offset, x_offset, d_factor] 530 | all_ret = render_rays_with_pdf(z_mids, weights, rays[x], render_info, chunk=chunk, **kwargs) 531 | 532 | for k in all_ret: 533 | end_dims = list(all_ret[k].shape[1:]) 534 | k_sh = list(sh[1:-1]) + end_dims 535 | all_ret[k] = tf.reshape(all_ret[k], k_sh) 536 | 537 | # initialized merged results 538 | # make zeros matrices in the correct shape for each type of returned data 539 | if x == 0: 540 | merged_ret[k] = np.zeros(list([H,W]) + end_dims) 541 | # y_shape = H//d_factor + (H%d_factor > 0) 542 | # x_shape = W//d_factor + (W%d_factor > 0) 543 | # z_mids = tf.reshape(all_ret['z_mids'], list([y_shape, x_shape]) + list(all_ret['z_mids'].shape[1:])) 544 | # weights = tf.reshape(all_ret['weights'], list([y_shape, x_shape]) + list(all_ret['weights'].shape[1:])) 545 | z_mids = all_ret['z_mids'] 546 | weights = all_ret['weights'] 547 | 548 | # add to the merged results 549 | for k in all_ret: 550 | i = x // d_factor 551 | j = x % d_factor 552 | h_size = small_h - (H%d_factor < (i+1) and (not H%d_factor == 0)) 553 | w_size = small_w - (W%d_factor < (j+1) and (not W%d_factor == 0)) 554 | merged_ret[k][i::d_factor, j::d_factor, ...] = all_ret[k][:h_size, :w_size] 555 | 556 | 557 | k_extract = ['rgb_map', 'disp_map', 'acc_map'] 558 | ret_list = [merged_ret[k] for k in k_extract] 559 | ret_dict = {k: merged_ret[k] for k in merged_ret if k not in k_extract} 560 | return ret_list + [ret_dict] 561 | 562 | 563 | def render_path(render_poses, hwf, chunk, render_kwargs, gt_imgs=None, savedir=None, render_factor=0): 564 | 565 | H, W, focal = hwf 566 | 567 | if render_factor != 0: 568 | # Render downsampled for speed 569 | H = H//render_factor 570 | W = W//render_factor 571 | focal = focal/render_factor 572 | 573 | rgbs = [] 574 | disps = [] 575 | 576 | t = time.time() 577 | for i, c2w in enumerate(render_poses): 578 | print(i, time.time() - t) 579 | t = time.time() 580 | rgb, disp, acc, _ = render( 581 | H, W, focal, chunk=chunk, c2w=c2w[:3, :4], **render_kwargs) 582 | rgbs.append(rgb.numpy()) 583 | disps.append(disp.numpy()) 584 | if i == 0: 585 | print(rgb.shape, disp.shape) 586 | 587 | if gt_imgs is not None and render_factor == 0: 588 | p = -10. * np.log10(np.mean(np.square(rgb - gt_imgs[i]))) 589 | print(p) 590 | 591 | if savedir is not None: 592 | rgb8 = to8b(rgbs[-1]) 593 | filename = os.path.join(savedir, '{:03d}.png'.format(i)) 594 | imageio.imwrite(filename, rgb8) 595 | 596 | rgbs = np.stack(rgbs, 0) 597 | disps = np.stack(disps, 0) 598 | 599 | return rgbs, disps 600 | 601 | 602 | def create_nerf(args): 603 | """Instantiate NeRF's MLP model.""" 604 | 605 | embed_fn, input_ch = get_embedder(args.multires, args.i_embed) 606 | 607 | input_ch_views = 0 608 | embeddirs_fn = None 609 | if args.use_viewdirs: 610 | embeddirs_fn, input_ch_views = get_embedder( 611 | args.multires_views, args.i_embed) 612 | output_ch = 4 613 | skips = [4] 614 | model = init_nerf_model( 615 | D=args.netdepth, W=args.netwidth, 616 | input_ch=input_ch, output_ch=output_ch, skips=skips, 617 | input_ch_views=input_ch_views, use_viewdirs=args.use_viewdirs) 618 | grad_vars = model.trainable_variables 619 | models = {'model': model} 620 | 621 | model_fine = None 622 | if args.N_importance > 0: 623 | model_fine = init_nerf_model( 624 | D=args.netdepth_fine, W=args.netwidth_fine, 625 | input_ch=input_ch, output_ch=output_ch, skips=skips, 626 | input_ch_views=input_ch_views, use_viewdirs=args.use_viewdirs) 627 | grad_vars += model_fine.trainable_variables 628 | models['model_fine'] = model_fine 629 | 630 | def network_query_fn(inputs, viewdirs, network_fn): return run_network( 631 | inputs, viewdirs, network_fn, 632 | embed_fn=embed_fn, 633 | embeddirs_fn=embeddirs_fn, 634 | netchunk=args.netchunk) 635 | 636 | render_kwargs_train = { 637 | 'network_query_fn': network_query_fn, 638 | 'perturb': args.perturb, 639 | 'N_importance': args.N_importance, 640 | 'network_fine': model_fine, 641 | 'N_samples': args.N_samples, 642 | 'network_fn': model, 643 | 'use_viewdirs': args.use_viewdirs, 644 | 'white_bkgd': args.white_bkgd, 645 | 'raw_noise_std': args.raw_noise_std, 646 | } 647 | 648 | # NDC only good for LLFF-style forward facing data 649 | if args.dataset_type != 'llff' or args.no_ndc: 650 | print('Not ndc!') 651 | render_kwargs_train['ndc'] = False 652 | render_kwargs_train['lindisp'] = args.lindisp 653 | 654 | render_kwargs_test = { 655 | k: render_kwargs_train[k] for k in render_kwargs_train} 656 | render_kwargs_test['perturb'] = False 657 | render_kwargs_test['raw_noise_std'] = 0. 658 | 659 | start = 0 660 | basedir = args.basedir 661 | expname = args.expname 662 | 663 | if args.ft_path is not None and args.ft_path != 'None': 664 | ckpts = [args.ft_path] 665 | else: 666 | ckpts = [os.path.join(basedir, expname, f) for f in sorted(os.listdir(os.path.join(basedir, expname))) if 667 | ('model_' in f and 'fine' not in f and 'optimizer' not in f)] 668 | print('Found ckpts', ckpts) 669 | if len(ckpts) > 0 and not args.no_reload: 670 | ft_weights = ckpts[-1] 671 | print('Reloading from', ft_weights) 672 | model.set_weights(np.load(ft_weights, allow_pickle=True)) 673 | start = int(ft_weights[-10:-4]) + 1 674 | print('Resetting step to', start) 675 | 676 | if model_fine is not None: 677 | ft_weights_fine = '{}_fine_{}'.format( 678 | ft_weights[:-11], ft_weights[-10:]) 679 | print('Reloading fine from', ft_weights_fine) 680 | model_fine.set_weights(np.load(ft_weights_fine, allow_pickle=True)) 681 | 682 | return render_kwargs_train, render_kwargs_test, start, grad_vars, models 683 | 684 | 685 | def config_parser(): 686 | 687 | import configargparse 688 | parser = configargparse.ArgumentParser() 689 | parser.add_argument('--config', is_config_file=True, 690 | help='config file path') 691 | parser.add_argument("--expname", type=str, help='experiment name') 692 | parser.add_argument("--basedir", type=str, default='./logs/', 693 | help='where to store ckpts and logs') 694 | parser.add_argument("--datadir", type=str, 695 | default='./data/llff/fern', help='input data directory') 696 | 697 | # training options 698 | parser.add_argument("--netdepth", type=int, default=8, 699 | help='layers in network') 700 | parser.add_argument("--netwidth", type=int, default=256, 701 | help='channels per layer') 702 | parser.add_argument("--netdepth_fine", type=int, 703 | default=8, help='layers in fine network') 704 | parser.add_argument("--netwidth_fine", type=int, default=256, 705 | help='channels per layer in fine network') 706 | parser.add_argument("--N_rand", type=int, default=32*32*4, 707 | help='batch size (number of random rays per gradient step)') 708 | parser.add_argument("--lrate", type=float, 709 | default=5e-4, help='learning rate') 710 | parser.add_argument("--lrate_decay", type=int, default=250, 711 | help='exponential learning rate decay (in 1000s)') 712 | parser.add_argument("--chunk", type=int, default=1024*32, 713 | help='number of rays processed in parallel, decrease if running out of memory') 714 | parser.add_argument("--netchunk", type=int, default=1024*64, 715 | help='number of pts sent through network in parallel, decrease if running out of memory') 716 | parser.add_argument("--no_batching", action='store_true', 717 | help='only take random rays from 1 image at a time') 718 | parser.add_argument("--no_reload", action='store_true', 719 | help='do not reload weights from saved ckpt') 720 | parser.add_argument("--ft_path", type=str, default=None, 721 | help='specific weights npy file to reload for coarse network') 722 | parser.add_argument("--random_seed", type=int, default=None, 723 | help='fix random seed for repeatability') 724 | 725 | # pre-crop options 726 | parser.add_argument("--precrop_iters", type=int, default=0, 727 | help='number of steps to train on central crops') 728 | parser.add_argument("--precrop_frac", type=float, 729 | default=.5, help='fraction of img taken for central crops') 730 | 731 | # rendering options 732 | parser.add_argument("--N_samples", type=int, default=64, 733 | help='number of coarse samples per ray') 734 | parser.add_argument("--N_importance", type=int, default=0, 735 | help='number of additional fine samples per ray') 736 | parser.add_argument("--perturb", type=float, default=1., 737 | help='set to 0. for no jitter, 1. for jitter') 738 | parser.add_argument("--use_viewdirs", action='store_true', 739 | help='use full 5D input instead of 3D') 740 | parser.add_argument("--i_embed", type=int, default=0, 741 | help='set 0 for default positional encoding, -1 for none') 742 | parser.add_argument("--multires", type=int, default=10, 743 | help='log2 of max freq for positional encoding (3D location)') 744 | parser.add_argument("--multires_views", type=int, default=4, 745 | help='log2 of max freq for positional encoding (2D direction)') 746 | parser.add_argument("--raw_noise_std", type=float, default=0., 747 | help='std dev of noise added to regularize sigma_a output, 1e0 recommended') 748 | 749 | parser.add_argument("--render_only", action='store_true', 750 | help='do not optimize, reload weights and render out render_poses path') 751 | parser.add_argument("--render_test", action='store_true', 752 | help='render the test set instead of render_poses path') 753 | parser.add_argument("--render_factor", type=int, default=0, 754 | help='downsampling factor to speed up rendering, set 4 or 8 for fast preview') 755 | 756 | # dataset options 757 | parser.add_argument("--dataset_type", type=str, default='llff', 758 | help='options: llff / blender / deepvoxels') 759 | parser.add_argument("--testskip", type=int, default=8, 760 | help='will load 1/N images from test/val sets, useful for large datasets like deepvoxels') 761 | 762 | # deepvoxels flags 763 | parser.add_argument("--shape", type=str, default='greek', 764 | help='options : armchair / cube / greek / vase') 765 | 766 | # blender flags 767 | parser.add_argument("--white_bkgd", action='store_true', 768 | help='set to render synthetic data on a white bkgd (always use for dvoxels)') 769 | parser.add_argument("--half_res", action='store_true', 770 | help='load blender synthetic data at 400x400 instead of 800x800') 771 | 772 | # llff flags 773 | parser.add_argument("--factor", type=int, default=8, 774 | help='downsample factor for LLFF images') 775 | parser.add_argument("--no_ndc", action='store_true', 776 | help='do not use normalized device coordinates (set for non-forward facing scenes)') 777 | parser.add_argument("--lindisp", action='store_true', 778 | help='sampling linearly in disparity rather than depth') 779 | parser.add_argument("--spherify", action='store_true', 780 | help='set for spherical 360 scenes') 781 | parser.add_argument("--llffhold", type=int, default=8, 782 | help='will take every 1/N images as LLFF test set, paper uses 8') 783 | 784 | # logging/saving options 785 | parser.add_argument("--i_print", type=int, default=100, 786 | help='frequency of console printout and metric loggin') 787 | parser.add_argument("--i_img", type=int, default=500, 788 | help='frequency of tensorboard image logging') 789 | parser.add_argument("--i_weights", type=int, default=10000, 790 | help='frequency of weight ckpt saving') 791 | parser.add_argument("--i_testset", type=int, default=50000, 792 | help='frequency of testset saving') 793 | parser.add_argument("--i_video", type=int, default=50000, 794 | help='frequency of render_poses video saving') 795 | 796 | return parser 797 | 798 | 799 | def train(): 800 | 801 | parser = config_parser() 802 | args = parser.parse_args() 803 | 804 | if args.random_seed is not None: 805 | print('Fixing random seed', args.random_seed) 806 | np.random.seed(args.random_seed) 807 | tf.compat.v1.set_random_seed(args.random_seed) 808 | 809 | # Load data 810 | 811 | if args.dataset_type == 'llff': 812 | images, poses, bds, render_poses, i_test = load_llff_data(args.datadir, args.factor, 813 | recenter=True, bd_factor=.75, 814 | spherify=args.spherify) 815 | hwf = poses[0, :3, -1] 816 | poses = poses[:, :3, :4] 817 | print('Loaded llff', images.shape, 818 | render_poses.shape, hwf, args.datadir) 819 | if not isinstance(i_test, list): 820 | i_test = [i_test] 821 | 822 | if args.llffhold > 0: 823 | print('Auto LLFF holdout,', args.llffhold) 824 | i_test = np.arange(images.shape[0])[::args.llffhold] 825 | 826 | i_val = i_test 827 | i_train = np.array([i for i in np.arange(int(images.shape[0])) if 828 | (i not in i_test and i not in i_val)]) 829 | 830 | print('DEFINING BOUNDS') 831 | if args.no_ndc: 832 | near = tf.reduce_min(bds) * .9 833 | far = tf.reduce_max(bds) * 1. 834 | else: 835 | near = 0. 836 | far = 1. 837 | print('NEAR FAR', near, far) 838 | 839 | elif args.dataset_type == 'blender': 840 | images, poses, render_poses, hwf, i_split = load_blender_data( 841 | args.datadir, args.half_res, args.testskip) 842 | print('Loaded blender', images.shape, 843 | render_poses.shape, hwf, args.datadir) 844 | i_train, i_val, i_test = i_split 845 | 846 | near = 2. 847 | far = 6. 848 | 849 | if args.white_bkgd: 850 | images = images[..., :3]*images[..., -1:] + (1.-images[..., -1:]) 851 | else: 852 | images = images[..., :3] 853 | 854 | elif args.dataset_type == 'deepvoxels': 855 | 856 | images, poses, render_poses, hwf, i_split = load_dv_data(scene=args.shape, 857 | basedir=args.datadir, 858 | testskip=args.testskip) 859 | 860 | print('Loaded deepvoxels', images.shape, 861 | render_poses.shape, hwf, args.datadir) 862 | i_train, i_val, i_test = i_split 863 | 864 | hemi_R = np.mean(np.linalg.norm(poses[:, :3, -1], axis=-1)) 865 | near = hemi_R-1. 866 | far = hemi_R+1. 867 | 868 | else: 869 | print('Unknown dataset type', args.dataset_type, 'exiting') 870 | return 871 | 872 | # Cast intrinsics to right types 873 | H, W, focal = hwf 874 | H, W = int(H), int(W) 875 | hwf = [H, W, focal] 876 | 877 | if args.render_test: 878 | render_poses = np.array(poses[i_test]) 879 | 880 | # Create log dir and copy the config file 881 | basedir = args.basedir 882 | expname = args.expname 883 | os.makedirs(os.path.join(basedir, expname), exist_ok=True) 884 | f = os.path.join(basedir, expname, 'args.txt') 885 | with open(f, 'w') as file: 886 | for arg in sorted(vars(args)): 887 | attr = getattr(args, arg) 888 | file.write('{} = {}\n'.format(arg, attr)) 889 | if args.config is not None: 890 | f = os.path.join(basedir, expname, 'config.txt') 891 | with open(f, 'w') as file: 892 | file.write(open(args.config, 'r').read()) 893 | 894 | # Create nerf model 895 | render_kwargs_train, render_kwargs_test, start, grad_vars, models = create_nerf( 896 | args) 897 | 898 | bds_dict = { 899 | 'near': tf.cast(near, tf.float32), 900 | 'far': tf.cast(far, tf.float32), 901 | } 902 | render_kwargs_train.update(bds_dict) 903 | render_kwargs_test.update(bds_dict) 904 | 905 | # Short circuit if only rendering out from trained model 906 | if args.render_only: 907 | print('RENDER ONLY') 908 | if args.render_test: 909 | # render_test switches to test poses 910 | images = images[i_test] 911 | else: 912 | # Default is smoother render_poses path 913 | images = None 914 | 915 | testsavedir = os.path.join(basedir, expname, 'renderonly_{}_{:06d}'.format( 916 | 'test' if args.render_test else 'path', start)) 917 | os.makedirs(testsavedir, exist_ok=True) 918 | print('test poses shape', render_poses.shape) 919 | 920 | rgbs, _ = render_path(render_poses, hwf, args.chunk, render_kwargs_test, 921 | gt_imgs=images, savedir=testsavedir, render_factor=args.render_factor) 922 | print('Done rendering', testsavedir) 923 | imageio.mimwrite(os.path.join(testsavedir, 'video.mp4'), 924 | to8b(rgbs), fps=30, quality=8) 925 | 926 | return 927 | 928 | # Create optimizer 929 | lrate = args.lrate 930 | if args.lrate_decay > 0: 931 | lrate = tf.keras.optimizers.schedules.ExponentialDecay(lrate, 932 | decay_steps=args.lrate_decay * 1000, decay_rate=0.1) 933 | optimizer = tf.keras.optimizers.Adam(lrate) 934 | models['optimizer'] = optimizer 935 | 936 | global_step = tf.compat.v1.train.get_or_create_global_step() 937 | global_step.assign(start) 938 | 939 | # Prepare raybatch tensor if batching random rays 940 | N_rand = args.N_rand 941 | use_batching = not args.no_batching 942 | if use_batching: 943 | # For random ray batching. 944 | # 945 | # Constructs an array 'rays_rgb' of shape [N*H*W, 3, 3] where axis=1 is 946 | # interpreted as, 947 | # axis=0: ray origin in world space 948 | # axis=1: ray direction in world space 949 | # axis=2: observed RGB color of pixel 950 | print('get rays') 951 | # get_rays_np() returns rays_origin=[H, W, 3], rays_direction=[H, W, 3] 952 | # for each pixel in the image. This stack() adds a new dimension. 953 | rays = [get_rays_np(H, W, focal, p) for p in poses[:, :3, :4]] 954 | rays = np.stack(rays, axis=0) # [N, ro+rd, H, W, 3] 955 | print('done, concats') 956 | # [N, ro+rd+rgb, H, W, 3] 957 | rays_rgb = np.concatenate([rays, images[:, None, ...]], 1) 958 | # [N, H, W, ro+rd+rgb, 3] 959 | rays_rgb = np.transpose(rays_rgb, [0, 2, 3, 1, 4]) 960 | rays_rgb = np.stack([rays_rgb[i] 961 | for i in i_train], axis=0) # train images only 962 | # [(N-1)*H*W, ro+rd+rgb, 3] 963 | rays_rgb = np.reshape(rays_rgb, [-1, 3, 3]) 964 | rays_rgb = rays_rgb.astype(np.float32) 965 | print('shuffle rays') 966 | np.random.shuffle(rays_rgb) 967 | print('done') 968 | i_batch = 0 969 | 970 | N_iters = 1000000 971 | print('Begin') 972 | print('TRAIN views are', i_train) 973 | print('TEST views are', i_test) 974 | print('VAL views are', i_val) 975 | 976 | # Summary writers 977 | writer = tf.contrib.summary.create_file_writer( 978 | os.path.join(basedir, 'summaries', expname)) 979 | writer.set_as_default() 980 | 981 | for i in range(start, N_iters): 982 | time0 = time.time() 983 | 984 | # Sample random ray batch 985 | 986 | if use_batching: 987 | # Random over all images 988 | batch = rays_rgb[i_batch:i_batch+N_rand] # [B, 2+1, 3*?] 989 | batch = tf.transpose(batch, [1, 0, 2]) 990 | 991 | # batch_rays[i, n, xyz] = ray origin or direction, example_id, 3D position 992 | # target_s[n, rgb] = example_id, observed color. 993 | batch_rays, target_s = batch[:2], batch[2] 994 | 995 | i_batch += N_rand 996 | if i_batch >= rays_rgb.shape[0]: 997 | np.random.shuffle(rays_rgb) 998 | i_batch = 0 999 | 1000 | else: 1001 | # Random from one image 1002 | img_i = np.random.choice(i_train) 1003 | target = images[img_i] 1004 | pose = poses[img_i, :3, :4] 1005 | 1006 | if N_rand is not None: 1007 | rays_o, rays_d = get_rays(H, W, focal, pose) 1008 | if i < args.precrop_iters: 1009 | dH = int(H//2 * args.precrop_frac) 1010 | dW = int(W//2 * args.precrop_frac) 1011 | coords = tf.stack(tf.meshgrid( 1012 | tf.range(H//2 - dH, H//2 + dH), 1013 | tf.range(W//2 - dW, W//2 + dW), 1014 | indexing='ij'), -1) 1015 | if i < 10: 1016 | print('precrop', dH, dW, coords[0,0], coords[-1,-1]) 1017 | else: 1018 | coords = tf.stack(tf.meshgrid( 1019 | tf.range(H), tf.range(W), indexing='ij'), -1) 1020 | coords = tf.reshape(coords, [-1, 2]) 1021 | select_inds = np.random.choice( 1022 | coords.shape[0], size=[N_rand], replace=False) 1023 | select_inds = tf.gather_nd(coords, select_inds[:, tf.newaxis]) 1024 | rays_o = tf.gather_nd(rays_o, select_inds) 1025 | rays_d = tf.gather_nd(rays_d, select_inds) 1026 | batch_rays = tf.stack([rays_o, rays_d], 0) 1027 | target_s = tf.gather_nd(target, select_inds) 1028 | 1029 | ##### Core optimization loop ##### 1030 | 1031 | with tf.GradientTape() as tape: 1032 | 1033 | # Make predictions for color, disparity, accumulated opacity. 1034 | rgb, disp, acc, extras = render( 1035 | H, W, focal, chunk=args.chunk, rays=batch_rays, 1036 | verbose=i < 10, retraw=True, **render_kwargs_train) 1037 | 1038 | # Compute MSE loss between predicted and true RGB. 1039 | img_loss = img2mse(rgb, target_s) 1040 | trans = extras['raw'][..., -1] 1041 | loss = img_loss 1042 | psnr = mse2psnr(img_loss) 1043 | 1044 | # Add MSE loss for coarse-grained model 1045 | if 'rgb0' in extras: 1046 | img_loss0 = img2mse(extras['rgb0'], target_s) 1047 | loss += img_loss0 1048 | psnr0 = mse2psnr(img_loss0) 1049 | 1050 | gradients = tape.gradient(loss, grad_vars) 1051 | optimizer.apply_gradients(zip(gradients, grad_vars)) 1052 | 1053 | dt = time.time()-time0 1054 | 1055 | ##### end ##### 1056 | 1057 | # Rest is logging 1058 | 1059 | def save_weights(net, prefix, i): 1060 | path = os.path.join( 1061 | basedir, expname, '{}_{:06d}.npy'.format(prefix, i)) 1062 | np.save(path, net.get_weights()) 1063 | print('saved weights at', path) 1064 | 1065 | if i % args.i_weights == 0: 1066 | for k in models: 1067 | save_weights(models[k], k, i) 1068 | 1069 | if i % args.i_video == 0 and i > 0: 1070 | 1071 | rgbs, disps = render_path( 1072 | render_poses, hwf, args.chunk, render_kwargs_test) 1073 | print('Done, saving', rgbs.shape, disps.shape) 1074 | moviebase = os.path.join( 1075 | basedir, expname, '{}_spiral_{:06d}_'.format(expname, i)) 1076 | imageio.mimwrite(moviebase + 'rgb.mp4', 1077 | to8b(rgbs), fps=30, quality=8) 1078 | imageio.mimwrite(moviebase + 'disp.mp4', 1079 | to8b(disps / np.max(disps)), fps=30, quality=8) 1080 | 1081 | if args.use_viewdirs: 1082 | render_kwargs_test['c2w_staticcam'] = render_poses[0][:3, :4] 1083 | rgbs_still, _ = render_path( 1084 | render_poses, hwf, args.chunk, render_kwargs_test) 1085 | render_kwargs_test['c2w_staticcam'] = None 1086 | imageio.mimwrite(moviebase + 'rgb_still.mp4', 1087 | to8b(rgbs_still), fps=30, quality=8) 1088 | 1089 | if i % args.i_testset == 0 and i > 0: 1090 | testsavedir = os.path.join( 1091 | basedir, expname, 'testset_{:06d}'.format(i)) 1092 | os.makedirs(testsavedir, exist_ok=True) 1093 | print('test poses shape', poses[i_test].shape) 1094 | render_path(poses[i_test], hwf, args.chunk, render_kwargs_test, 1095 | gt_imgs=images[i_test], savedir=testsavedir) 1096 | print('Saved test set') 1097 | 1098 | if i % args.i_print == 0 or i < 10: 1099 | 1100 | print(expname, i, psnr.numpy(), loss.numpy(), global_step.numpy()) 1101 | print('iter time {:.05f}'.format(dt)) 1102 | with tf.contrib.summary.record_summaries_every_n_global_steps(args.i_print): 1103 | tf.contrib.summary.scalar('loss', loss) 1104 | tf.contrib.summary.scalar('psnr', psnr) 1105 | tf.contrib.summary.histogram('tran', trans) 1106 | if args.N_importance > 0: 1107 | tf.contrib.summary.scalar('psnr0', psnr0) 1108 | 1109 | if i % args.i_img == 0: 1110 | 1111 | # Log a rendered validation view to Tensorboard 1112 | img_i = np.random.choice(i_val) 1113 | target = images[img_i] 1114 | pose = poses[img_i, :3, :4] 1115 | 1116 | rgb, disp, acc, extras = render(H, W, focal, chunk=args.chunk, c2w=pose, 1117 | **render_kwargs_test) 1118 | 1119 | psnr = mse2psnr(img2mse(rgb, target)) 1120 | 1121 | # Save out the validation image for Tensorboard-free monitoring 1122 | testimgdir = os.path.join(basedir, expname, 'tboard_val_imgs') 1123 | if i==0: 1124 | os.makedirs(testimgdir, exist_ok=True) 1125 | imageio.imwrite(os.path.join(testimgdir, '{:06d}.png'.format(i)), to8b(rgb)) 1126 | 1127 | with tf.contrib.summary.record_summaries_every_n_global_steps(args.i_img): 1128 | 1129 | tf.contrib.summary.image('rgb', to8b(rgb)[tf.newaxis]) 1130 | tf.contrib.summary.image( 1131 | 'disp', disp[tf.newaxis, ..., tf.newaxis]) 1132 | tf.contrib.summary.image( 1133 | 'acc', acc[tf.newaxis, ..., tf.newaxis]) 1134 | 1135 | tf.contrib.summary.scalar('psnr_holdout', psnr) 1136 | tf.contrib.summary.image('rgb_holdout', target[tf.newaxis]) 1137 | 1138 | if args.N_importance > 0: 1139 | 1140 | with tf.contrib.summary.record_summaries_every_n_global_steps(args.i_img): 1141 | tf.contrib.summary.image( 1142 | 'rgb0', to8b(extras['rgb0'])[tf.newaxis]) 1143 | tf.contrib.summary.image( 1144 | 'disp0', extras['disp0'][tf.newaxis, ..., tf.newaxis]) 1145 | tf.contrib.summary.image( 1146 | 'z_std', extras['z_std'][tf.newaxis, ..., tf.newaxis]) 1147 | 1148 | global_step.assign_add(1) 1149 | 1150 | 1151 | if __name__ == '__main__': 1152 | train() 1153 | -------------------------------------------------------------------------------- /run_nerf_helpers.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import tensorflow as tf 4 | import numpy as np 5 | import imageio 6 | import json 7 | 8 | 9 | # Misc utils 10 | 11 | def img2mse(x, y): return tf.reduce_mean(tf.square(x - y)) 12 | 13 | 14 | def mse2psnr(x): return -10.*tf.log(x)/tf.log(10.) 15 | 16 | 17 | def to8b(x): return (255*np.clip(x, 0, 1)).astype(np.uint8) 18 | 19 | 20 | # Positional encoding 21 | 22 | class Embedder: 23 | 24 | def __init__(self, **kwargs): 25 | 26 | self.kwargs = kwargs 27 | self.create_embedding_fn() 28 | 29 | def create_embedding_fn(self): 30 | 31 | embed_fns = [] 32 | d = self.kwargs['input_dims'] 33 | out_dim = 0 34 | if self.kwargs['include_input']: 35 | embed_fns.append(lambda x: x) 36 | out_dim += d 37 | 38 | max_freq = self.kwargs['max_freq_log2'] 39 | N_freqs = self.kwargs['num_freqs'] 40 | 41 | if self.kwargs['log_sampling']: 42 | freq_bands = 2.**tf.linspace(0., max_freq, N_freqs) 43 | else: 44 | freq_bands = tf.linspace(2.**0., 2.**max_freq, N_freqs) 45 | 46 | for freq in freq_bands: 47 | for p_fn in self.kwargs['periodic_fns']: 48 | embed_fns.append(lambda x, p_fn=p_fn, 49 | freq=freq: p_fn(x * freq)) 50 | out_dim += d 51 | 52 | self.embed_fns = embed_fns 53 | self.out_dim = out_dim 54 | 55 | def embed(self, inputs): 56 | return tf.concat([fn(inputs) for fn in self.embed_fns], -1) 57 | 58 | 59 | def get_embedder(multires, i=0): 60 | 61 | if i == -1: 62 | return tf.identity, 3 63 | 64 | embed_kwargs = { 65 | 'include_input': True, 66 | 'input_dims': 3, 67 | 'max_freq_log2': multires-1, 68 | 'num_freqs': multires, 69 | 'log_sampling': True, 70 | 'periodic_fns': [tf.math.sin, tf.math.cos], 71 | } 72 | 73 | embedder_obj = Embedder(**embed_kwargs) 74 | def embed(x, eo=embedder_obj): return eo.embed(x) 75 | return embed, embedder_obj.out_dim 76 | 77 | 78 | # Model architecture 79 | 80 | def init_nerf_model(D=8, W=256, input_ch=3, input_ch_views=3, output_ch=4, skips=[4], use_viewdirs=False): 81 | 82 | relu = tf.keras.layers.ReLU() 83 | def dense(W, act=relu): return tf.keras.layers.Dense(W, activation=act) 84 | 85 | print('MODEL', input_ch, input_ch_views, type( 86 | input_ch), type(input_ch_views), use_viewdirs) 87 | input_ch = int(input_ch) 88 | input_ch_views = int(input_ch_views) 89 | 90 | inputs = tf.keras.Input(shape=(input_ch + input_ch_views)) 91 | inputs_pts, inputs_views = tf.split(inputs, [input_ch, input_ch_views], -1) 92 | inputs_pts.set_shape([None, input_ch]) 93 | inputs_views.set_shape([None, input_ch_views]) 94 | 95 | print(inputs.shape, inputs_pts.shape, inputs_views.shape) 96 | outputs = inputs_pts 97 | for i in range(D): 98 | outputs = dense(W)(outputs) 99 | if i in skips: 100 | outputs = tf.concat([inputs_pts, outputs], -1) 101 | 102 | if use_viewdirs: 103 | alpha_out = dense(1, act=None)(outputs) 104 | bottleneck = dense(256, act=None)(outputs) 105 | inputs_viewdirs = tf.concat( 106 | [bottleneck, inputs_views], -1) # concat viewdirs 107 | outputs = inputs_viewdirs 108 | # The supplement to the paper states there are 4 hidden layers here, but this is an error since 109 | # the experiments were actually run with 1 hidden layer, so we will leave it as 1. 110 | for i in range(1): 111 | outputs = dense(W//2)(outputs) 112 | outputs = dense(3, act=None)(outputs) 113 | outputs = tf.concat([outputs, alpha_out], -1) 114 | else: 115 | outputs = dense(output_ch, act=None)(outputs) 116 | 117 | model = tf.keras.Model(inputs=inputs, outputs=outputs) 118 | return model 119 | 120 | 121 | # Ray helpers 122 | 123 | def get_rays(H, W, focal, c2w): 124 | """Get ray origins, directions from a pinhole camera.""" 125 | i, j = tf.meshgrid(tf.range(W, dtype=tf.float32), 126 | tf.range(H, dtype=tf.float32), indexing='xy') 127 | dirs = tf.stack([(i-W*.5)/focal, -(j-H*.5)/focal, -tf.ones_like(i)], -1) 128 | rays_d = tf.reduce_sum(dirs[..., np.newaxis, :] * c2w[:3, :3], -1) 129 | rays_o = tf.broadcast_to(c2w[:3, -1], tf.shape(rays_d)) 130 | return rays_o, rays_d 131 | 132 | 133 | def get_rays_np(H, W, focal, c2w): 134 | """Get ray origins, directions from a pinhole camera.""" 135 | i, j = np.meshgrid(np.arange(W, dtype=np.float32), 136 | np.arange(H, dtype=np.float32), indexing='xy') 137 | dirs = np.stack([(i-W*.5)/focal, -(j-H*.5)/focal, -np.ones_like(i)], -1) 138 | rays_d = np.sum(dirs[..., np.newaxis, :] * c2w[:3, :3], -1) 139 | rays_o = np.broadcast_to(c2w[:3, -1], np.shape(rays_d)) 140 | return rays_o, rays_d 141 | 142 | 143 | def ndc_rays(H, W, focal, near, rays_o, rays_d): 144 | """Normalized device coordinate rays. 145 | 146 | Space such that the canvas is a cube with sides [-1, 1] in each axis. 147 | 148 | Args: 149 | H: int. Height in pixels. 150 | W: int. Width in pixels. 151 | focal: float. Focal length of pinhole camera. 152 | near: float or array of shape[batch_size]. Near depth bound for the scene. 153 | rays_o: array of shape [batch_size, 3]. Camera origin. 154 | rays_d: array of shape [batch_size, 3]. Ray direction. 155 | 156 | Returns: 157 | rays_o: array of shape [batch_size, 3]. Camera origin in NDC. 158 | rays_d: array of shape [batch_size, 3]. Ray direction in NDC. 159 | """ 160 | # Shift ray origins to near plane 161 | t = -(near + rays_o[..., 2]) / rays_d[..., 2] 162 | rays_o = rays_o + t[..., None] * rays_d 163 | 164 | # Projection 165 | o0 = -1./(W/(2.*focal)) * rays_o[..., 0] / rays_o[..., 2] 166 | o1 = -1./(H/(2.*focal)) * rays_o[..., 1] / rays_o[..., 2] 167 | o2 = 1. + 2. * near / rays_o[..., 2] 168 | 169 | d0 = -1./(W/(2.*focal)) * \ 170 | (rays_d[..., 0]/rays_d[..., 2] - rays_o[..., 0]/rays_o[..., 2]) 171 | d1 = -1./(H/(2.*focal)) * \ 172 | (rays_d[..., 1]/rays_d[..., 2] - rays_o[..., 1]/rays_o[..., 2]) 173 | d2 = -2. * near / rays_o[..., 2] 174 | 175 | rays_o = tf.stack([o0, o1, o2], -1) 176 | rays_d = tf.stack([d0, d1, d2], -1) 177 | 178 | return rays_o, rays_d 179 | 180 | 181 | # Hierarchical sampling helper 182 | 183 | def sample_pdf(bins, weights, N_samples, det=False): 184 | 185 | # Get pdf 186 | weights += 1e-5 # prevent nans 187 | pdf = weights / tf.reduce_sum(weights, -1, keepdims=True) 188 | cdf = tf.cumsum(pdf, -1) 189 | cdf = tf.concat([tf.zeros_like(cdf[..., :1]), cdf], -1) 190 | 191 | # Take uniform samples 192 | if det: 193 | u = tf.linspace(0., 1., N_samples) 194 | u = tf.broadcast_to(u, list(cdf.shape[:-1]) + [N_samples]) 195 | else: 196 | u = tf.random.uniform(list(cdf.shape[:-1]) + [N_samples]) 197 | 198 | # Invert CDF 199 | inds = tf.searchsorted(cdf, u, side='right') 200 | below = tf.maximum(0, inds-1) 201 | above = tf.minimum(cdf.shape[-1]-1, inds) 202 | inds_g = tf.stack([below, above], -1) 203 | cdf_g = tf.gather(cdf, inds_g, axis=-1, batch_dims=len(inds_g.shape)-2) 204 | bins_g = tf.gather(bins, inds_g, axis=-1, batch_dims=len(inds_g.shape)-2) 205 | 206 | denom = (cdf_g[..., 1]-cdf_g[..., 0]) 207 | denom = tf.where(denom < 1e-5, tf.ones_like(denom), denom) 208 | t = (u-cdf_g[..., 0])/denom 209 | samples = bins_g[..., 0] + t * (bins_g[..., 1]-bins_g[..., 0]) 210 | 211 | return samples 212 | -------------------------------------------------------------------------------- /run_nerf_helpers_fast.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import tensorflow as tf 4 | import numpy as np 5 | import imageio 6 | import json 7 | import random 8 | 9 | 10 | # Misc utils 11 | 12 | def img2mse(x, y): return tf.reduce_mean(tf.square(x - y)) 13 | 14 | 15 | def mse2psnr(x): return -10.*tf.log(x)/tf.log(10.) 16 | 17 | 18 | def to8b(x): return (255*np.clip(x, 0, 1)).astype(np.uint8) 19 | 20 | 21 | # Positional encoding 22 | 23 | class Embedder: 24 | 25 | def __init__(self, **kwargs): 26 | 27 | self.kwargs = kwargs 28 | self.create_embedding_fn() 29 | 30 | def create_embedding_fn(self): 31 | 32 | embed_fns = [] 33 | d = self.kwargs['input_dims'] 34 | out_dim = 0 35 | if self.kwargs['include_input']: 36 | embed_fns.append(lambda x: x) 37 | out_dim += d 38 | 39 | max_freq = self.kwargs['max_freq_log2'] 40 | N_freqs = self.kwargs['num_freqs'] 41 | 42 | if self.kwargs['log_sampling']: 43 | freq_bands = 2.**tf.linspace(0., max_freq, N_freqs) 44 | else: 45 | freq_bands = tf.linspace(2.**0., 2.**max_freq, N_freqs) 46 | 47 | for freq in freq_bands: 48 | for p_fn in self.kwargs['periodic_fns']: 49 | embed_fns.append(lambda x, p_fn=p_fn, 50 | freq=freq: p_fn(x * freq)) 51 | out_dim += d 52 | 53 | self.embed_fns = embed_fns 54 | self.out_dim = out_dim 55 | 56 | def embed(self, inputs): 57 | return tf.concat([fn(inputs) for fn in self.embed_fns], -1) 58 | 59 | 60 | def get_embedder(multires, i=0): 61 | 62 | if i == -1: 63 | return tf.identity, 3 64 | 65 | embed_kwargs = { 66 | 'include_input': True, 67 | 'input_dims': 3, 68 | 'max_freq_log2': multires-1, 69 | 'num_freqs': multires, 70 | 'log_sampling': True, 71 | 'periodic_fns': [tf.math.sin, tf.math.cos], 72 | } 73 | 74 | embedder_obj = Embedder(**embed_kwargs) 75 | def embed(x, eo=embedder_obj): return eo.embed(x) 76 | return embed, embedder_obj.out_dim 77 | 78 | 79 | # Model architecture 80 | 81 | def init_nerf_model(D=8, W=256, input_ch=3, input_ch_views=3, output_ch=4, skips=[4], use_viewdirs=False): 82 | 83 | relu = tf.keras.layers.ReLU() 84 | def dense(W, act=relu): return tf.keras.layers.Dense(W, activation=act) 85 | 86 | print('MODEL', input_ch, input_ch_views, type( 87 | input_ch), type(input_ch_views), use_viewdirs) 88 | input_ch = int(input_ch) 89 | input_ch_views = int(input_ch_views) 90 | 91 | inputs = tf.keras.Input(shape=(input_ch + input_ch_views)) 92 | inputs_pts, inputs_views = tf.split(inputs, [input_ch, input_ch_views], -1) 93 | inputs_pts.set_shape([None, input_ch]) 94 | inputs_views.set_shape([None, input_ch_views]) 95 | 96 | print(inputs.shape, inputs_pts.shape, inputs_views.shape) 97 | outputs = inputs_pts 98 | for i in range(D): 99 | outputs = dense(W)(outputs) 100 | if i in skips: 101 | outputs = tf.concat([inputs_pts, outputs], -1) 102 | 103 | if use_viewdirs: 104 | alpha_out = dense(1, act=None)(outputs) 105 | bottleneck = dense(256, act=None)(outputs) 106 | inputs_viewdirs = tf.concat( 107 | [bottleneck, inputs_views], -1) # concat viewdirs 108 | outputs = inputs_viewdirs 109 | # The supplement to the paper states there are 4 hidden layers here, but this is an error since 110 | # the experiments were actually run with 1 hidden layer, so we will leave it as 1. 111 | for i in range(1): 112 | outputs = dense(W//2)(outputs) 113 | outputs = dense(3, act=None)(outputs) 114 | outputs = tf.concat([outputs, alpha_out], -1) 115 | else: 116 | outputs = dense(output_ch, act=None)(outputs) 117 | 118 | model = tf.keras.Model(inputs=inputs, outputs=outputs) 119 | return model 120 | 121 | # For scene vizualization 122 | def plot_scene(ax, points, poses, hwfs): 123 | ''' 124 | Plots camera centers, boxes showing their view directions, and 125 | 3D world points on a provided matplotlib axis. 126 | 127 | Arguments- 128 | ax - a matplotlib axis 129 | points - 3D world points 130 | poses - c2w camera matrices 131 | hwfs - height, width, focal length of camera views 132 | ''' 133 | 134 | ax.scatter(points[:,0], points[:,1], points[:,2], c='C0', alpha=0.8) 135 | for i in range(len(poses)): 136 | c2w = poses[i] 137 | H, W, focal = hwfs[i] 138 | os, ds = get_rays_np(H,W,focal, c2w) 139 | 140 | # plot center 141 | center = os[0,0] 142 | ax.scatter([center[0]], [center[1]], [center[2]], c='red', alpha=0.5) 143 | 144 | # plot view bounds 145 | xs = [ds[0,0,0], ds[0,-1,0],ds[-1,-1,0],ds[-1,0,0],ds[0,0,0]] 146 | ys = [ds[0,0,1], ds[0,-1,1],ds[-1,-1,1],ds[-1,0,1],ds[0,0,1]] 147 | zs = [ds[0,0,2], ds[0,-1,2],ds[-1,-1,2],ds[-1,0,2],ds[0,0,2]] 148 | ax.plot(xs, ys, zs, c='red') 149 | 150 | #fix the scaling 151 | ax.autoscale_view(tight=None, scalex=False, scaley=False, scalez=True) 152 | 153 | x_bound = ax.get_xlim() 154 | y_bound = ax.get_ylim() 155 | new_bound = (min(x_bound[0], y_bound[0]), max(x_bound[1], y_bound[1])) 156 | ax.set_xlim(left=new_bound[0], right=new_bound[1]) 157 | ax.set_ylim(bottom=new_bound[0], top=new_bound[1]) 158 | 159 | # Ray helpers 160 | 161 | def get_rays(H, W, focal, c2w, override_dirs=None): 162 | """Get ray origins, directions from a pinhole camera.""" 163 | i, j = tf.meshgrid(tf.range(W, dtype=tf.float32), 164 | tf.range(H, dtype=tf.float32), indexing='xy') 165 | dirs = tf.stack([(i-W*.5)/focal, -(j-H*.5)/focal, -tf.ones_like(i)], -1) 166 | if override_dirs is not None: 167 | dirs = override_dirs 168 | rays_d = tf.reduce_sum(dirs[..., np.newaxis, :] * c2w[:3, :3], -1) 169 | rays_o = tf.broadcast_to(c2w[:3, -1], tf.shape(rays_d)) 170 | return rays_o, rays_d 171 | 172 | 173 | def get_rays_np(H, W, focal, c2w, override_dirs=None): 174 | """Get ray origins, directions from a pinhole camera.""" 175 | i, j = np.meshgrid(np.arange(W, dtype=np.float32), 176 | np.arange(H, dtype=np.float32), indexing='xy') 177 | dirs = np.stack([(i-W*.5)/focal, -(j-H*.5)/focal, -np.ones_like(i)], -1) 178 | if override_dirs is not None: 179 | dirs = override_dirs 180 | rays_d = np.sum(dirs[..., np.newaxis, :] * c2w[:3, :3], -1) 181 | rays_o = np.broadcast_to(c2w[:3, -1], np.shape(rays_d)) 182 | # print(rays_d[1]) 183 | # print(rays_d[50]) 184 | return rays_o, rays_d 185 | 186 | 187 | def ndc_rays(H, W, focal, near, rays_o, rays_d): 188 | """Normalized device coordinate rays. 189 | 190 | Space such that the canvas is a cube with sides [-1, 1] in each axis. 191 | 192 | Args: 193 | H: int. Height in pixels. 194 | W: int. Width in pixels. 195 | focal: float. Focal length of pinhole camera. 196 | near: float or array of shape[batch_size]. Near depth bound for the scene. 197 | rays_o: array of shape [batch_size, 3]. Camera origin. 198 | rays_d: array of shape [batch_size, 3]. Ray direction. 199 | 200 | Returns: 201 | rays_o: array of shape [batch_size, 3]. Camera origin in NDC. 202 | rays_d: array of shape [batch_size, 3]. Ray direction in NDC. 203 | """ 204 | # Shift ray origins to near plane 205 | t = -(near + rays_o[..., 2]) / rays_d[..., 2] 206 | rays_o = rays_o + t[..., None] * rays_d 207 | 208 | # Projection 209 | o0 = -1./(W/(2.*focal)) * rays_o[..., 0] / rays_o[..., 2] 210 | o1 = -1./(H/(2.*focal)) * rays_o[..., 1] / rays_o[..., 2] 211 | o2 = 1. + 2. * near / rays_o[..., 2] 212 | 213 | d0 = -1./(W/(2.*focal)) * \ 214 | (rays_d[..., 0]/rays_d[..., 2] - rays_o[..., 0]/rays_o[..., 2]) 215 | d1 = -1./(H/(2.*focal)) * \ 216 | (rays_d[..., 1]/rays_d[..., 2] - rays_o[..., 1]/rays_o[..., 2]) 217 | d2 = -2. * near / rays_o[..., 2] 218 | 219 | rays_o = tf.stack([o0, o1, o2], -1) 220 | rays_d = tf.stack([d0, d1, d2], -1) 221 | print(rays_d[...,2]) 222 | return rays_o, rays_d 223 | 224 | def ndc_points(H, W, focal, near, points): 225 | ''' 226 | Convert 3d world points to normalized device coordinates 227 | ''' 228 | # Projection 229 | o0 = -1./(W/(2.*focal)) * points[..., 0] / points[..., 2] 230 | o1 = -1./(H/(2.*focal)) * points[..., 1] / points[..., 2] 231 | o2 = 1. + 2. * near / points[..., 2] 232 | 233 | points = tf.stack([o0, o1, o2], -1) 234 | return points 235 | 236 | 237 | def mat_bilinear_interpolation(known, H, W, x_offset, y_offset, gap): 238 | ''' 239 | Performs bilear interpolation for Z-2 dimensional points, given a set of Z dimensional vectors 240 | where the first two dimensions are x and y positions 241 | Arguments: 242 | known - the known points arranged in a grid with a distance of size between 243 | points in the x and y directions. Should start at (0,0) 244 | H - x upper bound for interpolated points 245 | W - y upper bound for interpolated points 246 | x_offset - the x index of the first desired point (subsequent points will be spaced by gap) 247 | y_offset - the y index of the first desired point (subsequent points will be spaced by gap) 248 | gap - The x and y gap size between points for the known grid and the desired grid of points 249 | 250 | Returns: 251 | A grid of points, gap distance apart in x and y, interpolated from the known points 252 | 253 | NOTE: No longer used 254 | ''' 255 | 256 | num_y = H // gap + (H%gap > y_offset) 257 | num_x = W // gap + (W%gap > x_offset) 258 | new_shape = (num_y, num_x) + tuple(known.shape[2:]) 259 | interpolated = np.zeros(new_shape) 260 | 261 | # interpolate in the x direction 262 | x_blend = (1 - x_offset/gap) * known[:, :-1, ...] + (x_offset/gap) * known[:, 1:, ...] 263 | 264 | # now blend in the y direction 265 | full_blend = (1 - y_offset/gap) * x_blend[:-1, ...] + (y_offset/gap) * x_blend[1:, ...] 266 | 267 | interpolated[:full_blend.shape[0], :full_blend.shape[1], ...] = full_blend 268 | 269 | # handle points that aren't between four other points 270 | 271 | # bottom row points 272 | if num_y > full_blend.shape[0]: 273 | bottom_row = (1 - x_offset/gap) * known[-1:, :-1, ...] + (x_offset/gap) * known[-1:, 1:, ...] 274 | interpolated[-1:, :bottom_row.shape[1], ...] = bottom_row 275 | 276 | # far right points 277 | if num_x > full_blend.shape[1]: 278 | right_column = (1 - y_offset/gap) * known[:-1, -1:, ...] + (y_offset/gap) * known[1:, -1:, ...] 279 | interpolated[:right_column.shape[0], -1:, ...] = right_column 280 | 281 | # bottom right corner 282 | if num_y > full_blend.shape[0] and num_x > full_blend.shape[1]: 283 | interpolated[-1,-1,...] = known[-1, -1, ...] 284 | 285 | return interpolated 286 | 287 | 288 | def weighted_sampling_interpolation(known_z_vals, weights, H, W, x_offset, y_offset, gap, samples, det=False): 289 | ''' 290 | Produces samples for a new ray by drawing from the pdfs of neighboring rays with known 291 | pdfs. Number of samples drawn from neighboring pdfs depends on their distance. 292 | Arguments: 293 | known_z_vals - an array with the z_values of the known pdfs 294 | weights - an array with the weights of the known pdfs 295 | H - height of the image being reconstructed (this method is a subroutine of render()) 296 | W - width of the image being reconstructed 297 | x_offset - the x coordinate of the first point being interpolated 298 | y_offset - the y coordinate of the first point being interpolated 299 | gap - the x and y gap between points to interpolate (and the known points) 300 | samples - the total number of samples to draw 301 | ''' 302 | 303 | num_y = H // gap + (H%gap > y_offset) 304 | num_x = W // gap + (W%gap > x_offset) 305 | new_shape = (num_y, num_x) + (samples,) 306 | interpolated = np.zeros(new_shape) 307 | 308 | vertical = (x_offset == 0) 309 | horizontal = (y_offset == 0) 310 | 311 | # duplicate the last row/column of the values if necessary 312 | # (we may want to interpolate somewhere where there aren't four surrounding points) 313 | far_bottom = (known_z_vals.shape[0] - 1 < num_y) 314 | far_right = (known_z_vals.shape[1] - 1 < num_x) 315 | 316 | # size of the known grid 317 | known_y = known_z_vals.shape[0] 318 | known_x = known_z_vals.shape[1] 319 | 320 | new_z_shape = (known_y + far_bottom, known_x + far_right) + tuple(known_z_vals.shape[2:]) 321 | new_weights_shape = (known_y + far_bottom, known_x + far_bottom) + tuple(weights.shape[2:]) 322 | new_zs = np.zeros(new_z_shape) 323 | new_weights = np.zeros(new_weights_shape) 324 | 325 | # fill the new, larger, arrays 326 | new_zs[:known_y, :known_x, ...] = known_z_vals 327 | new_weights[:known_y, :known_x, ...] = weights 328 | 329 | if far_bottom: 330 | new_zs[-1:, :known_x, ...] = known_z_vals[-1:, :, ...] 331 | new_weights[-1:, :known_x, ...] = weights[-1:, :, ...] 332 | 333 | if far_right: 334 | new_zs[:known_y, -1:, ...] = known_z_vals[:,-1:, ...] 335 | new_weights[:known_y, -1:, ...] = weights[:,-1:, ...] 336 | 337 | if far_bottom and far_right: 338 | new_zs[-1:, -1:, ...] = known_z_vals[-1:, -1:, ...] 339 | new_weights[-1:, -1:, ...] = weights[-1:, -1:, ...] 340 | 341 | known_z_vals = tf.convert_to_tensor(new_zs, dtype=tf.float32) 342 | weights = tf.convert_to_tensor(new_weights, dtype=tf.float32) 343 | 344 | 345 | # if directly between two points vertically 346 | if vertical: 347 | top_left = round((1 - y_offset/gap) * samples) 348 | bottom_left = round((y_offset/gap) * samples) 349 | elif horizontal: 350 | top_left = round((1 - x_offset/gap) * samples) 351 | top_right = round((x_offset/gap) * samples) 352 | else: 353 | top_left = int((1 - x_offset/gap) * (1 - y_offset/gap) * samples) 354 | top_right = int((x_offset/gap) * (1 - y_offset/gap) * samples) 355 | bottom_left = int((1 - x_offset/gap) * (y_offset/gap) * samples) 356 | bottom_right = int((x_offset/gap) * (y_offset/gap) * samples) 357 | 358 | # randomly assign remainders 359 | for i in range(samples - top_left - top_right - bottom_left - bottom_right): 360 | ray = random.randint(0,3) 361 | if ray == 0: 362 | top_left += 1 363 | elif ray == 1: 364 | top_right += 1 365 | elif ray == 2: 366 | bottom_left += 1 367 | elif ray == 3: 368 | bottom_right += 1 369 | 370 | # sample the four distributions 371 | top_left_samples = sample_pdf(known_z_vals[:-1, :-1, ...], weights[:-1,:-1,...], top_left, det=det) 372 | if not vertical: 373 | top_right_samples = sample_pdf(known_z_vals[:-1,1:, ...], weights[:-1,1:,...], top_right, det=det) 374 | if not horizontal: 375 | bottom_left_samples = sample_pdf(known_z_vals[1:, :-1, ...], weights[1:,:-1,...], bottom_left, det=det) 376 | if not (vertical or horizontal): 377 | bottom_right_samples = sample_pdf(known_z_vals[1:, 1:, ...], weights[1:,1:,...], bottom_right, det=det) 378 | 379 | # combine the samples 380 | if vertical: 381 | all_points = [top_left_samples, bottom_left_samples] 382 | elif horizontal: 383 | all_points = [top_left_samples, top_right_samples] 384 | else: 385 | all_points = [top_left_samples, top_right_samples, bottom_left_samples, bottom_right_samples] 386 | combined_samples = tf.sort(tf.concat(all_points, -1), -1) 387 | interpolated[:combined_samples.shape[0], :combined_samples.shape[1], ...] = combined_samples 388 | 389 | #This should be taken care of at the beginning now 390 | 391 | # # if we want to interpolate below the known points 392 | # far_bottom = combined_samples.shape[0] < num_y 393 | # if far_bottom: 394 | # left = round((1-x_offset/gap) * samples) 395 | # right = round((x_offset/gap) * samples) 396 | # left_samples = sample_pdf(known_z_vals[-1:,:-1, ...], weights[-1:,:-1,...], left, det=det) 397 | # right_samples = sample_pdf(known_z_vals[-1:,1:,...], weights[-1:,1:,...], right, det=det) 398 | # combined_samples = tf.sort(tf.concat([left_samples, right_samples], -1), -1) 399 | # interpolated[-1:, :combined_samples.shape[1], ...] = combined_samples 400 | 401 | # # if we want to interpolate to the right of the known points 402 | # far_right = combined_samples.shape[1] < num_x 403 | # if far_right: 404 | # top = round((1-y_offset/gap) * samples) 405 | # bottom = round((y_offset/gap) * samples) 406 | # top_samples = sample_pdf(known_z_vals[:-1,-1:, ...], weights[:-1,-1:,...], top, det=det) 407 | # bottom_samples = sample_pdf(known_z_vals[1:,-1:,...], weights[1:,-1:,...], bottom, det=det) 408 | # combined_samples = tf.sort(tf.concat([top_samples, bottom_samples], -1), -1) 409 | # interpolated[:combined_samples.shape[0],-1:,...] = combined_samples 410 | 411 | # # handle far bottom right point if necessary 412 | # if far_bottom and far_right: 413 | # interpolated[-1:,-1:,...] = sample_pdf(known_z_vals[-1:,-1:, ...], weights[-1:,-1:, ...], samples, det=det) 414 | 415 | return tf.cast(interpolated, dtype=tf.float32) 416 | 417 | 418 | 419 | # Hierarchical sampling helper 420 | 421 | def sample_pdf(bins, weights, N_samples, det=False):\ 422 | # NOTE: had to cast a bunch of stuff to tf.double for some reason 423 | 424 | weights = tf.cast(weights, dtype=tf.float64) 425 | # Get pdf 426 | weights += 1e-5 # prevent nans 427 | pdf = weights / tf.reduce_sum(weights, -1, keepdims=True) 428 | cdf = tf.cumsum(pdf, -1) 429 | cdf = tf.concat([tf.zeros_like(cdf[..., :1]), cdf], -1) 430 | cdf = tf.cast(cdf, dtype=tf.double) 431 | 432 | # Take uniform samples 433 | if det: 434 | u = tf.linspace(0., 1., N_samples) 435 | u = tf.broadcast_to(u, list(cdf.shape[:-1]) + [N_samples]) 436 | else: 437 | u = tf.random.uniform(list(cdf.shape[:-1]) + [N_samples]) 438 | 439 | u = tf.cast(u, dtype=tf.double) 440 | 441 | # Invert CDF 442 | inds = tf.searchsorted(cdf, u, side='right') 443 | below = tf.maximum(0, inds-1) 444 | above = tf.minimum(cdf.shape[-1]-1, inds) 445 | inds_g = tf.stack([below, above], -1) 446 | cdf_g = tf.gather(cdf, inds_g, axis=-1, batch_dims=len(inds_g.shape)-2) 447 | bins_g = tf.gather(bins, inds_g, axis=-1, batch_dims=len(inds_g.shape)-2) 448 | 449 | denom = (cdf_g[..., 1]-cdf_g[..., 0]) 450 | denom = tf.where(denom < 1e-5, tf.ones_like(denom), denom) 451 | t = (u-cdf_g[..., 0])/denom 452 | t = tf.cast(t, dtype=tf.double) 453 | bins_g = tf.cast(bins_g, dtype=tf.double) 454 | samples = bins_g[..., 0] + t * (bins_g[..., 1]-bins_g[..., 0]) 455 | 456 | # back to float 457 | samples = tf.cast(samples, dtype=tf.float32) 458 | return samples 459 | -------------------------------------------------------------------------------- /sandbox.py: -------------------------------------------------------------------------------- 1 | import os, sys 2 | # os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true' 3 | os.environ['CUDA_VISIBLE_DEVICES'] = '1' 4 | import tensorflow as tf 5 | tf.compat.v1.enable_eager_execution() 6 | 7 | import numpy as np 8 | import imageio 9 | import json 10 | import random 11 | import time 12 | import pprint 13 | 14 | import matplotlib 15 | # matplotlib.use('Qt4Agg') 16 | 17 | 18 | import run_nerf 19 | from run_nerf_helpers import * 20 | 21 | from load_llff import load_llff_data 22 | from load_deepvoxels import load_dv_data 23 | from load_blender import load_blender_data 24 | 25 | from mpl_toolkits.mplot3d import Axes3D 26 | import matplotlib.pyplot as plt 27 | 28 | import random 29 | 30 | 31 | points_3d = np.load("./newpointcloud.npy") 32 | centers = np.load("./camcenters.npy") 33 | 34 | # plot 35 | fig = plt.figure() 36 | ax = fig.add_subplot(111, projection='3d') 37 | 38 | # reduce points for visualization 39 | red_factor = 1 40 | inds = random.sample(range(points_3d.shape[0]), int(points_3d.shape[0]/ red_factor)) 41 | points_3d = np.take(points_3d, inds, axis=0) 42 | ax.scatter(points_3d[:, 1], points_3d[:, 0], points_3d[:, 2], c="C0", alpha=0.8) 43 | ax.scatter(centers[:, 1], centers[:,0], centers[:,2], c="red", alpha=0.5) 44 | # xs = [-1,-1,1,1,-1] 45 | # ys = [-1,1,1,-1,-1] 46 | # zs = [-1,-1,-1,-1,-1] 47 | # ax.plot(xs,ys,zs) 48 | ax.autoscale_view(tight=None, scalex=False, scaley=False, scalez=True) 49 | x_bound = ax.get_xlim() 50 | y_bound = ax.get_ylim() 51 | new_bound = (min(x_bound[0], y_bound[0]), max(x_bound[1], y_bound[1])) 52 | ax.set_xlim(left=new_bound[0], right=new_bound[1]) 53 | ax.set_ylim(bottom=new_bound[0], top=new_bound[1]) 54 | plt.show() 55 | -------------------------------------------------------------------------------- /sum.out: -------------------------------------------------------------------------------- 1 | ## SLURM PROLOG ############################################################### 2 | ## Job ID : 15828829 3 | ## Job Name : ccv_batchscript 4 | ## Nodelist : gpu1202 5 | ## CPUs : 1 6 | ## Mem/Node : 8192 MB 7 | ## Directory : /gpfs/home/bli31/2952k/FastNeRF 8 | ## Started : Sun Dec 6 12:38:58 EST 2020 9 | ############################################################################### 10 | module: unloading 'python/2.7.12' 11 | module: loading 'python/3.7.4' 12 | module: loading 'anaconda/2-4.3.0' 13 | 2020-12-06 12:39:07.424490: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 14 | 2020-12-06 12:39:07.474758: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 15 | name: TITAN V major: 7 minor: 0 memoryClockRate(GHz): 1.455 16 | pciBusID: 0000:af:00.0 17 | 2020-12-06 12:39:07.483866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 18 | 2020-12-06 12:39:07.523416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 19 | 2020-12-06 12:39:07.556895: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 20 | 2020-12-06 12:39:07.612671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 21 | 2020-12-06 12:39:07.672854: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 22 | 2020-12-06 12:39:07.698173: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 23 | 2020-12-06 12:39:07.840692: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 24 | 2020-12-06 12:39:07.842046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 25 | 2020-12-06 12:39:07.845415: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA 26 | 2020-12-06 12:39:07.859631: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz 27 | 2020-12-06 12:39:07.860281: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5630066d81b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 28 | 2020-12-06 12:39:07.860305: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 29 | 2020-12-06 12:39:07.861106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 30 | name: TITAN V major: 7 minor: 0 memoryClockRate(GHz): 1.455 31 | pciBusID: 0000:af:00.0 32 | 2020-12-06 12:39:07.861141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 33 | 2020-12-06 12:39:07.861154: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 34 | 2020-12-06 12:39:07.861164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 35 | 2020-12-06 12:39:07.861175: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 36 | 2020-12-06 12:39:07.861185: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 37 | 2020-12-06 12:39:07.861195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 38 | 2020-12-06 12:39:07.861206: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 39 | 2020-12-06 12:39:07.862373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 40 | 2020-12-06 12:39:07.862411: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 41 | 2020-12-06 12:39:07.974176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 42 | 2020-12-06 12:39:07.974223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 43 | 2020-12-06 12:39:07.974243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 44 | 2020-12-06 12:39:07.976711: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 45 | 2020-12-06 12:39:07.976768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11155 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:af:00.0, compute capability: 7.0) 46 | 2020-12-06 12:39:07.978966: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x563006feedb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 47 | 2020-12-06 12:39:07.978991: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): TITAN V, Compute Capability 7.0 48 | WARNING:tensorflow: 49 | The TensorFlow contrib module will not be included in TensorFlow 2.0. 50 | For more information, please see: 51 | * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md 52 | * https://github.com/tensorflow/addons 53 | * https://github.com/tensorflow/io (for I/O related ops) 54 | If you depend on functionality not listed there, please file an issue. 55 | 56 | 2020-12-06 12:39:17.223955: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 57 | WARNING:tensorflow:From /gpfs/home/bli31/2952k/FastNeRF/run_nerf_helpers.py:207: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. 58 | Instructions for updating: 59 | Use tf.where in 2.0, which has the same broadcast rule as np.where 60 | WARNING:tensorflow:From /gpfs/home/bli31/2952k/FastNeRF/run_nerf_helpers.py:14: The name tf.log is deprecated. Please use tf.math.log instead. 61 | 62 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 63 | return array(a, dtype, copy=False, order=order, subok=True) 64 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 65 | return array(a, dtype, copy=False, order=order, subok=True) 66 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 67 | return array(a, dtype, copy=False, order=order, subok=True) 68 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 69 | return array(a, dtype, copy=False, order=order, subok=True) 70 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 71 | return array(a, dtype, copy=False, order=order, subok=True) 72 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 73 | [swscaler @ 0x565316f9a400] Warning: data is not aligned! This can lead to a speed loss 74 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 75 | [swscaler @ 0x55f13424fb80] Warning: data is not aligned! This can lead to a speed loss 76 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 77 | [swscaler @ 0x5600b7526400] Warning: data is not aligned! This can lead to a speed loss 78 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 79 | return array(a, dtype, copy=False, order=order, subok=True) 80 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 81 | return array(a, dtype, copy=False, order=order, subok=True) 82 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 83 | return array(a, dtype, copy=False, order=order, subok=True) 84 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 85 | return array(a, dtype, copy=False, order=order, subok=True) 86 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 87 | return array(a, dtype, copy=False, order=order, subok=True) 88 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 89 | [swscaler @ 0x5583c5192400] Warning: data is not aligned! This can lead to a speed loss 90 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 91 | [swscaler @ 0x565346197b80] Warning: data is not aligned! This can lead to a speed loss 92 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 93 | [swscaler @ 0x55698d5a0400] Warning: data is not aligned! This can lead to a speed loss 94 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 95 | return array(a, dtype, copy=False, order=order, subok=True) 96 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 97 | return array(a, dtype, copy=False, order=order, subok=True) 98 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 99 | return array(a, dtype, copy=False, order=order, subok=True) 100 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 101 | return array(a, dtype, copy=False, order=order, subok=True) 102 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 103 | return array(a, dtype, copy=False, order=order, subok=True) 104 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 105 | [swscaler @ 0x55d39787a400] Warning: data is not aligned! This can lead to a speed loss 106 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 107 | [swscaler @ 0x55b67cd7bb80] Warning: data is not aligned! This can lead to a speed loss 108 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 109 | [swscaler @ 0x558eeecde400] Warning: data is not aligned! This can lead to a speed loss 110 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 111 | return array(a, dtype, copy=False, order=order, subok=True) 112 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 113 | return array(a, dtype, copy=False, order=order, subok=True) 114 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 115 | return array(a, dtype, copy=False, order=order, subok=True) 116 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 117 | return array(a, dtype, copy=False, order=order, subok=True) 118 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 119 | return array(a, dtype, copy=False, order=order, subok=True) 120 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 121 | [swscaler @ 0x5559392fe400] Warning: data is not aligned! This can lead to a speed loss 122 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 123 | [swscaler @ 0x55ab93b3cb80] Warning: data is not aligned! This can lead to a speed loss 124 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 125 | [swscaler @ 0x55b6fccd8400] Warning: data is not aligned! This can lead to a speed loss 126 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 127 | return array(a, dtype, copy=False, order=order, subok=True) 128 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 129 | return array(a, dtype, copy=False, order=order, subok=True) 130 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 131 | return array(a, dtype, copy=False, order=order, subok=True) 132 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 133 | return array(a, dtype, copy=False, order=order, subok=True) 134 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 135 | return array(a, dtype, copy=False, order=order, subok=True) 136 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 137 | [swscaler @ 0x564f1dff1400] Warning: data is not aligned! This can lead to a speed loss 138 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 139 | [swscaler @ 0x56053ab8db80] Warning: data is not aligned! This can lead to a speed loss 140 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 141 | [swscaler @ 0x560123d52400] Warning: data is not aligned! This can lead to a speed loss 142 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 143 | return array(a, dtype, copy=False, order=order, subok=True) 144 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 145 | return array(a, dtype, copy=False, order=order, subok=True) 146 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 147 | return array(a, dtype, copy=False, order=order, subok=True) 148 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 149 | return array(a, dtype, copy=False, order=order, subok=True) 150 | /users/bli31/anaconda/nerf/lib/python3.7/site-packages/numpy/core/_asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray 151 | return array(a, dtype, copy=False, order=order, subok=True) 152 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 153 | [swscaler @ 0x56139b5a3400] Warning: data is not aligned! This can lead to a speed loss 154 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 155 | [swscaler @ 0x55fb84232b80] Warning: data is not aligned! This can lead to a speed loss 156 | IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (504, 378) to (512, 384) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). 157 | [swscaler @ 0x5647b6db7400] Warning: data is not aligned! This can lead to a speed loss 158 | slurmstepd: error: *** JOB 15828829 ON gpu1202 CANCELLED AT 2020-12-07T04:53:45 DUE TO TIME LIMIT *** 159 | -------------------------------------------------------------------------------- /tiny_nerf_data.npz.1: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/houchenst/FastNeRF/05da0b0ff7c19767721ddc6bfd09b19268dfef48/tiny_nerf_data.npz.1 --------------------------------------------------------------------------------