├── README.md └── assets ├── Relighting_via_HDRI.gif ├── architecture.jpg ├── dynamic_field.png ├── dynamic_sequences.gif ├── light_sampling_field.png └── qualitative_lightstage.jpg /README.md: -------------------------------------------------------------------------------- 1 | 2 |

3 | 4 |

Light Sampling Field and BRDF Representation for Physically-based Neural Rendering

5 |

6 | Jing Yang 7 | · 8 | Hanyuan Xiao 9 | · 10 | Wenbin Teng 11 | · 12 | Yunxuan Cai 13 | · 14 | Yajie Zhao 15 |

16 |

ICLR 2023

17 |
18 | Logo 19 |
20 | 21 |

22 |
23 | 24 | Paper PDF 25 | 26 | 27 | Paper PDF 28 | 29 | 30 | Paper PDF 31 | 32 | 33 | Project Page 34 | youtube views 35 |

36 |

37 | 38 |
39 |
40 | 41 | 42 |
43 | Table of Contents 44 |
    45 |
  1. 46 | Abstract 47 |
    2. 48 | 51 | 54 |
  2. 55 | Light Sampling Fields 56 |
    3. 57 |
  3. 58 | More Qualitative Results 59 |
    4. 60 |
  4. 61 | Citation 62 |
    5. 63 |
64 |
65 |
66 |
67 | 68 | ## Abstract 69 | Physically-based rendering (PBR) is key for immersive rendering effects used widely in the industry to showcase detailed realistic scenes from computer graphics assets. A well-known caveat is that producing the same is computationally heavy and relies on complex capture devices. Inspired by the success in quality and efficiency of recent volumetric neural rendering, we want to develop a physically-based neural shader to eliminate device dependency and significantly boost performance. However, no existing lighting and material models in the current neural rendering approaches can accurately represent the comprehensive lighting models and BRDFs properties required by the PBR process. Thus, this paper proposes a novel lighting representation that models direct and indirect light locally through a light sampling strategy in a learned light sampling field. We also propose BRDF models to separately represent surface/subsurface scattering details to enable complex objects such as translucent material (i.e., skin, jade). We then implement our proposed representations with an end-to-end physically-based neural face skin shader, which takes a standard face asset (i.e., geometry, albedo map, and normal map) and an HDRI for illumination as inputs and generates a photo-realistic rendering as output. Extensive experiments showcase the quality and efficiency of our PBR face skin shader, indicating the effectiveness of our proposed lighting and material representations. 70 | 71 | ## Light Sampling Fields 72 | | ![Light Sampling Field](assets/light_sampling_field.png) | 73 | | :-------------------------------------------------------------: | 74 | | Light Sampling Field Visualization | 75 | | ![Local Lighting Changes](assets/dynamic_field.png) | 76 | | Local Lighting Variations in Real Capture | 77 | 78 | ## More Qualitative Results 79 | 80 | | ![Dynamic Sequences](assets/dynamic_sequences.gif) | 81 | | :-------------------------------------------------------------: | 82 | | Multiview Relighting on Dynamic Sequences | 83 | | ![Relighting via HDRI](assets/Relighting_via_HDRI.gif) | 84 | | Relighting via HDRI as Global Lighting | 85 | | ![Qualitative Results](assets/qualitative_lightstage.jpg) | 86 | | Qualitative Results over Light Stage Scans | 87 | 88 | ## Citation 89 | 90 | ```bibtex 91 | @inproceedings{ 92 | yang2023light, 93 | title = {Light Sampling Field and {BRDF} Representation for Physically-based Neural Rendering}, 94 | author = {Jing Yang and Hanyuan Xiao and Wenbin Teng and Yunxuan Cai and Yajie Zhao}, 95 | booktitle = {The Eleventh International Conference on Learning Representations }, 96 | year = {2023}, 97 | url = {https://openreview.net/forum?id=yYEb8v65X8} 98 | } 99 | ``` 100 | 101 | ## Acknowledgments 102 | 103 | We extend our gratitude to [Marcel Ramos](), [Pratusha Prasad](), [Bipin Koshore]() for their expert artistic contributions and technical assistance with hardware and capture tools. Additionally, our appreciation goes to the [ICON](https://github.com/YuliangXiu/ICON) for their impressive and visually stunning GitHub showcase page. 104 | 105 |
106 | 107 | 110 | -------------------------------------------------------------------------------- /assets/Relighting_via_HDRI.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingyangcarl/LightSamplingField/2391735e59f5478303ece5b63db1e1b00a4e220e/assets/Relighting_via_HDRI.gif -------------------------------------------------------------------------------- /assets/architecture.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingyangcarl/LightSamplingField/2391735e59f5478303ece5b63db1e1b00a4e220e/assets/architecture.jpg -------------------------------------------------------------------------------- /assets/dynamic_field.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingyangcarl/LightSamplingField/2391735e59f5478303ece5b63db1e1b00a4e220e/assets/dynamic_field.png -------------------------------------------------------------------------------- /assets/dynamic_sequences.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingyangcarl/LightSamplingField/2391735e59f5478303ece5b63db1e1b00a4e220e/assets/dynamic_sequences.gif -------------------------------------------------------------------------------- /assets/light_sampling_field.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingyangcarl/LightSamplingField/2391735e59f5478303ece5b63db1e1b00a4e220e/assets/light_sampling_field.png -------------------------------------------------------------------------------- /assets/qualitative_lightstage.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingyangcarl/LightSamplingField/2391735e59f5478303ece5b63db1e1b00a4e220e/assets/qualitative_lightstage.jpg --------------------------------------------------------------------------------