Embedding polygonal mesh assets within photorealistic Neural Radience Fields (NeRF) volumes, such that they can be rendered and their dynamics simulated in a physically consistent manner with the NeRF, is under-explored from the system perspective of integrating NeRF into the traditional graphics pipeline.
This paper designs a two-way coupling between mesh and NeRF during rendering and simulation. We first review the light transport equations for both mesh and NeRF, then distill them into a straightforward algorithm for updating radiance and throughput along a cast ray with an arbitrary number of bounces. To resolve the discrepancy between the linear color space that the path tracer assumes, versus the sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range (HDR) images. We also present a strategy to estimate light sources and cast shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric formulation can be efficiently integrated with a high-performance physics simulator that supports cloth, rigid and soft bodies. The full rendering and simulation system can be run on a GPU at interactive rates.
We show that a hybrid system approach outperforms alternatives in visual realism for mesh insertion, because it allows realistic light transport from volumetric NeRF media onto surfaces, which affects the appearance of reflective/refractive surfaces and illumination of diffuse surfaces informed by the scene.
Our method takes an optimized radiance field model and surface meshes as inputs. We can run physiscs simulation between the NeRF and meshes. The updated mesh vertices and NeRF transformations are synchronized to the renderer, which uses Monte Carlo simulation to sample ray paths. As the ray travels through space, it alternates between surface rendering (ray-tracing) and NeRF rendering (ray-marching), both updating its radiance.
he NeRF volume rendering is largely based on NVIDA's great work InstantNGP.
The physics part also runs with NVIDIA's Warp and/or DeepMind's Mujoco
.A more detailed tutorial for ray tracing can be found in Ray Tracing in One Weekend.
We also borrow Nerfies' beautiful html template and sincerely thank Nerfies!
@article{qiao2023dmrf,
author = {Yi-Ling Qiao and Alexander Gao and Yiran Xu and Yue Feng and Jia-Bin Huang and Ming C. Lin},
title = {Dynamic Mesh-Aware Radiance Fields},
journal = {ICCV},
year = {2023},
}