Multi-view stereo-regulated NeRF for urban scene novel view synthesis

Abstract

Neural radiance fields (NeRF), which encode a scene into a neural representation, have demonstrated impressive novel view synthesis quality on single object and small regions of space. However, when faced with urban outdoor environments, NeRF is limited by the capacity of a single MLP and insufficient input views, leading to incorrect geometries that hinder the production of realistic renderings. In this paper, we present MVSRegNeRF, an extension of neural radiance fields focused on large-scale autonomous driving scenario. We employ traditional patch-match based Multi-view stereo (MVS) method to generate dense depth maps, which we utilize to regulate the geometry optimization of NeRF. We also integrate multi-resolution hash encodings into our neural scene representation to accelerate the training process. Thanks to the relatively precise geometry constraint of our approach, we achieve high-quality novel view synthesis on real-world large-scale street scene. Our experiments on the KITTI-360 dataset demonstrate that MVSRegNeRF outperforms the state-of-the-art methods in Novel View Appearance Synthesis tasks.