Abstract
Learning coordinate-based volumetric 3D scene representations such as neuralradiance fields (NeRF) has been so far studied assuming RGB or RGB-D images asinputs. At the same time, it is known from the neuroscience literature thathuman visual system (HVS) is tailored to process asynchronous brightnesschanges rather than synchronous RGB images, in order to build and continuouslyupdate mental 3D representations of the surroundings for navigation andsurvival. Visual sensors that were inspired by HVS principles are eventcameras. Thus, events are sparse and asynchronous per-pixel brightness (orcolour channel) change signals. In contrast to existing works on neural 3Dscene representation learning, this paper approaches the problem from a newperspective. We demonstrate that it is possible to learn NeRF suitable fornovel-view synthesis in the RGB space from asynchronous event streams. Ourmodels achieve high visual accuracy of the rendered novel views of challengingscenes in the RGB space, even though they are trained with substantially fewerdata (i.e., event streams from a single event camera moving around the object)and more efficiently (due to the inherent sparsity of event streams) than theexisting NeRF models trained with RGB images. We will release our datasets andthe source code, see https://4dqv.mpi-inf.mpg.de/EventNeRF/.