Inferring representations of 3D scenes from 2D observations is a fundamentalproblem of computer graphics, computer vision, and artificial intelligence.Emerging 3D-structured neural scene representations are a promising approach to3D scene understanding. In this work, we propose a novel neural scenerepresentation, Light Field Networks or LFNs, which represent both geometry andappearance of the underlying 3D scene in a 360-degree, four-dimensional lightfield parameterized via a neural implicit representation. Rendering a ray froman LFN requires only a *single* network evaluation, as opposed to hundreds ofevaluations per ray for ray-marching or volumetric based renderers in3D-structured neural scene representations. In the setting of simple scenes, weleverage meta-learning to learn a prior over LFNs that enables multi-viewconsistent light field reconstruction from as little as a single imageobservation. This results in dramatic reductions in time and memory complexity,and enables real-time rendering. The cost of storing a 360-degree light fieldvia an LFN is two orders of magnitude lower than conventional methods such asthe Lumigraph. Utilizing the analytical differentiability of neural implicitrepresentations and a novel parameterization of light space, we furtherdemonstrate the extraction of sparse depth maps from LFNs.