Abstract
Today, most methods for image understanding tasks rely on feed-forward neuralnetworks. While this approach has allowed for empirical accuracy, efficiency,and task adaptation via fine-tuning, it also comes with fundamentaldisadvantages. Existing networks often struggle to generalize across differentdatasets, even on the same task. By design, these networks ultimately reasonabout high-dimensional scene features, which are challenging to analyze. Thisis true especially when attempting to predict 3D information based on 2Dimages. We propose to recast 3D multi-object tracking from RGB cameras as an\emph{Inverse Rendering (IR)} problem, by optimizing via a differentiablerendering pipeline over the latent space of pre-trained 3D objectrepresentations and retrieve the latents that best represent object instancesin a given input image. To this end, we optimize an image loss over generativelatent spaces that inherently disentangle shape and appearance properties. Weinvestigate not only an alternate take on tracking but our method also enablesexamining the generated objects, reasoning about failure situations, andresolving ambiguous cases. We validate the generalization and scalingcapabilities of our method by learning the generative prior exclusively fromsynthetic data and assessing camera-based 3D tracking on the nuScenes and Waymodatasets. Both these datasets are completely unseen to our method and do notrequire fine-tuning. Videos and code are available athttps://light.princeton.edu/inverse-rendering-tracking/.