Abstract
We present a novel deep learning architecture for probabilistic futureprediction from video. We predict the future semantics, geometry and motion ofcomplex real-world urban scenes and use this representation to control anautonomous vehicle. This work is the first to jointly predict ego-motion,static scene, and the motion of dynamic agents in a probabilistic manner, whichallows sampling consistent, highly probable futures from a compact latentspace. Our model learns a representation from RGB video with a spatio-temporalconvolutional module. The learned representation can be explicitly decoded tofuture semantic segmentation, depth, and optical flow, in addition to being aninput to a learnt driving policy. To model the stochasticity of the future, weintroduce a conditional variational approach which minimises the divergencebetween the present distribution (what could happen given what we have seen)and the future distribution (what we observe actually happens). Duringinference, diverse futures are generated by sampling from the presentdistribution.