Abstract
We study the task of predicting dynamic physical properties from videos. Morespecifically, we consider physical properties that require temporal informationto be inferred: elasticity of a bouncing object, viscosity of a flowing liquid,and dynamic friction of an object sliding on a surface. To this end, we makethe following contributions: (i) We collect a new video dataset for eachphysical property, consisting of synthetic training and testing splits, as wellas a real split for real world evaluation. (ii) We explore three ways to inferthe physical property from videos: (a) an oracle method where we supply thevisual cues that intrinsically reflect the property using classical computervision techniques; (b) a simple read out mechanism using a visual prompt andtrainable prompt vector for cross-attention on pre-trained video generative andself-supervised models; and (c) prompt strategies for Multi-modal LargeLanguage Models (MLLMs). (iii) We show that video foundation models trained ina generative or self-supervised manner achieve a similar performance, thoughbehind that of the oracle, and MLLMs are currently inferior to the othermodels, though their performance can be improved through suitable prompting.