Abstract
Controlling the movements of dynamic objects and the camera within generatedvideos is a meaningful yet challenging task. Due to the lack of datasets withcomprehensive motion annotations, existing algorithms can not simultaneouslycontrol the motions of both camera and objects, resulting in limitedcontrollability over generated contents. To address this issue and facilitatethe research in this field, we introduce a Synthetic Dataset for Free-FormMotion Control (SynFMC). The proposed SynFMC dataset includes diverse objectsand environments and covers various motion patterns according to specificrules, simulating common and complex real-world scenarios. The complete 6D poseinformation facilitates models learning to disentangle the motion effects fromobjects and the camera in a video. To validate the effectiveness andgeneralization of SynFMC, we further propose a method, Free-Form Motion Control(FMC). FMC enables independent or simultaneous control of object and cameramovements, producing high-fidelity videos. Moreover, it is compatible withvarious personalized text-to-image (T2I) models for different content styles.Extensive experiments demonstrate that the proposed FMC outperforms previousmethods across multiple scenarios.