Abstract
Text-Image-to-Video (TI2V) generation aims to generate a video from an imagefollowing a text description, which is also referred to as text-guided imageanimation. Most existing methods struggle to generate videos that align wellwith the text prompts, particularly when motion is specified. To overcome thislimitation, we introduce MotiF, a simple yet effective approach that directsthe model's learning to the regions with more motion, thereby improving thetext alignment and motion generation. We use optical flow to generate a motionheatmap and weight the loss according to the intensity of the motion. Thismodified objective leads to noticeable improvements and complements existingmethods that utilize motion priors as model inputs. Additionally, due to thelack of a diverse benchmark for evaluating TI2V generation, we propose TI2VBench, a dataset consists of 320 image-text pairs for robust evaluation. Wepresent a human evaluation protocol that asks the annotators to select anoverall preference between two videos followed by their justifications. Througha comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourcedmodels, achieving an average preference of 72%. The TI2V Bench is released inhttps://wang-sj16.github.io/motif/.