Abstract
Generating sound effects for videos often requires creating artistic soundeffects that diverge significantly from real-life sources and flexible controlin the sound design. To address this problem, we introduce MultiFoley, a modeldesigned for video-guided sound generation that supports multimodalconditioning through text, audio, and video. Given a silent video and a textprompt, MultiFoley allows users to create clean sounds (e.g., skateboard wheelsspinning without wind noise) or more whimsical sounds (e.g., making a lion'sroar sound like a cat's meow). MultiFoley also allows users to choose referenceaudio from sound effects (SFX) libraries or partial videos for conditioning. Akey novelty of our model lies in its joint training on both internet videodatasets with low-quality audio and professional SFX recordings, enablinghigh-quality, full-bandwidth (48kHz) audio generation. Through automatedevaluations and human studies, we demonstrate that MultiFoley successfullygenerates synchronized high-quality sounds across varied conditional inputs andoutperforms existing methods. Please see our project page for video results:https://ificl.github.io/MultiFoley/