Abstract
Modern visual effects (VFX) software has made it possible for skilled artiststo create imagery of virtually anything. However, the creation process remainslaborious, complex, and largely inaccessible to everyday users. In this work,we present AutoVFX, a framework that automatically creates realistic anddynamic VFX videos from a single video and natural language instructions. Bycarefully integrating neural scene modeling, LLM-based code generation, andphysical simulation, AutoVFX is able to provide physically-grounded,photorealistic editing effects that can be controlled directly using naturallanguage instructions. We conduct extensive experiments to validate AutoVFX'sefficacy across a diverse spectrum of videos and instructions. Quantitative andqualitative results suggest that AutoVFX outperforms all competing methods by alarge margin in generative quality, instruction alignment, editing versatility,and physical plausibility.