Abstract
Although diffusion-based models can generate high-quality and high-resolutionvideo sequences from textual or image inputs, they lack explicit integration ofgeometric cues when controlling scene lighting and visual appearance acrossframes. To address this limitation, we propose IllumiCraft, an end-to-enddiffusion framework accepting three complementary inputs: (1)high-dynamic-range (HDR) video maps for detailed lighting control; (2)synthetically relit frames with randomized illumination changes (optionallypaired with a static background reference image) to provide appearance cues;and (3) 3D point tracks that capture precise 3D geometry information. Byintegrating the lighting, appearance, and geometry cues within a unifieddiffusion architecture, IllumiCraft generates temporally coherent videosaligned with user-defined prompts. It supports background-conditioned andtext-conditioned video relighting and provides better fidelity than existingcontrollable video generation methods. Project Page:https://yuanze-lin.me/IllumiCraft_page