Abstract
Despite significant progress in diffusion-based image generation,subject-driven generation and instruction-based editing remain challenging.Existing methods typically treat them separately, struggling with limitedhigh-quality data and poor generalization. However, both tasks requirecapturing complex visual variations while maintaining consistency betweeninputs and outputs. Therefore, we propose MIGE, a unified framework thatstandardizes task representations using multimodal instructions. It treatssubject-driven generation as creation on a blank canvas and instruction-basedediting as modification of an existing image, establishing a sharedinput-output formulation. MIGE introduces a novel multimodal encoder that mapsfree-form multimodal instructions into a unified vision-language space,integrating visual and semantic features through a feature fusion mechanism.This unification enables joint training of both tasks, providing two keyadvantages: (1) Cross-Task Enhancement: By leveraging shared visual andsemantic representations, joint training improves instruction adherence andvisual consistency in both subject-driven generation and instruction-basedediting. (2) Generalization: Learning in a unified format facilitatescross-task knowledge transfer, enabling MIGE to generalize to novelcompositional tasks, including instruction-based subject-driven editing.Experiments show that MIGE excels in both subject-driven generation andinstruction-based editing while setting a state-of-the-art in the new task ofinstruction-based subject-driven editing. Code and model have been publiclyavailable at https://github.com/Eureka-Maggie/MIGE.