Abstract
Updating diffusion models in an incremental setting would be practical inreal-world applications yet computationally challenging. We present a novellearning strategy of Concept Neuron Selection (CNS), a simple yet effectiveapproach to perform personalization in a continual learning scheme. CNSuniquely identifies neurons in diffusion models that are closely related to thetarget concepts. In order to mitigate catastrophic forgetting problems whilepreserving zero-shot text-to-image generation ability, CNS finetunes conceptneurons in an incremental manner and jointly preserves knowledge learned ofprevious concepts. Evaluation of real-world datasets demonstrates that CNSachieves state-of-the-art performance with minimal parameter adjustments,outperforming previous methods in both single and multi-concept personalizationworks. CNS also achieves fusion-free operation, reducing memory storage andprocessing time for continual personalization.