Cones: Concept Neurons in Diffusion Models for Customized Generation

  • 2023-03-09 09:16:04
  • Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, Yang Cao
  • 343


Human brains respond to semantic features of presented stimuli with differentneurons. It is then curious whether modern deep neural networks admit a similarbehavior pattern. Specifically, this paper finds a small cluster of neurons ina diffusion model corresponding to a particular subject. We call those neuronsthe concept neurons. They can be identified by statistics of network gradientsto a stimulation connected with the given subject. The concept neuronsdemonstrate magnetic properties in interpreting and manipulating generationresults. Shutting them can directly yield the related subject contextualized indifferent scenes. Concatenating multiple clusters of concept neurons canvividly generate all related concepts in a single image. A few steps of furtherfine-tuning can enhance the multi-concept capability, which may be the first tomanage to generate up to four different subjects in a single image. Forlarge-scale applications, the concept neurons are environmentally friendly aswe only need to store a sparse cluster of int index instead of dense float32values of the parameters, which reduces storage consumption by 90\% comparedwith previous subject-driven generation methods. Extensive qualitative andquantitative studies on diverse scenarios show the superiority of our method ininterpreting and manipulating diffusion models.


Quick Read (beta)

loading the full paper ...