Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion

  • 2023-01-27 14:52:53
  • Flavio Schneider, Zhijing Jin, Bernhard Schölkopf
  • 47


The recent surge in popularity of diffusion models for image generation hasbrought new attention to the potential of these models in other areas of mediasynthesis. One area that has yet to be fully explored is the application ofdiffusion models to music generation. Music generation requires to handlemultiple aspects, including the temporal dimension, long-term structure,multiple layers of overlapping sounds, and nuances that only trained listenerscan detect. In our work, we investigate the potential of diffusion models fortext-conditional music generation. We develop a cascading latent diffusionapproach that can generate multiple minutes of high-quality stereo music at48kHz from textual descriptions. For each model, we make an effort to maintainreasonable inference speed, targeting real-time on a single consumer GPU. Inaddition to trained models, we provide a collection of open-source librarieswith the hope of facilitating future work in the field. We open-source the following: - Music samples for this paper: - All music samples for all models: - Codes:


Quick Read (beta)

loading the full paper ...