Abstract
Text-to-audio (TTA) system has recently gained attention for its ability tosynthesize general audio based on text descriptions. However, previous studiesin TTA have limited generation quality with high computational costs. In thisstudy, we propose AudioLDM, a TTA system that is built on a latent space tolearn the continuous audio representations from contrastive language-audiopretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMswith audio embedding while providing text embedding as a condition duringsampling. By learning the latent representations of audio signals and theircompositions without modeling the cross-modal relationship, AudioLDM isadvantageous in both generation quality and computational efficiency. Trainedon AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTAperformance measured by both objective and subjective metrics (e.g., frechetdistance). Moreover, AudioLDM is the first TTA system that enables varioustext-guided audio manipulations (e.g., style transfer) in a zero-shot fashion.Our implementation and demos are available at https://audioldm.github.io.