Abstract
We present Bit Diffusion: a simple and generic approach for generatingdiscrete data with continuous diffusion models. The main idea behind ourapproach is to first represent the discrete data as binary bits, and then traina continuous diffusion model to model these bits as real numbers which we callanalog bits. To generate samples, the model first generates the analog bits,which are then thresholded to obtain the bits that represent the discretevariables. We further propose two simple techniques, namely Self-Conditioningand Asymmetric Time Intervals, which lead to a significant improvement insample quality. Despite its simplicity, the proposed approach can achievestrong performance in both discrete image generation and image captioningtasks. For discrete image generation, we significantly improve previousstate-of-the-art on both CIFAR-10 (which has 3K discrete 8-bit tokens) andImageNet-64x64 (which has 12K discrete 8-bit tokens), outperforming the bestautoregressive model in both sample quality (measured by FID) and efficiency.For image captioning on MS-COCO dataset, our approach achieves competitiveresults compared to autoregressive models.