Abstract
Masked diffusion models (MDMs) have emerged as a popular research topic forgenerative modeling of discrete data, thanks to their superior performance overother discrete diffusion models, and are rivaling the auto-regressive models(ARMs) for language modeling tasks. The recent effort in simplifying the maskeddiffusion framework further leads to alignment with continuous-space diffusionmodels and more principled training and sampling recipes. In this paper,however, we reveal that both training and sampling of MDMs are theoreticallyfree from the time variable, arguably the key signature of diffusion models,and are instead equivalent to masked models. The connection on the samplingaspect is drawn by our proposed first-hitting sampler (FHS). Specifically, weshow that the FHS is theoretically equivalent to MDMs' original generationprocess while significantly alleviating the time-consuming categorical samplingand achieving a 20$\times$ speedup. In addition, our investigation challengesprevious claims that MDMs can surpass ARMs in generative perplexity. Weidentify, for the first time, an underlying numerical issue, even with the32-bit floating-point precision, which results in inaccurate categoricalsampling. We show that the numerical issue lowers the effective temperatureboth theoretically and empirically, leading to unfair assessments of MDMs'generation results in the previous literature.