Abstract
Despite their fundamental role, it remains unclear what properties could makevisual tokenizers more effective for generative modeling. We observe thatmodern generative models share a conceptually similar training objective --reconstructing clean signals from corrupted inputs such as Gaussian noise ormasking -- a process we term denoising. Motivated by this insight, we proposealigning tokenizer embeddings directly with the downstream denoising objective,encouraging latent embeddings to be more easily reconstructed even when heavilycorrupted. To achieve this, we introduce the Latent Denoising Tokenizer(l-DeTok), a simple yet effective tokenizer trained to reconstruct clean imagesfrom latent embeddings corrupted by interpolative noise and random masking.Extensive experiments on ImageNet 256x256 demonstrate that our tokenizerconsistently outperforms standard tokenizers across six representativegenerative models. Our findings highlight denoising as a fundamental designprinciple for tokenizer development, and we hope it could motivate newperspectives for future tokenizer design.