Abstract
We introduce Hybrid Autoregressive Transformer (HART), an autoregressive (AR)visual generation model capable of directly generating 1024x1024 images,rivaling diffusion models in image generation quality. Existing AR models facelimitations due to the poor image reconstruction quality of their discretetokenizers and the prohibitive training costs associated with generating 1024pximages. To address these challenges, we present the hybrid tokenizer, whichdecomposes the continuous latents from the autoencoder into two components:discrete tokens representing the big picture and continuous tokens representingthe residual components that cannot be represented by the discrete tokens. Thediscrete component is modeled by a scalable-resolution discrete AR model, whilethe continuous component is learned with a lightweight residual diffusionmodule with only 37M parameters. Compared with the discrete-only VAR tokenizer,our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K,leading to a 31% generation FID improvement from 7.85 to 5.38. HART alsooutperforms state-of-the-art diffusion models in both FID and CLIP score, with4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourcedat https://github.com/mit-han-lab/hart.