Abstract
Recent progress in multimodal generation has increasingly combinedautoregressive (AR) and diffusion-based approaches, leveraging theircomplementary strengths: AR models capture long-range dependencies and producefluent, context-aware outputs, while diffusion models operate in continuouslatent spaces to refine high-fidelity visual details. However, existing hybridsoften lack systematic guidance on how and why to allocate model capacitybetween these paradigms. In this work, we introduce MADFormer, a MixedAutoregressive and Diffusion Transformer that serves as a testbed for analyzingAR-diffusion trade-offs. MADFormer partitions image generation into spatialblocks, using AR layers for one-pass global conditioning across blocks anddiffusion layers for iterative local refinement within each block. Throughcontrolled experiments on FFHQ-1024 and ImageNet, we identify two key insights:(1) block-wise partitioning significantly improves performance onhigh-resolution images, and (2) vertically mixing AR and diffusion layersyields better quality-efficiency balances--improving FID by up to 75% underconstrained inference compute. Our findings offer practical design principlesfor future hybrid generative models.