Abstract
Conventional wisdom suggests that autoregressive models are used to processdiscrete data. When applied to continuous modalities such as visual data,Visual AutoRegressive modeling (VAR) typically resorts to quantization-basedapproaches to cast the data into a discrete space, which can introducesignificant information loss. To tackle this issue, we introduce a ContinuousVAR framework that enables direct visual autoregressive generation withoutvector quantization. The underlying theoretical foundation is strictly properscoring rules, which provide powerful statistical tools capable of evaluatinghow well a generative model approximates the true distribution. Within thisframework, all we need is to select a strictly proper score and set it as thetraining objective to optimize. We primarily explore a class of trainingobjectives based on the energy score, which is likelihood-free and thusovercomes the difficulty of making probabilistic predictions in the continuousspace. Previous efforts on continuous autoregressive generation, such as GIVTand diffusion loss, can also be derived from our framework using other strictlyproper scores. Source code: https://github.com/shaochenze/EAR.