Abstract
The success of autoregressive (AR) language models in text generation hasinspired the computer vision community to adopt Large Language Models (LLMs)for image generation. However, considering the essential differences betweentext and image modalities, the design space of language models for imagegeneration remains underexplored. We observe that image tokens exhibit greaterrandomness compared to text tokens, which presents challenges when trainingwith token prediction. Nevertheless, AR models demonstrate their potential byeffectively learning patterns even from a seemingly suboptimal optimizationproblem. Our analysis also reveals that while all models successfully grasp theimportance of local information in image generation, smaller models struggle tocapture the global context. In contrast, larger models showcase improvedcapabilities in this area, helping to explain the performance gains achievedwhen scaling up model size. We further elucidate the design space of languagemodels for vision generation, including tokenizer choice, model choice, modelscalability, vocabulary design, and sampling strategy through extensivecomparative experiments. Our work is the first to analyze the optimizationbehavior of language models in vision generation, and we believe it can inspiremore effective designs when applying LMs to other domains. Finally, ourelucidated language model for image generation, termed as ELM, achievesstate-of-the-art performance on the ImageNet 256*256 benchmark. The code isavailable at https://github.com/Pepperlll/LMforImageGeneration.git.