Abstract
This work presents a Fully BInarized Large Language Model (FBI-LLM),demonstrating for the first time how to train a large-scale binary languagemodel from scratch (not the partial binary or ternary LLM like BitNet b1.58) tomatch the performance of its full-precision counterparts (e.g., FP16 or BF16)in transformer-based LLMs. It achieves this by employing an autoregressivedistillation (AD) loss with maintaining equivalent model dimensions (130M,1.3B, 7B) and training data volume as regular LLM pretraining, while deliveringcompetitive results in terms of perplexity and task-specific effectiveness.Intriguingly, by analyzing the training trajectory, we find that the pretrainedweight is not necessary for training binarized LLMs from scratch. This researchencourages a new computational framework and may facilitate the future designof specialized hardware tailored for fully 1-bit LLMs. We make all models,code, and training dataset fully accessible and transparent to support furtherresearch (Code: https://github.com/LiqunMa/FBI-LLM. Model:https://huggingface.co/LiqunMa/).