Abstract
Large language models (LLMs) have demonstrated remarkable reasoningcapabilities across diverse domains. Recent studies have shown that increasingtest-time computation enhances LLMs' reasoning capabilities. This typicallyinvolves extensive sampling at inference time guided by an external LLMverifier, resulting in a two-player system. Despite external guidance, theeffectiveness of this system demonstrates the potential of a single LLM totackle complex tasks. Thus, we pose a new research problem: Can we internalizethe searching capabilities to fundamentally enhance the reasoning abilities ofa single LLM? This work explores an orthogonal direction focusing onpost-training LLMs for autoregressive searching (i.e., an extended reasoningprocess with self-reflection and self-exploration of new strategies). Toachieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and atwo-stage training paradigm: 1) a small-scale format tuning stage tointernalize the COAT reasoning format and 2) a large-scale self-improvementstage leveraging reinforcement learning. Our approach results in Satori, a 7BLLM trained on open-source models and data. Extensive empirical evaluationsdemonstrate that Satori achieves state-of-the-art performance on mathematicalreasoning benchmarks while exhibits strong generalization to out-of-domaintasks. Code, data, and models are fully open-sourced.