Abstract
Training tool-augmented LLMs has emerged as a promising approach to enhancinglanguage models' capabilities for complex tasks. The current supervisedfine-tuning paradigm relies on constructing extensive domain-specific datasetsto train models. However, this approach often struggles to generalizeeffectively to unfamiliar or intricate tool-use scenarios. Recently,reinforcement learning (RL) paradigm can endow LLMs with superior reasoning andgeneralization abilities. In this work, we address a key question: Can the pureRL be used to effectively elicit a model's intrinsic reasoning capabilities andenhance the tool-agnostic generalization? We propose a dynamicgeneralization-guided reward design for rule-based RL, which progressivelyshifts rewards from exploratory to exploitative tool-use patterns. Based onthis design, we introduce the Tool-Zero series models. These models are trainedto enable LLMs to autonomously utilize general tools by directly scaling up RLfrom Zero models (i.e., base models without post-training). Experimentalresults demonstrate that our models achieve over 7% performance improvementcompared to both SFT and RL-with-SFT models under the same experimentalsettings. These gains are consistently replicated across cross-dataset andintra-dataset evaluations, validating the effectiveness and robustness of ourmethods.