Abstract
The alignments of reasoning abilities between smaller and larger LanguageModels are largely conducted via Supervised Fine-Tuning (SFT) usingdemonstrations generated from robust Large Language Models (LLMs). Althoughthese approaches deliver more performant models, they do not show sufficientlystrong generalization ability as the training only relies on the provideddemonstrations. In this paper, we propose the Self-refine Instruction-tuning method thatelicits Smaller Language Models to self-refine their abilities. Our approach isbased on a two-stage process, where reasoning abilities are first transferredbetween LLMs and Small Language Models (SLMs) via Instruction-tuning ondemonstrations provided by LLMs, and then the instructed models Self-refinetheir abilities through preference optimization strategies. In particular, thesecond phase operates refinement heuristics based on the Direct PreferenceOptimization algorithm, where the SLMs are elicited to deliver a series ofreasoning paths by automatically sampling the generated responses and providingrewards using ground truths from the LLMs. Results obtained on commonsense andmath reasoning tasks show that this approach significantly outperformsInstruction-tuning in both in-domain and out-domain scenarios, aligning thereasoning abilities of Smaller and Larger Language Models.