Self-Training Meets Consistency: Improving LLMs' Reasoning With Consistency-Driven Rationale Evaluation

  • 2024-11-22 08:54:17
  • Jaehyeok Lee, Keisuke Sakaguchi, JinYeong Bak
  • 0

Abstract

Self-training approach for large language models (LLMs) improves reasoningabilities by training the models on their self-generated rationales. Previousapproaches have labeled rationales that produce correct answers for a givenquestion as appropriate for training. However, a single measure risksmisjudging rationale quality, leading the models to learn flawed reasoningpatterns. To address this issue, we propose CREST (Consistency-driven RationaleEvaluation for Self-Training), a self-training framework that further evaluateseach rationale through follow-up questions and leverages this evaluation toguide its training. Specifically, we introduce two methods: (1) filtering outrationales that frequently result in incorrect answers on follow-up questionsand (2) preference learning based on mixed preferences from rationaleevaluation results of both original and follow-up questions. Experiments onthree question-answering datasets using open LLMs show that CREST not onlyimproves the logical robustness and correctness of rationales but also improvesreasoning abilities compared to previous self-training approaches.

 

Quick Read (beta)

loading the full paper ...