Abstract
As models increasingly leverage multi-step reasoning strategies to solvecomplex problems, supervising the logical validity of these intermediate stepshas become a critical research challenge. Process reward models address this byproviding step-by-step feedback, but current approaches have two majordrawbacks: they typically function as classifiers without providingexplanations, and their reliance on supervised fine-tuning with static datasetslimits generalization. Inspired by recent advances, we reframe stepwise rewardmodeling from a classification task to a reasoning task itself. We thus proposea generative judge that reasons about the policy model's reasoning steps (i.e.,meta-reasons), outputting thinking tokens before delivering a final verdict.Our model, StepWiser, is trained by reinforcement learning using relativeoutcomes of rollouts. We show it provides (i) better judgment accuracy onintermediate steps than existing methods; (ii) can be used to improve thepolicy model at training time; and (iii) improves inference-time search.