Abstract
Reinforcement learning from human feedback (RLHF) is widely used to trainlarge language models (LLMs). However, it is unclear whether LLMs accuratelylearn the underlying preferences in human feedback data. We coin the term\textit{Learned Feedback Pattern} (LFP) for patterns in an LLM's activationslearned during RLHF that improve its performance on the fine-tuning task. Wehypothesize that LLMs with LFPs accurately aligned to the fine-tuning feedbackexhibit consistent activation patterns for outputs that would have receivedsimilar feedback during RLHF. To test this, we train probes to estimate thefeedback signal implicit in the activations of a fine-tuned LLM. We thencompare these estimates to the true feedback, measuring how accurate the LFPsare to the fine-tuning feedback. Our probes are trained on a condensed, sparseand interpretable representation of LLM activations, making it easier tocorrelate features of the input with our probe's predictions. We validate ourprobes by comparing the neural features they correlate with positive feedbackinputs against the features GPT-4 describes and classifies as related to LFPs.Understanding LFPs can help minimize discrepancies between LLM behavior andtraining objectives, which is essential for the safety of LLMs.