Abstract
Many capable large language models (LLMs) are developed via self-supervisedpre-training followed by a reinforcement-learning fine-tuning phase, oftenbased on human or AI feedback. During this stage, models may be guided by theirinductive biases to rely on simpler features which may be easier to extract, ata cost to robustness and generalisation. We investigate whether principlesgoverning inductive biases in the supervised fine-tuning of LLMs also applywhen the fine-tuning process uses reinforcement learning. Following Lovering etal (2021), we test two hypotheses: that features more $\textit{extractable}$after pre-training are more likely to be utilised by the final policy, and thatthe evidence for/against a feature predicts whether it will be utilised.Through controlled experiments on synthetic and natural language tasks, we findstatistically significant correlations which constitute strong evidence forthese hypotheses.