Abstract
Large language models (LLMs) are demonstrably capable of cross-lingualtransfer, but can produce inconsistent output when prompted with the samequeries written in different languages. To understand how language models areable to generalize knowledge from one language to the others, we apply thelogit lens to interpret the implicit steps taken by LLMs to solve multilingualmulti-choice reasoning questions. We find LLMs predict inconsistently and areless accurate because they rely on subspaces of individual languages, ratherthan working in a shared semantic space. While larger models are moremultilingual, we show their hidden states are more likely to dissociate fromthe shared representation compared to smaller models, but are nevertheless morecapable of retrieving knowledge embedded across different languages. Finally,we demonstrate that knowledge sharing can be modulated by steering the models'latent processing towards the shared semantic space. We find reinforcingutilization of the shared space improves the models' multilingual reasoningperformance, as a result of more knowledge transfer from, and better outputconsistency with English.