Abstract
Neurons in the brain are spatially organized such that neighbors on tissueoften exhibit similar response profiles. In the human language system,experimental studies have observed clusters for syntactic and semanticcategories, but the mechanisms underlying this functional organization remainunclear. Here, building on work from the vision literature, we develop TopoLM,a transformer language model with an explicit two-dimensional spatialrepresentation of model units. By combining a next-token prediction objectivewith a spatial smoothness loss, representations in this model assemble intoclusters that correspond to semantically interpretable groupings of text andclosely match the functional organization in the brain's language system.TopoLM successfully predicts the emergence of the spatio-functionalorganization of a cortical language system as well as the organization offunctional clusters selective for fine-grained linguistic features empiricallyobserved in human cortex. Our results suggest that the functional organizationof the human language system is driven by a unified spatial objective, andprovide a functionally and spatially aligned model of language processing inthe brain.