Natural language understanding tasks such as open-domain question answeringoften require retrieving and assimilating factual information from multiplesources. We propose to address this problem by integrating a semi-parametricrepresentation of a large text corpus into a Transformer model as a source offactual knowledge. Specifically, our method represents knowledge with `mentionmemory', a table of dense vector representations of every entity mention in acorpus. The proposed model - TOME - is a Transformer that accesses theinformation through internal memory layers in which each entity mention in theinput passage attends to the mention memory. This approach enables synthesis ofand reasoning over many disparate sources of information within a singleTransformer model. In experiments using a memory of 150 million Wikipediamentions, TOME achieves strong performance on several open-domainknowledge-intensive tasks, including the claim verification benchmarks HoVerand FEVER and several entity-based QA benchmarks. We also show that the modellearns to attend to informative mentions without any direct supervision.Finally we demonstrate that the model can generalize to new unseen entities byupdating the memory without retraining.