Abstract
In order for language models to aid physics research, they must first encoderepresentations of mathematical and natural language discourse which lead tocoherent explanations, with correct ordering and relevance of statements. Wepresent a collection of datasets developed to evaluate the performance oflanguage models in this regard, which measure capabilities with respect tosentence ordering, position, section prediction, and discourse coherence.Analysis of the data reveals equations and sub-disciplines which are mostcommon in physics discourse, as well as the sentence-level frequency ofequations and expressions. We present baselines that demonstrate howcontemporary language models are challenged by coherence related tasks inphysics, even when trained on mathematical natural language objectives.