Abstract
Pre-trained transformer language models (TLMs) have recently refashionednatural language processing (NLP): Most state-of-the-art NLP models now operateon top of TLMs to benefit from contextualization and knowledge induction. Toexplain their success, the scientific community conducted numerous analyses.Besides other methods, syntactic agreement tests were utilized to analyse TLMs.Most of the studies were conducted for the English language, however. In thiswork, we analyse German TLMs. To this end, we design numerous agreement tasks,some of which consider peculiarities of the German language. Our experimentalresults show that state-of-the-art German TLMs generally perform well onagreement tasks, but we also identify and discuss syntactic structures thatpush them to their limits.