Abstract
Both humans and neural language models are able to perform subject-verbnumber agreement (SVA). In principle, semantics shouldn't interfere with thistask, which only requires syntactic knowledge. In this work we test whethermeaning interferes with this type of agreement in English in syntacticstructures of various complexities. To do so, we generate both semanticallywell-formed and nonsensical items. We compare the performance of BERT-base tothat of humans, obtained with a psycholinguistic online crowdsourcingexperiment. We find that BERT and humans are both sensitive to our semanticmanipulation: They fail more often when presented with nonsensical items,especially when their syntactic structure features an attractor (a noun phrasebetween the subject and the verb that has not the same number as the subject).We also find that the effect of meaningfulness on SVA errors is stronger forBERT than for humans, showing higher lexical sensitivity of the former on thistask.