Abstract
The latest work on language representations carefully integratescontextualized features into language model training, which enables a series ofsuccess especially in various machine reading comprehension and naturallanguage inference tasks. However, the existing language representation modelsincluding ELMo, GPT and BERT only exploit plain context-sensitive features suchas character or word embeddings. They rarely consider incorporating structuredsemantic information which can provide rich semantics for languagerepresentation. To promote natural language understanding, we propose toincorporate explicit contextual semantics from pre-trained semantic rolelabeling, and introduce an improved language representation model,Semantics-aware BERT (SemBERT), which is capable of explicitly absorbingcontextual semantics over a BERT backbone. SemBERT keeps the convenientusability of its BERT precursor in a light fine-tuning way without substantialtask-specific modifications. Compared with BERT, semantics-aware BERT is assimple in concept but more powerful. It obtains new state-of-the-art orsubstantially improves results on ten reading comprehension and languageinference tasks.