Abstract
The newly emerged transformer technology has a tremendous impact on NLPresearch. In the general English domain, transformer-based models have achievedstate-of-the-art performances on various NLP benchmarks. In the clinicaldomain, researchers also have investigated transformer models for clinicalapplications. The goal of this study is to systematically explore three widelyused transformer-based models (i.e., BERT, RoBERTa, and XLNet) for clinicalrelation extraction and develop an open-source package with clinicalpre-trained transformer-based models to facilitate information extraction inthe clinical domain. We developed a series of clinical RE models based on threetransformer architectures, namely BERT, RoBERTa, and XLNet. We evaluated thesemodels using 2 publicly available datasets from 2018 MADE1.0 and 2018 n2c2challenges. We compared two classification strategies (binary vs. multi-classclassification) and investigated two approaches to generate candidate relationsin different experimental settings. In this study, we compared threetransformer-based (BERT, RoBERTa, and XLNet) models for relation extraction. Wedemonstrated that the RoBERTa-clinical RE model achieved the best performanceon the 2018 MADE1.0 dataset with an F1-score of 0.8958. On the 2018 n2c2dataset, the XLNet-clinical model achieved the best F1-score of 0.9610. Ourresults indicated that the binary classification strategy consistentlyoutperformed the multi-class classification strategy for clinical relationextraction. Our methods and models are publicly available athttps://github.com/uf-hobi-informatics-lab/ClinicalTransformerRelationExtraction.We believe this work will improve current practice on clinical relationextraction and other related NLP tasks in the biomedical domain.