Abstract
Deep learning-based Natural Language Processing methods, especiallytransformers, have achieved impressive performance in the last few years.Applying those state-of-the-art NLP methods to legal activities to automate orsimplify some simple work is of great value. This work investigates the valueof domain adaptive pre-training and language adapters in legal NLP tasks. Bycomparing the performance of language models with domain adaptive pre-trainingon different tasks and different dataset splits, we show that domain adaptivepre-training is only helpful with low-resource downstream tasks, thus far frombeing a panacea. We also benchmark the performance of adapters in a typicallegal NLP task and show that they can yield similar performance to full modeltuning with much smaller training costs. As an additional result, we releaseLegalRoBERTa, a RoBERTa model further pre-trained on legal corpora.