Abstract
Despite recent advances in deep learning-based language modelling, manynatural language processing (NLP) tasks in the financial domain remainchallenging due to the paucity of appropriately labelled data. Other issuesthat can limit task performance are differences in word distribution betweenthe general corpora - typically used to pre-train language models - andfinancial corpora, which often exhibit specialized language and symbology.Here, we investigate two approaches that may help to mitigate these issues.Firstly, we experiment with further language model pre-training using largeamounts of in-domain data from business and financial news. We then applyaugmentation approaches to increase the size of our dataset for modelfine-tuning. We report our findings on an Environmental, Social and Governance(ESG) controversies dataset and demonstrate that both approaches are beneficialto accuracy in classification tasks.