Abstract
Language models (LMs) capabilities have grown with a fast pace over the pastdecade leading researchers in various disciplines, such as biomedical research,to increasingly explore the utility of LMs in their day-to-day applications.Domain specific language models have already been in use for biomedical naturallanguage processing (NLP) applications. Recently however, the interest hasgrown towards medical language models and their understanding capabilities. Inthis paper, we investigate the medical conceptual grounding of various languagemodels against expert clinicians for identification of hallmarks ofimmunotherapy in breast cancer abstracts. Our results show that pre-trainedlanguage models have potential to outperform large language models inidentifying very specific (low-level) concepts.