Abstract
Understanding pragmatics-the use of language in context-is crucial fordeveloping NLP systems capable of interpreting nuanced language use. Despiterecent advances in language technologies, including large language models,evaluating their ability to handle pragmatic phenomena such as implicatures andreferences remains challenging. To advance pragmatic abilities in models, it isessential to understand current evaluation trends and identify existinglimitations. In this survey, we provide a comprehensive review of resourcesdesigned for evaluating pragmatic capabilities in NLP, categorizing datasets bythe pragmatics phenomena they address. We analyze task designs, data collectionmethods, evaluation approaches, and their relevance to real-world applications.By examining these resources in the context of modern language models, wehighlight emerging trends, challenges, and gaps in existing benchmarks. Oursurvey aims to clarify the landscape of pragmatic evaluation and guide thedevelopment of more comprehensive and targeted benchmarks, ultimatelycontributing to more nuanced and context-aware NLP models.