Abstract
Classifiers are an important and defining feature of the Chinese language,and their correct prediction is key to numerous educational applications. Yet,whether the most popular Large Language Models (LLMs) possess proper knowledgethe Chinese classifiers is an issue that has largely remain unexplored in theNatural Language Processing (NLP) literature. To address such a question, we employ various masking strategies to evaluatethe LLMs' intrinsic ability, the contribution of different sentence elements,and the working of the attention mechanisms during prediction. Besides, weexplore fine-tuning for LLMs to enhance the classifier performance. Our findings reveal that LLMs perform worse than BERT, even with fine-tuning.The prediction, as expected, greatly benefits from the information about thefollowing noun, which also explains the advantage of models with abidirectional attention mechanism such as BERT.