Advancing the Evaluation of Traditional Chinese Language Models: Towards a Comprehensive Benchmark Suite

  • 2023-09-15 15:52:23
  • Chan-Jan Hsu, Chang-Le Liu, Feng-Ting Liao, Po-Chun Hsu, Yi-Chang Chen, Da-shan Shiu
  • 0


The evaluation of large language models is an essential task in the field oflanguage understanding and generation. As language models continue to advance,the need for effective benchmarks to assess their performance has becomeimperative. In the context of Traditional Chinese, there is a scarcity ofcomprehensive and diverse benchmarks to evaluate the capabilities of languagemodels, despite the existence of certain benchmarks such as DRCD, TTQA, CMDQA,and FGC dataset. To address this gap, we propose a novel set of benchmarks thatleverage existing English datasets and are tailored to evaluate language modelsin Traditional Chinese. These benchmarks encompass a wide range of tasks,including contextual question-answering, summarization, classification, andtable understanding. The proposed benchmarks offer a comprehensive evaluationframework, enabling the assessment of language models' capabilities acrossdifferent tasks. In this paper, we evaluate the performance of GPT-3.5,Taiwan-LLaMa-v1.0, and Model 7-C, our proprietary model, on these benchmarks.The evaluation results highlight that our model, Model 7-C, achievesperformance comparable to GPT-3.5 with respect to a part of the evaluatedcapabilities. In an effort to advance the evaluation of language models inTraditional Chinese and stimulate further research in this field, we haveopen-sourced our benchmark and opened the model for trial.


Quick Read (beta)

loading the full paper ...