Abstract
Novelty is a crucial criterion in the peer review process for evaluatingacademic papers. Traditionally, it's judged by experts or measure by uniquereference combinations. Both methods have limitations: experts have limitedknowledge, and the effectiveness of the combination method is uncertain.Moreover, it's unclear if unique citations truly measure novelty. The largelanguage model (LLM) possesses a wealth of knowledge, while human expertspossess judgment abilities that the LLM does not possess. Therefore, ourresearch integrates the knowledge and abilities of LLM and human experts toaddress the limitations of novelty assessment. One of the most common types ofnovelty in academic papers is the introduction of new methods. In this paper,we propose leveraging human knowledge and LLM to assist pretrained languagemodels (PLMs, e.g. BERT etc.) in predicting the method novelty of papers.Specifically, we extract sentences related to the novelty of the academic paperfrom peer review reports and use LLM to summarize the methodology section ofthe academic paper, which are then used to fine-tune PLMs. In addition, we havedesigned a text-guided fusion module with novel Sparse-Attention to betterintegrate human and LLM knowledge. We compared the method we proposed with alarge number of baselines. Extensive experiments demonstrate that our methodachieves superior performance.