Abstract
Knowledge editing aims to correct outdated or inaccurate knowledge in neuralnetworks. In this paper, we explore knowledge editing using easily accessibledocuments instead of manually labeled factual triples employed in earlierresearch. To advance this field, we establish the first evaluation benchmark,\textit{DocTER}, featuring Documents containing counterfactual knowledge forediting. A comprehensive four-perspective evaluation is introduced: EditSuccess, Locality, Reasoning, and Cross-lingual Transfer. To adapt conventionaltriplet-based knowledge editing methods for this task, we develop anExtract-then-Edit pipeline that extracts triples from documents before applyingexisting methods. Experiments on popular knowledge editing methods demonstratethat editing with documents presents significantly greater challenges thanusing triples. In document-based scenarios, even the best-performing in-contextediting approach still lags behind by 10 points in editing success whencompared to using gold triples. This observation also holds for both reasoningand cross-lingual test sets. We further analyze key factors influencing taskperformance, including the quality of extracted triples, the frequency andposition of edited knowledge in documents, various methods for enhancingreasoning, and performance differences across various directions incross-lingual knowledge editing, which provide valuable insights for futureresearch.