Abstract
Intrinsic evaluations of OIE systems are carried out either manually -- withhuman evaluators judging the correctness of extractions -- or automatically, onstandardized benchmarks. The latter, while much more cost-effective, is lessreliable, primarily because of the incompleteness of the existing OIEbenchmarks: the ground truth extractions do not include all acceptable variantsof the same fact, leading to unreliable assessment of models' performance.Moreover, the existing OIE benchmarks are available for English only. In thiswork, we introduce BenchIE: a benchmark and evaluation framework forcomprehensive evaluation of OIE systems for English, Chinese and German. Incontrast to existing OIE benchmarks, BenchIE takes into account informationalequivalence of extractions: our gold standard consists of fact synsets,clusters in which we exhaustively list all surface forms of the same fact. Webenchmark several state-of-the-art OIE systems using BenchIE and demonstratethat these systems are significantly less effective than indicated by existingOIE benchmarks. We make BenchIE (data and evaluation code) publicly available.