Abstract
The paper gives an overview of the Russian Semantic Similarity Evaluation(RUSSE) shared task held in conjunction with the Dialogue 2015 conference.There exist a lot of comparative studies on semantic similarity, yet noanalysis of such measures was ever performed for the Russian language.Exploring this problem for the Russian language is even more interesting,because this language has features, such as rich morphology and free wordorder, which make it significantly different from English, German, and otherwell-studied languages. We attempt to bridge this gap by proposing a sharedtask on the semantic similarity of Russian nouns. Our key contribution is anevaluation methodology based on four novel benchmark datasets for the Russianlanguage. Our analysis of the 105 submissions from 19 teams reveals thatsuccessful approaches for English, such as distributional and skip-gram models,are directly applicable to Russian as well. On the one hand, the best resultsin the contest were obtained by sophisticated supervised models that combineevidence from different sources. On the other hand, completely unsupervisedapproaches, such as a skip-gram model estimated on a large-scale corpus, wereable score among the top 5 systems.