Abstract
Access to word-sentiment associations is useful for many applications,including sentiment analysis, stance detection, and linguistic analysis.However, manually assigning fine-grained sentiment association scores to wordshas many challenges with respect to keeping annotations consistent. We applythe annotation technique of Best-Worst Scaling to obtain real-valued sentimentassociation scores for words and phrases in three different domains: generalEnglish, English Twitter, and Arabic Twitter. We show that on all three domainsthe ranking of words by sentiment remains remarkably consistent even when theannotation process is repeated with a different set of annotators. We also, forthe first time, determine the minimum difference in sentiment association thatis perceptible to native speakers of a language.