Abstract
Deaf and hard of hearing individuals regularly rely on captioning whilewatching live TV. Live TV captioning is evaluated by regulatory agencies usingvarious caption evaluation metrics. However, caption evaluation metrics areoften not informed by preferences of DHH users or how meaningful the captionsare. There is a need to construct caption evaluation metrics that take therelative importance of words in a transcript into account. We conductedcorrelation analysis between two types of word embeddings and human-annotatedlabeled word-importance scores in existing corpus. We found that normalizedcontextualized word embeddings generated using BERT correlated better withmanually annotated importance scores than word2vec-based word embeddings. Wemake available a pairing of word embeddings and their human-annotatedimportance scores. We also provide proof-of-concept utility by training wordimportance models, achieving an F1-score of 0.57 in the 6-class word importanceclassification task.