Abstract
Systems that can efficiently search collections of sign language videos havebeen highlighted as a useful application of sign language technology. However,the problem of searching videos beyond individual keywords has received limitedattention in the literature. To address this gap, in this work we introduce thetask of sign language retrieval with free-form textual queries: given a writtenquery (e.g., a sentence) and a large collection of sign language videos, theobjective is to find the signing video in the collection that best matches thewritten query. We propose to tackle this task by learning cross-modalembeddings on the recently introduced large-scale How2Sign dataset of AmericanSign Language (ASL). We identify that a key bottleneck in the performance ofthe system is the quality of the sign video embedding which suffers from ascarcity of labeled training data. We, therefore, propose SPOT-ALIGN, aframework for interleaving iterative rounds of sign spotting and featurealignment to expand the scope and scale of available training data. We validatethe effectiveness of SPOT-ALIGN for learning a robust sign video embeddingthrough improvements in both sign recognition and the proposed video retrievaltask.