Abstract
We address the problem of American Sign Language fingerspelling recognitionin the wild, using videos collected from websites. We introduce the largestdata set available so far for the problem of fingerspelling recognition, andthe first using naturally occurring video data. Using this data set, we presentthe first attempt to recognize fingerspelling sequences in this challengingsetting. Unlike prior work, our video data is extremely challenging due to lowframe rates and visual variability. To tackle the visual challenges, we train aspecial-purpose signing hand detector using a small subset of our data. Giventhe hand detector output, a sequence model decodes the hypothesizedfingerspelled letter sequence. For the sequence model, we exploreattention-based recurrent encoder-decoders and CTC-based approaches. As thefirst attempt at fingerspelling recognition in the wild, this work is intendedto serve as a baseline for future work on sign language recognition inrealistic conditions. We find that, as expected, letter error rates are muchhigher than in previous work on more controlled data, and we analyze thesources of error and effects of model variants.