Abstract
Character-level models have been used extensively in recent years in NLPtasks as both supplements and replacements for closed-vocabulary token-levelword representations. In one popular architecture, character-level LSTMs areused to feed token representations into a sequence tagger predictingtoken-level annotations such as part-of-speech (POS) tags. In this work, weexamine the behavior of POS taggers across languages from the perspective ofindividual hidden units within the character LSTM. We aggregate the behavior ofthese units into language-level metrics which quantify the challenges thattaggers face on languages with different morphological properties, and identifylinks between synthesis and affixation preference and emergent behavior of thehidden tagger layer. In a comparative experiment, we show how modifying thebalance between forward and backward hidden units affects model arrangement andperformance in these types of languages.