Data-hungry deep neural networks have established themselves as the standardfor many NLP tasks including the traditional sequence tagging ones. Despitetheir state-of-the-art performance on high-resource languages, they still fallbehind of their statistical counter-parts in low-resource scenarios. Onemethodology to counter attack this problem is text augmentation, i.e.,generating new synthetic training data points from existing data. Although NLPhas recently witnessed a load of textual augmentation techniques, the fieldstill lacks a systematic performance analysis on a diverse set of languages andsequence tagging tasks. To fill this gap, we investigate three categories oftext augmentation methodologies which perform changes on the syntax (e.g.,cropping sub-sentences), token (e.g., random word insertion) and character(e.g., character swapping) levels. We systematically compare them onpart-of-speech tagging, dependency parsing and semantic role labeling for adiverse set of language families using various models including thearchitectures that rely on pretrained multilingual contextualized languagemodels such as mBERT. Augmentation most significantly improves dependencyparsing, followed by part-of-speech tagging and semantic role labeling. We findthe experimented techniques to be effective on morphologically rich languagesin general rather than analytic languages such as Vietnamese. Our resultssuggest that the augmentation techniques can further improve over strongbaselines based on mBERT. We identify the character-level methods as the mostconsistent performers, while synonym replacement and syntactic augmentersprovide inconsistent improvements. Finally, we discuss that the results mostheavily depend on the task, language pair, and the model type.