Abstract
Task transfer, transferring knowledge contained in related tasks, holds thepromise of reducing the quantity of labeled data required to fine-tune languagemodels. Dialogue understanding encompasses many diverse tasks, yet tasktransfer has not been thoroughly studied in conversational AI. This workexplores conversational task transfer by introducing FETA: a benchmark forfew-sample task transfer in open-domain dialogue. FETA contains two underlyingsets of conversations upon which there are 10 and 7 tasks annotated, enablingthe study of intra-dataset task transfer; task transfer without domainadaptation. We utilize three popular language models and three learningalgorithms to analyze the transferability between 132 source-target task pairsand create a baseline for future work. We run experiments in the single- andmulti-source settings and report valuable findings, e.g., most performancetrends are model-specific, and span extraction and multiple-choice tasksbenefit the most from task transfer. In addition to task transfer, FETA can bea valuable resource for future research into the efficiency andgeneralizability of pre-training datasets and model architectures, as well asfor learning settings such as continual and multitask learning.