Abstract
Incremental Learning (IL) has been a long-standing problem in both vision andNatural Language Processing (NLP) communities. In recent years, as Pre-trainedLanguage Models (PLMs) have achieved remarkable progress in various NLPdownstream tasks, utilizing PLMs as backbones has become a common practice inrecent research of IL in NLP. Most assume that catastrophic forgetting is thebiggest obstacle to achieving superior IL performance and propose varioustechniques to overcome this issue. However, we find that this assumption isproblematic. Specifically, we revisit more than 20 methods on fourclassification tasks (Text Classification, Intent Classification, RelationExtraction, and Named Entity Recognition) under the two most popular ILsettings (Class-Incremental and Task-Incremental) and reveal that most of themseverely underestimate the inherent anti-forgetting ability of PLMs. Based onthe observation, we propose a frustratingly easy method called SEQ* for IL withPLMs. The results show that SEQ* has competitive or superior performancecompared to state-of-the-art (SOTA) IL methods and requires considerably lesstrainable parameters and training time. These findings urge us to revisit theIL with PLMs and encourage future studies to have a fundamental understandingof the catastrophic forgetting in PLMs. The data, code and scripts are publiclyavailable athttps://github.com/zzz47zzz/codebase-for-incremental-learning-with-llm.