Abstract
Recently, foundation language models (LMs) have marked significantachievements in the domains of natural language processing (NLP) and computervision (CV). Unlike traditional neural network models, foundation LMs obtain agreat ability for transfer learning by acquiring rich commonsense knowledgethrough pre-training on extensive unsupervised datasets with a vast number ofparameters. However, they still can not emulate human-like continuous learningdue to catastrophic forgetting. Consequently, various continual learning(CL)-based methodologies have been developed to refine LMs, enabling them toadapt to new tasks without forgetting previous knowledge. However, a systematictaxonomy of existing approaches and a comparison of their performance are stilllacking, which is the gap that our survey aims to fill. We delve into acomprehensive review, summarization, and classification of the existingliterature on CL-based approaches applied to foundation language models, suchas pre-trained language models (PLMs), large language models (LLMs) andvision-language models (VLMs). We divide these studies into offline CL andonline CL, which consist of traditional methods, parameter-efficient-basedmethods, instruction tuning-based methods and continual pre-training methods.Offline CL encompasses domain-incremental learning, task-incremental learning,and class-incremental learning, while online CL is subdivided into hard taskboundary and blurry task boundary settings. Additionally, we outline thetypical datasets and metrics employed in CL research and provide a detailedanalysis of the challenges and future work for LMs-based continual learning.