Abstract
Drawing inspiration from human learning behaviors, this work proposes a novelapproach to mitigate catastrophic forgetting in Prompt-based Continual Learningmodels by exploiting the relationships between continuously emerging classdata. We find that applying human habits of organizing and connectinginformation can serve as an efficient strategy when training deep learningmodels. Specifically, by building a hierarchical tree structure based on theexpanding set of labels, we gain fresh insights into the data, identifyinggroups of similar classes could easily cause confusion. Additionally, we delvedeeper into the hidden connections between classes by exploring the originalpretrained model's behavior through an optimal transport-based approach. Fromthese insights, we propose a novel regularization loss function that encouragesmodels to focus more on challenging knowledge areas, thereby enhancing overallperformance. Experimentally, our method demonstrated significant superiorityover the most robust state-of-the-art models on various benchmarks.