CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning

  • 2022-11-23 18:57:11
  • James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, Zsolt Kira
  • 0


Computer vision models suffer from a phenomenon known as catastrophicforgetting when learning novel concepts from continuously shifting trainingdata. Typical solutions for this continual learning problem require extensiverehearsal of previously seen data, which increases memory costs and may violatedata privacy. Recently, the emergence of large-scale pre-trained visiontransformer models has enabled prompting approaches as an alternative todata-rehearsal. These approaches rely on a key-query mechanism to generateprompts and have been found to be highly resistant to catastrophic forgettingin the well-established rehearsal-free continual learning setting. However, thekey mechanism of these methods is not trained end-to-end with the tasksequence. Our experiments show that this leads to a reduction in theirplasticity, hence sacrificing new task accuracy, and inability to benefit fromexpanded parameter capacity. We instead propose to learn a set of promptcomponents which are assembled with input-conditioned weights to produceinput-conditioned prompts, resulting in a novel attention-based end-to-endkey-query scheme. Our experiments show that we outperform the current SOTAmethod DualPrompt on established benchmarks by as much as 5.4% in averageaccuracy. We also outperform the state of art by as much as 6.6% accuracy on acontinual learning benchmark which contains both class-incremental anddomain-incremental task shifts, corresponding to many practical settings.


Quick Read (beta)

loading the full paper ...