Abstract
Fully fine-tuning pretrained large-scale transformer models has become apopular paradigm for video-language modeling tasks, such as temporal languagegrounding and video-language summarization. With a growing number of tasks andlimited training data, such full fine-tuning approach leads to costly modelstorage and unstable training. To overcome these shortcomings, we introducelightweight adapters to the pre-trained model and only update them atfine-tuning time. However, existing adapters fail to capture intrinsic temporalrelations among video frames or textual words. Moreover, they neglect thepreservation of critical task-related information that flows from the rawvideo-language input into the adapter's low-dimensional space. To address theseissues, we first propose a novel REcurrent ADapter (READ) that employsrecurrent computation to enable temporal modeling capability. Second, wepropose Partial Video-Language Alignment (PVLA) objective via the use ofpartial optimal transport to maintain task-related information flowing into ourREAD modules. We validate our READ framework through extensive experimentswhere READ significantly outperforms all existing fine-tuning strategies onmultiple low-resource temporal language grounding and video-languagesummarization benchmarks. The code, model, and data have been made available athttps://nguyentthong.github.io/READ.