Abstract
Test-time adaptation (TTA) has demonstrated significant potential inaddressing distribution shifts between training and testing data. Open-settest-time adaptation (OSTTA) aims to adapt a source pre-trained model online toan unlabeled target domain that contains unknown classes. This task becomesmore challenging when multiple modalities are involved. Existing methods haveprimarily focused on unimodal OSTTA, often filtering out low-confidence sampleswithout addressing the complexities of multimodal data. In this work, wepresent Adaptive Entropy-aware Optimization (AEO), a novel frameworkspecifically designed to tackle Multimodal Open-set Test-time Adaptation(MM-OSTTA) for the first time. Our analysis shows that the entropy differencebetween known and unknown samples in the target domain strongly correlates withMM-OSTTA performance. To leverage this, we propose two key components:Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive ModalityPrediction Discrepancy Optimization (AMP). These components enhance the abilityof model to distinguish unknown class samples during online adaptation byamplifying the entropy difference between known and unknown samples. Tothoroughly evaluate our proposed methods in the MM-OSTTA setting, we establisha new benchmark derived from existing datasets. This benchmark includes twodownstream tasks and incorporates five modalities. Extensive experiments acrossvarious domain shift situations demonstrate the efficacy and versatility of theAEO framework. Additionally, we highlight the strong performance of AEO inlong-term and continual MM-OSTTA settings, both of which are challenging andhighly relevant to real-world applications. Our source code is available athttps://github.com/donghao51/AEO.