Abstract
Multimodal question answering (QA) often requires identifying which video,audio, or sensor tokens are relevant to the question. Yet modalitydisagreements are common: off-camera speech, background noise, or motionoutside the field of view often mislead fusion models that weight all streamsequally. We present RAVEN, a unified QA architecture whose core is QuART, aquery-conditioned cross-modal gating module that assigns scalar relevancescores to each token across modalities, enabling the model to amplifyinformative signals and suppress distractors before fusion. RAVEN is trainedthrough a three-stage pipeline comprising unimodal pretraining, query-alignedfusion, and disagreement-oriented fine-tuning -- each stage targeting adistinct challenge in multi-modal reasoning: representation quality,cross-modal relevance, and robustness to modality mismatch. To support trainingand evaluation, we release AVS-QA, a dataset of 300K synchronizedAudio--Video-Sensor streams paired with automatically generated question-answerpairs. Experimental results on seven multi-modal QA benchmarks -- includingegocentric and exocentric tasks -- show that RAVEN achieves up to 14.5\% and8.0\% gains in accuracy compared to state-of-the-art multi-modal large languagemodels, respectively. Incorporating sensor data provides an additional 16.4\%boost, and the model remains robust under modality corruption, outperformingSOTA baselines by 50.23\%. Our code and dataset are available athttps://github.com/BASHLab/RAVEN.