Abstract
Collaborative perception aims to mitigate the limitations of single-agentperception, such as occlusions, by facilitating data exchange among multipleagents. However, most current works consider a homogeneous scenario where allagents use identity sensors and perception models. In reality, heterogeneousagent types may continually emerge and inevitably face a domain gap whencollaborating with existing agents. In this paper, we introduce a new openheterogeneous problem: how to accommodate continually emerging newheterogeneous agent types into collaborative perception, while ensuring highperception performance and low integration cost? To address this problem, wepropose HEterogeneous ALliance (HEAL), a novel extensible collaborativeperception framework. HEAL first establishes a unified feature space withinitial agents via a novel multi-scale foreground-aware Pyramid Fusion network.When heterogeneous new agents emerge with previously unseen modalities ormodels, we align them to the established unified space with an innovativebackward alignment. This step only involves individual training on the newagent type, thus presenting extremely low training costs and highextensibility. To enrich agents' data heterogeneity, we bring OPV2V-H, a newlarge-scale dataset with more diverse sensor types. Extensive experiments onOPV2V-H and DAIR-V2X datasets show that HEAL surpasses SOTA methods inperformance while reducing the training parameters by 91.5% when integrating 3new agent types. We further implement a comprehensive codebase at:https://github.com/yifanlu0227/HEAL