Collaborative deep reinforcement learning (CDRL) algorithms in which multipleagents can coordinate over a wireless network is a promising approach to enablefuture intelligent and autonomous systems that rely on real-timedecision-making in complex dynamic environments. Nonetheless, in practicalscenarios, CDRL faces many challenges due to the heterogeneity of agents andtheir learning tasks, different environments, time constraints of the learning,and resource limitations of wireless networks. To address these challenges, inthis paper, a novel semantic-aware CDRL method is proposed to enable a group ofheterogeneous untrained agents with semantically-linked DRL tasks tocollaborate efficiently across a resource-constrained wireless cellularnetwork. To this end, a new heterogeneous federated DRL (HFDRL) algorithm isproposed to select the best subset of semantically relevant DRL agents forcollaboration. The proposed approach then jointly optimizes the training lossand wireless bandwidth allocation for the cooperating selected agents in orderto train each agent within the time limit of its real-time task. Simulationresults show the superior performance of the proposed algorithm compared tostate-of-the-art baselines.