Abstract
This paper proposes a reinforcement learning (RL)-aided cognitive frameworkfor massive MIMO-based integrated sensing and communication (ISAC) systemsemploying a uniform planar array (UPA). The focus is on enhancing radar sensingperformance in environments with unknown and dynamic disturbancecharacteristics. A Wald-type detector is employed for robust target detectionunder non-Gaussian clutter, while a SARSA-based RL algorithm enables adaptiveestimation of target positions without prior environmental knowledge. Based onthe RL-derived sensing information, a joint waveform optimization strategy isformulated to balance radar sensing accuracy and downlink communicationthroughput. The resulting design provides an adaptive trade-off betweendetection performance and achievable sum rate through an analytically derivedclosed-form solution. Monte Carlo simulations demonstrate that the proposedcognitive ISAC framework achieves significantly improved detection probabilitycompared to orthogonal and non-learning adaptive baselines, while maintainingcompetitive communication performance. These results underline the potential ofRL-assisted sensing for robust and spectrum-efficient ISAC in next-generationwireless networks.