Abstract
Preference-based reinforcement learning has gained prominence as a strategyfor training agents in environments where the reward signal is difficult tospecify or misaligned with human intent. However, its effectiveness is oftenlimited by the high cost and low availability of reliable human input,especially in domains where expert feedback is scarce or errors are costly. Toaddress this, we propose a novel framework that combines two complementarystrategies: probabilistic crowd modelling to handle noisy, multi-annotatorfeedback, and active learning to prioritize feedback on the most informativeagent actions. We extend the Advise algorithm to support multiple trainers,estimate their reliability online, and incorporate entropy-based queryselection to guide feedback requests. We evaluate our approach in a set ofenvironments that span both synthetic and real-world-inspired settings,including 2D games (Taxi, Pacman, Frozen Lake) and a blood glucose control taskfor Type 1 Diabetes using the clinically approved UVA/Padova simulator. Ourpreliminary results demonstrate that agents trained with feedback on uncertaintrajectories exhibit faster learning in most tasks, and we outperform thebaselines for the blood glucose control task.