Abstract
Model-based reinforcement learning (RL) is anticipated to exhibit highersample efficiency compared to model-free RL by utilizing a virtual environmentmodel. However, it is challenging to obtain sufficiently accuraterepresentations of the environmental dynamics due to uncertainties in complexsystems and environments. An inaccurate environment model may degrade thesample efficiency and performance of model-based RL. Furthermore, whilemodel-based RL can improve sample efficiency, it often still requiressubstantial training time to learn from scratch, potentially limiting itsadvantages over model-free approaches. To address these challenges, this paperintroduces a knowledge-informed model-based residual reinforcement learningframework aimed at enhancing learning efficiency by infusing established expertknowledge into the learning process and avoiding the issue of beginning fromzero. Our approach integrates traffic expert knowledge into a virtualenvironment model, employing the Intelligent Driver Model (IDM) for basicdynamics and neural networks for residual dynamics, thus ensuring adaptabilityto complex scenarios. We propose a novel strategy that combines traditionalcontrol methods with residual RL, facilitating efficient learning and policyoptimization without the need to learn from scratch. The proposed approach isapplied to CAV trajectory control tasks for the dissipation of stop-and-gowaves in mixed traffic flow. Experimental results demonstrate that our proposedapproach enables the CAV agent to achieve superior performance in trajectorycontrol compared to the baseline agents in terms of sample efficiency, trafficflow smoothness and traffic mobility. The source code and supplementarymaterials are available at https://github.com/zihaosheng/traffic-expertise-RL/.