Abstract
In reinforcement learning with sparse rewards, demonstrations can acceleratelearning, but determining when to imitate them remains challenging. We proposeSmooth Policy Regularisation from Demonstrations (SPReD), a framework thataddresses the fundamental question: when should an agent imitate ademonstration versus follow its own policy? SPReD uses ensemble methods toexplicitly model Q-value distributions for both demonstration and policyactions, quantifying uncertainty for comparisons. We develop two complementaryuncertainty-aware methods: a probabilistic approach estimating the likelihoodof demonstration superiority, and an advantage-based approach scaling imitationby statistical significance. Unlike prevailing methods (e.g. Q-filter) thatmake binary imitation decisions, SPReD applies continuous,uncertainty-proportional regularisation weights, reducing gradient varianceduring training. Despite its computational simplicity, SPReD achievesremarkable gains in experiments across eight robotics tasks, outperformingexisting approaches by up to a factor of 14 in complex tasks while maintainingrobustness to demonstration quality and quantity. Our code is available athttps://github.com/YujieZhu7/SPReD.