Abstract
Reinforcement Learning (RL) offers promising solutions for control tasks inindustrial cyber-physical systems (ICPSs), yet its real-world adoption remainslimited. This paper demonstrates how seemingly small but well-designedmodifications to the RL problem formulation can substantially improveperformance, stability, and sample efficiency. We identify and investigate keyelements of RL problem formulation and show that these enhance both learningspeed and final policy quality. Our experiments use a one-degree-of-freedom(1-DoF) helicopter testbed, the Quanser Aero~2, which features non-lineardynamics representative of many industrial settings. In simulation, theproposed problem design principles yield more reliable and efficient training,and we further validate these results by training the agent directly onphysical hardware. The encouraging real-world outcomes highlight the potentialof RL for ICPS, especially when careful attention is paid to the designprinciples of problem formulation. Overall, our study underscores the crucialrole of thoughtful problem formulation in bridging the gap between RL researchand the demands of real-world industrial systems.