Abstract
Many real-world control problems involve both discrete decision variables -such as the choice of control modes, gear switching or digital outputs - aswell as continuous decision variables - such as velocity setpoints, controlgains or analogue outputs. However, when defining the corresponding optimalcontrol or reinforcement learning problem, it is commonly approximated withfully continuous or fully discrete action spaces. These simplifications aim attailoring the problem to a particular algorithm or solver which may onlysupport one type of action space. Alternatively, expert heuristics are used toremove discrete actions from an otherwise continuous space. In contrast, wepropose to treat hybrid problems in their 'native' form by solving them withhybrid reinforcement learning, which optimizes for discrete and continuousactions simultaneously. In our experiments, we first demonstrate that theproposed approach efficiently solves such natively hybrid reinforcementlearning problems. We then show, both in simulation and on robotic hardware,the benefits of removing possibly imperfect expert-designed heuristics. Lastly,hybrid reinforcement learning encourages us to rethink problem definitions. Wepropose reformulating control problems, e.g. by adding meta actions, to improveexploration or reduce mechanical wear and tear.