Abstract
We introduce a new approach to hierarchy formation and task decomposition inhierarchical reinforcement learning. Our method is based on the Hierarchy OfAbstract Machines (HAM) framework because HAM approach is able to designefficient controllers that will realize specific behaviors in real robots. Thekey to our algorithm is the introduction of the internal or "mental"environment in which the state represents the structure of the HAM hierarchy.The internal action in this environment leads to changes the hierarchy of HAMs.We propose the classical Q-learning procedure in the internal environment whichallows the agent to obtain an optimal hierarchy. We extends the HAM frameworkby adding on-model approach to select the appropriate sub-machine to executeaction sequences for certain class of external environment states. Preliminaryexperiments demonstrated the prospects of the method.