Abstract
In standard Reinforcement Learning (RL) settings, the interaction between theagent and the environment is typically modeled as a Markov Decision Process(MDP), which assumes that the agent observes the system state instantaneously,selects an action without delay, and executes it immediately. In real-worlddynamic environments, such as cyber-physical systems, this assumption oftenbreaks down due to delays in the interaction between the agent and the system.These delays can vary stochastically over time and are typically unobservable,meaning they are unknown when deciding on an action. Existing methods deal withthis uncertainty conservatively by assuming a known fixed upper bound on thedelay, even if the delay is often much lower. In this work, we introduce theinteraction layer, a general framework that enables agents to adaptively andseamlessly handle unobservable and time-varying delays. Specifically, the agentgenerates a matrix of possible future actions to handle both unpredictabledelays and lost action packets sent over networks. Building on this framework,we develop a model-based algorithm, Actor-Critic with Delay Adaptation (ACDA),which dynamically adjusts to delay patterns. Our method significantlyoutperforms state-of-the-art approaches across a wide range of locomotionbenchmark environments.