Homeostasis is a prevalent process by which living beings maintain theirinternal milieu around optimal levels. Multiple lines of evidence suggest thatliving beings learn to act to predicatively ensure homeostasis (allostasis). Aclassical theory for such regulation is drive reduction, where a function ofthe difference between the current and the optimal internal state. The recentlyintroduced homeostatic regulated reinforcement learning theory (HRRL), bydefining within the framework of reinforcement learning a reward function basedon the internal state of the agent, makes the link between the theories ofdrive reduction and reinforcement learning. The HRRL makes it possible toexplain multiple eating disorders. However, the lack of continuous change inthe internal state of the agent with the discrete-time modeling has been so fara key shortcoming of the HRRL theory. Here, we propose an extension of thehomeostatic reinforcement learning theory to a continuous environment in spaceand time, while maintaining the validity of the theoretical results and thebehaviors explained by the model in discrete time. Inspired by theself-regulating mechanisms abundantly present in biology, we also introduce amodel for the dynamics of the agent internal state, requiring the agent tocontinuously take actions to maintain homeostasis. Based on theHamilton-Jacobi-Bellman equation and function approximation with neuralnetworks, we derive a numerical scheme allowing the agent to learn directly howits internal mechanism works, and to choose appropriate action policies viareinforcement learning and an appropriate exploration of the environment. Ournumerical experiments show that the agent does indeed learn to behave in a waythat is beneficial to its survival in the environment, making our frameworkpromising for modeling animal dynamics and decision-making.