Homeostatically Regulated Reinforcement Learning

• Homeostatically Regulated Reinforcement Learning is a framework where reward is derived from reducing deviations in internal physiological states.
• It employs mathematical formulations, including HJB equations and neural function approximators, to ensure predictive control under uncertainty.
• The paradigm extends to continuous-space settings, deep RL, robotics, and biomedical applications such as blood-glucose regulation.

Searching arXiv for the cited HRRL papers to ground the article in the referenced literature. Homeostatically Regulated Reinforcement Learning (HRRL) is a normative reinforcement-learning framework in which reward is grounded in interoceptive regulation rather than in externally specified task utilities. In HRRL, a biological or artificial agent learns policies that control internal physiological variables by reducing deviation from homeostatic setpoints; reward is therefore identified with drive reduction, and optimal behavior becomes predictive control of internal state under environmental uncertainty. Subsequent work has extended this idea from discrete-time formulations to continuous-time, continuous-space settings through Hamilton–Jacobi–Bellman (HJB) methods and neural function approximation, while related lines of research have adapted the framework to deep RL, intrinsic motivation, robotics, and constrained physiological control such as blood-glucose regulation (Yoshida et al., 7 Jul 2025, Laurençon et al., 2021, Laurencon et al., 2024, Mguni et al., 5 Aug 2025).

1. Conceptual basis

HRRL formalizes a long-standing biological intuition: survival depends on maintaining internal physiological variables such as hydration, temperature, sodium, or glucose within viable bounds, and motivated behavior exists to stabilize those variables. In the framework summarized in "Linking Homeostasis to Reinforcement Learning: Internal State Control of Motivated Behavior" (Yoshida et al., 7 Jul 2025), homeostasis regulates essential physiological variables around a homeostatic optimum, while allostasis denotes predictive, anticipatory regulation that tolerates short-term deviations to reduce long-term cumulative error. HRRL links these ideas to RL by defining what counts as reward in terms of interoceptive improvement.

The standard formalization augments the agent’s state with both exteroceptive and interoceptive components. Exteroceptive state is written as $x_t \in X$, interoceptive or homeostatic state as $h_t \in H \subset \mathbb{R}^n$, and action as $a_t \in A$. The full state is therefore an augmented state $s_t = (x_t, h_t)$, or, under noisy sensing, a POMDP state estimate rather than a fully observed MDP state (Yoshida et al., 7 Jul 2025). This structure is central: actions primarily affect environmental trajectories, but the optimization objective is defined by internal-state regulation.

Within this perspective, drive-reduction theory is not treated as a purely reactive doctrine. HRRL is explicitly predictive: agents learn forward-looking strategies that exploit cues, resource layouts, and environmental contingencies to minimize future deviations from physiological setpoints. The framework is therefore not limited to immediate need satisfaction; it is intended to explain anticipatory regulation, risk aversion, adaptive movement, and other motivated behaviors reviewed in the HRRL perspective (Yoshida et al., 7 Jul 2025).

2. Canonical mathematical formulation

The canonical HRRL construction begins with a homeostatic setpoint $H^* = (h_1^*, \dots, h_n^*)$ and deviation $e_t = h_t - h^*$. A drive function $d(h_t)$ is then defined so that it is minimal at the setpoint and increases with deviation. The perspective review gives both a generalized norm-like form and a representative example, such as

$d(h_t) = \|h_t - h^*\|_2^2,$

with reward defined by drive reduction:

$r_t = d(H_t) - d(H_{t+1}).$

Under the simplified internal update $H_{t+1} = H_t + K_t$, this becomes $h_t \in H \subset \mathbb{R}^n$0, so outcomes are rewarding exactly insofar as they move the agent toward homeostatic optimum (Yoshida et al., 7 Jul 2025).

The optimization objective is standard discounted RL with homeostatic reward:

$h_t \in H \subset \mathbb{R}^n$1

Value and action-value functions obey standard Bellman recursions, and the temporal-difference signal becomes

$h_t \in H \subset \mathbb{R}^n$2

A central theoretical result stated in the perspective is the equivalence

$h_t \in H \subset \mathbb{R}^n$3

where $h_t \in H \subset \mathbb{R}^n$4 is the discounted sum of drives and $h_t \in H \subset \mathbb{R}^n$5 is the discounted sum of drive reductions. Maximizing cumulative reward is therefore equivalent to minimizing cumulative discounted homeostatic deviation (Yoshida et al., 7 Jul 2025).

In discrete-time summaries of HRRL, states $h_t \in H \subset \mathbb{R}^n$6 include internal homeostatic variables and relevant external variables, while actions influence both internal and external dynamics. A common reward is written either as $h_t \in H \subset \mathbb{R}^n$7 or as $h_t \in H \subset \mathbb{R}^n$8, where $h_t \in H \subset \mathbb{R}^n$9 is a scalar penalty for homeostatic deviation (Laurencon et al., 2024). These equivalent viewpoints emphasize either transition-wise drive reduction or state-wise deviation penalty.

The derivative-based behavioral properties reviewed in the perspective give the framework its distinctive explanatory force. Reward increases with outcome magnitude in the relevant dimension, deprivation enhances the reward value of appropriate outcomes, unrelated deficits can inhibit value in another dimension, and concavity of reward in outcome magnitude yields risk aversion through Jensen’s inequality. The review explicitly associates these properties with incentive salience, cross-need competition, and prospect-theoretic risk sensitivity (Yoshida et al., 7 Jul 2025).

3. Continuous-time and continuous-space HRRL

A major limitation of early discrete HRRL is that internal state changes only at discrete decision steps unless additional dynamics are introduced. "Continuous Homeostatic Reinforcement Learning for Self-Regulated Autonomous Agents" extends HRRL to continuous time and continuous space by making internal variables drift continuously even when the agent is idle, thereby forcing ongoing self-regulation rather than intermittent correction (Laurençon et al., 2021). The observable state is written as $a_t \in A$0, where $a_t \in A$1 is internal deviation and $a_t \in A$2 denotes the environment within the agent’s view-field.

In the deterministic continuous formulation used for learning, drive is defined as

$a_t \in A$3

with practical regularization $a_t \in A$4 near $a_t \in A$5, and instantaneous reward becomes the negative time derivative of drive:

$a_t \in A$6

The discounted reward value $a_t \in A$7 and discounted deviation functional $a_t \in A$8 remain equivalent through

$a_t \in A$9

implying

$s_t = (x_t, h_t)$0

This preserves the discrete HRRL equivalence in continuous time (Laurençon et al., 2021).

The continuous optimality condition is expressed as a discounted HJB equation on $s_t = (x_t, h_t)$1:

$s_t = (x_t, h_t)$2

The numerical scheme learns both a transition model $s_t = (x_t, h_t)$3 and a deviation function $s_t = (x_t, h_t)$4 with neural networks. Action selection is $s_t = (x_t, h_t)$5-greedy with respect to HJB minimization; training minimizes a transition-model loss and an HJB residual loss,

$s_t = (x_t, h_t)$6

and

$s_t = (x_t, h_t)$7

This is a model-based RL scheme driven directly by continuous-time optimal control structure (Laurençon et al., 2021).

"Continuous Time Continuous Space Homeostatic Reinforcement Learning (CTCS-HRRL)" advances the same linkage to continuous time-space and validates it in simulation with a model that mimics the homeostatic mechanisms in a real-world biological agent. The paper states that CTCS-HRRL uses the Hamilton-Jacobian Bellman Equation, function approximation based on neural networks and Reinforcement Learning, and demonstrates that the agent can dynamically choose policies that favor homeostasis in a continuously changing internal-state milieu. Its reported result is qualitative but central: the agent learns homeostatic behavior in a CTCS environment (Laurencon et al., 2024).

4. Algorithmic realizations and variants

The HRRL perspective explicitly extends the framework to deep RL by feeding the augmented state $s_t = (x_t, h_t)$8 into neural function approximators. The review gives representative value-based and actor–critic losses, including DQN-style critic updates and PPO-style actor–critic objectives in which the advantage is computed from homeostatic reward. It also notes that HRRL can be complemented by intrinsic motivation, for example through a prediction-error bonus $s_t = (x_t, h_t)$9 or a generic exploration bonus $H^* = (h_1^*, \dots, h_n^*)$0, while keeping homeostatic regulation as the core reward principle (Yoshida et al., 7 Jul 2025).

The same review presents hierarchical and modular HRRL as a multi-drive arbitration scheme:

$H^* = (h_1^*, \dots, h_n^*)$1

This formulation separates multiple drives into subpolicies and uses gating weights for arbitration. The review attributes improved efficiency in complex, changing environments to modular HRRL as reported by Dulberg et al. (Yoshida et al., 7 Jul 2025).

A distinct but related line appears in "Curiosity-driven reinforcement learning with homeostatic regulation" (Abril et al., 2018). In that paper, homeostasis is not implemented as explicit physiological set-points or bounded intervals. Instead, it is instantiated as a familiarity-seeking bonus within an information-theoretic intrinsic reward. The method uses a forward model $H^* = (h_1^*, \dots, h_n^*)$2, an extended forward model $H^* = (h_1^*, \dots, h_n^*)$3, and the intrinsic reward

$H^* = (h_1^*, \dots, h_n^*)$4

followed by per-episode $H^* = (h_1^*, \dots, h_n^*)$5-normalization. The heterostatic component rewards novelty through forward-model error, while the homeostatic component rewards states for which prediction improves when the next action is known (Abril et al., 2018).

Its implementation uses DDPG, continuous actions, and a 3-room continuous 2D environment of size $H^* = (h_1^*, \dots, h_n^*)$6. The experiments use episode length $H^* = (h_1^*, \dots, h_n^*)$7, $H^* = (h_1^*, \dots, h_n^*)$8 random-action probability, $H^* = (h_1^*, \dots, h_n^*)$9 episodes per setting, and $e_t = h_t - h^*$0. Increasing $e_t = h_t - h^*$1 improves sampling efficiency for learning the forward model relative to $e_t = h_t - h^*$2, and when starts are restricted to the bottom room, every non-random policy with any $e_t = h_t - h^*$3 outperforms the pure random baseline, which reaches the top room only $e_t = h_t - h^*$4 times on average over $e_t = h_t - h^*$5k episodes. In the global-start experiment, a pure random sampler achieves validation MSE $e_t = h_t - h^*$6, slightly better than the best $e_t = h_t - h^*$7 run at $e_t = h_t - h^*$8, although the authors explicitly note that the comparison is unfair because the random policy diffuses over many starting positions while the learned policy concentrates on complex regions near doors (Abril et al., 2018).

5. Applications and empirical demonstrations

The HRRL perspective surveys several classes of demonstrations. Predictive shivering is used as an example of anticipatory regulation: a conditioned stimulus preceding a thermal challenge leads the agent to raise temperature in advance, thereby reducing long-term cumulative drive. The review also states that autonomous movement can emerge as a side effect of resource acquisition, and that HRRL has been proposed for robotics and continuous-control settings with multimodal inputs and continuous actions (Yoshida et al., 7 Jul 2025).

The continuous self-regulated-agent study provides a concrete embodied demonstration. The environment is a continuous 2D closed space with fixed resource locations, continuous spatial coordinates and orientation, and a discrete action repertoire that includes walking, running, rotating, consuming, and sleeping. The internal dynamics include two fatigue variables: “muscle” fatigue accumulates with movement and can disable running or even walking above thresholds, while “sleep” fatigue accumulates when the agent does not recover and can make sleep the only admissible action. The learned deviation function $e_t = h_t - h^*$9 is reported to be minimized near the resource supplying the currently deficient internal variable, and the resulting behavior interleaves periodic resource-seeking with rest to maintain admissible internal ranges (Laurençon et al., 2021).

CTCS-HRRL reports a simulation-based experiment with a model that “mimics the homeostatic mechanisms in a real-world biological agent.” The main finding is that the agent learns homeostatic behavior in a continuous-time continuous-space environment and can dynamically choose policies that favor homeostasis under continuously changing internal-state conditions (Laurencon et al., 2024).

A more domain-specific instantiation appears in "Reinforcement Learning for Target Zone Blood Glucose Control" (Mguni et al., 5 Aug 2025), which formulates regulation of blood glucose level as a constrained Markov decision process. The regulated variable is $d(h_t)$0, the clinical target zone is $d(h_t)$1–$d(h_t)$2 mg/dL in evaluation, and the theoretical target band is $d(h_t)$3. The reward includes a target-zone term $d(h_t)$4, a penalty for long-acting activations, and a quadratic penalty for fast-acting impulses. The action space combines impulse control for fast-acting boluses and switching control for long-acting basal activation, while safety is imposed through a hypoglycemia-aware dosing constraint, a target-band violation budget, and an intervention-count budget (Mguni et al., 5 Aug 2025).

The learning architecture in that study uses state augmentation in the style of Sauté RL, with remaining-budget variables $d(h_t)$5, a large penalty $d(h_t)$6 upon budget violation, three coordinated policies, and model predictive shielding. The fast-acting and long-acting policies are trained with PPO, and the switcher policy $d(h_t)$7 is trained with SAC. The paper also states a Q-learning convergence theorem for the constrained problem under standard assumptions (Mguni et al., 5 Aug 2025).

Empirically, the target-zone controller is evaluated in the GlucoEnv simulator implementing the UVA/Padova T1DM model. In the CMP scenario, the method achieves TIR $d(h_t)$8, TBR $d(h_t)$9, and AIME $d(h_t) = \|h_t - h^*\|_2^2,$0; in AGVP, TIR $d(h_t) = \|h_t - h^*\|_2^2,$1, TBR $d(h_t) = \|h_t - h^*\|_2^2,$2, and AIME $d(h_t) = \|h_t - h^*\|_2^2,$3; in PHC, TIR $d(h_t) = \|h_t - h^*\|_2^2,$4, TBR $d(h_t) = \|h_t - h^*\|_2^2,$5, and AIME $d(h_t) = \|h_t - h^*\|_2^2,$6. The abstract separately reports reducing blood-glucose-level violations from $d(h_t) = \|h_t - h^*\|_2^2,$7 to as low as $d(h_t) = \|h_t - h^*\|_2^2,$8 in a stylized task. The paper states explicitly that the method is not intended for clinical deployment (Mguni et al., 5 Aug 2025).

6. Biological interpretation, misconceptions, and open problems

HRRL is frequently presented as biologically plausible because it maps reward prediction to internal regulation rather than to arbitrary task labels. The perspective review links dopaminergic reinforcement signals in VTA to TD-like errors modulated by interoceptive state, writes a putative neural TD-error proportional to

$d(h_t) = \|h_t - h^*\|_2^2,$9

and relates this to hypothalamic and interoceptive pathways, taste valuation, and social homeostasis. The same review argues that distorted drive functions or altered internal-state representations could help explain consummatory anhedonia, addiction, and broader computational psychosomatics (Yoshida et al., 7 Jul 2025).

The framework also stands in a specific relation to neighboring theories. Relative to classical drive-reduction theory, HRRL supplies explicit computational machinery: value functions, TD errors, planning, and Bellman recursions. Relative to allostasis, it provides a predictive-control interpretation in which agents minimize future deviations rather than merely reacting to present ones. Relative to active inference, the perspective contrasts objectives directly: active inference minimizes expected free energy, whereas HRRL maximizes cumulative discounted drive reduction (Yoshida et al., 7 Jul 2025).

A recurrent misconception is that all uses of “homeostatic regulation” in RL refer to explicit physiological setpoints. That is true for canonical HRRL and for continuous-time formulations centered on internal variables and setpoints, but not for the 2018 curiosity paper, where “homeostasis” is instantiated as a preference for familiar state-action configurations rather than bounded physiological parameters (Abril et al., 2018). The blood-glucose CMDP, by contrast, reinstates explicit target zones, deviation penalties, and resource/safety budgets in a highly structured physiological domain (Mguni et al., 5 Aug 2025).

Open problems are repeatedly identified across the literature. Continuous formulations note sensitivity to architecture, learning rates, and smoothness assumptions when solving HJB equations approximately with neural residual minimization, and they highlight partial observability, stochastic dynamics, and multi-agent extensions as unresolved directions (Laurençon et al., 2021, Laurencon et al., 2024). The perspective emphasizes uncertainty in interoceptive state estimation, the difficulty of designing multi-need drive functions, and the challenge of integrating learned forward models, hierarchical arbitration, and intrinsic motivation at scale (Yoshida et al., 7 Jul 2025). The blood-glucose study adds a safety-critical perspective: it remains an in silico system, and broader personalization, unobserved exogenous factors, offline-safe RL, and richer clinical constraints remain future work rather than solved problems (Mguni et al., 5 Aug 2025).