HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learning (2510.22832v1)

Abstract: The Hierarchical Reasoning Model (HRM) has impressive reasoning abilities given its small size, but has only been applied to supervised, static, fully-observable problems. One of HRM's strengths is its ability to adapt its computational effort to the difficulty of the problem. However, in its current form it cannot integrate and reuse computation from previous time-steps if the problem is dynamic, uncertain or partially observable, or be applied where the correct action is undefined, characteristics of many real-world problems. This paper presents HRM-Agent, a variant of HRM trained using only reinforcement learning. We show that HRM can learn to navigate to goals in dynamic and uncertain maze environments. Recent work suggests that HRM's reasoning abilities stem from its recurrent inference process. We explore the dynamics of the recurrent inference process and find evidence that it is successfully reusing computation from earlier environment time-steps.

• The paper demonstrates that carrying forward the recurrent latent state enhances planning efficiency, yielding a near 99% success rate in maze navigation tasks.
• The paper employs a dual recurrent architecture with a Deep Q-Network head to enable unsupervised incremental solution construction in dynamic environments.
• The paper reveals that persistent state dynamics accelerate convergence and training efficiency, highlighting the advantage of minimizing state resets in adaptive tasks.

HRM-Agent: Reinforcement Learning for Recurrent Reasoning in Dynamic Environments

Introduction

The HRM-Agent paper presents a variant of the Hierarchical Reasoning Model (HRM) adapted for reinforcement learning (RL) in dynamic, fully-observable maze environments. The original HRM demonstrated efficient reasoning in static, supervised settings, leveraging a dual recurrent architecture with high-level (H) and low-level (L) modules. This work extends HRM to RL, enabling the agent to incrementally construct solutions and adapt to environmental changes without supervised labels or explicit intermediate supervision. The central research question is whether HRM's recurrent reasoning process can be exploited in dynamic, uncertain, or partially observable environments, where the agent must continually update its internal plan as the environment evolves.

Model Architecture and Training

The HRM-Agent replaces the supervised output head of HRM with a Deep Q-Network (DQN) head, producing $Q$-values for each action. The model is trained off-policy using standard DQN loss, with actions selected via an epsilon-greedy strategy. The architecture maintains the dual recurrent modules ($z_H$, $z_L$) and disables Adaptive Computation Time (ACT) to facilitate analysis of recurrent state convergence. Two variants are evaluated:

• Carry Z: The recurrent latent state $z$ is initialized from its final value in the previous environment time-step, promoting continuity of planning.
• Reset Z: $z$ is reinitialized to a random vector at each environment time-step, serving as an ablation to test the utility of state persistence.

No gradients are propagated across environment time-steps, and recurrent convergence is performed without gradient accumulation, mirroring the original HRM training regime.

Figure 1: Maze-navigation tasks used to test the HRM-Agent model in the Nethack Learning Environment (NLE) with MiniHack.

Experimental Environments

Two maze environments derived from MiniHack are used:

• Four-rooms: A classic sparse-reward navigation task with dynamic doors that randomly open and close, requiring the agent to re-plan paths as obstacles change.
• Dynamic Random Maze: A procedurally generated maze with random walls and doors, where the set of viable paths changes stochastically during each episode. A* path planning ensures at least one solution exists when all doors are open.

These environments are designed to require non-trivial planning and adaptation, testing the agent's ability to generalize and reason about unseen configurations.

Empirical Results

Navigation Performance

After training, HRM-Agent achieves approximately 99% success in reaching the goal in both environments, with mean episode lengths approaching the theoretical optimum (≈10 steps, accounting for detours due to closed doors). The results indicate that the agent learns efficient, generalized path-planning policies rather than memorizing specific solutions.

Recurrent State Dynamics

Analysis of the recurrent latent state $z$ reveals that the "Carry Z" variant converges more rapidly and with smaller mean-square-error (MSE) between initial and final recurrent states, especially when the environment has not materially changed. When the environment changes (e.g., doors open/close), the initial $z$ diverges more from the final $z$, consistent with the need to update the plan. The "Reset Z" variant shows larger initial divergence and slower convergence, supporting the hypothesis that carrying forward $z$ enables reuse of prior computation and continuity of intent.

Figure 2: Four-rooms environment.

Figure 3: Four-rooms environment.

Figure 4: Analysis of the convergence of the recurrent latent state $z$ to its final values in the four-rooms environment. The left plot shows $z_L$ and the right plot $z_H$.

Figure 5: Analysis of the convergence of the recurrent latent state $z$ to its final values, in the dynamic random maze environment.

Figure 6: Analysis of the divergence of the recurrent latent state $z$ from the output of the first forward pass $z^1$ to its final values, in the dynamic random maze environment.

Theoretical and Practical Implications

The results demonstrate that HRM's recurrent reasoning process can be effectively leveraged in RL settings, supporting incremental solution construction and adaptation to dynamic changes. Carrying forward the latent state $z$ is beneficial for both convergence speed and overall training efficiency, suggesting that persistent internal representations are critical for reasoning in temporally extended tasks. However, the authors note potential risks: excessive persistence may reduce latent state entropy, potentially harming exploration and generalization in more complex domains. Stochastic state resets (analogous to dropout) may mitigate this issue.

The findings have implications for the design of agentic architectures in RL, particularly in environments requiring long-horizon planning, hierarchical reasoning, and adaptation to non-stationary dynamics. The approach is extensible to partially observable settings, multi-agent scenarios, and tasks requiring continual or few-shot learning.

Future Directions

The paper outlines several avenues for future research:

• On-policy variants: Adapting HRM-Agent to A2C or similar on-policy algorithms.
• Restoring ACT: Re-enabling adaptive computation time to allow the agent to learn optimal "thinking time" per step.
• Simplified architectures: Investigating the Tiny Recursive Model (TRM) for further architectural efficiency.
• Complex environments: Extending to richer NLE tasks with monsters, lockable doors, and agent-centered observations.
• Streaming and continual learning: Exploring online learning frameworks for continual adaptation.

Conclusion

HRM-Agent demonstrates that recurrent reasoning models can be trained via reinforcement learning to solve dynamic, fully-observable maze navigation tasks, achieving high success rates and efficient planning with modest model capacity. Carrying forward the recurrent latent state enables reuse of prior computation and continuity of intent, supporting rapid convergence and efficient training. The approach is promising for broader applications in RL, especially in domains requiring hierarchical, long-horizon reasoning and adaptation to non-stationary environments. Future work will address scaling to more complex tasks, on-policy training, and continual learning scenarios.