When to Re-Plan: Subgoal Persistence in Hierarchical Latent Reasoning

Abstract: Long-horizon reasoning requires a system to commit to medium-horizon intent without becoming rigid: re-plan too often and computation never coheres into multi-step structure; commit too long and the plan goes stale. We study this stability-adaptivity tradeoff in the latent reasoning setting, where multi-step computation occurs inside hidden state rather than externalized token traces. We extend the Hierarchical Reasoning Model (HRM) with a feudal-style manager-worker interface: a slow high-level module periodically emits a normalized directional subgoal that persists for P low-level steps, biasing the worker's hidden-state updates and supplying an intrinsic cosine alignment loss. On ARC and ConceptARC, we find that subgoal persistence -- not subgoal injection alone -- is the central knob: moderate periods P in [3, 6] consistently outperform both very frequent (P=1) and very long horizons, with a clear minimum LM loss at P=3 (1.544 vs. 1.674 at P=1, 1.640 baseline; replicated over 5 seeds at mean 1.595, std 0.045). The intrinsic alignment weight lambda shows a complementary narrow optimum (lambda approximately 0.05). A controlled ablation at past-sweet-spot lambda isolates learned directional structure -- not architectural capacity or auxiliary loss alone -- as the source of interference when the alignment signal exceeds its optimum. Together these findings implicate a design principle for compositional planning in latent reasoning systems: medium-horizon intent must be coherent across enough computational steps for compositional structure to form.

• The paper identifies subgoal persistence as the key design variable in hierarchical latent reasoning, with manager periods of P=3–6 outperforming no-subgoal and P=1 settings on ARC-AGI.
• A sweep finds the best performance at P=3, reaching mean language-model loss of 1.595 across five seeds, while overly frequent revision destabilizes computation and longer commitments degrade only gradually.
• The method works best as a lightweight planning prior, with alignment weight λ≈0.05; stronger weighting harms performance, and ablations show that learned directional structure—not added capacity—causes this interference.

Motivation and problem statement

Latent reasoning architectures such as the Hierarchical Reasoning Model (HRM) perform multi-step computation inside hidden state rather than as externalized token traces, gaining speed and compactness at the cost of the implicit commitment structure that token-level chain-of-thought provides for free. In an externalized trace, each emitted token constrains subsequent deliberation; in a latent reasoner, nothing forces medium-horizon intent to persist across micro-steps. The paper asks a precise question: how long should a latent reasoner commit to an intent before revising it? The answer is operationalized through subgoal persistence — the number of low-level steps $P$ that a manager-emitted directional subgoal remains in force — and the resulting stability–adaptivity tradeoff is characterized empirically on ARC-AGI and ConceptARC.

The framing imports the commitment-duration lens from feudal reinforcement learning and the options framework, where commitment duration is already recognized as a first-class design choice. The transfer is nontrivial: the worker's "actions" are hidden-state updates rather than environment interactions, and the cost of a stale plan is representational rather than behavioral.

Method: Subgoal-Augmented HRM

The method extends HRM with a feudal manager–worker interface. Every $P$ low-level micro-steps, the high-level state $z^H$ is projected through a learned matrix $W_g$ and $\ell_2$-normalized to produce a directional subgoal $g_k$. The authors argue explicitly for direction rather than target-state goals: a target would be brittle under nonstationary recurrent dynamics, whereas direction encodes "where to move" without overspecifying execution. A learned scalar gate $\alpha_k = \sigma(w_\alpha^\top z^H_{t_k})$ modulates commitment strength.

During its persistence window, the active goal enters the worker's update as an additive bias via a projection $V_L$, optionally also into the high-level update. Injection alone does not guarantee progress along the issued direction, so a cosine alignment loss rewards net low-level displacement over the window:

$$\mathcal{L}^{(k)}_{\text{align}} = 1 - \cos(\Delta z^L_k,\, g_k), \qquad \Delta z^L_k = z^L_{t_k+P} - z^L_{t_k}$$

added to the HRM objective (task loss plus ACT halting losses) with weight $\lambda$. Because HRM detaches state between segments, alignment gradients do not cross segment boundaries. The backbone follows the standard HRM configuration: hidden size 512, 4+4 transformer layers, $P$0 high-level update period, ACT with up to 16 internal steps.

Persistence is the central knob

Sweeping $P$1 at fixed $P$2 yields three findings that together constitute the paper's strongest claim:

Manager period $P$3	LM loss
1	1.674 (worst; above baseline)
2	1.638
3	1.544 (best)
4	1.564
5	1.590
6	1.564
8	1.568
no-subgoal baseline	1.640

First, persistence is necessary, not optional: at $P$4 the full subgoal infrastructure underperforms the no-subgoal baseline ($P$5 vs.\ $P$6). This is presented as the cleanest evidence that the persistence mechanism, not the injection mechanism, carries the benefit — a subgoal overwritten every step provides no temporal coherence and effectively injects noise. Second, the transition from $P$7 to $P$8 produces the largest single-step improvement in the sweep ($P$9 relative to baseline), suggesting a minimum coherence horizon below which compositional structure cannot form. Third, decay beyond the optimum is gradual: loss stays within $z^H$0 out to $z^H$1. The asymmetry supports the interpretation that staleness is a soft failure mode while absence of commitment is not.

Replication across 5 seeds at the best configurations gives mean LM loss $z^H$2 (std $z^H$3) at $z^H$4 and $z^H$5 (std $z^H$6) at $z^H$7; seed variability is small relative to the $z^H$8-vs-$z^H$9 gap, ruling out a single-seed artifact. One caveat the authors flag: cross-task validation on ConceptARC-mini shows only a marginal improvement ($W_g$0 vs.\ $W_g$1 baseline, roughly $W_g$2), which they themselves downgrade to a directionally consistent secondary observation rather than independent benchmark evidence.

Narrow optimum in the alignment weight

Fixing $W_g$3 and sweeping $W_g$4 reveals a narrow optimum at $W_g$5: LM loss $W_g$6 there, versus $W_g$7 at $W_g$8, $W_g$9 at $\ell_2$0 (essentially matching baseline), and degradation below baseline for $\ell_2$1. Token-level accuracy tracks the same optimum ($\ell_2$2 at $\ell_2$3). The interpretation is that the alignment signal functions best as a lightweight planning prior rather than a dominant geometric constraint; when it competes with the task gradient on comparable terms, it reduces the worker's representational flexibility. Removing the $\ell_2$4 normalization of $\ell_2$5 or the commitment gate degrades performance, confirming the importance of directional semantics and the soft-prior role.

Past-sweet-spot ablation isolates learned directional structure

A three-cell controlled ablation at $\ell_2$6, $\ell_2$7 (batch size 64 on a single L4 GPU, num_aug=100) distinguishes architectural capacity, auxiliary loss, and learned directional content:

Cell	Train LM loss	Δ vs. baseline
A_full (learned directions)	1.327	+0.100
B_baseline (vanilla HRM)	1.227	—
E_random (random unit directions)	1.230	+0.003

Two contrasts are decisive. Random directions with identical architecture and auxiliary loss match baseline within $\ell_2$8 — an order of magnitude below the main study's seed-level std of $\ell_2$9 — so neither added capacity nor the intrinsic term alone causes harm. The full mechanism's $g_k$0 gap over E_random therefore attributes past-sweet-spot interference specifically to learned directional content: when over-weighted, real directional structure captures representational capacity otherwise used for the task objective. As the authors put it, a mechanism that does nothing past its optimum cannot do anything at it either — the fact that learned directions help at $g_k$1 and hurt at $g_k$2 is two sides of the same representational work.

A methodological caveat deserves emphasis: absolute loss values are not comparable between the main study (CPU, batch size 768, num_aug=1000) and the ablation (GPU, batch size 64, num_aug=100); the same nominal configuration yields $g_k$3 versus $g_k$4. All claims rest on within-regime contrasts, and the ablation uses a single fixed seed evaluated at one optimization step.

Limitations and open questions

The paper is candid about scope. Evaluation is restricted to ARC-style grid puzzles, and the held-out ConceptARC-mini gain is too small to serve as generalization evidence. The ablation's single-seed design is bounded by main-study variance but not directly replicated. Most substantively, the paper offers no representation-level analysis of how persistent subgoals shape worker hidden-state geometry — the analysis that would directly probe the claimed compositional substrate — because late-training checkpoints were not retained; this is left explicitly open. Whether the identified sweet spot $g_k$5 transfers beyond this task family and backbone scale also remains untested.

Conclusion

This work establishes subgoal persistence, not subgoal injection, as the operative variable in feudal-style latent reasoning: moderate manager periods ($g_k$6) yield consistent gains, $g_k$7 actively harms performance relative to no subgoals, and staleness degrades far more gracefully than instability. Combined with the narrow $g_k$8 optimum and the ablation attributing past-sweet-spot interference to learned directional structure, the results support a concrete design principle: medium-horizon intent must remain coherent across enough computational steps for compositional structure to form, while remaining soft enough not to override task optimization.