Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning (2512.20605v2)

Abstract: Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. During RL, these models explore by generating new outputs, one token at a time. However, sampling actions token-by-token can result in highly inefficient learning, particularly when rewards are sparse. Here, we show that it is possible to overcome this problem by acting and exploring within the internal representations of an autoregressive model. Specifically, to discover temporally-abstract actions, we introduce a higher-order, non-causal sequence model whose outputs control the residual stream activations of a base autoregressive model. On grid world and MuJoCo-based tasks with hierarchical structure, we find that the higher-order model learns to compress long activation sequence chunks onto internal controllers. Critically, each controller executes a sequence of behaviorally meaningful actions that unfold over long timescales and are accompanied with a learned termination condition, such that composing multiple controllers over time leads to efficient exploration on novel tasks. We show that direct internal controller reinforcement, a process we term "internal RL", enables learning from sparse rewards in cases where standard RL finetuning fails. Our results demonstrate the benefits of latent action generation and reinforcement in autoregressive models, suggesting internal RL as a promising avenue for realizing hierarchical RL within foundation models.

• The paper demonstrates that latent temporal abstractions naturally emerge within pretrained autoregressive models through linear probing and intervention analysis.
• The metacontroller leverages unsupervised variational inference to orchestrate abstract subgoal policies, enhancing exploration in sparse-reward RL tasks.
• Freezing the autoregressive backbone preserves compositional structure, leading to improved sample efficiency and robust, out-of-distribution generalization.

Emergent Temporal Abstractions in Autoregressive Models Enable Hierarchical Reinforcement Learning

Introduction

This paper presents a mechanistic and empirical investigation of the emergence of temporal abstraction in the internal representations of large-scale autoregressive models and its utility for hierarchical reinforcement learning (HRL) (2512.20605). The core hypothesis is that autoregressive models pretrained on behavioral data from goal-directed agents not only encode actionable priors but also implicitly form temporally extended abstractions in their intermediate activations. These emergent structures are leveraged to design a metacontroller that steers the base model’s activations, yielding highly effective, unsupervised latent-space control and efficient exploration in sparse-reward RL domains.

Emergence of Temporally-Abstract Action Representations

The study first demonstrates that internal states in autoregressive sequence models (transformers and SSMs) encode latent variables corresponding to agent subgoals, despite the absence of explicit supervision for compositional or hierarchical behavior. Through linear probing and causal intervention analysis, the authors reveal that belief states about latent, temporally extended goals are linearly decodable from residual activations at mid to late depths.

Figure 1: Linear decodability of abstract subgoals from residual stream activations increases with model depth, supporting the emergence of internal belief representations of temporally-abstract actions.

These probe results generalize to both discrete (grid world) and continuous (MuJoCo ant) environments, underscoring the versatility of autoregressive pretraining for pattern abstraction. The decodability and controllability peak at mid-depth layers, consistent with recent findings on transfer and future-predictive capacity in LLMs, suggesting an optimal locus for intervention within model hierarchies.

Figure 2: Success rate as a function of controller insertion depth confirms that mid-layer intervention supports compositional generalization in both grid world and ant environments.

Self-Supervised Discovery and Control via Metacontroller

The authors introduce a metacontroller—an unsupervised, recurrent, stochastic hypernetwork—that learns to intervene in the base model’s residual stream via sequences of sparse, temporally-abstract controllers without end-to-end supervision. The metacontroller generates controller codes $z_t$ and temporal switching gates $\beta_t$, which yield temporally persistent interventions corresponding to subgoal policies.

Figure 3: Overview of the approach. A metacontroller dynamically orchestrates sparse internal controller sequences in the residual stream, enabling abstract, temporally extended control and addressing hierarchical exploration via internal RL.

Figure 4: Metacontroller architecture details the sequential generation of controller codes, temporal switching, and execution of interventions in the residual stream.

Through variational inference, the model’s internal representation of controller codes aligns with underlying subgoal structure; switches in $\beta_t$ correspond almost perfectly to ground-truth abstract action boundaries, both in discrete and continuous control domains, even in the absence of reward or subgoal label data.

Rate-Distortion Analysis and the Importance of Pretrained Freezing

A comprehensive rate-distortion analysis reveals that freezing the base autoregressive model is essential to the emergence of compositional latent policies. Training the metacontroller alone (with a frozen base) yields a characteristic rate-distortion curve with a sharp distortion reduction at subgoal-aligned switching rates, which is absent if the sequence model parameters are co-trained.

Figure 5: Rate-distortion tradeoff illustrates that temporally aligned and compositional switching only emerges under a frozen autoregressive backbone; co-training results in degenerate, collapsed segmentations.

This finding highlights the critical role of the autoregressive model’s representational prior—self-supervised pretraining on goal-directed behaviors imparts a structural bias that supports abstraction and composition, which is lost if the backbone is adaptively modified during metacontroller training.

Internal RL: Efficient Sparse-Reward Exploration in Latent Action Space

Capitalizing on the sparse and compositional abstract actions uncovered by the metacontroller, the authors propose an internal RL paradigm. Here, RL is conducted not in the raw token or action space, but within the discovered low-dimensional latent controller code space. The autoregressive model is subsumed as part of the environment, and the RL policy emits interventions at significantly reduced temporal frequency.

This approach solves challenging, sparse-reward, compositional tasks where token-level exploration and strong hierarchical RL baselines (e.g., CompILE) fail. Empirical curves show many orders of magnitude improvement in sample efficiency, initial success rates, and credit assignment.

(Figure 3) (re-shown for context)

Figure 3: Acting in the abstract action space via internal RL—agent operates at contracted timescales, closing the gap in sparse-reward domains by leveraging discovered latent abstractions.

Robustness, Compositional Generalization, and Interpretable Latents

The learned controllers generalize out-of-distribution to new subgoal orderings, timings, and environmental configurations. Latent codes corresponding to abstract actions, such as "go to blue," when activated, robustly redirect policy execution to novel contexts, supporting the claim of compositional, context-agnostic policy primitives.

Figure 6: Latent controller codes—such as one corresponding to "go to blue"—are effective when activated arbitrarily in novel contexts, steering the agent’s behavior toward the desired subgoal.

Further, the model demonstrates robustness to moderate pretraining noise and the presence of suboptimal demonstrations, and its compositional generalization is supported across variations in training hyperparameters (Figure 7).

Figure 7: Compositional generalization persists across variations in pretraining steps, weight decay, auxiliary losses, and demonstration suboptimality.

Theoretical and Practical Implications

The results demonstrate that large-scale autoregressive sequence models encode hierarchically meaningful abstractions that can be exploited for HRL. This establishes a formal and empirical connection between in-context latent variable inference, ascribed to next-token predictors, and the classical "options" framework in RL. The internal RL methodology shows that intervening within internal residual stream activations opens a novel path for rapid, sample-efficient, and robust strategy acquisition in sparse-reward, compositional domains.

The requirement that the base model be frozen after pretraining is both theoretically interesting and practically significant—this phase separation protects the emergence of compositional structure, regularizing abstract action discovery and facilitating efficient credit assignment at higher temporal abstraction levels.

Prospective Directions

This framework is immediately relevant for future research on controllable foundation models, scalable interpretability methods, and reasoning LLMs. Internal RL could be extended toward more complex reasoning tasks (e.g., mathematical proofs, multi-step planning in language), multi-agent compositionality, and integration with other forms of recurrent or continual reasoning modules.

A further theoretical implication is elucidating the correspondence between information bottlenecks in variational sequence models and the emergence of discrete, interpretable, temporally abstract policies. This opens up connections with information-theoretic analyses of policy structure and may inform architectures for general-purpose, compositional intelligence.

Conclusion

This work establishes that temporally-abstract, compositional action representations robustly emerge in the internal activations of pretrained autoregressive models. By leveraging a metacontroller to unsupervisedly discover and sequence these latents, the authors realize an internal RL paradigm capable of hierarchical exploration and credit assignment at scales where traditional methods fail. Both the architectural insights—on where to intervene and how to structure abstractions—and the empirical successes—in continuous and discrete RL—mark this approach as a promising foundation for interpretable, steerable, and efficient AI agents.