Agentic Transformer (AT) Overview

• Agentic Transformer (AT) is a framework that augments transformer LLMs with explicit modules for state encoding, persistent memory, planning, and tool execution.
• It employs a reasoning–action–reflection loop enabling iterative planning, adaptive behavior, and test-time performance improvements.
• Instantiations in RL and MoE architectures demonstrate superior benchmark performance, advancing tool integration, decision making, and autonomous capabilities.

Agentic Transformer (AT) denotes a class of models and architectural frameworks that extend standard transformer-based LLMs into autonomous, goal-driven agentic systems. ATs transition beyond passive sequence prediction by integrating modules for perception, memory, planning, tool use, and iterative self-reflection, operating via explicit reasoning–action–reflection loops to achieve adaptive and verifiable autonomous behavior in dynamic environments (Sibai et al., 6 Jan 2026, NVIDIA et al., 23 Dec 2025, Liu et al., 2023).

1. Formal Architecture and Operational Paradigm

Agentic Transformers build upon a pre-trained transformer LLM $f_\theta$, which consumes temporal context $c_t$ and produces structured outputs. The canonical AT architecture augments $f_\theta$ with explicit representations for agent state $s_t$, persistent memory $M_t$, action selection $a_t \in \mathcal{A}$, and a planning policy $\pi(a_t|s_t,M_t)$ that leverages these components for decision making. ATs operate according to a reasoning–action–reflection loop:

1. Perceive: Receive raw observation $o_t \in \mathcal{O}$.
2. State Encoding: Compute embedding $x_t = P(o_t)$ and update state $s_t = \mathrm{Enc}(s_{t-1}, x_t)$.
3. Plan / Reason: Sample action $a_t \sim \pi(a|s_t, M_{t-1})$ via chain-of-thought tracing or policy output.
4. Act / Execute: Apply $a_t$ through tool API or environment interface $E(a_t)$, yielding result $r_t$.
5. Reflect / Update: Persist knowledge via memory update $M_t = U(M_{t-1}, x_t, a_t, r_t)$ and transition to $s_{t+1} = R(s_t, x_t, a_t, r_t)$.

These stages formalize the AT core loop, enabling persistent knowledge accumulation, context-driven adaptation, and iterative policy refinement (Sibai et al., 6 Jan 2026).

2. Modular Integration: Perception, Memory, Planning, Tool Execution

The AT abstraction decomposes into four principal modules:

• Perception ($P$): Maps observations to structured embeddings ($P:\mathcal{O}\to\mathbb{X}$). This includes parsing unstructured text, structured JSON, or images to appropriate latent representations.
• Persistent Memory ($U$): Maintains long- and short-term knowledge, hierarchically storing recent context, episodic events, and semantic summaries. Update rule: $M_t = U(M_{t-1}, x_t, a_t, r_t)$, typically via concatenation and relevance-weighted retrieval.
• Planning / Policy ($\pi$): Produces next-step actions through $f_\theta$-generated reasoning traces ("chain-of-thought"), from which discrete/continuous actions are extracted. $\pi(a_t | s_t, M_{t-1})$ may be implemented via softmax over action representations generated by neural network modules.
• Tool Execution ($E$): Applies agent actions to the external environment. This can entail API calls, code execution, or actuation in simulated or physical spaces. The execution step is potentially non-deterministic, returning noisy or probabilistic results.

The high-level operational iteration is summarized as:

$$(o_t \overset{P}{\to} x_t \overset{\mathrm{Enc}}{\to} s_t \overset{\pi}{\to} a_t \overset{E}{\to} r_t \overset{U}{\to} M_t \overset{R}{\to} s_{t+1})$$

This design supports verifiable planning, scalable coordination, and persistent memory architectures as formal interfaces (Sibai et al., 6 Jan 2026).

3. Instantiations: ATs in RL and MoE Hybrid Architectures

Distinct instantiations of the Agentic Transformer paradigm have emerged in both reinforcement learning (RL) and scalable LLM architectures.

a) Chain-of-Hindsight Agentic Transformers

(Liu et al., 2023) proposes training a GPT-style policy transformer on "chains" of trajectories relabeled via return ordering. Given $n$ trajectories $(\tau^1,\dots,\tau^n)$ sorted by ascending total return, each is relabeled so return-to-go at time $t$ is set to the best-achieved value $\bar R$ minus cumulative reward so far. The input to the transformer at each group is ($\widehat R$, $s$, $a$, $r$, $d$), supporting test-time improvement via sequential self-conditioning:

• At test, AT rolls out $n$ repeated trials, each referencing and improving on the prior, eventually achieving higher returns than models relying on only demonstration or vanilla autoregression.
• Empirically, such ATs outperform Decision Transformers and both TD-learning and imitation learning RL baselines on benchmarks like D4RL and ExoRL (Liu et al., 2023).

b) Mixture-of-Experts Mamba-Transformer Agentic Transformers

Nemotron 3 Nano 30B-A3B (NVIDIA et al., 23 Dec 2025) operationalizes the AT abstraction via a mixture-of-experts (MoE), hybrid Mamba-Transformer. Key architecture features:

• 52 transformer layers, alternating between Mamba-2 SSM blocks (state dimension 128) and Grouped-Query Attention (GQA) blocks.
• Sparse-FFN MoE layers with 128 experts, of which 6 are active per pass ($\alpha ≈ 4.7\%$), coordinated via learned MLP router with sigmoid gating.
• No positional embeddings; sequence structure is encoded through state-space pathways.
• RL fine-tuning, multi-environment curriculum, and reinforcement learning from human feedback (RLHF) support agentic reasoning and adaptive tool use.

AT-specific agentic capabilities are realized by internal critic modules, discrete tool call sampling/execution, and iterative chain-of-thought planning (NVIDIA et al., 23 Dec 2025).

4. Training Methodologies and Self-Improvement Properties

AT training regimes exploit multi-stage pipelines integrating: (1) large-scale pretraining, (2) supervised fine-tuning (SFT) with tool-traced or multi-modal datasets, (3) multi-environment RL with curriculum and utility-driven updates, (4) RLHF and preference-based optimization targeting reasoning proficiency and format adherence.

In chain-of-hindsight formulations, staged relabeling and autoregressive modeling enable test-time agency—i.e., adaptive improvement across sequential trials. Empirically, ATs:

• Show monotonic performance improvement in repeated trials at test time, unlike contemporaneous models such as Decision Transformers which plateau after the initial attempt.
• Exhibit scaling benefits: larger model variants consistently outperform smaller ones given sufficient trajectory chains and context (Liu et al., 2023).

In the MoE Mamba-Transformer paradigm, agentic behaviors emerge via chat templates that preserve multi-step “> …” segments, internal critics for on-the-fly reward estimation, and preference optimization to iteratively minimize hallucinated tool emissions and maximize benchmark accuracy (NVIDIA et al., 23 Dec 2025).

5. Performance Benchmarks, Applications, and Practical Considerations

Evaluations consistently demonstrate that Agentic Transformers excel at decision making, tool integration, and handling extreme-length reasoning tasks.

Benchmark Domain	Metric / Result (AT)	Reference
Reasoning (AIME25)	No tools: 89.06%, With tools: 99.17%	(NVIDIA et al., 23 Dec 2025)
RL (D4RL avg)	85.21 (AT) vs. 84.83 (TD3+BC), 77.49 (DT)	(Liu et al., 2023)
TerminalBench (hard)	8.51%	(NVIDIA et al., 23 Dec 2025)
Long-context QA	RULER-100 @1M: 86.34%	(NVIDIA et al., 23 Dec 2025)

Key application domains include research automation, embodied robotics, enterprise and financial planning, tutoring, and code assistance (Sibai et al., 6 Jan 2026, NVIDIA et al., 23 Dec 2025).

ATs introduce increased inference and energy costs due to persistent, multi-step reasoning and tool execution. Mitigations such as dynamic model selection, sparse activation, and adaptive inference are explored to contain resource usage.

6. Persistent Challenges and Future Research Directions

Despite empirical progress, Agentic Transformers pose unresolved technical and governance challenges:

• Verifiable Planning: There is a critical need for formally specified, constraint-aware planning so that $\pi$ is auditable and verifiable. Hybrid symbolic–neural methods and runtime safety checks are among proposed directions (Sibai et al., 6 Jan 2026).
• Multi-Agent Coordination: Scaling to multi-agent deployments requires robust protocols for negotiation, role assignment, and shared memory (e.g., contract-net auctions, blackboard architectures, decentralized consensus).
• Persistent Memory: Ensuring long-horizon consistency and bias mitigation motivates research into hierarchical episodic–semantic memory, relevance-weighted retrieval, and decay/sanitization pipelines.
• Safety, Alignment, Reliability: Risks of tool misuse, misalignment from user intent, and accumulation of planning errors necessitate: chain-of-thought audit logging; permissioned execution frameworks; agentic task benchmarks extending MMLU (e.g., "ToolUse-QA"); and human-in-the-loop safeguards for high-impact actions.

Responsible deployment is predicated on advances in interpretability, technical robustness, and the development of governance frameworks for agentic systems (Sibai et al., 6 Jan 2026).

7. Historical and Conceptual Context

The Agentic Transformer paradigm represents a shift from traditional autoregressive language modeling to integrated cognitive architectures capable of autonomous, goal-directed operation. The synthesis of multi-modal perception, episodic memory, explicit policy planning, and environment/tool interaction positions ATs at the frontier of next-generation AI agents. Open research trajectories include theoretical analysis of agency emergence, scalable multi-agent consensus, and the principled evaluation of persistent memory and goal-planning mechanisms (Sibai et al., 6 Jan 2026, NVIDIA et al., 23 Dec 2025, Liu et al., 2023).