Pangu-Agent Framework: Modular LLM Agents

• The paper introduces a modular architecture where intrinsic reasoning modules and extrinsic action functions are integrated via reinforcement learning to improve agent performance.
• It leverages neuro-symbolic integration and memory augmentation to enable structured reasoning and dynamic, context-aware decision pipelines.
• Empirical benchmarks demonstrate significant gains in sample efficiency and cross-domain generalization across single- and multi-agent tasks.

The Pangu-Agent Framework encompasses a family of modular, LLM-based agent paradigms emphasizing structured reasoning, neuro-symbolic integration, and highly adaptive decision pipelines across a wide array of environments and tasks. By interleaving intrinsic reasoning modules with extrinsic action selection under reinforcement learning (RL) objectives, Pangu-Agent aims to address the limitations of earlier monolithic RL agents and direct LLM policies, enabling improved sample efficiency, interpretability, and cross-domain generalization in both single- and multi-agent settings (Christianos et al., 2023, Gu et al., 2022, Shi et al., 30 May 2025).

1. Architectural Overview and Core Principles

Pangu-Agent's architecture is fundamentally modular, inspired by the cognitive modularity of the human brain. It decomposes decision-making into sequences of intrinsic ("thinking") functions and extrinsic ("acting") functions within an overarching RL loop. The key architectural components are:

• Memory Module (MEM): Stores episodic traces: sequences of observations $o_{1:t}$, actions $a_{1:t-1}$, internal thoughts, tool outputs, and results.
• Intrinsic Functions $\{\mu_k\}$: Parameterized modules transforming memory, such as Plan, Reflect, Communicate, ToolUse, and search operators (e.g., BFS, DFS, MCTS). Each $\mu_k: (o_{1:t}, a_{1:t-1}, M_{t-1}) \mapsto M_t'$ acts as a distinct reasoning primitive.
• Extrinsic Function (Act) $\pi_\mathrm{outer}$: Converts the output of all intrinsic operations for a time step into environment-facing actions via LLM prompting and subsequent parsing.
• Prompt Builder $\phi^\ell(\cdot)$ and Parser $\mathcal{F}^\ell(\cdot)$: Task-dependent modules that construct LLM inputs and extract environment-compatible actions.
• Fine-Tuning Module: Supports both supervised fine-tuning (SFT) via language modeling loss and RL-based fine-tuning (RLFT) with Proximal Policy Optimization (PPO), including tailored credit assignment for actions spanning multiple tokens or modules.

The agent’s operational loop involves observing, updating memory through possibly nested intrinsic calls, constructing prompts, sampling from the LLM, parsing actions, executing in the environment, and updating via either SFT or RLFT (Christianos et al., 2023).

2. Formal Definitions and Learning Objectives

Pangu-Agent's policy stack and recursive memory updating are formalized as follows:

• Policy Representation:

$$\pi_{\mathrm{Pangu}}(a_t^\ell|o_{1:t}^\ell, M_t^\ell) = \mathcal{F}^\ell(\cdot \sim \mathrm{LLM}(\phi^\ell(o_{1:t}^\ell, M_t^\ell)))$$

• Memory Update:

$$M_t^\ell = \mu_K^\ell(\dots\mu_1^\ell(o_{1:t}^\ell, a_{1:t-1}^\ell, M_{t-1}^\ell)\dots)$$

• Global RL objective:

$$\max_{\pi, \{\mu_k\}} \mathbb{E}_{\pi, \{\mu_k\}} \left[\sum_{t=0}^{T} \gamma^t r_t\right]$$

• SFT loss (LoRA-supported):

$a_{1:t-1}$0

• RLFT (PPO) loss and value head:

Includes custom Generalized Advantage Estimation and rewards aligned to modular action boundaries.

Intrinsic functions encode prior knowledge and structural reasoning patterns—chain-of-thought (CoT), planning, tool-use—by transforming memory before the extrinsic (Act) stage, enforcing cognitive modularity and reusability. Human-designed and learned pipelines can be instantiated or composed arbitrarily during agent construction or runtime (Christianos et al., 2023).

3. Learning, Fine-Tuning, and Meta-Control Procedures

3.1 Supervised Fine-Tuning (SFT)

• High-quality trajectories are gathered (often via intrinsic prompting or rejection sampling).
• Multiple rounds of SFT can be iteratively applied, each yielding improved data.
• Both full-rank and low-rank adaptation (LoRA) of LLM parameters are supported.

3.2 Reinforcement Learning Fine-Tuning (RLFT)

• PPO is used with continuous batching for sample-efficient optimization.
• Accelerated implementations employ fused xFormers kernels and PagedAttention mechanisms to achieve 3x inference speed-ups.
• Action credit assignment precisely tracks modular action spans (from token-to-action).

3.3 Hierarchical and Meta-Learning

• Composite, multi-step methods (ReAct, SwiftSage, Least-to-Most, Self-Consistency) naturally emerge via nesting of intrinsic function calls, allowing dynamically assembled, high-level control flows.
• The DecisionFlow mechanism enables LLM-guided routing among possible intrinsic function modules during runtime, supporting dynamic, meta-level adaptation (Christianos et al., 2023).

4. Empirical Evaluation and Benchmarking

Pangu-Agent and its variants have been empirically validated across an extensive range of single- and multi-agent environments:

• Single-agent tasks: ALFWorld, BabyAI, GSM8K, HotpotQA, WebShop, HumanEval.
• Multi-agent tasks: Iterated Prisoner’s Dilemma, teacher–student GSM8K/HumanEval.

Structured Reasoning Ablations

• FS-CoT consistently outperforms Direct, ZS-CoT, and FS prompting baselines across multiple LLMs (GPT-3.5, Llama2, Vicuna, OpenChat, Mistral).
• Composite strategies (FS-CoT-SC, FS-CoT-React, Fast Sage, Least-to-Most) achieve a further 5–15 point performance gain over FS-CoT (Christianos et al., 2023).

Fine-Tuning and Generalization

Task	Direct	SFT	SFT+RL
ALFWorld	0.04	0.50	0.88
BabyAI GoToObj	0.28	0.75	0.91
GoToRedBlueBall	0.04	0.21	0.77

• Multi-task SFT+RL on Llama2-7B: ALFWorld 0.82, and BabyAI average 0.74 over 11 distinct tasks, indicating substantial cross-domain generalization.
• Same-model agent pairs show only marginal drops in multi-agent settings compared to single-agent, and mixed LLM pairs demonstrate robust, configurable multi-agent communication (Christianos et al., 2023).

5. Extensibility, Implementation, and Engineering Considerations

• Intrinsic/Extrinsic Functions: Easily extensible in Python via subclassing pangu.commands.Command. New reasoning modules and environment actions can be incorporated rapidly.
• Pipeline/Nesting: Arbitrary composite agent structures defined via YAML config files, supporting both nesting and DecisionFlow meta-control with no code changes.
• Infrastructure: Training pipelines facilitate parallel trajectory collection (16–64 workers), LoRA ranks 4–16, RL batch sizes of 64, leveraging 4–8× NVIDIA A100s or V100s.
• Inference speed is optimized via advanced kernel and memory techniques.
• Training timelines: SFT (4 epochs/1k trajectories) ~2hr/LLM/task; RLFT (1M steps) ~24–48hr/8 A100s.
• Open adaptation directions: End-to-end differentiability, enhanced memory retrieval, advanced planning, and tightly coupled tool integration are identified as promising extensions (Christianos et al., 2023).

6. Connections and Evolution in the Pangu Lineage

The Pangu line of agent architectures spans several related frameworks and instantiations:

• Discriminative Search (Pangu for KBQA): Decomposes grounded language tasks into a symbolic agent (plan generator) and neural LMs (plausibility scorer). Candidate plans are generated explicitly and scored, enabling high sample efficiency and faithfulness in real-world environments (Gu et al., 2022).
• DeepDiver Module (Adaptive Search): Augments the Pangu-7B-Reasoner with reinforcement learning under open-web information-seeking constraints (WebPuzzle), introducing the notion of Search Intensity Scaling (SIS) by teaching agents to dynamically scale search effort in noisy, ambiguous retrieval environments. The training regimen leverages curriculum learning (SFT → RL), group relative policy optimization (GRPO), and advanced reward engineering, and demonstrates performance on par with models an order of magnitude larger (Shi et al., 30 May 2025).
• Critical Analysis: Validity by construction, discriminative scoring, modular extensibility, and direct controllability are strengths. Inference cost (LM must score every candidate for beam search), reliance on gold plans, and prompt engineering for few-shot learning are acknowledged as open limitations (Gu et al., 2022).

7. Significance and Outlook

Pangu-Agent demonstrates that modular, LLM-based agent architectures equipped with structured reasoning primitives and explicit memory augmentation dramatically improve generalization, adaptation, and transparency over both direct policy and standalone LLM approaches. The extensive empirical validation on challenging reasoning, planning, and real-world search tasks, together with the robust engineering pipeline, substantiates its impact on the field of generalist AI agents (Christianos et al., 2023, Gu et al., 2022, Shi et al., 30 May 2025). Emerging directions include deepened differentiability, improved multi-modal integration, weak/partial supervision, and higher-order meta-learning. Such advances point towards more robust, interpretable, and powerful generalist agent systems.