Tertiary Memory and Reflection in Neural Agents

• Tertiary memory is a structured, persistent external storage layer that consolidates past experiences and skills to support verifiable, long-term decision-making.
• Reflection refers to the self-assessment and error-correction processes that update and refine outputs based on retrieved memory, ensuring accuracy and coherence.
• Together, these mechanisms enhance neural agent performance by enabling cross-task generalization, dynamic memory updates, and robust rule-based governance.

Tertiary memory and reflection are foundational concepts in contemporary neural agent systems—particularly LLM architectures—that designate a stratum of persistent, interpretable, and actionable memory external to both transient context (“short-term memory”) and static model parameters (“long-term memory”). Tertiary memory systematically consolidates past episodes, skills, insights, or rules into structured stores, while reflection refers to mechanisms for self-assessment, correction, and continual adaptation using such stores. These mechanisms facilitate cross-task generalization, correct error propagation, support verifiability, and enable longitudinal coherence in both dialogic and sequential decision processes.

1. Foundations and Definitions

Tertiary memory denotes a persistent, structured external storage layer that augments an agent’s “short-term” contextual buffer (chat window or context window) and “long-term” parametric encoding (weights $\theta$). Its primary function is to capture, summarize, and retrieve distilled forms of experience—ranging from citation evidence (Sun et al., 2023), diagnostic skills (Lan et al., 20 Sep 2024), dialogue topic summaries (Tan et al., 11 Mar 2025), meta-policy rules (Wu et al., 4 Sep 2025), to contextual rationales and history (Wedel, 28 May 2025). Reflection is the organizational mechanism by which agents interrogate, evaluate, or regenerate outputs in light of tertiary memory, often via explicit verification, human-in-the-loop auditing, or rule-based admissibility checks.

Tertiary memory structures are foundational in frameworks such as:

• Evolving memory for verifiable text generation (Sun et al., 2023)
• Hierarchical memory banks in dialog agents (Tan et al., 11 Mar 2025)
• Meta-policy memory for action selection (Wu et al., 4 Sep 2025)
• Contextual memory intelligence with human reflection (Wedel, 28 May 2025)
• Distilled skills and electronic records in clinical simulation (Lan et al., 20 Sep 2024)

2. Tertiary Memory Architectures

Hierarchical and Modular Designs

Tertiary memory is architected in multi-level hierarchical organizations:

Paper	Short-Term	Mid-Term	Tertiary Memory Layer
(Lan et al., 20 Sep 2024)	CR (sessions)	EMR (summaries)	Diagnostic Skills (distilled reflective nodes)
(Tan et al., 11 Mar 2025)	Utterances	Turns/Sessions	Topic-based Memory Bank (summarized episodes)
(Sun et al., 2023)	Dₛ (new docs)	Dₗ (retained docs)	M⁽ᵗ⁾ = Dₛ ∪ Dₗ (verified evidence corpus)
(Wu et al., 4 Sep 2025)	Prompt stack	Model weights	MPM (symbolic predicate→action rules)
(Wedel, 28 May 2025)	Session info	N/A	Insight Layer records (with rationale/context markers)

• (Lan et al., 20 Sep 2024) operationalizes CR→EMR→DS, with skills extracted post hoc via a reflection step conditioned on diagnostic mistakes.
• (Tan et al., 11 Mar 2025) employs a memory bank with dynamic granularity, storing topic summaries cross-referenced to raw dialogs.
• (Sun et al., 2023) maintains sets of validated external documents as long- and short-term evidence, conceptually a tertiary layer above both context window and weights.
• (Wu et al., 4 Sep 2025) defines a meta-policy memory (MPM) as a set of state predicates and corrective rules for action, positioned above all other memory forms.
• (Wedel, 28 May 2025) represents each insight as a triplet $m_i = (v_i, C_i, R_i)$ combining embedding, contextual metadata, and rationale, indexed in a vector/graph store.

3. Reflection Mechanisms and Algorithms

Reflection procedures operate via a spectrum of mechanisms for error correction, knowledge consolidation, and inference-time guidance:

Self-Reflection and Verification Loops

• Two-Tier Verifier (Sun et al., 2023): Comprises a generation verifier (entailment between claim and proposed citations) and a memory verifier (entailment between claim and the entirety of the memory bank).
  • $GV(s_t, C_t)$ returns $1$ if $NLI(C_t \vDash s_t)$ is true.
  • $MV(s_t, M^{(t)})$ performs the same check over $M^{(t)}$.
  • Claims are accepted, pruned, or regenerated based on these checks, enforcing factual consistency and dynamically updating the evidence memory.

Prospective and Retrospective Reflection

• Prospective Reflection (Tan et al., 11 Mar 2025): Dynamic summarization at the end of each interaction session, extracting semantically-coherent topics across varying granularities (utterance, turn, session). New summaries are either merged with or added as fresh memory entries, with subsequent updates to avoid fragmentation.
• Retrospective Reflection (Tan et al., 11 Mar 2025): Online RL-based reranking of candidate retrievals using per-response citation feedback. Cited memories receive positive rewards and the memory retriever/reranker is updated via REINFORCE to emphasize high-utility entries.

Supervisory Plugins and Corrective Skills

• Supervisor Plugin (Lan et al., 20 Sep 2024): Monitors dialogue stage, tracks symptom status, and issues turn-by-turn instructions. When a misdiagnosis is detected, triggers extraction of a diagnostic “Skills” node that abstracts over/under-emphasized symptoms and guides future questioning.

Meta-Policy Reflection and Admissibility

• Meta-Policy Reflexion (Wu et al., 4 Sep 2025): Online accumulation of corrective rules in the form $\{(p_i \implies a_i, w_i)\}$, where $p_i$ is a predicate over state, $a_i$ an action, and $w_i$ a confidence. Extracted rules bias future LLM outputs via prompt augmentation (“soft” influence) and can be enforced as action admissibility constraints (“hard” veto), yielding robust transfer and error correction in sequential agents.

Human-in-the-Loop Reflection

• Insight Layer (Wedel, 28 May 2025): Integrates a Reflection Interface that halts workflows at designated “insight repair points.” A human expert reviews surfaced context/rationales, addresses drift warnings, and annotates corrections, which are then embedded and traced in the memory layer.

4. Memory Update, Retrieval, and Scoring

Tertiary memory maintenance involves explicit update rules and retrieval policies:

• Update: Memory entries (evidence, skills, topic summaries, rules) are appended, merged, or pruned based on session completion, reflection outcomes, and performance feedback (Lan et al., 20 Sep 2024, Tan et al., 11 Mar 2025, Sun et al., 2023, Wu et al., 4 Sep 2025).
  • Example: After dialogue $t$, $m_{\mathrm{EMR}}^t$ and, if needed, $m_{\mathrm{DS}}^t$ are added, with importance scores updated and normalized (Lan et al., 20 Sep 2024).
  • MPM is expanded with rules $\{(p_i \implies a_i, w_i)\}$ upon observed failures (Wu et al., 4 Sep 2025).
• Retrieval and Scoring: Queries are processed by similarity functions (e.g., embedding dot product or cosine), relevance normalization, and combined with node importance:
  • $$\text{rel}_m = \mathrm{NORM}(\mathbf{e}_m \cdot E(q))$$
  • $$\text{score}_m = \alpha_1\,\text{rel}_m + \alpha_2\,\mathit{imp}_m$$ (Lan et al., 20 Sep 2024)
  • Sampling allows for exploration by occasionally retrieving nodes of intermediate relevance (Lan et al., 20 Sep 2024).
• RL-based Refinement: Retrospective reranking in RMM employs Gumbel-Softmax stochastic sampling and is updated by REINFORCE with per-memory rewards based on citation in LLM output (Tan et al., 11 Mar 2025).
• Constraint Enforcement: Washed through both soft prompt augmentation (biasing next-token sampling) and hard admissibility (vetoes on invalid actions) (Wu et al., 4 Sep 2025).

5. Empirical Results and System-Level Gains

Empirical evidence across varied domains demonstrates the efficacy of combining tertiary memory with robust reflection protocols:

• Citation and Answer Quality: In verifiable text generation, evolving memory and self-reflection raise citation F1 from 50.09 to 60.61 and answer EM from ~32.7% to 41.5% on 2WikiMultihopQA—relative gains of +21% and +27% (Sun et al., 2023). Removal of memory or verifier components reduces these gains by 4-8 F1 points, demonstrating their necessity.
• Dialogue Personalization and Accuracy: RMM achieves 6–10% absolute improvement over RAG and baseline systems on metrics such as MSC METEOR and LongMemEval Answer Accuracy, confirming that hierarchical memory and reflection produce more coherent long-term dialogue (Tan et al., 11 Mar 2025).
• Clinical Simulation: Adding both EMR (session summaries) and Skills (distilled lessons) yields 4–7% absolute gains for depression and suicide risk classification, with the supervisor plugin alone boosting scores by 1-3% depending on evaluation setting (Lan et al., 20 Sep 2024).
• Policy Transfer and Robustness: Meta-Policy Reflexion achieves substantially faster adaptation (from 70.0% to 83.9% after one round, and 100% by round three on training tasks), and superior generalization on test tasks (MPR 87.8%, MPR+HAC 91.4%, outperforming Reflexion at 86.9%) (Wu et al., 4 Sep 2025).
• Explainability and Governance: CMI yields traceable, auditable records for every memory edit, reflection, and drift alert, with production targets outlined for recall latency (<250ms), regeneration time (<1s), and explicit compliance protocols (Wedel, 28 May 2025).

6. Theoretical Context and Principled Extensions

Contemporary tertiary memory and reflection systems are informed by cognitive-science, organization theory, and systems epistemology:

• Double-Loop Learning: Systems such as CMI extend beyond simple behavioral correction to support revision of decision rationales/assumptions, storing not only “what worked” but also rejected alternatives (Wedel, 28 May 2025).
• Distributed Memory: Externalization of contextual cues (procedural steps, social interaction) enables longitudinal reasoning over traces distributed across people, artifacts, and digital environments.
• Autopoiesis and Irreducibility: Preserving full rationale traces ($R_i$) acknowledges that complex decision histories are often non-compressible, necessitating memory systems that store both outcomes and supporting context (Wedel, 28 May 2025).

Tertiary memory frameworks contrast with and address the limitations of RAG, vector-only stores, and purely parametric or prompt-level approaches. They enable explainability, responsible governance, and resilience in evolving LLM-driven systems, with applications extending from conversational agents to scientific reasoning, healthcare workflows, and resource-constrained agent environments (Sun et al., 2023, Lan et al., 20 Sep 2024, Tan et al., 11 Mar 2025, Wedel, 28 May 2025, Wu et al., 4 Sep 2025).

7. Limitations, Failure Modes, and Future Directions

Tertiary memory systems may face challenges regarding scalability (rule pruning, redundancy, and growing memory size), maintenance of retrieval quality (drift detection, relevance decay), and interpretability (noisy or conflicting rules). Approaches such as confidence weighting, automated abstraction, distributed memory organization, and expansion to multimodal predicates are under investigation (Wu et al., 4 Sep 2025, Wedel, 28 May 2025).

A plausible implication is that future agent architectures will tightly integrate structured reflective memory, dynamic reasoning, and human oversight as foundational capabilities, enabling robust adaptation, safety, and cross-domain generalization without reliance on frequent model re-training or rigid pre-defined retrieval schemas.