HaluMem: AI Benchmark & Spintronic Memory

• HaluMem is a dual-purpose concept that defines an AI dialogue memory evaluation benchmark and a spintronic Hall-memristor using antiferromagnetic materials.
• The AI benchmarking suite operationalizes memory extraction, updating, and query diagnostics with concrete metrics like recall, precision, and QA accuracy to mitigate hallucinations.
• The spintronic component exploits the nonlinear Hall effect for ultrafast, energy-efficient, and durable nonvolatile memory with distinct four-terminal device design.

HaluMem refers to two distinct, technically rigorous concepts: (1) a benchmarking suite and framework for diagnosing and quantifying hallucinations in AI dialog agent memory systems (Chen et al., 5 Nov 2025), and (2) an antiferromagnetic Hall-memristor, a novel class of spintronic memory devices that encode information via nonlinear Hall effects in certain materials (Barrera et al., 24 Jul 2025). Both concepts are foundational for their respective domains: the former for advancing long-term consistency in LLM–based agents, the latter for enabling ultrafast, energy-efficient nonvolatile physical memory.

1. HaluMem: Hallucination Evaluation Benchmark for AI Memory Systems

1.1 Formal Structure and Metrics

HaluMem conceptualizes an agent’s longitudinal dialog memory as a sequence of plaintext memory points $m_i$, extracted, maintained, and queried over a multi-turn conversation $D = \{(u_1, a_1), ..., (u_N, a_N)\}$. It decomposes the memory system $S$ into three core operations:

• Memory Extraction $E$: $$G^{ext}_s = \{m_i^s\},\, \widehat{M}^{ext}_s = E(D^s) = \{\hat m_j^s\}$$
• Memory Updating $U$: $G^{upd}_s = \{(m^{old} \rightarrow m^{new})\}$, with system output $\widehat{G}^{upd}_s = U(\widehat M^{ext}_s, D^s)$
• Memory Question Answering $Q$: Given query $q_j$, produces $\hat y_j = A(\widehat R_j, q_j)$ using retrieved memories $\widehat R_j = R(\widehat M, q_j)$

Stage-wise metrics are defined for each operation, including recall, precision, weighted recall (importance and partial-extraction-aware), accuracy, false memory resistance (FMR), update accuracy, hallucination rate, omission rate, and downstream QA metrics: $$\mathrm{Recall} = \frac{N_\mathrm{correct}}{N_\mathrm{should}}, \quad \mathrm{Precision}_T = \frac{\sum_{j \in M_T} s_j}{|M_T|}, \ldots$$ with further variants adjusting scores by memory type, importance, and update operation.

2. Benchmark Architecture and Task Workflow

The HaluMem evaluation suite operationalizes the above metrics over a standardized API:

1. AddDialogue(D^s) triggers in-session memory extraction.
2. GetDialogueMemory(s) returns session-level extracted memories $\widehat M^{ext}_s$.
3. RetrieveMemory(q) identifies top-$K$ relevant memories for query response.

Memory points record content, type (persona, event, relationship), timestamps, update status, and provenance, enabling fine-grained error tracing across extraction, update, and retrieval. Each operational stage (extraction, update, QA) is evaluated both in isolation (operation-local hallucinations) and end-to-end, providing localization of error sources (Chen et al., 5 Nov 2025).

3. Dataset Composition: HaluMem-Medium and HaluMem-Long

HaluMem comprises two user-centric, multi-turn dialog datasets:

• HaluMem-Medium: 20 users, 30,073 turns, $\sim$15K gold memories, $>3.5$K multi-type queries, average context $\sim160$K tokens/user; memory points are split among persona, event, and relationship types, with distractors and updates explicitly labeled.
• HaluMem-Long: Extends Medium with inserted irrelevant (ELI5, mathematical, synthetic) dialog, context lengths $>1$M tokens/user, longer-range dependencies, and identical gold memory/query structure.

Both splits uniformly distribute memory types and questions across fabrication, omission, conflict, inference, and generalization, stressing both precision and recall at every operational layer. Distractor memories and multi-hop inference queries are embedded to systematically probe both anti-hallucination and anti-amnesia (Chen et al., 5 Nov 2025, Hu et al., 3 Jan 2026).

4. Evaluation Methodology: Hallucination Diagnosis and Scoring

Hallucinations in HaluMem are algorithmically and manually labeled per operation:

• Extraction: Fabrications ($\hat m \notin G^{ext}_s$), omissions ($m \in G^{ext}_s$ missing from $\widehat M^{ext}_s$), and partials (weighted scores).
• Update: Hallucinations (fabricated/erroneous updates), omissions (missed updates), correct (gold-consistent transforms).
• QA: Hallucinations (answers unsupported by memory), omissions (missing answer elements), QA-accuracy (semantic equivalence to gold answer given memory access).

Automated judgment uses GPT-4o with controlled prompt templates; final metrics are averaged across all users, sessions, and question types. Error propagation (low extraction recall $\rightarrow$ poor update trigger $\rightarrow$ high QA omission) is quantified, revealing that extraction-stage hallucinations dominate overall system unreliability (Chen et al., 5 Nov 2025).

5. Empirical Findings and Baseline Recommendations

Controlled ablations and baseline evaluations (Chen et al., 5 Nov 2025, Hu et al., 3 Jan 2026) yield the following results:

Setting	Mem-Recall	Mem-Precision	MemUpdate-Correct	QA-Accuracy
Separate keys, add-only	0.7332	0.9360	–	0.2815
Separate keys, add/up/noop	0.7121	0.9811	0.1095	0.4800
Merged key, add-only	0.7332	0.9360	–	0.4785
Merged key, add/up/noop	0.7124	0.9808	0.1537	0.5535

Flat vector indexes with merged session-level keys ([S,F,K], Editor's term) and explicit Add/Update/Noop maintenance maximize QA-accuracy ($\approx55\%$), extraction precision ($\approx98\%$), and recall ($\approx71\%$), outperforming both graph-based memory indexes (lower recall, higher hallucination rate) and non-updating architectures (Hu et al., 3 Jan 2026). Hallucination rates in QA reach 17–30% even for the best systems; Medium versus Long splits reveal severe drops in recall and accuracy under extreme context lengths.

Flat baselines outperform graph-based (entity-centric description graph) approaches, which, while achieving nearly perfect precision, drop to 47% recall and 49% QA-accuracy due to missing fact granularity.

6. Diagnosis, Constraints, and Future Directions

The propagation and amplification of hallucinations in LLM memory systems arises predominantly from:

• Over-extraction or misinterpretation during memory extraction (fabrication, inclusion of low-value or false memories)
• Insufficient linking of extraction and update stages, leading to outdated facts or memory conflicts
• Noisy retrieval under ultra-long context, introducing distractor memories and compounding QA hallucinations

Proposed countermeasures include:

• Interpretable, constrained extraction: Rule-based scoring, importance weighting, and interactive self-verification to only admit high-confidence, user-confirmed facts
• Structured update logic: Versioning, explicit conflict detection and resolution, and deletion-tracking of memory points
• Controlled retrieval: Hybrid text/graph indexing with strict relevance thresholds and dynamic context trimming
• End-to-end traceability: Provenance and confidence tagging for each memory point, supporting downstream hallucination detection and exclusion
• Scalable efficiency: Batch extraction, incremental indexing for extreme context length, minimizing latency without precision loss

The evidence suggests that reliable long-term memory for LLM-driven agents will require an overview of symbolic/logical rigor and learned flexibility, with interpretability and error localization accessible at every operation stage (Chen et al., 5 Nov 2025).

7. HaluMem in Spintronics: The Antiferromagnetic Hall-Memristor

In condensed matter physics, HaluMem also designates the “Hall-memristor,” a four-terminal spintronic device leveraging antiferromagnetic (AF) materials with a nonlinear Hall effect for memory storage (Barrera et al., 24 Jul 2025). Its critical properties include:

• State encoding: Memory is encoded in the orientation of the Néel vector $\mathbf n$ of the AF material (e.g., CuMnAs); the nonlinear Hall conductance $\sigma_H(\mathbf n)$ enables nonvolatile electrical readout
• Reading channel: Second-order (nonlinear) Hall effect, $j_a = \sigma_{a,bc}(\mathbf n) E_b E_c$, with $\sigma_H$ dependent on $\mathbf n$. The Hall voltage persists after removal of the writing field
• Writing/erasing channel: Nonlinear Edelstein Effect (NLEE) generates a staggered spin polarization $\delta s_x \propto \chi_{x,xx} E_x^2$ that exerts exchange torque, reorienting $\mathbf n$
• Device operation: Four-terminal geometry enables separate writing (current pulse along $x$-axis) and reading (transverse Hall voltage via $y$-axis measurement)
• Switching and retention: Device exhibits $\sim$10–30% Hall conductance modulation upon state switching, ultrafast writing ($\sim$1 ps relaxation), %%%%37$\hat y_j = A(\widehat R_j, q_j)$38%%%% endurance cycles, and nonvolatility linked to AF anisotropy

This HaluMem implementation exploits specific symmetry constraints (spatial inversion and mirror-plane breaking, PT invariance) to enable memory storage and retrieval in AFs without net magnetization. The device-level realization in tetragonal CuMnAs sets a concrete basis for Hall-memristive memory and may underpin future “all-electrical” spin-memory technologies (Barrera et al., 24 Jul 2025).

---

In summary, HaluMem identifies critical research frontiers for both AI hallucination evaluation and AF spintronics memory. In computational memory evaluation, HaluMem benchmarks establish operation-local, end-to-end, and type-sensitive measurement of hallucination dynamics, with actionable baselines for extracting, updating, and querying dialog memory at scale (Chen et al., 5 Nov 2025, Hu et al., 3 Jan 2026). In quantum materials, HaluMem denotes a framework for fast, nonvolatile electrical memory via nonlinear Hall responses to antiferromagnetic order (Barrera et al., 24 Jul 2025).