Learning What to Remember: A Cognitively Grounded Multi-Factor Value Model for Agentic Memory

Abstract: Long-running LLM agents accumulate interaction histories far larger than any context window, forcing a standing decision: what to encode deeply, what to forget, and what to retrieve under a fixed memory budget. Production systems answer with semantic similarity or recency -- both mis-specified for the forgetting decision, which is made at consolidation time before the future query is known. We propose a multi-factor memory value function V(m)=\sum_i w_i f_i(m) over seven interpretable factors (emotional intensity, goal relevance, value alignment, self/user relevance, task utility, reliability, and usage history) drawn from cognitive psychology, whose weights are learned from a downstream objective by a gradient-free optimiser, and whose single scalar uniformly controls encoding depth, forget risk, and retrieval rank. We make a methodological point: on LongMemEval, scoring goal relevance against the held-out evaluation question saturates gold-evidence retention at \approx 0.98 -- this measures retrieval, not forgetting. In the realistic blind regime, a learned multi-factor value retains 0.770 \pm 0.011 of gold evidence across 479 usable cases, versus 0.657 for uniform weights, 0.518 for the best single factor, and 0.368 for recency; every paired gap's 95% bootstrap CI is above zero, and a neural network over the same factors ties the linear model. The learned weights are interpretable -- reliability, emotional intensity, and self/user relevance dominate, while query-time goal similarity is correctly down-weighted for the forgetting decision. A controlled synthetic task with planted confounds confirms the learner recovers a separating weighting (1.00 retention) where uniform weighting fails (0.62). The substrate is open-source; all experiments run on a single CPU with no API calls.

• The paper proposes a learned linear value function using seven cognitive factors—emotion, goal relevance, value alignment, self-relevance, task utility, reliability, and usage history—to control encoding, forgetting, and retrieval.
• The paper shows that honest blind evaluation matters: on LongMemEval-S, the multi-factor model retained 0.770 ± 0.011 of gold evidence versus 0.286 for blind goal-similarity and 0.368 for recency.
• The paper finds that interpretable linear scoring matches a neural ablation, with reliability and emotional intensity receiving the largest learned weights, while cautioning that retention is only a proxy for downstream question-answering quality.

The paper "Learning What to Remember: A Cognitively Grounded Multi-Factor Value Model for Agentic Memory" (2606.12945) addresses a structural deficiency in how LLM agents manage persistent memory: the decision of what to retain, forget, and retrieve is typically governed by a single factor — semantic similarity to the current query or recency — even though the forgetting decision is necessarily made at consolidation time, before the future query exists. The authors propose a learned, seven-factor memory value function grounded in cognitive psychology, demonstrate that a widely used evaluation practice conflates retrieval with retention, and show that the learned multi-factor value substantially outperforms single-factor baselines once the forgetting decision is evaluated honestly.

The mis-specification argument

The core premise is that production memory stacks (RAG-style similarity ranking (2606.12945), MemGPT's recency-driven eviction, MemoryBank's hand-set Ebbinghaus decay) answer the memory-triage question with a signal that is either temporally mis-specified or informationally impoverished. Similarity-to-query is undefined at consolidation time; recency is query-agnostic but structurally discards old-but-durable facts in favor of recent low-value content. Neither encodes an estimate of a memory's expected future usefulness.

Human memory research supplies the alternative: value-directed remembering (Castel), levels-of-processing and the self-reference effect (Craik & Lockhart; Rogers et al.), emotional modulation of consolidation (McGaugh), and the rational analysis of memory, in which the forgetting curve tracks environmental need-probability (Anderson & Schooler). The paper operationalizes these determinants as seven interpretable factors — emotional intensity, goal relevance, value alignment, self/user relevance, task utility, reliability, and usage history — combined in a learned linear value

$V(m) = \sum_{i=1}^{7} w_i\, f_i(m).$

Linearity is defended on three grounds: capacity match to sparse indirect supervision, auditability (the weights are the explanation), and the decision-theoretic reading of $V(m)$ as a first-order additive-utility approximation. A neural ablation over the same factors empirically confirms near-additive combination (see below).

One scalar, three operations, learned weights

A distinctive architectural claim is that a single scalar $V(m)$ uniformly controls all three memory operations: a four-tier encoding-depth mapping (shallow through meta), a multiplicative forget score combining age decay, usage damping, and value resistance $\frac{1}{1+\beta V(m)}$, and retrieval ranking. This replaces the separately hand-tuned rules of prior systems, including the fixed, LLM-rated importance term in Generative Agents, which governs retrieval only. Because the encode–forget–retrieve–answer pipeline is non-differentiable, the weights are fit by a gradient-free stochastic hill-climb over a seven-dimensional search space, maximizing a downstream return $R(w)$ — gold-evidence retention under a fixed keep budget in this work, with downstream QA accuracy as the general objective. The authors are explicit that the hill-climb is a lightweight stand-in for CMA-ES, and that the objective here is a proxy for task return.

The oracle/blind methodological contrast

The paper's sharpest contribution is methodological. On LongMemEval-S, gold evidence is defined relative to the evaluation question. If the goal-relevance factor is computed as similarity between a stored turn and the held-out question — an oracle unavailable at consolidation time — similarity alone retains $0.979$ of gold, and no multi-factor advantage is possible. This measures retrieval, not forgetting. In the realistic blind regime, where goal relevance is computed only against the ongoing session's user-turn centroid, goal-only similarity collapses to $0.286$ (the $0.30$ chance floor) — a $0.69$-point swing that a careless query-defined metric would entirely hide. The paper argues, plausibly, that any retention metric built on query-defined gold must report both regimes, and that this observation applies beyond this benchmark.

Headline results

The central experiment annotates all turns of the $479$ usable LongMemEval-S cases with four API-free factors (emotional intensity, goal relevance, self/user relevance, reliability), keeps the top $V(m)$0 by $V(m)$1, and measures gold-evidence retention over $V(m)$2 resampled $V(m)$3 splits:

Policy	Oracle	Blind
Learned multi-factor	$V(m)$4	$V(m)$5
Uniform weights	$V(m)$6	$V(m)$7
Self-relevance only	$V(m)$8	$V(m)$9
Reliability only	$V(m)$0	$V(m)$1
Recency only	—	$V(m)$2
Random (30% keep)	—	$V(m)$3

Per-case bootstrap ($V(m)$4) places every paired gap's 95% CI above zero ($V(m)$5 vs. uniform, $V(m)$6 vs. recency, $V(m)$7 vs. reliability-only), and the learned policy wins on all $V(m)$8 splits. The advantage persists across keep fractions $V(m)$9 and closes only at the near-saturated $\frac{1}{1+\beta V(m)}$0.

Two secondary results strengthen the claim. First, a small MLP trained with a pairwise ranking loss over the same factors retains $\frac{1}{1+\beta V(m)}$1 — a statistical tie ($\frac{1}{1+\beta V(m)}$2) — so the interpretable linear value is not a capacity compromise on these factors. Second, the learned weights are diagnostic: reliability ($\frac{1}{1+\beta V(m)}$3), emotional intensity ($\frac{1}{1+\beta V(m)}$4), and self/user relevance ($\frac{1}{1+\beta V(m)}$5) dominate, while blind goal relevance is driven to exactly $\frac{1}{1+\beta V(m)}$6 — the optimizer independently discovered, from retention alone, that session-topic similarity is uninformative about the future needle. A controlled synthetic study with planted confounds confirms the learner recovers a separating weighting ($\frac{1}{1+\beta V(m)}$7 retention vs. $\frac{1}{1+\beta V(m)}$8 for uniform), though the authors correctly frame this as a sanity check rather than a central claim.

Limitations

The paper is unusually candid about scope. Retention is a proxy: no answerer or LLM judge is run, so the claim concerns what survives forgetting, not answer quality — high retention is necessary but not sufficient for QA correctness. Three of seven factors are inert: value alignment, task utility, and usage history require a value profile, an LLM judge, and access logs respectively, and are held at zero; notably, task utility is arguably the most direct expected-utility signal, so the headline result is a conservative four-factor estimate whose direction under the full factor set is not established. Single benchmark and modality: LongMemEval-S, English multi-session chat only; tool-use, coding, and navigation workloads are untested. The blind anchor is one choice: the session-turn centroid is a proxy for the consolidation-time goal, and better query-agnostic anchors could shift the goal-relevance term. Optimization is simple: a random-restart hill-climb rather than CMA-ES, though results are stable across splits and bootstrap intervals.

Conclusion

This paper makes two durable contributions: a methodological one — that query-defined gold retention metrics measure retrieval ceilings, not forgetting quality, and honest evaluation requires a blind regime reported alongside the oracle — and an empirical one, that a learned, cognitively grounded multi-factor value retains substantially more gold evidence than any single-factor policy in the realistic regime, with interpretable weights that double as a workload diagnostic. The immediate open question the paper itself poses is whether the advantage survives the full pipeline: fitting the weights to downstream QA accuracy (which would activate the task-utility factor) and measuring answer quality rather than retention, followed by transfer tests to tool-use and coding workloads.