LycheeMemory V2: Efficient Long-Term Memory for LLM Agents via Semantic Segment-Level Consolidation

Abstract: Long-horizon LLM agents must preserve information from past interactions to support future tasks. Existing memory systems typically rely on eager consolidation, invoking LLMs after each interaction to extract, summarize, or update memories. This design makes memory construction increasingly costly as conversations grow. Coarse summarization can reduce construction cost but risks discarding fine-grained contextual evidence, whereas larger retrieval contexts or multi-hop LLM reasoning shift the overhead to query time. We present LycheeMemory V2, an efficient long-term memory framework that replaces turn-level consolidation with semantic segment-level consolidation. Instead of consolidating every interaction, LycheeMemory batches multiple exchanges into segments and encodes each finalized segment into context-independent typed memory records. Segment-level batching lowers LLM encoding frequency, while semantic boundary detection helps preserve coherent event-level and temporal evidence compared with fixed-window batching. The resulting records are organized with lightweight structured indexes for query-planned evidence retrieval. Experiments using GPT-4.1-Mini show that LycheeMemory achieves state-of-the-art performance, reaching 89.22% on LoCoMo and 92.20% on LongMemEval-S. Compared with A-Mem, it reduces construction tokens by 86.0% on LoCoMo and 75.9% on LongMemEval-S without increasing query-time token usage. More broadly, our results suggest that the accuracy--cost trade-off of long-term agent memory depends not only on what information is retained, but also on the granularity at which it is consolidated.

• The paper introduces semantic segment-level consolidation, which batches coherent dialogue exchanges before encoding typed, self-contained records and reduces construction tokens by 86.0% on LoCoMo and 75.9% on LongMemEval-S versus A-Mem.
• LycheeMemory V2 combines embedding-based segmentation, structured evidence graphs, and plan-guided multi-route retrieval to achieve 89.22% accuracy on LoCoMo and 92.20% on LongMemEval-S with GPT-4.1-Mini.
• The ablations show that semantic boundaries preserve fine-grained temporal and multi-hop evidence, while structured retrieval and diversity-aware fusion materially improve accuracy without increasing query-time token usage.

LycheeMemory V2 addresses a specific inefficiency in long-term memory systems for LLM agents: the cost of eager, turn-level memory consolidation. The paper's central claim is that the accuracy–cost trade-off of agent memory depends not only on what information is retained but on the granularity at which consolidation is performed. By batching multiple dialogue exchanges into semantically coherent segments and invoking the LLM once per finalized segment rather than once per turn, the system achieves state-of-the-art accuracy on LoCoMo (89.22%) and LongMemEval-S (92.20%) with GPT-4.1-Mini, while reducing construction tokens by 86.0% and 75.9% respectively relative to A-Mem, without increasing query-time token usage (2608.12990).

Motivation and problem framing

The authors identify three failure modes in existing designs. Eager turn-level consolidation, used by Mem0 and A-Mem, invokes an LLM after each exchange to extract facts or update memories, making construction cost grow linearly with conversation length. Coarse summarization reduces write cost but discards fine-grained evidence—entities, temporal expressions, coreference relations—that long-term QA questions frequently require. Finally, compensating at query time through larger top-$k$ retrieval or iterative LLM-driven multi-hop search shifts overhead to inference. The key design question is therefore how to preserve evidence at the right granularity while controlling both construction and query budgets.

Method

LycheeMemory consists of four stages: online semantic segmentation, segment-level encoding, structured evidence organization, and plan-guided multi-route retrieval.

Online semantic segmentation. Each incoming exchange is embedded, and a boundary probability $p_t$ is computed from four signals: semantic surprise $s_t$ (one minus the maximum cosine similarity between the new exchange embedding and either the active segment centroid or the most recent exchange embedding), cohesion drop $d_t$ (the decrease in mean intra-segment cosine similarity caused by appending the exchange), token-length pressure, and turn-count pressure. A segment is finalized when $p_t$ exceeds a threshold $\delta = 0.50$ or a hard cap (900 tokens, 10 exchanges) is reached. Crucially, this boundary decision is entirely embedding-based; mid-segment exchanges are buffered without any generative LLM call, so the number of encoding calls scales with the number of segments rather than turns.

Segment-level memory encoding. Each finalized segment is encoded in a single LLM pass that simultaneously extracts atomic information units, resolves coreference and elliptical mentions, and normalizes relative dates using the session timestamp. Each record carries a type from a finite schema (fact, preference, event, constraint, procedure, failure pattern, tool affordance), a self-contained natural-language statement, entity and topic tags, normalized temporal scopes, and provenance links to source turns. To maintain continuity without re-reading full history, the encoder also emits a bounded disambiguation state (aliases, canonical names, reference relations) that is carried into the next segment's prompt under a fixed character budget.

Structured evidence organization. Records are stored in an append-only store with a vector index plus five classes of structured nodes: entity, topic, entity-topic, temporal (day- and month-level), and event-frame nodes grouping records from the same finalized segment. All nodes derive from encoder metadata, so indexing requires no additional LLM calls.

Plan-guided multi-route retrieval. A single LLM planning call decomposes the question into typed recall routes, each executing four non-generative channels: direct record recall, evidence-node recall with expansion, temporal filtering, and raw-turn recall. Candidates are fused via reciprocal-rank fusion ($\kappa = 60$), optionally reranked by bge-reranker-v2-m3, and passed through diversity-aware (MMR-style) selection. The only generative query-time operation is the planning call.

Main results

With GPT-4.1-Mini, LycheeMemory reaches 89.22% overall on LoCoMo (1,540 retained questions) versus 84.80% for Full Context and 83.77% for TiMem, the strongest baselines. On LongMemEval-S it reaches 92.20% versus 66.20% for Full Context and 75.80% for TiMem. With GPT-4o-Mini it attains 78.90% and 78.80%, exceeding the strongest baselines by 2.47 and 1.00 percentage points.

Category-level gains over A-Mem are largest precisely where fine-grained evidence matters: multi-hop (+27.3 pp) and open-domain (+25.0 pp) on LoCoMo; temporal reasoning (+34.59 pp), preference tracking (+26.67 pp), and multi-session reasoning (+26.32 pp) on LongMemEval-S. Notably, LycheeMemory outperforms Full Context on LoCoMo even though Full Context sees every token—an indication that write-time disambiguation and structured retrieval add value beyond raw context availability. One counterpoint the authors report candidly: MemoryOS reaches 100.00% on LongMemEval-S preference tracking versus LycheeMemory's 90.00%, suggesting dedicated user-profile modules retain an advantage for preference-intensive queries.

Accuracy–cost trade-off

The efficiency results are the paper's strongest quantitative contribution:

Benchmark	LycheeMemory	A-Mem	Reduction
LoCoMo construction tokens	204.1K	1459.9K	−86.0%
LongMemEval-S construction tokens	304.7K	1264.3K	−75.9%
LoCoMo query tokens	4.01K	5.56K	−27.9%
LongMemEval-S query tokens	8.88K	15.46K	−42.6%

Construction savings stem directly from batching: segments average 5.8 turns on LoCoMo, so encoding calls scale with $|\mathcal{S}|$ rather than $T$. Because query tokens also fall relative to A-Mem and TiMem, the accuracy gains cannot be attributed to query-time context expansion—the single planning call plus deterministic recall keeps generative query cost low. Token accounting excludes embeddings, indexing, reranking, fusion, and judge calls, and separates write-side from read-side cost, which makes the trade-off comparison interpretable.

Ablations

The ablations cleanly separate two effects. Batching drives cost reduction: replacing semantic segmentation with eager construction raises construction tokens from 204.1K to 849.9K (+316%) and drops overall accuracy to 81.88%. Semantic boundaries drive accuracy: fixed-window batching uses slightly fewer tokens (174.7K) but loses 6.8 pp overall, with the largest losses on multi-hop and open-domain questions—evidence that mechanically split windows fragment referentially coherent evidence. On the representation side, summary-level records cut construction tokens to 99.7K but lose 8.4 pp overall (−12.8 pp on temporal reasoning), supporting typed self-contained records over coarse summaries; removing cross-segment reference context costs 7.7 pp at nearly unchanged token cost. On the retrieval side, record-vector-only retrieval loses 7.5 pp, removing the planner loses 5.8 pp (while saving 1.75K query tokens, exposing a genuine planner-versus-cost trade-off), and jointly disabling fusion/reranking/diversity selection causes the single largest degradation: −22.6 pp. Boundary-threshold sensitivity is modest—overall accuracy varies only 1.04 pp as $\delta$ ranges from 0.30 to 0.70.

Limitations and open questions

The evaluation is confined to text-only conversational QA; multimodal memories, production latency, storage growth, privacy governance, and online user feedback are explicitly out of scope. The preference-tracking weakness noted above remains unresolved—a combination of segment-level evidence construction with dedicated profile modeling is proposed but not evaluated. All experiments use hosted APIs (GPT-4.1-Mini/GPT-4o-Mini, text-embedding-3-small); substituting open-source components would change both capability and token accounting, so the reported numbers do not transfer automatically. Additionally, the append-only store performs no supersession detection or validity-interval closing, deferring all conflict resolution to answer generation; whether this scales to histories with dense knowledge updates is untested. The boundary detector's hand-tuned pressure functions and threshold were validated only within the evaluated range on LoCoMo.

Conclusion

LycheeMemory V2 demonstrates that consolidation granularity is a first-order efficiency lever for long-term agent memory. Segment-level batching reduces write-side LLM calls by roughly an order of magnitude, semantic boundary detection preserves the event coherence that fixed windows destroy, and typed self-contained records with structured indexes enable accurate retrieval through a single generative planning call per query. The result is a Pareto improvement over representative baselines on both benchmarks—higher accuracy at lower construction and query cost—with ablations attributing the gains to specific design choices rather than tuning artifacts.