TokenMizer: Graph-Structured Session Memory for Long-Horizon LLM Context Management

Abstract: LLM deployments for long-horizon tasks face a fundamental constraint: context windows are finite while productive work sessions are not. When history exceeds the Maximum Effective Context Window (MECW), critical structured information - architectural decisions, task transitions, file histories - is silently discarded. Existing mitigations treat history as flat text, destroying the relational structure that makes sessions resumable. We present TokenMizer, an open-source proxy system that models LLM session history as a typed knowledge graph. The schema defines 14 node types and 7 edge types. A hybrid extraction pipeline populates the graph incrementally, while a three-tier checkpoint system serializes it into compact resume blocks. An 8-layer compression pipeline reduces context overhead, and a semantic cache reduces repeated-query latency. Evaluated on a controlled benchmark of 21 sessions spanning 5 domains, TokenMizer demonstrates significant token economy. It produces resume blocks averaging 78 tokens (range: 42-124) - 2x smaller than evaluated baselines (159-170 tokens) - while achieving higher decision recall (+9-17 percentage points). Crucially, baselines only preserve that a technology was mentioned; TokenMizer preserves the rationale. Across all sessions, TokenMizer achieves mean task recall 51.0%, decision recall 46.6%, and file recall 58.7%. Variance reflects domain heterogeneity: explicit imperative phrasing (software engineering) scores higher than implicit reasoning (research). Ablation studies show fuzzy label matching is the dominant improvement factor (+33 pp task recall). The heuristic compression achieves 47.3% token reduction with zero external dependencies. TokenMizer provides a queryable alternative to text-retention baselines at half the token cost.

• The paper demonstrates that modeling session history as a typed knowledge graph increases decision recall by up to 17 percentage points over flat text approaches.
• It employs a hybrid extraction pipeline and three-tier checkpointing that compress session data by 47.3% and reduce context overhead by 2×.
• The system integrates semantic caching and HTTP reverse proxy deployment to support efficient, queryable session resumption for developer-centric workflows.

TokenMizer: Graph-Structured Session Memory for Long-Horizon LLM Context Management

Motivation and Problem Statement

TokenMizer addresses the core limitation inherent in current LLM deployments for long-horizon, developer-centric tasks: the finite context window. The Maximum Effective Context Window (MECW), as empirically delineated in prior work, is a practical constraint far below the nominal or MCW, with real-world degradation in multi-step task fidelity and persistent under-attention to middle-context content. Traditional context mitigation strategies—truncation, summarization, and retrieval—treat the session history as flat text, obliterating typed, relational, and status structure that is critical for session resumability and developer workflow continuity.

The paper models session history as a typed knowledge graph, enabling more efficient, structured, and queryable representations of session state and markedly improved recall of decision rationale and state transitions. The objective is not merely to maximize lexical recall but to maximize recovery of session-critical structural information—particularly architectural decision rationales—that is not attainable from flattened, text-retention paradigms.

System Design and Architecture

TokenMizer is implemented as a transparent HTTP reverse proxy (OpenAI-style API compatibility), requiring only an endpoint substitution for deployment. Session state is captured in a relational, typed knowledge graph with 14 carefully engineered node types spanning action, decision, and context categories and 7 directed, semantic edge types encoding dependency, implementation, and resolution relations.

Extraction Pipeline

The extraction framework is centered on a hybrid pipeline, with a high-throughput heuristic extractor (0.5 ms/per session avg latency) supporting LLM-based extraction for implicit and indirect phrasing as an upgrade path. Extraction is tightly coupled with a validation and scoring module to reduce noise—incorporating attributes such as an importance score, status tracking (monotonic progressions enforce consistency), and confidence gating. Fuzzy label matching is the dominant factor in recall improvement, contributing +33 percentage points in task recall over naïve, strict match protocols.

Checkpointing and Compression

A three-tier checkpointing system serializes the most salient nodes—ordered by importance—into budget-constrained resume blocks (mean: 78 tokens; range: 42–124), regularly outperforming baseline token budgets by a factor of two. The structured serialization maintains session continuity across context boundaries, supporting rapid recovery and session resumption.

The compression pipeline comprises eight layers, with the first six relying on lightweight, heuristic content compression (including de-duplication, filler removal, and normalization), and the final two optionally leveraging neural compressive scoring models (LLMLingua-2/LongLLMLingua). The pipeline achieves 47.3% mean token reduction prior to invoking any external model inference.

Semantic Caching

TokenMizer introduces a sentence-embedding-driven semantic cache for LLM response reuse. Embedding-level cache hits are achieved with a cosine similarity threshold of 0.92 (MiniLM-L6-v2 backbone), reporting a 70% hit rate on controlled, clustered synthetic workloads. This reduces inference costs and latency on repeated queries within the same semantic subspace.

Experimental Evaluation

Benchmarking and Metrics

All quantitative claims are grounded in a 21-session synthetic benchmark, spanning five domains: software engineering, data science, DevOps, research/writing, and debugging. Session annotation captures granular task, decision, and file-level entity ground truth; recall is measured via a robust fuzzy-matching protocol. The key metrics are mean recall (task/decision/file), token efficiency (normalized recall per 100 resume tokens), and information loss (mean unweighted recall deficit).

Numerical Results

TokenMizer achieves:

• Task recall: 51.0% (±32.7%)
• Decision recall: 46.6% (±32.0%)—+9–17 percentage points over all baselines
• File recall: 58.7% (±46.7%)
• Resume block: 78 tokens (±21.4) on average, 2× smaller than baselines
• Token efficiency: Debugging sessions, in particular, achieve η=0.91, the highest observed in the corpus
• Compression: Heuristic pipeline yields 47.3% mean token reduction with minimal computational cost

The strongest relative improvement is observed in decision recall—a byproduct of explicit rationale-typed nodes. No evaluated text-retention baseline preserves the causal and status structure of decision-making, only shallow mention frequency.

Ablation and Domain Analysis

Fuzzy label matching is the decisive ablation factor. Expanded pattern triggers or direct label splitting yield minor or ambiguous gains without fuzzy matching. Domain-level analysis demonstrates that sessions with explicit imperative phrasing—software engineering, data science—are systematically more recoverable than domains with implicit reasoning—research/writing, planning.

Correlation and Robustness

Session length is moderately anti-correlated with information loss ($r = -0.59$): longer sessions derive greater benefit from structured checkpointing, confirming the hypothesis that structural methods amplify in value as session length grows. Compression ratio is uncorrelated with session verbosity, indicating that information structure, not token count, governs context management efficiency.

Implications and Limitations

Practical and Theoretical Impacts

Representing LLM session memory as a structured graph rather than plain text unlocks principled, loss-bound, and queryable preservation of session state, enabling richer long-horizon workflows in developer tooling, DevOps, and research assistance. The approach minimizes context bloat, halves token costs, and maintains decision rationale continuity lost in baseline approaches. Additionally, the modular, proxy-based deployment ensures zero integration friction and makes this immediately actionable by production LLM consumers.

Limitations

This evaluation is subject to several caveats:

• The benchmark is synthetic, and all annotations are by the author; real-world and multi-annotator validation is essential for conclusive generalization
• Heuristic extraction and compression, while efficient, cannot fully capture implicit or non-imperative phrasing, leading to zero-recall outliers in research and planning domains—a limitation addressable via the LLM-based extraction path
• No statistical hypothesis testing due to limited sample size; observed differences provide descriptive trends only
• Edge-linking and cross-session memory mechanisms are only partially realized in the current release

Future Directions

Advancing TokenMizer entails extending LLM-based semantic extraction, implementing session-spanning graph queries (e.g., persistent architectural decision retrieval), developing domain-adaptive weighting of information loss metrics, and scaling evaluation to longitudinal, real-user datasets. There is substantial conceptual room for integrating embedding-based graph augmentation and exploring learned graph validation and compression strategies.

Conclusion

TokenMizer establishes graph-structured context memory as a superior paradigm for LLM session state management on long-horizon, developer-centric workflows. By leveraging fine-grained node typing, status tracking, and aggressive, non-inferential compression, it achieves both higher structural recall—especially for rationale and status– and a 2× reduction in resume context overhead compared to prevailing baselines. The approach enables more interpretable, efficient, and robust context management for future AI development agents, with future work demanding validation on live, heterogeneous corpora and integration of more powerful extraction and summarization pathways.