OpenMementos: Efficient LLM Memory & Reasoning

• OpenMementos is a dual resource featuring a large-scale dataset for block-based chain-of-thought segmentation and a blueprint for symbolic note-taking with Prolog integration.
• The dataset pipeline uses iterative memento generation and dynamic programming segmentation to achieve up to 2–3× KV-cache reduction and enhanced inference efficiency.
• The system design employs LLM-driven plan decomposition and symbolic execution to enable accurate multi-hop QA while maintaining lower memory overhead.

OpenMementos encompasses two major, independently developed resources: (1) a large-scale, publicly available dataset designed for training LLMs to self-manage reasoning context through block segmentation and dense intermediate summaries (Kontonis et al., 10 Apr 2026), and (2) a blueprint for an open-source system architecture bridging LLM reasoning and symbolic Prolog execution for multi-hop question answering (Wan et al., 25 Jun 2025). Both share the core conceptual goal of advancing explicit, memory- and context-efficient intermediate state management in LLM-driven workflows, but address different modalities—one via supervised block-level annotation and the other via symbolic planning and database construction.

1. Objectives and Core Motivation

OpenMementos, in the context of dataset design (Kontonis et al., 10 Apr 2026), is created to enable LLMs to partition long chain-of-thought (CoT) traces into semantically coherent "thinking blocks" and to summarize each block into a compact "memento." This supports models that attend exclusively to the succession of mementos rather than the full raw reasoning history, with the following overarching goals:

• Train models to explicitly break and structure their own reasoning streams.
• Achieve 2–3× reduction in KV-cache footprint during inference, yielding lower memory consumption and increased token throughput.
• Support self-management of context in LLMs tackling extended or agentive tasks.
• Maintain strong accuracy across mathematical, scientific, and code reasoning benchmarks.

As a system design (Wan et al., 25 Jun 2025), OpenMementos refers to an open-source framework that embodies the Memento prompting and symbolic-execution paradigm for multi-hop question answering. The approach decomposes complex queries into symbolic sub-queries (Prolog predicates), populates a fact database incrementally via LLM-driven extraction/validation, and relies on symbolic execution for efficient and deterministic answer computation.

2. Dataset Schema, Corpus, and Construction Pipeline

Dataset Specifications

The OpenMementos dataset (Kontonis et al., 10 Apr 2026) consists of 228,000 fully segmented and summarized CoT traces, mined from OpenThoughts-v3 (outputs of QwQ-32B), with the following domain distribution:

• 54% mathematics
• 19% coding
• 27% science

Each record in the dataset corresponds to a single problem trace and includes:

Field Name	Type	Description
problem_id	string	Unique identifier for the problem
domain	enum (math/code/science)	Problem domain
prompt	string	Original input prompt
blocks	array of objects	Sequence of K reasoning blocks; each block has structured metadata as follows:

Each block contains:

• block_index (int)
• block_text (string): raw CoT fragment
• block_token_count, block_char_count (int)
• summary_text (string): memento (dense state summary)
• summary_token_count, summary_char_count (int)
• compression_ratio (float): summary_char_count / block_char_count

A minimal JSON sample:

{ "problem_id": "AIME24_P2", "domain": "math", "prompt": "Find the sum of all bases b > 9 such that 17_b divides 97_b.", "blocks": [ { "block_index": 0, "block_text": "Convert 17_b = b+7; 97_b = 9b+7; require (b+7)|(9b+7)…", "block_token_count": 45, "block_char_count": 120, "summary_text": "Need (b+7)|(9b+7) ⇒ (b+7)|56.", "summary_token_count": 9, "summary_char_count": 30, "compression_ratio": 0.25 }, ... ] }

Annotation Pipeline

The dataset is produced through a four-stage hybrid pipeline:

1. Seed Selection: 228K traces selected from OpenThoughts-v3.
2. Sentence Splitting: Code/math blocks are protected; sentences split on terminal punctuation with fragments merged by heuristic rules (e.g., fragments <5 tokens are attached to their neighbors), resulting in ≈187 candidate sentences per trace.
3. Boundary Scoring: Each candidate boundary between sentences is scored by a GPT-5.x model from 0 to 3, with scoring guided by syntactic and semantic rules (e.g., avoid boundaries mid-derivation).
4. Segmentation: Optimal block boundaries are chosen to maximize mean boundary strength minus block-size non-uniformity, subject to block token-count ≥200, using dynamic programming or beam search.
5. Iterative Memento Generation: Each block is summarized by a GPT-5.x "STATE-COMPRESSOR." Mementos are scored by a "JUDGE" on a six-criterion rubric (formulas, values, methods, validation, hallucination avoidance, result-first structure, τ≥8/10 threshold). If needed, feedback is given and the compressor re-tries (max 2 iterations); two-shot pass rate reaches 92%.

No human annotation or IAA metrics are employed: consistency and quality are enforced entirely by iterative LLM scoring and feedback.

3. Access, Format, and Licensing

OpenMementos is released via the HuggingFace Dataset hub under the MIT License. The dataset is available in both Apache Arrow and JSONL formats and can be accessed programmatically:

from datasets import load_dataset ds = load_dataset("microsoft/OpenMementos", split="train") for example in ds: prompt = example["prompt"] blocks = example["blocks"] # Use <|block_start|>…<|summary_end|> to assemble chat inputs

Default split is "train" (228K examples); there are no validation or test splits provided.

4. Statistical Profile and Representative Example

Aggregate corpus statistics:

Domain	Median Blocks/Trace	Median Block Size (chars)	Median Memento Size (chars)	Compression Ratio (chars)	Avg Block Tokens	Avg Memento Tokens	Overall Trace Compression
math	9	3800	509	0.16×	1150	194	6× (10.9K→1.85K tokens)
code	9	3200	603	0.18×
science	7	2300	578	0.23×

Example (AIME’24 P2):

• Prompt: "Find the sum of all integer bases b > 9 for which 17_b divides 97_b."
• Block 0 (raw): "Convert 17_b = b+7, 97_b = 9b+7; require (b+7)|(9b+7); rewrite 9b+7 = 9(b+7) – 56, so (b+7)|56."
  • Memento: "Need (b+7)|(9b+7) ⇒ (b+7)|56."
• Block 1 (raw): "List positive divisors of 56: ...; b+7 ∈ {28,56} ⇒ b ∈ {21,49}. Verify both satisfy divisibility."
  • Memento: "Valid bases: b=21, 49."
• Block 2 (raw): "Compute answer = 21 + 49 = 70."
  • Memento: "Sum of valid b = 21+49=70."

5. Usage in Model Training and Inference

A two-stage supervised fine-tuning (SFT) regimen on OpenMementos enables models (e.g., Qwen3, Phi-4, Olmo 3, 8B–32B parameters) to:

• Segment and summarize reasoning traces in a manner that preserves downstream accuracy.
• Reduce peak KV-cache memory by approximately 2.5× during inference.
• Achieve up to 1.75× throughput increase when deployed using custom inference modifications to the vLLM stack.
• Enable reinforcement learning (RL) for further accuracy improvements.

A key observation is that the information flow in trained models exploits a dual channel: explicit memento text and the corresponding KV cache state. Eliminating access to the KV state corresponding to a reasoning block causes an absolute accuracy drop of 15 pp on the AIME24 dataset, indicating non-redundant implicit retention of context.

6. Blueprint for Symbolic Note-Taking: OpenMementos as System Architecture

(Wan et al., 25 Jun 2025) details OpenMementos as the open-source instantiation of the Memento system—a pipeline that bridges LLMs and symbolic execution engines (e.g., SWI-Prolog) for multi-hop QA:

1. Plan Generation: Input question $Q$ is decomposed by an LLM into a Prolog query plan $(q_1, ..., q_T)$, with accompanying question and statement templates for each predicate.
2. Database Construction: For each plan step, the system:
   • Retrieves partial candidate bindings from the evolving database.
   • Extracts missing entities with LLM calls using question templates and in-context evidence, or verifies candidates (factual NLI).
   • Populates the Prolog note-taking buffer (database) with grounded facts.
3. Final Query Execution: The complete plan is executed symbolically over the database to yield the answer.

The note-taking buffer, implemented as a Prolog database, maintains only the distilled, necessary facts—enabling linear scaling with the number of reasoning steps and avoiding context overload.

Implementation stack:

• Python 3.10, HuggingFace Transformers or OpenAI API, BM25 retrievers, SWI-Prolog (e.g., via pyswip).
• Modular design: separate planner, executor, and Prolog interface modules.

Cost and performance: for $T\approx$ 5–10, an end-to-end QA requires ≈$1–2 USD/1k$ questions (API), with traceably low memory/storage overhead.

7. Limitations, Failure Modes, and Future Directions

Noted limitations of OpenMementos-style approaches (Wan et al., 25 Jun 2025, Kontonis et al., 10 Apr 2026):

• Reliance on LLMs for fact extraction/verification can introduce hallucinated or incorrect facts, polluting the note-taking buffer.
• Errors in plan generation or segmentation propagate, with no built-in backtracking or plan revision at present.
• Multiple LLM calls per question incur additional latency.
• For large $T$ or heavily branched queries, the number of LLM calls may become prohibitive without parallelization.

Proposed future extensions include:

• Plan verification passes and small Prolog checkers.
• Dynamic backtracking with plan revision or evidence expansion.
• Enhanced fact grounding (evidence highlighting, paragraph IDs).
• Hierarchical memory structures for persistent summaries.
• Hybrid execution with external symbolic databases.
• Parallelized extraction and execution when candidate set sizes are large.

A plausible implication is that future versions of OpenMementos may achieve even greater context efficiency and robustness in agentic LLM applications by integrating advanced backtracking, hybrid symbolic extraction, and persistent memory methods.

---

OpenMementos defines the state-of-the-art for explicit, scalable context management in LLM reasoning—both as a high-quality dataset for block-structured summarization (Kontonis et al., 10 Apr 2026) and as a blueprint for open-source, symbolically instrumented QA pipelines (Wan et al., 25 Jun 2025). Its dual instantiations continue to inform efficiency-driven research in LLM architecture, symbolic memory integration, and multi-step reasoning.