GitOfThoughts: Version-Controlled Reasoning and Agent Memory You Can Replay, Diff, and Merge

Abstract: LLM reasoning is ephemeral: chains of thought vanish with the context window, pruned search branches leave no record, and memory buffers cannot be diffed, merged, or audited. Every other complex software process (code, infrastructure, data, experiments) is version-controlled; reasoning is not. We introduce GitOfThoughts, which stores an agent's reasoning tree as a git repository: every scored thought is a commit, scores are notes, outcomes are tags, and retrieval is "git log" over the agent's own history. This makes reasoning replayable, auditable, and mergeable across agents at near-zero engineering cost. We then ask the harder question: does memory, in any substrate, actually improve accuracy? Across five substrates (none, markdown, vector, graph, git), two benchmarks, two model scales, and pre-registered replications, the answer for novel problems is no. No memory format reliably helps, and a promising early result collapsed under its own pre-registered replication. Memory pays only above what we call the copyability threshold: when the retrieved case is a near-duplicate of the current problem (similarity >~ 0.8), accuracy jumps sharply; below it, nothing. The gain is answer retrieval, not method transfer: a 4.5x larger model doubles the near-duplicate payoff yet still cannot extract a transferable method from a worked example. The only general lever we find is test-time sampling. The case for git-as-substrate is therefore auditability, provenance, and mergeability at accuracy parity. We document a retracted result and a refuted hypothesis to model the evaluation standard we hold ourselves to.

• The paper introduces a Git-based reasoning system that stores scored agent thoughts as content-addressed commits, enabling replay, line-level diffs, branching, signed provenance, and cross-agent merging at roughly 15 ms per write.
• The paper finds no reliable accuracy improvement from markdown, vector, graph, or Git memory on novel GPQA and MATH-500 problems across two model families and samples up to 500 questions, while self-consistency delivers the only clear general gain at +3.4 percentage points.
• The paper identifies a copyability threshold near cosine similarity 0.8: near-duplicate examples improve accuracy by 12–13.5 points at 7B scale and up to 28.5 points at 32B, but non-copyable examples do not transfer solution methods.

Overview and motivation

GitOfThoughts (2606.14470) addresses two problems that are usually treated separately. The first is infrastructural: LLM reasoning traces are ephemeral. Chain-of-thought prompts expire with the context window, pruned tree-search branches leave no record, and agent memory buffers (Reflexion-style verbal RL stores, skill libraries, vector caches) expose no diff, merge, signed authorship, or third-party verifiability. The authors argue this ephemerality is structural rather than cosmetic: it blocks reproducibility ("what did the agent think at step 17?"), audit (detecting train–test leakage or gold-answer memorization), memory transfer across agents, and incident review of confident wrong answers.

The second problem is empirical: does cross-problem memory, in any substrate, actually improve accuracy on novel problems? The paper's answer is a robust null, with one sharply bounded exception — a "copyability threshold" above which memory pays off substantially but only as answer retrieval, not method transfer.

Design: reasoning as a versioned DAG

The core observation is that a reasoning tree shares every invariant git was designed for: immutable nodes, parent links, content addressing by hash, labelling, branching, and search. The mapping is direct:

Reasoning concept	git primitive
Scored reasoning node	commit
Node identity	SHA-1 hash
Refinement / parent edge	commit parent
Combined score	git notes
Validation outcome	git tag (success_*, failed_*)
Exploration path	branch
Retrieval	git log --grep, -S (pickaxe)
Cross-agent merge	git fetch + git merge
Reproducible artefact / signed trace	git bundle / git commit -S

Each problem is solved in its own ephemeral repository; a commit-on-score hook writes four files per node (thought, scores, trace, metadata), attaches the score as a note, and tags the outcome. A long-lived memory branch accumulates cross-problem insights; retrieval uses only stock git, ranked by a confidence-weighted score $$\rho = 0.7\,s_{\text{comb}} + 0.2\,\text{tag} + 0.1\,\text{recency}$$. The search machinery itself is deliberately conventional: depth-1 tree-of-thoughts with branching factor 4 over an inner ReAct loop (max 3 steps) with calculator/sympy/LP tools, scored as $s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$.

Against the natural objection that SQLite or an append-only JSONL log plus ripgrep could replicate this cheaply, the authors concede the JSONL strawman reproduces much of git's function, and identify git's residual differentiators precisely: tested three-way merge with conflict surfacing, signed authorship, bit-identical reproduction via content addressing, pack-file deduplication, and tooling (blame, bisect, hosting). These are operational arguments, and the paper treats them as such rather than claiming retrieval superiority.

Auditability properties fall out of the substrate without added engineering: any SHA reconstructs full state via checkout; git diff success_X failed_Y -- thought.md yields line-level diffs over actual reasoning text; pre-registrations are themselves commits, so the decision rules for the stronger-model replication were committed before the 32B model served its first token. The fairness audit on GPQA is performed with the framework's own primitives: zero hits for gold answers in any commit, zero verbatim gold letters in traces, fresh memory per question verified, and bit-identical re-runs.

The memory null

All experiments flow through a pluggable MemoryBackend interface with five implementations — none, markdown, vector (MiniLM embeddings in Chroma), graph (networkx with spread-activation retrieval), and git — holding the agent fixed while swapping only the substrate. Costs are small everywhere: git pays ~15 ms/write and ~48 ms/read, the same order as the vector index's read.

Three escalating experiments establish the null:

• Controlled transfer ($n{=}40$, exploratory): all backends ingest identical lessons; no substrate beats none on GPQA or MATH-500, and every backend lands below the no-memory control on MATH-500.
• Memory inside the agent: a promising $+15$pp git trend at $n{=}40$ was pre-registered and confirmed at $n{\approx}100$, where it collapsed to $+1.0$pp with the entire backend ranking reshuffled. This retraction is reported prominently and is the paper's central methodological lesson.
• MATH-500 at $n{=}500$ (Qwen2.5-7B-Instruct, leakage-audited corpus of 2,000 worked solutions): self-consistency is the only arm whose CI clears zero ($+3.4$pp); every memory substrate, static few-shot, subject-filtered retrieval, and answer-only/answer-free ablations sit within noise. No online learning curve emerges even as subject-relevant retrievals accumulate.

A cross-model replication on GPQA ($n{=}98$, second backbone) reproduces the null. The scope is stated carefully: two adjacent open-weight backbones from different families, up to $s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$0 — a robustness check, not a scaling law. ScienceWorld is excluded from the headline claim because the scaffolded 9B sits at ~12% absolute versus SwiftSage's 84.7%, making it a floored agent whose null says "could not demonstrate a benefit," not "memory does not help cross-episode."

The copyability threshold

Varying test↔memory similarity directly over 200 hard MATH seeds produces the paper's first positive memory effect and its sharpest boundary. Near-duplicate retrieval (cosine $s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$1) gives $s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$2 to $s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$3pp at 7B scale; same-subject or unrelated examples give nothing. The threshold $s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$4 explains every prior null, since cross-problem exemplars sit at cosine ~0.1–0.5.

Critically, a dedicated method-transfer arm — same method, different numbers, non-copyable answer (cosine 0.72) — is null at both scales. A pre-registered run on Qwen2.5-32B-Instruct replicates the transfer null almost exactly ($s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$5pp vs. the 7B's $s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$6pp) while amplifying the near-duplicate control to $s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$7 to $s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$8pp, reaching 86% where the 7B plateaued at 65%. Scale steepens the copyability step without unlocking abstraction. This connects to prior evidence that demonstrations act through task recognition and surface format rather than method content, and bounds when agent memory pays off: recurring workloads, not novel problems.

What moves accuracy, and what does not

Self-consistency is the single general lever ($s = 0.6\,s_{\text{local}} + 0.4\,s_{\text{cross}}$9pp* at $n{=}40$0). Direct tests of test-time architectures on GPQA found none reliably beat greedy CoT; self-consistency and revision can hurt by breaking already-correct answers, and verifier best-of-$n{=}40$1 gains $n{=}40$2pp (one problem) at $n{=}40$3 tokens — within noise. The selector/aggregator emerges as the bottleneck.

The system headline (47.0% vs. ReTreVal's 34.0% and vanilla's 33.0% on GPQA) is reported with its confound stated explicitly: wall-clock budgets differ across arms (60 s / 180 s / 600 s), and the gain decomposes into MCQ-aware expansion plus compute budget, not git and not memory. Of 53 misses, 46 are timeouts. Durable execution via Hatchet adds ~249 ms/node dispatch latency with results identical by construction — durability, not accuracy. A functional distributed merge demonstrated that injected contradictory lessons surface as genuine git conflicts while concatenation silently retains them, though the caveat is important: with content-hashed filenames merges are conflict-free by construction, so conflict surfacing requires a keyed layout encoding "same slot, different content."

Limitations and open questions

The paper is explicit about its boundaries. The headline system result lacks a compute-matched control. The null covers short distilled lessons and worked-example retrieval on two adjacent open-weight backbones; it does not cover frontier-class models, rich episodic memory (full reasoning traces rather than one-shot distilled lessons), or multi-agent settings. ScienceWorld remains inconclusive pending a non-floored agent. Wall-clock budgets use a single seed; the 32B runs AWQ int4 quantization (affecting absolute level, not paired deltas). Three pre-registered experiments follow: merged-memory accuracy against concatenation, a compute-matched system baseline, and independently varying frontier models and episodic richness to test whether either lowers $n{=}40$4. Whether merged multi-agent memory helps accuracy is stated up front as expected to be small or null given the copyability threshold.

Conclusion

GitOfThoughts makes every scored reasoning node a git commit, buying provenance, replay, line-level diff, merge, and reproducibility at near-zero cost (~15 ms/write). Its empirical contribution is equally deliberate: a pre-registered evaluation establishing that no memory substrate reliably improves accuracy on novel problems across two benchmarks, two backbones, and up to $n{=}40$5; that memory helps only above a copyability threshold ($n{=}40$6) as answer retrieval rather than method transfer, a boundary that survives a $n{=}40$7 scale-up; and that test-time sampling is the one general lever. The case for git-as-substrate rests on auditability at accuracy parity, and the open questions handed forward are precise: whether frontier models with rich episodic memory lower $n{=}40$8, and whether git's structurally unique merge operation delivers accuracy value where single-agent retrieval cannot.