---
title: 'KAPSO: Knowledge-grounded Program Synthesis'
url: https://www.emergentmind.com/topics/kapso
type: topic
---

# KAPSO: Knowledge-grounded Program Synthesis

Searching arXiv for KAPSO and closely related benchmark/system papers to ground the article and contextual comparisons.
I’m checking the available research tools and arXiv records now.
KAPSO, short for **Knowledge-grounded framework for Autonomous Program Synthesis and Optimization**, is a modular framework for autonomous program synthesis and optimization. Given a natural-language goal and an evaluation method, it iteratively performs ideation, code synthesis and editing, execution, evaluation, and learning to improve a runnable artifact toward measurable objectives. Its defining premise is that program synthesis is not the endpoint of the system; rather, synthesis functions as one operator inside a broader long-horizon optimization loop in which progress is defined by evaluator outcomes. The framework is designed to address long-horizon failures in coding agents, including lost experimental state, brittle debugging, and weak reuse of domain expertise, through a git-native experimentation engine, a structured knowledge system, and a cognitive memory layer [2601.21526].

## 1. Problem setting and conceptual scope

KAPSO is introduced for settings in which successful autonomous programming requires repeated cycles of proposing ideas, editing code, running the artifact, measuring outcomes, debugging failures, and learning from prior attempts. The framework targets tasks where improvement can be measured through executable outcomes, including ML competitions, heuristic optimization, algorithm engineering, and benchmarked software-building tasks [2601.21526].

A central distinction in the KAPSO formulation is between code generation as a one-shot act and autonomous optimization over executable artifacts. The framework is explicitly built around an evaluator-driven regime in which a candidate repository state is executed, judged, and then revised. This makes the evaluator contract structurally primary: the system improves only insofar as evaluator outcomes improve. In that sense, KAPSO reframes autonomous programming as optimization over runnable artifacts rather than synthesis of static code [2601.21526].

The paper also defines KAPSO in terms of long-horizon reliability. The motivating failure modes include lost state across iterations, repeated integration errors, failure to reuse prior knowledge, brittle debugging, and weak management of environment contracts. KAPSO’s architecture is therefore organized to preserve provenance, expose execution traces, and make both external knowledge and internally accumulated lessons retrievable during subsequent iterations [2601.21526].

## 2. Architectural organization: knowledge plane, execution plane, and git-native experimentation

KAPSO separates its system into a **knowledge plane** and an **execution plane**. The knowledge plane stores and retrieves reusable information from repositories, internal playbooks, documentation, scientific papers, web-derived notes, benchmark artifacts, and prior experiment traces. The execution plane is where code is run and judged, with an explicit evaluator boundary defined by a **ProblemHandler** for execution and an optional **Evaluator** for measurement. The evaluator may be fully automated, stochastic, preference-based, or LLM-assisted [2601.21526].

The framework’s first core component is the **git-native experimentation engine**. Each experiment is represented as a git branch in an experiment workspace. A branch can contain code diffs, commit identifiers, evaluator configuration, rollout count if stochastic evaluation is used, logs, structured diagnostics, and output files or traces. The workflow is: start from a parent branch, create a new branch for the candidate attempt, apply code edits, run the evaluator, commit artifacts and outputs, and push the branch back into the local experiment workspace so it can become a parent for future attempts. This design is intended to solve lost state across iterations, untraceable edits, brittle debugging, difficulty reusing successful intermediate states, and unclear provenance of results [2601.21526].

The engine is evaluator-agnostic. Default execution is local via subprocess in the generic handler, while some evaluators require containers; the ALE evaluation uses Docker through the benchmark harness. The paper states that the engine can support remote backends later, which suggests an intentionally decoupled runtime substrate rather than a benchmark-specific implementation [2601.21526].

This architecture makes the experiment branch, rather than the prompt or the model completion, the canonical unit of optimization. A plausible implication is that KAPSO is designed to make software state first-class in the control loop, which is materially different from agent frameworks that center only on conversational context.

## 3. Knowledge grounding and cognitive memory

The second core component is the **knowledge system**, which converts heterogeneous sources into a structured, typed, queryable base. In the reference implementation, knowledge is stored in **MediaWiki** for human review and curation and indexed for machine retrieval using a graph backend and a vector backend. The released knowledge base is populated from **over 2,000 widely used data and ML repositories**. The four primary page types are **Principle**, **Implementation**, **Environment**, and **Heuristic**, connected by typed edges such as `IMPLEMENTED_BY`, `USES_HEURISTIC`, `REQUIRES_ENV`, and cross-references [2601.21526].

The paper formalizes knowledge access as a two-stage retrieval process. First, repository retrieval optionally selects a seed repository:
$$
\mathrm{RepoRetrieve}(g) \rightarrow (\hat{r}, \rho)
$$
where \(\hat{r}\) is a candidate seed repository and \(\rho \in [0,1]\) is a confidence score. Initialization then follows:
$$
c_{\mathrm{init}} := \begin{cases}
\mathrm{RepoState}(\hat{r}) & \text{if } \rho \ge \tau, \\
\mathrm{Scaffold}(g) & \text{otherwise.}
\end{cases}
$$
Second, typed knowledge is retrieved conditioned on the goal \(g\), the optional seed repository \(\hat{r}\), and an optional failure signal \(s\):
$$
\mathrm{RetrieveKnowledge}(g, \hat{r}, s) \rightarrow K
$$
The framework distinguishes **KR** as base knowledge retrieval and **ERA** as error-recovery augmentation:
$$
K_{\mathrm{base}} = \mathrm{KR}(g,\hat{r})
$$
$$
K := \begin{cases}
\mathrm{ERA}(g, s, \hat{r}, K_{\mathrm{base}}) & \text{if } s \neq \emptyset, \\
K_{\mathrm{base}} & \text{otherwise.}
\end{cases}
$$
ERA attaches failure-conditioned heuristics, diagnostics, and alternative implementations. The returned knowledge packet includes the optional seed repository, retrieved pages grouped by type, confidence scores, source pages, query used, and recovery attachments. The reference implementation uses **Neo4j** for graph indexing and **Weaviate** for vector retrieval [2601.21526].

The third core component is the **cognitive memory layer**. It stores episodic memory entries derived from run logs, diffs, evaluator feedback, traces, and judge rationales. Each memory item includes a trigger description, a generalized lesson, a recommended action, and provenance back to the experiment branch and artifacts. After each experiment, failures are generalized into reusable fix patterns, successes yield best-practice lessons, relevant memories are retrieved, and those memories are inserted into the next context. Memory is stored in a vector database, defaulting to Weaviate, with a JSON fallback [2601.21526].

The memory controller decides whether to **Retry**, **Pivot**, or **Complete**:
$$
\pi(C_i) \in \{Retry, Pivot, Complete\}
$$
where the controller state \(C_i\) includes the goal, current knowledge packet, latest experiment, retrieved episodic insights, and meta statistics such as repeated failures. If pivoting occurs, knowledge retrieval is re-run excluding the current workflow. This gives KAPSO a mechanism for converting experiment history into policy-relevant state, rather than treating prior runs as passive logs [2601.21526].

## 4. Orchestration, evaluator contracts, and optimization loop

The top-level loop is implemented by an **OrchestratorAgent** composed of four pluggable subsystems: **SearchStrategy**, **ContextManager**, **KnowledgeSearch**, and **CodingAgent**. SearchStrategy proposes candidate solution specifications and chooses what to evaluate next; ContextManager assembles the task description, constraints, retrieved knowledge, episodic memory, and experiment history; KnowledgeSearch retrieves workflows, implementations, heuristics, and environment constraints; CodingAgent applies code edits and debugging patches [2601.21526].

The solve loop begins from
$$
H_0 = \emptyset
$$
and, for each iteration, computes budget progress \(\beta_i\), stops if the budget is exhausted or a stop condition is met, renders context \(x_i\), runs the search strategy, collects new experiments, and appends them to history. The paper summarizes the operational logic as building context from evaluation state, knowledge, memory, and history; generating candidate experiments; executing them in isolated branches; and keeping the best artifact under the evaluator rule [2601.21526].

A KAPSO run is defined by a natural-language goal \(g\), a budget \(B\), and an evaluator contract \(\mathcal{E}\). The evaluator contract defines execution via \(\mathrm{Run}(c;\theta)\), utility \(U\) or a preference relation \(\succ\), stochastic aggregation operators, and stopping rules. A code artifact \(c \in \mathcal{C}\) is an executable repository state, and execution returns
$$
R(c) = (\text{status}, M(c), F(c), A(c))
$$
where `status` is success or error, \(M(c)\) are metrics, \(F(c)\) is free-form feedback or diagnostics, and \(A(c)\) are auxiliary artifacts such as logs or traces [2601.21526].

For scalar utilities, the conceptual target is
$$
J^\star(c) := \mathbb{E}\!\left[U(R(c))\right]
$$
which the paper emphasizes is an expectation over full records, not just scalar scores. With \(K\) rollouts, the aggregation operators are
$$
\widehat{R}_{K}(c) := \mathrm{Agg}^R_K(R^{(1)}(c),\dots,R^{(K)}(c))
$$
and
$$
\widehat{J}_{K}(c) := \mathrm{Agg}^J_K(U(R^{(1)}(c)),\dots,U(R^{(K)}(c)))
$$
often with \(\widehat{J}_K\) as the sample mean. Errors are treated as infeasible artifacts, and selection rules ensure feasible outputs dominate infeasible ones, for example by assigning \(-\infty\) utility or preferring feasibility in \(\succ\) [2601.21526].

At branch level, KAPSO uses an implement-and-debug loop in which it creates an experiment session, implements the candidate, runs it, optionally debugs it up to a fixed budget, and finalizes the session. The debug loop is restricted to execution errors, contract violations, missing files, and wrong output format; it is not the mechanism for higher-level objective optimization. One instantiation of the search layer uses LLM-steered tree search, pruning leaf nodes, expanding with LLM-generated child specifications, and selecting top-\(k\) leaf nodes for execution [2601.21526].

## 5. Runtime interface, benchmark regimes, and empirical performance

KAPSO includes a unified deployment interface through `deploy()`, which returns a `Software` handle with `run(inputs)` and lifecycle methods such as `start`, `stop`, `logs`, and `is_healthy`. The supported strategy options are **LOCAL**, **DOCKER**, **MODAL**, **BENTOML**, and **LANGGRAPH**. Under **LOCAL**, the default is import-and-call through `main.predict`; **DOCKER** exposes a containerized HTTP endpoint; **MODAL** uses a remote Modal function; **BENTOML** uses a BentoML service or BentoCloud; and **LANGGRAPH** targets LangGraph Platform deployment. The design goal is that the caller sees the same `run()` contract even though the runtime backend differs [2601.21526].

The framework is evaluated on two benchmarks representing different agentic software regimes: **MLE-Bench**, a Kaggle-style machine learning engineering benchmark, and **ALE-Bench**, a benchmark for AtCoder heuristic optimization contests [2601.21526].

| Benchmark | Evaluation protocol | Reported KAPSO results |
|---|---|---|
| MLE-Bench | `python main.py --debug`, submission validation, `python main.py`, grading on public dataset | Medal rate: Low 68.18%, Medium 44.74%, Hard 40.00%, All 50.67% |
| ALE-Bench | Compile and run in Docker, average public scores, private evaluation for best-public-score solution | Final performance 1909.4, rank percentile 6.1%, total cost \$914.8 |

For **MLE-Bench**, the task is to produce a Python solution that trains on provided competition data and writes `final_submission.csv` in the expected format. The stop condition is **24 hours**, **\$200 budget**, or early stopping if any medal is achieved. The reported metrics are **Private score** and **Medal rate**, with medal rate defined as the fraction of competitions where any medal is achieved. The reported KAPSO medal rates are **68.18%** for Low, **44.74%** for Medium, **40.00%** for Hard, and **50.67%** for All. The comparison systems listed are **R&D-Agent** with **68.18 / 21.05 / 22.22 / 35.11**, **AIRA-dojo** with **55.00 / 21.97 / 21.67 / 31.60**, **ML-Master** with **48.48 / 20.18 / 24.44 / 29.33**, and **AIDE** with **35.91 / 8.45 / 11.67 / 17.12**. The paper’s main finding is that the Leeroo/KAPSO system is competitive on easier tasks and much stronger on medium and hard tasks [2601.21526].

For **ALE-Bench**, the task is to produce a **C++23** solution in `main.cpp` that optimizes a contest-defined score under a strict time limit. The reported metrics are **Final performance**, defined as final ELO rating; **Rank percentile (RP)**, given by
$$
\text{RP} = \frac{\text{final rank}}{\text{number of contestants}}
$$
**Private absolute score**; and **Cost**, defined as cumulative LLM usage cost. Aggregated results report **Leeroo/KAPSO** at **1909.4** final performance, **6.1%** rank percentile, and **\$914.8** total cost, compared with **ALE-Agent** at **1879.3**, **6.8%**, and **\$1003.3**. Additional baselines are **ALE Sequential** at **1198 / 54.1% / \$111.0** and **ALE one-shot** at **832 / 88.4% / \$4.7**. Per-competition observations include **ahc016: 2022 vs 1457**, **ahc026: 2040 vs 1965**, **ahc027: 1839 vs 1740**, and **ahc046: 2194 vs 2153**. The paper states that KAPSO is not uniformly better on every contest, but is generally stronger and more cost-efficient [2601.21526].

## 6. Limitations, related positioning, and broader significance

The paper identifies several limitations and caveats. **Stochastic benchmark noise** is explicitly noted for ALE-Bench because the number of competitions is relatively small, so performance may vary across tasks and seeds. **Evaluation dependence** is structural: if the evaluator is weak, noisy, or poorly aligned with the true objective, the optimization loop inherits those limitations. **Retrieval quality matters**, particularly repository selection quality, retrieval relevance, and whether typed knowledge captures useful workflow constraints. **Debugging is bounded**: the implement-and-debug loop repairs failures only within a fixed budget and does not guarantee recovery from all synthesis or execution issues [2601.21526].

The paper does not present a detailed ablation suite in the excerpt, but it emphasizes several design insights. **Git-native provenance** is presented as helping reproducibility and reuse. **Typed knowledge retrieval** is presented as better than unstructured retrieval for surfacing workflow-relevant information. **Episodic memory** is presented as helping prevent repeated mistakes. **Evaluator-grounded optimization** is presented as more practical than treating code generation as a one-shot synthesis problem. **Modularity** is presented as necessary because coding agents, evaluators, and knowledge sources vary across domains [2601.21526].

KAPSO is positioned relative to systems such as **R&D-Agent**, **AIRA-dojo**, **ML-Master**, **AIDE**, **ALE-Agent**, **MLE-STAR**, and **InternAgent**. Its stated contribution is not merely a new agent policy, but a framework organized around evaluator-grounded search, git-based experiment management, structured knowledge grounding, episodic memory from traces, and modular pluggable strategy, evaluator, and backend design. Within that framing, KAPSO emphasizes the full lifecycle of autonomous optimization—search, implementation, execution, evaluation, learning, and provenance—rather than code generation or debugging in isolation [2601.21526].

The practical significance claimed for KAPSO is that it provides a systematic engineering loop for autonomous coding systems in environments where progress must be measured, failures are frequent, environment constraints matter, previous runs should be reusable, and performance improves over many iterations. This suggests a general program for autonomous software systems in which the central optimization object is the executable repository state, and the central control signal is evaluator-grounded empirical feedback rather than textual plausibility alone [2601.21526].

Source: https://www.emergentmind.com/topics/kapso