KAPSO: Knowledge-grounded Program Synthesis

• KAPSO is a modular framework for autonomous program synthesis and optimization that refines executable artifacts iteratively based on evaluator outcomes.
• It employs a git-native experimentation engine to preserve state, support debugging, and reuse domain expertise during long-horizon coding tasks.
• The framework integrates a structured knowledge system and cognitive memory layer to retrieve insights and avoid repeated mistakes across iterations.

Searching arXiv for KAPSO and closely related benchmark/system papers to ground the article and contextual comparisons. I’m checking the available research tools and arXiv records now. KAPSO, short for Knowledge-grounded framework for Autonomous Program Synthesis and Optimization, is a modular framework for autonomous program synthesis and optimization. Given a natural-language goal and an evaluation method, it iteratively performs ideation, code synthesis and editing, execution, evaluation, and learning to improve a runnable artifact toward measurable objectives. Its defining premise is that program synthesis is not the endpoint of the system; rather, synthesis functions as one operator inside a broader long-horizon optimization loop in which progress is defined by evaluator outcomes. The framework is designed to address long-horizon failures in coding agents, including lost experimental state, brittle debugging, and weak reuse of domain expertise, through a git-native experimentation engine, a structured knowledge system, and a cognitive memory layer (Nadaf et al., 29 Jan 2026).

1. Problem setting and conceptual scope

KAPSO is introduced for settings in which successful autonomous programming requires repeated cycles of proposing ideas, editing code, running the artifact, measuring outcomes, debugging failures, and learning from prior attempts. The framework targets tasks where improvement can be measured through executable outcomes, including ML competitions, heuristic optimization, algorithm engineering, and benchmarked software-building tasks (Nadaf et al., 29 Jan 2026).

A central distinction in the KAPSO formulation is between code generation as a one-shot act and autonomous optimization over executable artifacts. The framework is explicitly built around an evaluator-driven regime in which a candidate repository state is executed, judged, and then revised. This makes the evaluator contract structurally primary: the system improves only insofar as evaluator outcomes improve. In that sense, KAPSO reframes autonomous programming as optimization over runnable artifacts rather than synthesis of static code (Nadaf et al., 29 Jan 2026).

The paper also defines KAPSO in terms of long-horizon reliability. The motivating failure modes include lost state across iterations, repeated integration errors, failure to reuse prior knowledge, brittle debugging, and weak management of environment contracts. KAPSO’s architecture is therefore organized to preserve provenance, expose execution traces, and make both external knowledge and internally accumulated lessons retrievable during subsequent iterations (Nadaf et al., 29 Jan 2026).

2. Architectural organization: knowledge plane, execution plane, and git-native experimentation

KAPSO separates its system into a knowledge plane and an execution plane. The knowledge plane stores and retrieves reusable information from repositories, internal playbooks, documentation, scientific papers, web-derived notes, benchmark artifacts, and prior experiment traces. The execution plane is where code is run and judged, with an explicit evaluator boundary defined by a ProblemHandler for execution and an optional Evaluator for measurement. The evaluator may be fully automated, stochastic, preference-based, or LLM-assisted (Nadaf et al., 29 Jan 2026).

The framework’s first core component is the git-native experimentation engine. Each experiment is represented as a git branch in an experiment workspace. A branch can contain code diffs, commit identifiers, evaluator configuration, rollout count if stochastic evaluation is used, logs, structured diagnostics, and output files or traces. The workflow is: start from a parent branch, create a new branch for the candidate attempt, apply code edits, run the evaluator, commit artifacts and outputs, and push the branch back into the local experiment workspace so it can become a parent for future attempts. This design is intended to solve lost state across iterations, untraceable edits, brittle debugging, difficulty reusing successful intermediate states, and unclear provenance of results (Nadaf et al., 29 Jan 2026).

The engine is evaluator-agnostic. Default execution is local via subprocess in the generic handler, while some evaluators require containers; the ALE evaluation uses Docker through the benchmark harness. The paper states that the engine can support remote backends later, which suggests an intentionally decoupled runtime substrate rather than a benchmark-specific implementation (Nadaf et al., 29 Jan 2026).

This architecture makes the experiment branch, rather than the prompt or the model completion, the canonical unit of optimization. A plausible implication is that KAPSO is designed to make software state first-class in the control loop, which is materially different from agent frameworks that center only on conversational context.

3. Knowledge grounding and cognitive memory

The second core component is the knowledge system, which converts heterogeneous sources into a structured, typed, queryable base. In the reference implementation, knowledge is stored in MediaWiki for human review and curation and indexed for machine retrieval using a graph backend and a vector backend. The released knowledge base is populated from over 2,000 widely used data and ML repositories. The four primary page types are Principle, Implementation, Environment, and Heuristic, connected by typed edges such as IMPLEMENTED_BY, USES_HEURISTIC, REQUIRES_ENV, and cross-references (Nadaf et al., 29 Jan 2026).

The paper formalizes knowledge access as a two-stage retrieval process. First, repository retrieval optionally selects a seed repository:

$$\mathrm{RepoRetrieve}(g) \rightarrow (\hat{r}, \rho)$$

where $\hat{r}$ is a candidate seed repository and $\rho \in [0,1]$ is a confidence score. Initialization then follows:

$$c_{\mathrm{init}} := \begin{cases} \mathrm{RepoState}(\hat{r}) & \text{if } \rho \ge \tau, \ \mathrm{Scaffold}(g) & \text{otherwise.} \end{cases}$$

Second, typed knowledge is retrieved conditioned on the goal $g$, the optional seed repository $\hat{r}$, and an optional failure signal $s$:

$$\mathrm{RetrieveKnowledge}(g, \hat{r}, s) \rightarrow K$$

The framework distinguishes KR as base knowledge retrieval and ERA as error-recovery augmentation:

$K_{\mathrm{base}} = \mathrm{KR}(g,\hat{r})$

$$K := \begin{cases} \mathrm{ERA}(g, s, \hat{r}, K_{\mathrm{base}}) & \text{if } s \neq \emptyset, \ K_{\mathrm{base}} & \text{otherwise.} \end{cases}$$

ERA attaches failure-conditioned heuristics, diagnostics, and alternative implementations. The returned knowledge packet includes the optional seed repository, retrieved pages grouped by type, confidence scores, source pages, query used, and recovery attachments. The reference implementation uses Neo4j for graph indexing and Weaviate for vector retrieval (Nadaf et al., 29 Jan 2026).

The third core component is the cognitive memory layer. It stores episodic memory entries derived from run logs, diffs, evaluator feedback, traces, and judge rationales. Each memory item includes a trigger description, a generalized lesson, a recommended action, and provenance back to the experiment branch and artifacts. After each experiment, failures are generalized into reusable fix patterns, successes yield best-practice lessons, relevant memories are retrieved, and those memories are inserted into the next context. Memory is stored in a vector database, defaulting to Weaviate, with a JSON fallback (Nadaf et al., 29 Jan 2026).

The memory controller decides whether to Retry, Pivot, or Complete:

$\hat{r}$0

where the controller state $\hat{r}$1 includes the goal, current knowledge packet, latest experiment, retrieved episodic insights, and meta statistics such as repeated failures. If pivoting occurs, knowledge retrieval is re-run excluding the current workflow. This gives KAPSO a mechanism for converting experiment history into policy-relevant state, rather than treating prior runs as passive logs (Nadaf et al., 29 Jan 2026).

4. Orchestration, evaluator contracts, and optimization loop

The top-level loop is implemented by an OrchestratorAgent composed of four pluggable subsystems: SearchStrategy, ContextManager, KnowledgeSearch, and CodingAgent. SearchStrategy proposes candidate solution specifications and chooses what to evaluate next; ContextManager assembles the task description, constraints, retrieved knowledge, episodic memory, and experiment history; KnowledgeSearch retrieves workflows, implementations, heuristics, and environment constraints; CodingAgent applies code edits and debugging patches (Nadaf et al., 29 Jan 2026).

The solve loop begins from

$\hat{r}$2

and, for each iteration, computes budget progress $\hat{r}$3, stops if the budget is exhausted or a stop condition is met, renders context $\hat{r}$4, runs the search strategy, collects new experiments, and appends them to history. The paper summarizes the operational logic as building context from evaluation state, knowledge, memory, and history; generating candidate experiments; executing them in isolated branches; and keeping the best artifact under the evaluator rule (Nadaf et al., 29 Jan 2026).

A KAPSO run is defined by a natural-language goal $\hat{r}$5, a budget $\hat{r}$6, and an evaluator contract $\hat{r}$7. The evaluator contract defines execution via $\hat{r}$8, utility $\hat{r}$9 or a preference relation $\rho \in [0,1]$0, stochastic aggregation operators, and stopping rules. A code artifact $\rho \in [0,1]$1 is an executable repository state, and execution returns

$\rho \in [0,1]$2

where status is success or error, $\rho \in [0,1]$3 are metrics, $\rho \in [0,1]$4 is free-form feedback or diagnostics, and $\rho \in [0,1]$5 are auxiliary artifacts such as logs or traces (Nadaf et al., 29 Jan 2026).

For scalar utilities, the conceptual target is

$\rho \in [0,1]$6

which the paper emphasizes is an expectation over full records, not just scalar scores. With $\rho \in [0,1]$7 rollouts, the aggregation operators are

$\rho \in [0,1]$8

and

$\rho \in [0,1]$9

often with $$c_{\mathrm{init}} := \begin{cases} \mathrm{RepoState}(\hat{r}) & \text{if } \rho \ge \tau, \ \mathrm{Scaffold}(g) & \text{otherwise.} \end{cases}$$0 as the sample mean. Errors are treated as infeasible artifacts, and selection rules ensure feasible outputs dominate infeasible ones, for example by assigning $$c_{\mathrm{init}} := \begin{cases} \mathrm{RepoState}(\hat{r}) & \text{if } \rho \ge \tau, \ \mathrm{Scaffold}(g) & \text{otherwise.} \end{cases}$$1 utility or preferring feasibility in $$c_{\mathrm{init}} := \begin{cases} \mathrm{RepoState}(\hat{r}) & \text{if } \rho \ge \tau, \ \mathrm{Scaffold}(g) & \text{otherwise.} \end{cases}$$2 (Nadaf et al., 29 Jan 2026).

At branch level, KAPSO uses an implement-and-debug loop in which it creates an experiment session, implements the candidate, runs it, optionally debugs it up to a fixed budget, and finalizes the session. The debug loop is restricted to execution errors, contract violations, missing files, and wrong output format; it is not the mechanism for higher-level objective optimization. One instantiation of the search layer uses LLM-steered tree search, pruning leaf nodes, expanding with LLM-generated child specifications, and selecting top-$$c_{\mathrm{init}} := \begin{cases} \mathrm{RepoState}(\hat{r}) & \text{if } \rho \ge \tau, \ \mathrm{Scaffold}(g) & \text{otherwise.} \end{cases}$$3 leaf nodes for execution (Nadaf et al., 29 Jan 2026).

5. Runtime interface, benchmark regimes, and empirical performance

KAPSO includes a unified deployment interface through deploy(), which returns a Software handle with run(inputs) and lifecycle methods such as start, stop, logs, and is_healthy. The supported strategy options are LOCAL, DOCKER, MODAL, BENTOML, and LANGGRAPH. Under LOCAL, the default is import-and-call through main.predict; DOCKER exposes a containerized HTTP endpoint; MODAL uses a remote Modal function; BENTOML uses a BentoML service or BentoCloud; and LANGGRAPH targets LangGraph Platform deployment. The design goal is that the caller sees the same run() contract even though the runtime backend differs (Nadaf et al., 29 Jan 2026).

The framework is evaluated on two benchmarks representing different agentic software regimes: MLE-Bench, a Kaggle-style machine learning engineering benchmark, and ALE-Bench, a benchmark for AtCoder heuristic optimization contests (Nadaf et al., 29 Jan 2026).

Benchmark	Evaluation protocol	Reported KAPSO results
MLE-Bench	python main.py --debug, submission validation, python main.py, grading on public dataset	Medal rate: Low 68.18%, Medium 44.74%, Hard 40.00%, All 50.67%
ALE-Bench	Compile and run in Docker, average public scores, private evaluation for best-public-score solution	Final performance 1909.4, rank percentile 6.1%, total cost \$914.8

For MLE-Bench, the task is to produce a Python solution that trains on provided competition data and writes final_submission.csv in the expected format. The stop condition is 24 hours, \$200 budget, or early stopping if any medal is achieved. The reported metrics are Private score and Medal rate, with medal rate defined as the fraction of competitions where any medal is achieved. The reported KAPSO medal rates are 68.18% for Low, 44.74% for Medium, 40.00% for Hard, and 50.67% for All. The comparison systems listed are R&D-Agent with 68.18 / 21.05 / 22.22 / 35.11, AIRA-dojo with 55.00 / 21.97 / 21.67 / 31.60, ML-Master with 48.48 / 20.18 / 24.44 / 29.33, and AIDE with 35.91 / 8.45 / 11.67 / 17.12. The paper’s main finding is that the Leeroo/KAPSO system is competitive on easier tasks and much stronger on medium and hard tasks (Nadaf et al., 29 Jan 2026).

For ALE-Bench, the task is to produce a C++23 solution in main.cpp that optimizes a contest-defined score under a strict time limit. The reported metrics are Final performance, defined as final ELO rating; Rank percentile (RP), given by

$$c_{\mathrm{init}} := \begin{cases} \mathrm{RepoState}(\hat{r}) & \text{if } \rho \ge \tau, \ \mathrm{Scaffold}(g) & \text{otherwise.} \end{cases}$$4

Private absolute score; and Cost, defined as cumulative LLM usage cost. Aggregated results report Leeroo/KAPSO at 1909.4 final performance, 6.1% rank percentile, and \$c_{\mathrm{init}} := \begin{cases} \mathrm{RepoState}(\hat{r}) & \text{if } \rho \ge \tau, \ \mathrm{Scaffold}(g) & \text{otherwise.} \end{cases}$$51003.3</strong>. Additional baselines are <strong>ALE Sequential</strong> at <strong>1198 / 54.1$$c_{\mathrm{init}} := \begin{cases} \mathrm{RepoState}(\hat{r}) & \text{if } \rho \ge \tau, \ \mathrm{Scaffold}(g) & \text{otherwise.} \end{cases}$64.7. Per-competition observations include ahc016: 2022 vs 1457, ahc026: 2040 vs 1965, ahc027: 1839 vs 1740, and ahc046: 2194 vs 2153. The paper states that KAPSO is not uniformly better on every contest, but is generally stronger and more cost-efficient (Nadaf et al., 29 Jan 2026).

6. Limitations, related positioning, and broader significance

The paper identifies several limitations and caveats. Stochastic benchmark noise is explicitly noted for ALE-Bench because the number of competitions is relatively small, so performance may vary across tasks and seeds. Evaluation dependence is structural: if the evaluator is weak, noisy, or poorly aligned with the true objective, the optimization loop inherits those limitations. Retrieval quality matters, particularly repository selection quality, retrieval relevance, and whether typed knowledge captures useful workflow constraints. Debugging is bounded: the implement-and-debug loop repairs failures only within a fixed budget and does not guarantee recovery from all synthesis or execution issues (Nadaf et al., 29 Jan 2026).

The paper does not present a detailed ablation suite in the excerpt, but it emphasizes several design insights. Git-native provenance is presented as helping reproducibility and reuse. Typed knowledge retrieval is presented as better than unstructured retrieval for surfacing workflow-relevant information. Episodic memory is presented as helping prevent repeated mistakes. Evaluator-grounded optimization is presented as more practical than treating code generation as a one-shot synthesis problem. Modularity is presented as necessary because coding agents, evaluators, and knowledge sources vary across domains (Nadaf et al., 29 Jan 2026).

KAPSO is positioned relative to systems such as R&D-Agent, AIRA-dojo, ML-Master, AIDE, ALE-Agent, MLE-STAR, and InternAgent. Its stated contribution is not merely a new agent policy, but a framework organized around evaluator-grounded search, git-based experiment management, structured knowledge grounding, episodic memory from traces, and modular pluggable strategy, evaluator, and backend design. Within that framing, KAPSO emphasizes the full lifecycle of autonomous optimization—search, implementation, execution, evaluation, learning, and provenance—rather than code generation or debugging in isolation (Nadaf et al., 29 Jan 2026).

The practical significance claimed for KAPSO is that it provides a systematic engineering loop for autonomous coding systems in environments where progress must be measured, failures are frequent, environment constraints matter, previous runs should be reusable, and performance improves over many iterations. This suggests a general program for autonomous software systems in which the central optimization object is the executable repository state, and the central control signal is evaluator-grounded empirical feedback rather than textual plausibility alone (Nadaf et al., 29 Jan 2026).