---
title: 'SolidCoder: Execution-Grounded Code Generation'
url: https://www.emergentmind.com/topics/solidcoder
type: topic
---

# SolidCoder: Execution-Grounded Code Generation

Searching arXiv for SolidCoder and its comparison baseline to ground the article in the relevant literature.
SolidCoder is a code generation framework introduced in "SolidCoder: Bridging the Mental-Reality Gap in LLM Code Generation through Concrete Execution" [2604.19825]. It is organized around a simple principle—“don’t imagine — execute”—and is designed to address a failure mode in large language model code synthesis in which models internally simulate program behavior, hallucinate execution traces, and validate buggy code as correct. The framework characterizes this failure mode as the *Mental–Reality Gap*, decomposes it into the *Specification Gap* and the *Verification Gap*, and proposes a S.O.L.I.D. architecture that forces edge-case awareness before algorithm design and replaces imagined traces with sandboxed execution using property-based oracles [2604.19825].

## 1. Mental–Reality Gap and its two dimensions

The paper defines the *Mental–Reality Gap* as the disconnect between how a language model believes a program behaves and how it actually behaves at runtime [2604.19825]. In code generation systems that rely on internal reasoning, the model may produce a plausible internal trace and then mistake that trace for evidence of correctness. The paper attributes this to three sources: long traces compound state-tracking errors, natural-language descriptions under-specify invariants, and internal consistency checks reinforce imagined traces rather than reality [2604.19825].

The first dimension is the *Specification Gap*, described as edge-case blindness. In the paper’s semi-formal view, let $P$ be a natural-language problem specification and $f$ the synthesized program. The model’s internal plan $\pi$ may fail to include edge cases $E \subset X$ that exercise boundary conditions or corner patterns. The resulting program satisfies $P$ on typical inputs but fails on $E$ [2604.19825]. The paper illustrates this with a rotation problem in which a naive plan ignores $k=0$, $k \ge n$, $n=1$, or mixed types, so the code passes sample I/O but fails on inputs such as $x=[]$ or $k=n$ [2604.19825].

The second dimension is the *Verification Gap*, defined as hallucinated execution. Given code $f$ and a test $x$, mental simulation tries to infer $f(x)$ without running $f$. The model accumulates a hypothetical trace $\hat{\tau}$ and concludes correctness if $\hat{\tau}$ aligns with anticipated behavior. When $\hat{\tau} \ne \tau$, where $\tau$ is the real trace, the model validates incorrect behavior [2604.19825]. In the paper’s Figure 2 example, mental simulation declares “PASS” for a list-rotation implementation even though live execution produces `AssertionError: [3,1,2] != [2,3,1]` [2604.19825].

This decomposition is central because the two gaps are treated as orthogonal. The paper argues that improving specification alone does not eliminate hallucinated verification, and execution grounding alone does not ensure that the generated tests exercise the relevant edge cases [2604.19825]. A plausible implication is that robust code synthesis requires simultaneous control over pre-coding problem analysis and post-coding verification.

## 2. S.O.L.I.D. architecture

SolidCoder instantiates the principle “don’t imagine — execute” through five components that are assigned distinct roles [2604.19825]. The architecture moves edge-case reasoning ahead of algorithm design, uses property-based tests instead of imagined traces, and persists failing tests so that later repairs cannot regress [2604.19825].

| Component | Name | Function |
|---|---|---|
| S | Shift-left Planning | Elicit “killer edge cases” before plan generation |
| O | Oracle-based Assertions | Translate specifications and invariants into property-based tests |
| L | Live Execution | Execute code and tests in a sandbox; runtime is authoritative verdict |
| I | Intermediate Simulation | Cheap mental pre-filter after code generation |
| D | Defensive Accumulation | Persist failing tests and require future fixes to pass all of them |

*Shift-left Planning* closes the Specification Gap early by requiring edge-case enumeration before design begins [2604.19825]. The planning prompt asks: “Identify 3 potential edge cases that could break a naive solution: 1) Empty/minimal input, 2) Maximum constraint input, 3) Special pattern/boundary.” It then requires: “Write a detailed plan that handles these edge cases. The plan should be step-by-step and easy to implement” [2604.19825]. The plan must incorporate the enumerated edge cases, algorithmic approach, complexity notes, precision handling, and specific checks such as `k=0` and `n<2` [2604.19825].

*Oracle-based Assertions* address what the paper calls the Missing Oracle problem for novel tasks [2604.19825]. Rather than predicting exact outputs, SolidCoder translates specifications and domain invariants into property-based tests. The “Red Team” prompt instructs the model to identify a weak assumption—Type, Value, Structure, or Relationship—and produce a Python test script containing an `assert` that fails when the assumption is violated [2604.19825].

*Live Execution* is the authoritative verifier. The current implementation targets Python and runs code in a sandbox with a 5-second timeout, blocked `input()`, network blocking, isolated `exec_globals` per run, and builtins filtering [2604.19825]. Outcomes are classified as `PASS`, `FAIL_ASSERT`, or `FAIL_CRASH` [2604.19825].

*Intermediate Simulation* is retained, but only as an advisory pre-filter. The paper explicitly states that $I$ is not the final verdict; $L$ provides the final verdict [2604.19825]. This is a direct contrast with frameworks that rely on text-based verification.

*Defensive Accumulation* ensures that every failing test discovered during verification is retained in `accumulated_tests`, and every subsequent fix must pass all accumulated tests [2604.19825]. This prevents regression and turns the verification loop into an expanding set of behavioral constraints.

## 3. Formalization and oracle semantics

The formal core of the framework is the property-based oracle. Let $X$ be the input domain and $f: X \rightarrow Y$ the synthesized function. The paper defines a property-based oracle
$$
O: X \times Y \rightarrow \{0,1\},
$$
which evaluates whether the output $y$ for input $x$ satisfies required properties [2604.19825]. In many test wrappers,
$$
O(f, x) \coloneqq O(x, f(x)).
$$
If $p(x)$ is a test distribution over $X$, expected correctness is
$$
\mathbb{E}_{x \sim p}[O(f, x)].
$$
The paper emphasizes that in code, $O$ is implemented via `assert` statements over properties of $f(x)$, not exact outputs [2604.19825].

The evaluation metric is the standard pass@k estimator used broadly in code generation. If $n$ independent samples are generated for a task and $m$ are correct, the unbiased estimator of the probability that at least one out of $k$ sampled solutions is correct, without replacement, is
$$
\mathrm{pass@}k = 1 - \frac{\binom{n-m}{k}}{\binom{n}{k}}.
$$
For $k=1$, $\mathrm{pass@}1 = m/n$ [2604.19825]. In this work, pass@1 is measured as “first solution must pass all hidden tests” [2604.19825].

The paper also introduces an edge-case coverage concept for analysis. Let $C$ be a finite set of edge-case classes and $C_{\text{hit}} \subseteq C$ the classes addressed explicitly in the plan or covered by the test suite. Then
$$
\mathrm{cov} = \frac{|C_{\text{hit}}|}{|C|}.
$$
The paper does not report $\mathrm{cov}$ numerically, but states that the $S$ component is designed to maximize $C_{\text{hit}}$ ahead of coding [2604.19825].

This formalization is significant because it shifts correctness estimation away from internal textual plausibility and toward externally checkable properties. The paper’s framing suggests that even when exact reference outputs are unavailable, partial semantic constraints can still be operationalized as executable oracles.

## 4. Pipeline, oracle construction, and execution loop

The end-to-end control flow begins with planning iterations, proceeds through code generation and intermediate simulation, and then enters an assumption-breaking verification phase [2604.19825]. The paper gives the following high-level pseudocode structure: initialize `accumulated ← ∅`; for each planning iteration, identify edge cases, generate a plan, optionally refine the plan after advisory simulation, generate code, and optionally run `IntermediateSimulation`; then, for assumption-breaking rounds, generate an oracle test, have a Judge validate it as `VALID/INVALID`, execute the test live, accumulate failing tests, repair the code, and continue; if the candidate passes all sample I/O and accumulated tests, return it; otherwise enter a traditional debugging phase [2604.19825].

Oracle design is based on four classes of testable constraints [2604.19825]. *Functional properties* specify conditions independent of exact outputs, such as non-decreasing sorted output, length preservation, or permutation of input. *Invariants* encode structural relations such as idempotence or `rotate_left(x, n) = x`. *Equivalence to reference* is used when a simple trusted reference transform exists, such as Python slicing for rotation. *Metamorphic relations* assert predictable behavior under controlled transformations, for example:
- `rotate_left(x, k1 + k2) = rotate_left(rotate_left(x, k1), k2)`
- sorting preserves multiset and length [2604.19825]

The paper lists several test generation strategies: boundary cases, type/value anomalies such as `NaN`, `±inf`, duplicates, and negative values where allowed, structural properties such as permutation, stability, and idempotence, and metamorphic relations [2604.19825]. Shrinking or minimization is not implemented; tests are single-shot scripts [2604.19825].

False positives and false negatives are treated as a first-order risk of property-based verification. The paper notes that an oracle may hallucinate an incorrect property, so SolidCoder includes a Judge prompt to validate test cases for constraints, logical correctness, and fairness [2604.19825]. It also distinguishes `FAIL_ASSERT`, which indicates a violated property, from `FAIL_CRASH`, which indicates an exception; both outcomes are actionable but guide different fixes [2604.19825].

The concrete execution wrapper is integral to the pipeline. The implementation disables `input()`, blocks network access, isolates builtins and globals, and enforces a 5-second timeout per test, returning `PASS`, `FAIL_ASSERT`, or `FAIL_CRASH` [2604.19825]. This design makes runtime feedback the authoritative signal rather than a descriptive explanation produced by the model.

## 5. Empirical results, ablations, and comparison to mental simulation

The experimental evaluation uses GPT-4o (`gpt-4o-2024-08-06`) for direct comparison with prior work, and also evaluates GPT-OSS-120B and Grok-4.1-Fast [2604.19825]. The benchmark suite consists of HumanEval with 164 function-level problems with docstrings, CodeContests with 165 Codeforces problems and approximately 200 hidden tests per problem, and APPS with 150 problems and complex, ambiguous descriptions [2604.19825].

For GPT-4o, the reported pass@1 results are 95.7% on HumanEval, 77.0% on CodeContests, and 26.7% on APPS [2604.19825]. The paper reports these as improvements over CodeSIM of +0.6%p, +4.3%p, and +3.4%p respectively [2604.19825]. It also states that SolidCoder achieves state-of-the-art pass@1 performance with GPT-4o on these benchmarks [2604.19825].

The multi-model results indicate that the gains generalize to RL post-trained models [2604.19825]. On CodeContests, the paper reports +4.2%p for GPT-OSS-120B (87.9 → 92.1) and +3.0%p for Grok-4.1-Fast (95.2 → 98.2). On APPS, it reports +1.4%p for GPT-OSS-120B (39.3 → 40.7) and +0.7%p for Grok-4.1-Fast (41.3 → 42.0). HumanEval is described as saturating, with smaller or matched gains, including GPT-OSS-120B at 98.2% with CodeSIM and SolidCoder [2604.19825].

The ablation on GPT-4o over CodeContests is especially informative. Removing *Shift-left Planning* reduces performance from 77.0% to 53.3% (−23.7%p); removing *Intermediate Simulation* reduces it to 64.0% (−13.0%p); removing *Oracle-based Assertions* reduces it to 65.4% (−11.6%p); removing *Live Execution* reduces it to 69.1% (−7.9%p); removing *Defensive Accumulation* reduces it to 70.3% (−6.7%p) [2604.19825]. The paper’s stated takeaway is that edge-case awareness provides the largest individual gain, while execution grounding catches a different, complementary error category—hallucinated verification [2604.19825].

The comparison to mental-simulation baselines is explicit. CodeSIM is described as relying on mental simulation in planning and debugging, with text-based verification susceptible to hallucination [2604.19825]. The paper further reports that self-consistency-generated I/O led to −9.3%p performance, and interprets this as evidence of the oracle problem when exact outputs are predicted rather than checked through properties [2604.19825]. This suggests that execution-grounded verification is not merely a stronger checker, but a different validation regime.

## 6. Case study, limitations, and broader implications

The rotation example in the paper serves as a compact demonstration of the framework’s mechanism [2604.19825]. The simplified task is to implement `rotate_left(x, k)` for a list `x`. The illustrative buggy code normalizes `k` by `n` and then returns `x[-k:] + x[:-k]`, which rotates right instead of left [2604.19825]. Mental simulation traces the code as if it were a left rotation and concludes that the code is correct for `x=[1,2,3]`, `k=1` [2604.19825]. Under SolidCoder, *Shift-left Planning* explicitly elicits minimal input (`n=0`, `n=1`), `k=0`, `k \ge n`, negative `k`, and patterned data such as all-same or alternating values; the plan requires normalization modulo `n`, correct handling for `k=0`, handling for `n<2`, and preservation of length and permutation [2604.19825].

The corresponding oracle test checks length preservation, multiset preservation, and equality with the expected output `[2,3,1]` for the reference transform [2604.19825]. Live execution produces `FAIL_ASSERT` with the message `[3,1,2] != [2,3,1]` [2604.19825]. The failing test is then added to `accumulated_tests`, and the repaired code returns `x[k:] + x[:k]`, after which all accumulated tests pass [2604.19825]. The paper presents this as an example in which live, property-grounded verification prevents a hallucination-driven pass and drives targeted repair.

The paper also gives a complete passing example from HumanEval #0 `has_close_elements`, where the plan is to sort and check adjacent differences while handling empty and singleton inputs and considering floating-point precision [2604.19825]. A “Red Team” test using `NaN` and `inf` is validated by the Judge, live execution returns `PASS`, and no fix is required [2604.19825].

The framework’s limitations are also clearly stated. Oracle design can be difficult because some tasks lack strong property characterizations, and properties may be incomplete or wrong [2604.19825]. The Judge mitigates invalid tests, but oracle quality remains a constraint. The current implementation supports Python only, and extending to other languages requires secure sandboxing such as containers or microVMs together with syscall filtering [2604.19825]. The framework incurs sizable cost and latency overhead: for GPT-4o on HumanEval, tokens are approximately 7.5K versus CodeSIM 3.8K (+97%) with 8 API calls versus 4; for GPT-4o on CodeContests, tokens are approximately 31.9K versus 20.9K (+~50%) with 16 API calls versus 11, while pass@1 improves by +4.3%p [2604.19825].

The paper suggests difficulty-aware routing as a practical response to this overhead: lighter paths can be used on easy tasks, while the full S.O.L.I.D. loop is reserved for medium or hard tasks [2604.19825]. It also notes that the same model currently generates both code and properties, so model biases may propagate; heterogeneous model combinations are identified as a natural extension [2604.19825]. Repository-level tasks such as SWE-bench are outside the immediate scope because they introduce multi-file edits and dependency resolution [2604.19825].

The broader implication advanced by the paper is that bridging both dimensions of the Mental–Reality Gap is essential [2604.19825]. Specification improvements alone do not prevent hallucinated verification, and execution grounding alone will often test only the “happy path” unless edge cases are deliberately surfaced [2604.19825]. In that sense, SolidCoder is not only a benchmarked framework for function-level code synthesis, but also a proposal about where the bottleneck in future LLM coding systems lies: as generation quality improves, robust self-evaluation and executable verification become increasingly central [2604.19825].

Source: https://www.emergentmind.com/topics/solidcoder