SolidCoder: Execution-Grounded Code Generation

• SolidCoder is a code generation framework that replaces imagined execution with live, sandboxed testing to close the mental–reality gap.
• It employs a S.O.L.I.D. architecture, integrating shift-left planning, oracle-based tests, live execution, intermediate simulation, and defensive accumulation.
• The framework improves reliability by emphasizing edge-case awareness and executing property-based verifications, leading to state-of-the-art pass@1 performance.

Searching arXiv for SolidCoder and its comparison baseline to ground the article in the relevant literature. SolidCoder is a code generation framework introduced in "SolidCoder: Bridging the Mental-Reality Gap in LLM Code Generation through Concrete Execution" (Lee et al., 20 Apr 2026). It is organized around a simple principle—“don’t imagine — execute”—and is designed to address a failure mode in LLM code synthesis in which models internally simulate program behavior, hallucinate execution traces, and validate buggy code as correct. The framework characterizes this failure mode as the Mental–Reality Gap, decomposes it into the Specification Gap and the Verification Gap, and proposes a S.O.L.I.D. architecture that forces edge-case awareness before algorithm design and replaces imagined traces with sandboxed execution using property-based oracles (Lee et al., 20 Apr 2026).

1. Mental–Reality Gap and its two dimensions

The paper defines the Mental–Reality Gap as the disconnect between how a LLM believes a program behaves and how it actually behaves at runtime (Lee et al., 20 Apr 2026). In code generation systems that rely on internal reasoning, the model may produce a plausible internal trace and then mistake that trace for evidence of correctness. The paper attributes this to three sources: long traces compound state-tracking errors, natural-language descriptions under-specify invariants, and internal consistency checks reinforce imagined traces rather than reality (Lee et al., 20 Apr 2026).

The first dimension is the Specification Gap, described as edge-case blindness. In the paper’s semi-formal view, let $P$ be a natural-language problem specification and $f$ the synthesized program. The model’s internal plan $\pi$ may fail to include edge cases $E \subset X$ that exercise boundary conditions or corner patterns. The resulting program satisfies $P$ on typical inputs but fails on $E$ (Lee et al., 20 Apr 2026). The paper illustrates this with a rotation problem in which a naive plan ignores $k=0$, $k \ge n$, $n=1$, or mixed types, so the code passes sample I/O but fails on inputs such as $x=[]$ or $f$0 (Lee et al., 20 Apr 2026).

The second dimension is the Verification Gap, defined as hallucinated execution. Given code $f$1 and a test $f$2, mental simulation tries to infer $f$3 without running $f$4. The model accumulates a hypothetical trace $f$5 and concludes correctness if $f$6 aligns with anticipated behavior. When $f$7, where $f$8 is the real trace, the model validates incorrect behavior (Lee et al., 20 Apr 2026). In the paper’s Figure 1 example, mental simulation declares “PASS” for a list-rotation implementation even though live execution produces AssertionError: [3,1,2] != [2,3,1] (Lee et al., 20 Apr 2026).

This decomposition is central because the two gaps are treated as orthogonal. The paper argues that improving specification alone does not eliminate hallucinated verification, and execution grounding alone does not ensure that the generated tests exercise the relevant edge cases (Lee et al., 20 Apr 2026). A plausible implication is that robust code synthesis requires simultaneous control over pre-coding problem analysis and post-coding verification.

2. S.O.L.I.D. architecture

SolidCoder instantiates the principle “don’t imagine — execute” through five components that are assigned distinct roles (Lee et al., 20 Apr 2026). The architecture moves edge-case reasoning ahead of algorithm design, uses property-based tests instead of imagined traces, and persists failing tests so that later repairs cannot regress (Lee et al., 20 Apr 2026).

Component	Name	Function
S	Shift-left Planning	Elicit “killer edge cases” before plan generation
O	Oracle-based Assertions	Translate specifications and invariants into property-based tests
L	Live Execution	Execute code and tests in a sandbox; runtime is authoritative verdict
I	Intermediate Simulation	Cheap mental pre-filter after code generation
D	Defensive Accumulation	Persist failing tests and require future fixes to pass all of them

Shift-left Planning closes the Specification Gap early by requiring edge-case enumeration before design begins (Lee et al., 20 Apr 2026). The planning prompt asks: “Identify 3 potential edge cases that could break a naive solution: 1) Empty/minimal input, 2) Maximum constraint input, 3) Special pattern/boundary.” It then requires: “Write a detailed plan that handles these edge cases. The plan should be step-by-step and easy to implement” (Lee et al., 20 Apr 2026). The plan must incorporate the enumerated edge cases, algorithmic approach, complexity notes, precision handling, and specific checks such as k=0 and n\<2 (Lee et al., 20 Apr 2026).

Oracle-based Assertions address what the paper calls the Missing Oracle problem for novel tasks (Lee et al., 20 Apr 2026). Rather than predicting exact outputs, SolidCoder translates specifications and domain invariants into property-based tests. The “Red Team” prompt instructs the model to identify a weak assumption—Type, Value, Structure, or Relationship—and produce a Python test script containing an assert that fails when the assumption is violated (Lee et al., 20 Apr 2026).

Live Execution is the authoritative verifier. The current implementation targets Python and runs code in a sandbox with a 5-second timeout, blocked input(), network blocking, isolated exec_globals per run, and builtins filtering (Lee et al., 20 Apr 2026). Outcomes are classified as PASS, FAIL_ASSERT, or FAIL_CRASH (Lee et al., 20 Apr 2026).

Intermediate Simulation is retained, but only as an advisory pre-filter. The paper explicitly states that $f$9 is not the final verdict; $\pi$0 provides the final verdict (Lee et al., 20 Apr 2026). This is a direct contrast with frameworks that rely on text-based verification.

Defensive Accumulation ensures that every failing test discovered during verification is retained in accumulated_tests, and every subsequent fix must pass all accumulated tests (Lee et al., 20 Apr 2026). This prevents regression and turns the verification loop into an expanding set of behavioral constraints.

3. Formalization and oracle semantics

The formal core of the framework is the property-based oracle. Let $\pi$1 be the input domain and $\pi$2 the synthesized function. The paper defines a property-based oracle

$\pi$3

which evaluates whether the output $\pi$4 for input $\pi$5 satisfies required properties (Lee et al., 20 Apr 2026). In many test wrappers,

$\pi$6

If $\pi$7 is a test distribution over $\pi$8, expected correctness is

$\pi$9

The paper emphasizes that in code, $E \subset X$0 is implemented via assert statements over properties of $E \subset X$1, not exact outputs (Lee et al., 20 Apr 2026).

The evaluation metric is the standard pass@k estimator used broadly in code generation. If $E \subset X$2 independent samples are generated for a task and $E \subset X$3 are correct, the unbiased estimator of the probability that at least one out of $E \subset X$4 sampled solutions is correct, without replacement, is

$E \subset X$5

For $E \subset X$6, $E \subset X$7 (Lee et al., 20 Apr 2026). In this work, pass@1 is measured as “first solution must pass all hidden tests” (Lee et al., 20 Apr 2026).

The paper also introduces an edge-case coverage concept for analysis. Let $E \subset X$8 be a finite set of edge-case classes and $E \subset X$9 the classes addressed explicitly in the plan or covered by the test suite. Then

$P$0

The paper does not report $P$1 numerically, but states that the $P$2 component is designed to maximize $P$3 ahead of coding (Lee et al., 20 Apr 2026).

This formalization is significant because it shifts correctness estimation away from internal textual plausibility and toward externally checkable properties. The paper’s framing suggests that even when exact reference outputs are unavailable, partial semantic constraints can still be operationalized as executable oracles.

4. Pipeline, oracle construction, and execution loop

The end-to-end control flow begins with planning iterations, proceeds through code generation and intermediate simulation, and then enters an assumption-breaking verification phase (Lee et al., 20 Apr 2026). The paper gives the following high-level pseudocode structure: initialize accumulated ← ∅; for each planning iteration, identify edge cases, generate a plan, optionally refine the plan after advisory simulation, generate code, and optionally run IntermediateSimulation; then, for assumption-breaking rounds, generate an oracle test, have a Judge validate it as VALID/INVALID, execute the test live, accumulate failing tests, repair the code, and continue; if the candidate passes all sample I/O and accumulated tests, return it; otherwise enter a traditional debugging phase (Lee et al., 20 Apr 2026).

Oracle design is based on four classes of testable constraints (Lee et al., 20 Apr 2026). Functional properties specify conditions independent of exact outputs, such as non-decreasing sorted output, length preservation, or permutation of input. Invariants encode structural relations such as idempotence or rotate_left(x, n) = x. Equivalence to reference is used when a simple trusted reference transform exists, such as Python slicing for rotation. Metamorphic relations assert predictable behavior under controlled transformations, for example:

• rotate_left(x, k1 + k2) = rotate_left(rotate_left(x, k1), k2)
• sorting preserves multiset and length (Lee et al., 20 Apr 2026)

The paper lists several test generation strategies: boundary cases, type/value anomalies such as NaN, ±inf, duplicates, and negative values where allowed, structural properties such as permutation, stability, and idempotence, and metamorphic relations (Lee et al., 20 Apr 2026). Shrinking or minimization is not implemented; tests are single-shot scripts (Lee et al., 20 Apr 2026).

False positives and false negatives are treated as a first-order risk of property-based verification. The paper notes that an oracle may hallucinate an incorrect property, so SolidCoder includes a Judge prompt to validate test cases for constraints, logical correctness, and fairness (Lee et al., 20 Apr 2026). It also distinguishes FAIL_ASSERT, which indicates a violated property, from FAIL_CRASH, which indicates an exception; both outcomes are actionable but guide different fixes (Lee et al., 20 Apr 2026).

The concrete execution wrapper is integral to the pipeline. The implementation disables input(), blocks network access, isolates builtins and globals, and enforces a 5-second timeout per test, returning PASS, FAIL_ASSERT, or FAIL_CRASH (Lee et al., 20 Apr 2026). This design makes runtime feedback the authoritative signal rather than a descriptive explanation produced by the model.

5. Empirical results, ablations, and comparison to mental simulation

The experimental evaluation uses GPT-4o (gpt-4o-2024-08-06) for direct comparison with prior work, and also evaluates GPT-OSS-120B and Grok-4.1-Fast (Lee et al., 20 Apr 2026). The benchmark suite consists of HumanEval with 164 function-level problems with docstrings, CodeContests with 165 Codeforces problems and approximately 200 hidden tests per problem, and APPS with 150 problems and complex, ambiguous descriptions (Lee et al., 20 Apr 2026).

For GPT-4o, the reported pass@1 results are 95.7% on HumanEval, 77.0% on CodeContests, and 26.7% on APPS (Lee et al., 20 Apr 2026). The paper reports these as improvements over CodeSIM of +0.6%p, +4.3%p, and +3.4%p respectively (Lee et al., 20 Apr 2026). It also states that SolidCoder achieves state-of-the-art pass@1 performance with GPT-4o on these benchmarks (Lee et al., 20 Apr 2026).

The multi-model results indicate that the gains generalize to RL post-trained models (Lee et al., 20 Apr 2026). On CodeContests, the paper reports +4.2%p for GPT-OSS-120B (87.9 → 92.1) and +3.0%p for Grok-4.1-Fast (95.2 → 98.2). On APPS, it reports +1.4%p for GPT-OSS-120B (39.3 → 40.7) and +0.7%p for Grok-4.1-Fast (41.3 → 42.0). HumanEval is described as saturating, with smaller or matched gains, including GPT-OSS-120B at 98.2% with CodeSIM and SolidCoder (Lee et al., 20 Apr 2026).

The ablation on GPT-4o over CodeContests is especially informative. Removing Shift-left Planning reduces performance from 77.0% to 53.3% (−23.7%p); removing Intermediate Simulation reduces it to 64.0% (−13.0%p); removing Oracle-based Assertions reduces it to 65.4% (−11.6%p); removing Live Execution reduces it to 69.1% (−7.9%p); removing Defensive Accumulation reduces it to 70.3% (−6.7%p) (Lee et al., 20 Apr 2026). The paper’s stated takeaway is that edge-case awareness provides the largest individual gain, while execution grounding catches a different, complementary error category—hallucinated verification (Lee et al., 20 Apr 2026).

The comparison to mental-simulation baselines is explicit. CodeSIM is described as relying on mental simulation in planning and debugging, with text-based verification susceptible to hallucination (Lee et al., 20 Apr 2026). The paper further reports that self-consistency-generated I/O led to −9.3%p performance, and interprets this as evidence of the oracle problem when exact outputs are predicted rather than checked through properties (Lee et al., 20 Apr 2026). This suggests that execution-grounded verification is not merely a stronger checker, but a different validation regime.

6. Case study, limitations, and broader implications

The rotation example in the paper serves as a compact demonstration of the framework’s mechanism (Lee et al., 20 Apr 2026). The simplified task is to implement rotate_left(x, k) for a list x. The illustrative buggy code normalizes k by n and then returns x[-k:] + x[:-k], which rotates right instead of left (Lee et al., 20 Apr 2026). Mental simulation traces the code as if it were a left rotation and concludes that the code is correct for x=[1,2,3], k=1 (Lee et al., 20 Apr 2026). Under SolidCoder, Shift-left Planning explicitly elicits minimal input (n=0, n=1), k=0, k \ge n, negative k, and patterned data such as all-same or alternating values; the plan requires normalization modulo n, correct handling for k=0, handling for n\<2, and preservation of length and permutation (Lee et al., 20 Apr 2026).

The corresponding oracle test checks length preservation, multiset preservation, and equality with the expected output [2,3,1] for the reference transform (Lee et al., 20 Apr 2026). Live execution produces FAIL_ASSERT with the message [3,1,2] != [2,3,1] (Lee et al., 20 Apr 2026). The failing test is then added to accumulated_tests, and the repaired code returns x[k:] + x[:k], after which all accumulated tests pass (Lee et al., 20 Apr 2026). The paper presents this as an example in which live, property-grounded verification prevents a hallucination-driven pass and drives targeted repair.

The paper also gives a complete passing example from HumanEval #0 has_close_elements, where the plan is to sort and check adjacent differences while handling empty and singleton inputs and considering floating-point precision (Lee et al., 20 Apr 2026). A “Red Team” test using NaN and inf is validated by the Judge, live execution returns PASS, and no fix is required (Lee et al., 20 Apr 2026).

The framework’s limitations are also clearly stated. Oracle design can be difficult because some tasks lack strong property characterizations, and properties may be incomplete or wrong (Lee et al., 20 Apr 2026). The Judge mitigates invalid tests, but oracle quality remains a constraint. The current implementation supports Python only, and extending to other languages requires secure sandboxing such as containers or microVMs together with syscall filtering (Lee et al., 20 Apr 2026). The framework incurs sizable cost and latency overhead: for GPT-4o on HumanEval, tokens are approximately 7.5K versus CodeSIM 3.8K (+97%) with 8 API calls versus 4; for GPT-4o on CodeContests, tokens are approximately 31.9K versus 20.9K (+~50%) with 16 API calls versus 11, while pass@1 improves by +4.3%p (Lee et al., 20 Apr 2026).

The paper suggests difficulty-aware routing as a practical response to this overhead: lighter paths can be used on easy tasks, while the full S.O.L.I.D. loop is reserved for medium or hard tasks (Lee et al., 20 Apr 2026). It also notes that the same model currently generates both code and properties, so model biases may propagate; heterogeneous model combinations are identified as a natural extension (Lee et al., 20 Apr 2026). Repository-level tasks such as SWE-bench are outside the immediate scope because they introduce multi-file edits and dependency resolution (Lee et al., 20 Apr 2026).

The broader implication advanced by the paper is that bridging both dimensions of the Mental–Reality Gap is essential (Lee et al., 20 Apr 2026). Specification improvements alone do not prevent hallucinated verification, and execution grounding alone will often test only the “happy path” unless edge cases are deliberately surfaced (Lee et al., 20 Apr 2026). In that sense, SolidCoder is not only a benchmarked framework for function-level code synthesis, but also a proposal about where the bottleneck in future LLM coding systems lies: as generation quality improves, robust self-evaluation and executable verification become increasingly central (Lee et al., 20 Apr 2026).