Klear-CodeTest: Scalable Code Testing

• Klear-CodeTest is a scalable, rigorously-verified framework for synthesizing test cases targeting both typical and edge case behaviors in LLM-based code reinforcement learning.
• It automates the process from data curation to secure sandbox execution, using dual LLM agents for generating and validating comprehensive test suites.
• Empirical evaluations show improved RL reward shaping with higher test precision and coverage, addressing ambiguous specifications and unreliable evaluation oracles.

Klear-CodeTest is a scalable, rigorously verified test-case synthesis framework designed to improve LLM training in the domain of code reinforcement learning. At its core is a Generator-Validation (G-V) methodology that produces comprehensive test suites—spanning both typical and pathological program behaviors—by leveraging gold-standard solutions and a multi-layered security sandbox. Klear-CodeTest aims to address fundamental obstacles in LLM code evaluation: ambiguous specifications, lack of reliable oracles, and the difficulty of obtaining high-coverage, discriminative test sets crucial for precise RL reward shaping (Fu et al., 7 Aug 2025).

1. Motivation and Challenges in Code RL

Modern LLM-based code generation, as demonstrated by systems like GitHub Copilot, OpenAI Codex, and Cursor, is inherently stochastic and subject to semantic defects that elude trivial compilation checks. In reinforcement learning (RL) fine-tuning for code, reward assignment relies on unit test outcomes, highlighting the criticality of exact, comprehensive test sets.

Two principal challenges drive the design of Klear-CodeTest:

• Ambiguity in problem specifications: Natural-language prompts rarely enumerate complete boundary conditions, omitting critical corner cases.
• The Oracle Problem: Accurate verification is infeasible absent a trusted gold solution, especially when implementations are unavailable.

Additionally, synthesizing both broad-coverage and non-redundant unit tests remains unsolved at scale. Poor test set quality introduces reward noise in RL, leading to slower convergence and suboptimal policy learning.

2. Pipeline Architecture and Data Flow

Klear-CodeTest implements a fully automated pipeline, encompassing problem curation, diverse test-case generation, robust validation, and secured execution. The process comprises five primary stages:

1. Data Curation: Aggregates problem statements from competitive programming sources (Codeforces, TACO-verified, CodeContests). Deduplication uses n-gram matching; only problems in STDIN format with at least two gold solutions are retained.
2. Generator Module: Dual LLM-based agents emit "generator programs" to cover both regular and corner cases—targeting typical behavior and edge conditions, respectively. Each generator outputs 80 regular and 20 corner candidate inputs per run.
3. Validation Module: Candidate inputs are executed via both gold solutions within a controlled sandbox; only inputs producing matching outputs (per exact or special-judge equivalence) are accepted.
4. Special Judge Construction: For outputs with nontrivial equivalence (floating-point tolerance, permutations), checker scripts are auto-generated, validated, and repaired using a two-stage LLM workflow.
5. Sandbox System (Judge): Ensures secure, efficient code execution across C/C++, Python2, and Python3 via multi-layered isolation.

High-Level G-V Loop Pseudocode

def generator_validation(problem, gold_solutions): T = set() for mode in ['regular', 'corner']: for iter in range(3): generator = LLM_prompt(problem, mode) try: inputs = sandbox_run(generator) except CompileOrRuntimeError as log: LLM_feedback(generator, log) continue break input_sets[mode] = inputs for i in input_sets['regular'] + input_sets['corner']: o1 = sandbox_run(gold_solutions[0], i) o2 = sandbox_run(gold_solutions[1], i) if o1 == o2 or (special_judge and special_judge(o1, o2)): T.add((i, o1)) return T

This is an editorially simplified summary of the pseudocode provided in the source data; the actual system uses specialized handling for sandbox calls, special-judge logic, and iterative LLM prompting.

3. Generator-Validation Methodology

Generator Design

Two separate prompts are provided to LLMs (e.g., GPT-4) to elicit generator code in Python:

• Regular generators target typical program domains and standard constraint satisfaction.
• Corner generators are constructed to focus on boundaries, pathological, and extreme edge cases.

Raw generator code is sandboxed to capture syntax and import failures, with a feedback loop to the LLM—up to three iterations—achieving approximately 80% first-pass success.

Validation and Acceptance

Every generated candidate input $i$ is executed against two independent gold solutions $S_1, S_2$ within user-defined constraints $(T_\text{max}, M_\text{max})$. The consistency criterion is

$\|S_1(i) - S_2(i)\| = 0$

where "output equality" may be further determined by a "special judge" script for ambiguous value formats. Correctness rates $S_\text{correctness} = |T| / |I|$ and branch/constraint coverage $C_\text{coverage}$ (fraction of unique behaviors) are measurable outcomes.

Automatic special judge generation is implemented for complex equivalence scenarios and validated via a two-stage LLM process, increasing checker success (334 → 426 of 700, Table 3 in (Fu et al., 7 Aug 2025)).

4. Security Sandbox: Judge System

Existing general-purpose sandboxing systems (e.g., Firejail) pose challenges regarding performance and permissiveness necessary for test generation at scale. Judge is a purpose-built, six-layer sandbox providing:

• System-call filtering by ptrace with precise whitelists—78 syscalls for C/C++, 312 for Python.
• Resource limits set via rlimit (CPU time, VM size, stack ≤ 256 MB, file size ≤ 64 MB).
• Privilege dropping using setuid/setgid to a restricted user (UID 1536).
• Network isolation via new network namespaces (unshare(CLONE_NEWNET)).
• Filesystem isolation (new FS namespace, RO root, writable workdir, RO /tmp).
• Process monitoring (illegal syscalls, memory spikes, signals trapped via ptrace).

Performance benchmarks demonstrate that, for 8,234 CodeContests problems, Judge processes 7,058 (vs. Firejail’s 6,851) in 902 seconds (vs. 1,006 seconds), reducing inference time by 44.6% at 100 tests/problem (Tables 4 and 5).

5. Dataset Characteristics and Implementation

• Problems: 27,965 unique competition-style tasks, each with at least two gold solutions.
• Test Cases: Mean of 86 per problem (80 regular + 20 corner, minus rejected cases).
• Languages Supported: C, C++, Python2, Python3.
• Open Access: Source code, dataset, and Judge system are hosted at https://github.com/Kwai-Klear/CodeTest.

Canonical Usage Example

• Install Judge sandbox (requires Docker, minimal root privileges).
• Generate tests:

  python generate_tests.py --problems data/problems.json

• Output: JSON files per problem, each listing $(\text{input}, \text{output})$ pairs.

6. Experimental Evaluation and RL Results

Test-Case Quality Metrics

Klear-CodeTest assesses test-case effectiveness using True Positive Rate (TPR, acceptance of correct solutions) and True Negative Rate (TNR, rejection of incorrect solutions). Baselines include public and CodeContests official test sets. Averaged over DeepSeek-Distilled-Qwen-7B, DeepSeek-V3-0324, and DeepSeek-R1, Klear-CodeTest achieves (TPR, TNR) = (91.4%, 87.8%) versus CodeContests' (89.1%, 84.3%) (Table 6).

RL with DAPO

• Model: Qwen3-4B ("thinking mode")
• Reward: +1/pass, -1/fail per test set
• Training: 3,000 problems × 16 test cases, public vs. synthetic comparison
• Evaluation: LiveCodeBench-v5 (Aug 2024–Feb 2025)
• Results: Pass@1 improves from 54.1% (pre-RL) to 57.3% (RL + public tests) and 59.1% (RL + CodeTest). Improvement is concentrated on medium (+2.6 pp) and hard (+1.5 pp) problems. Training curves are smoother, indicating lower reward noise and greater stability (Table 7, Figure 1).

7. Strengths, Limitations, and Future Directions

Strengths

• Automated pipeline encompassing curation, test synthesis, verification, and execution.
• Rigorously validated oracle: gold solutions replace unreliable LLM-generated evaluators.
• Multi-round LLM generator refinement accelerates high-coverage test discovery.
• Custom sandbox scales to tens of thousands of problems with reduced latency and increased throughput.

Limitations

• RL evaluation limited to one model and DAPO algorithm; broader generalization (PPO, actor-critic) is yet untested.
• Coverage currently based strictly on solution agreement; deeper structural metrics (e.g., branch/path coverage) remain future work.
• Some problem types (interactive, randomized) still need manual handling.

Prospects

• Integrating code-coverage instrumentation into the validation loop for further reward signal granularity.
• Expanding to interactive, multi-threaded, or adversarial test generation domains.
• Exploring end-to-end co-training of generator and checker agents to minimize manual prompt engineering.

Klear-CodeTest provides a scalable, rigorously verified pipeline for test-case synthesis, offering demonstrable improvements in LLM-based code RL by enhancing test coverage, correctness discrimination, and execution security (Fu et al., 7 Aug 2025).