OpenCodeReasoning-II Benchmark

• OpenCodeReasoning-II is a large-scale benchmark suite and dataset that evaluates advanced code reasoning, generation, and self-critique across Python and C++ challenges.
• It employs a two-stage supervised fine-tuning strategy, first optimizing code generation and then jointly training code with critique for enhanced solution selection.
• The framework uses test-time self-critique with parallel sampling to measure LLM performance on long-range code reasoning and competitive programming benchmarks.

OpenCodeReasoning-II is a large-scale benchmark suite and dataset, as well as an associated framework for evaluating and training LLMs on advanced code reasoning, code generation, and automated critique. It serves as a reference point for progress in test-time scaling, multi-programming-language coverage, critique-aided solution selection, and robust measurement of long-range code reasoning. The suite was introduced in "OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique" (Ahmad et al., 11 Jul 2025), which consolidates and extends prior efforts in competitive coding and program critique.

1. Dataset Composition and Construction

OpenCodeReasoning-II consists of approximately 2.5 million (question, solution, critique) triples, covering roughly 35,000 unique programming challenges drawn from sources such as TACO, APPS, CodeContests, and CodeForces. The dataset is balanced across Python (≈1.4M samples) and C++ (≈1.1M samples), with each triple including:

• A programming problem (with associated unit tests)
• A chain-of-thought (CoT) style solution
• A CoT-style critique with a binary (right/wrong) outcome
• For ~60% of samples, the empirical pass rate over 5–50 unit tests

The pipeline for dataset construction employs multiple LLMs (DeepSeek-R1 for solutions, QwQ-32B for critiques), rigorous deduplication, contamination filtering (cosine similarity < 0.7 against major benchmarks), and automated as well as human-verified filtering for label and content quality.

Source	Python #Questions	Python #Samples	C++ #Questions	C++ #Samples
CodeForces	15,641	834,523	11,887	355,180
...	...	...	...	...
Total	34,125	1,398,166	30,092	1,174,475

This scale nearly doubles the previously largest open code-reasoning benchmark (Ahmad et al., 11 Jul 2025).

2. Supervised Fine-Tuning Strategy

The supervised fine-tuning (SFT) procedure for OpenCodeReasoning-II is structured as a two-stage process:

Stage I: Code Generation SFT

• Objective: Minimize the negative log-likelihood for generating CoT-style solutions given questions.
• Training lasts three epochs, using AdamW, BF16 precision, and context windows up to 32k tokens.

Stage II: Joint Code + Critique SFT

• Objective: Extend each training sample to include (question, solution, critique), and jointly minimize loss for code and critique generation:

$$\mathcal{L}_{\mathrm{joint}}(\theta) = \mathcal{L}_{\mathrm{code}}(\theta) + \mathcal{L}_{\mathrm{critique}}(\theta)$$

• Trained for one epoch with settings matching Stage I.

The SFT is applied to Qwen2.5-Instruct models at 7B, 14B, and 32B parameters. This approach enables the resulting models to both generate solutions and self-critique, providing a foundation for test-time scaling methods (Ahmad et al., 11 Jul 2025).

3. Test-Time Self-Critique and Selection

At inference, OpenCodeReasoning-II leverages a test-time scaling technique: self-critique with parallel sampling. For each question, $k$ candidate solutions are generated. Each solution is critiqued by the model, yielding:

• A binary judgment ($y_j \in \{0,1\}$)
• The chain-of-thought (CoT) trace length ($\ell_j$)

The selection heuristic is:

1. Discard candidate solutions with $y_j = 0$ ("wrong").
2. Among remaining solutions with $y_j = 1$, select the one with the shortest critique trace (i.e., $\arg\min_{j : y_j=1} \ell_j$).

Formally, the scoring function is

$$\mathrm{score}(s_j) = \frac{y_j}{\ell_j}, \quad s^* = \arg\max_j \mathrm{score}(s_j)$$

The main metrics are:

• pass@k: Probability that at least one of $k$ samples passes all unit tests.
• pass@1|select@k: Probability that the final selected solution passes, following the self-critique selection.

Empirical results show significant gains in pass@1|select@k over naïve selection, particularly in Python; e.g., for OCR-2-32B, $pass@1 = 61.3\%$, $k$0, $k$1 for Python; with similar but lower gains for C++ (Ahmad et al., 11 Jul 2025).

4. Evaluation Framework and Benchmark Extension

OpenCodeReasoning-II evaluates models using competitive programming benchmarks, with a particular focus on LiveCodeBench. The benchmark is extended to support C++ by incorporating 279 C++ problems (175 from AtCoder, 104 from LeetCode) and adjusting harnesses for language compatibility. This dual-language structure enhances cross-linguistic evaluation and transfer.

Tested models include DeepSeek-R1, Qwen3-32B, and multiple configurations of OCR-2 (7B, 14B, 32B). Pass rates are state-of-the-art for open-weight models, with OCR-2-32B achieving 62.1% (Python) and 58.1% (C++) pass@1.

Critique accuracy (the ability of the model to judge its own outputs) reaches 47% (Python) and 40.5% (C++) for OCR-2-32B, highlighting the limitations of current self-critique modules in reliably filtering solutions, especially for medium and hard problems.

5. Methodology: Pipeline, Metrics, and Ablations

The entire OpenCodeReasoning-II data and training pipeline is modular and adheres to rigorous quality assurance:

• Deduplication and contamination removal via fuzzy matching and cosine similarity.
• Syntactic validation via Tree-Sitter.
• CoT prompts with explicit > and <judgment> tags.

  • Binary-only critique labeling for consistency, though recognized as a limitation.

  Ablation studies presented include:

  • Effect of increasing $k$2 in parallel sampling (pass@k saturates, pass@1|select@k plateaus due to critique error rate).

  • Model temperature sensitivity (negligible effect up to reasonable $k$3 values).
  • Language transfer (Python-trained models underperform in C++; dual-LLMs exhibit positive but asymmetric transfer).
  • Data scaling, showing continued improvement for moderate-sized models up to dataset limits (Ahmad et al., 11 Jul 2025).

  6. Limitations and Recommendations for Future Work

  Identified limitations include:

  • Critique labels are only binary, lacking partial credit or granular error localization.
  • The shortest-trace heuristic for solution selection is simplistic; more advanced ranking models are likely to yield greater gains.
  • Self-critique accuracy remains sub-50% for challenging tasks, restricting the efficiency of pass@1|select@k gains at larger $k$4.
  • The C++ extension in LiveCodeBench is modest in scale.

  Suggested directions for future work:

  • Development of finer-grained or multi-label critique models with confidence estimation.
  • Use of advanced fine-tuning strategies: contrastive, RL, or generative reward modeling.
  • Richer critique signals combining model judgment and preference data.
  • Scaling to additional programming languages (Java, JavaScript, etc.).
  • Dynamic sampling and adaptive critique selection loops (Ahmad et al., 11 Jul 2025).

  7. Context: Position Relative to Other Reasoning Benchmarks

  OpenCodeReasoning-II synthesizes and extends the methodology of earlier works. Unlike single-language or single-format datasets, it establishes comprehensive coverage in both Python and C++. In contrast to black-box repository-level evaluation (as seen in RepoReason (Li et al., 7 Jan 2026)), OpenCodeReasoning-II is built around explicit CoT and critique, emphasizing fine-grained, scalable, and practical competitive programming constraints.

  Distinct approaches to code reasoning and critique—such as neuro-symbolic methods (CoreThink (Vaghasiya et al., 31 Aug 2025)), ontology-driven perturbations (HUMANEVAL-CORE (Hong et al., 2024)), and executable/Multi-modal LLMs (CodeDance (Song et al., 19 Dec 2025))—are recognized as complementary, suggesting that future iterations may benefit from richer diagnostic metrics, more adversarial data generation, and multi-dimensional critique signals.

  OpenCodeReasoning-II provides both an extensible dataset and an open evaluation protocol, serving as a foundation for empirical research on the strengths and failure modes of state-of-the-art code reasoners, and offers actionable baselines for the design of future, more robust code reasoning benchmarks (Ahmad et al., 11 Jul 2025).