BigCodeReward: Code Generation Benchmark

• BigCodeReward is a benchmark that evaluates automated reward models for code generation by comparing model outputs to human pairwise preferences and execution traces.
• It integrates multimodal inputs, including full conversational context and execution outputs, to provide comprehensive feedback on code correctness and practical performance.
• The benchmark employs rigorous metrics like accuracy and macro F1, using statistical methods such as bootstrap resampling and the Bradley-Terry model to highlight performance gaps among models.

BigCodeReward is a benchmark and evaluation resource developed to systematically assess the alignment between automated reward models and human preferences in code generation tasks. Originating from the BigCodeArena platform, which integrates large-scale, crowdsourced human evaluation with full code execution, BigCodeReward provides detailed pairwise preference judgments over model outputs, as well as a multimodal evaluation context including execution traces. Its aim is to quantify how reliably reward models—key components in reinforcement learning from human feedback (RLHF) for code—can replicate human preferences when scoring or ranking generated code, especially when execution outputs and multimodal cues are available.

1. Benchmark Construction and Motivations

BigCodeReward derives from the BigCodeArena platform, which collected over 14,000 code-centric conversation sessions involving 10 LLMs, 10 programming languages, and 8 execution environments. A targeted subset of around 4,700 multi-turn conversations was selected based on the presence of high-quality, human-provided pairwise preferences. The construction protocol involves:

• Concatenating all user inputs and final model responses in each multi-turn coding session to form a unified evaluation instance.
• Using human votes on pairs of model responses (“Model A is better”, “Model B is better”, “Tie”/unsatisfactory) as ground-truth preference labels.
• Including code execution outputs such as logs or UI renderings to enable richer multimodal evaluation.

The central motivation is to provide a rigorous testbed for evaluating how well automated reward models—those that assign preference or numeric scores to code snippets—replicate human preferences, especially when they have access to execution results that may reveal correctness or subtle failures not visible through static code inspection.

2. Data Protocol and Labeling Methodology

BigCodeReward adopts a post-processing pipeline that synthesizes multi-turn session data for high-quality benchmarking:

• Each evaluation instance consists of the entire conversational context (user prompts and both model-generated code snippets) paired with a human preference label.
• Human annotators select their preferred output or indicate a tie, establishing a three-class ground-truth label for every sample.
• Inclusion of execution outputs is a distinguishing feature: for each code snippet, execution traces (logs, error messages, visual results) are stored alongside the source and textual context, directly addressing the gap between theoretical code output and practical, observable behavior.

This approach ensures that evaluation captures not just syntactic or static code properties but also real-world execution outcomes, especially important in tasks where code correctness and UX are not easily recognized in static text.

3. Evaluation Protocol and Metrics

The primary goal in BigCodeReward is to quantify the consistency with which a reward model’s preferences agree with human votes. For each sample, the reward model is provided with the aggregated instance—including code, full context, and, when available, execution results—and outputs a class label (A, B, or Tie).

Evaluation is performed under two conditions:

• Without execution outputs: The reward model receives only code and text (for establishing a baseline of static understanding).
• With execution outputs: The reward model also receives runtime logs, UI screenshots, or other execution artifacts.

Metrics include:

• Accuracy: Proportion of samples for which the reward model’s prediction matches the human-provided label,

$$\text{Accuracy} = \frac{\text{\# correct predictions}}{\text{\# total samples}}$$

• Macro F1: The mean F1-score across the three possible classes (A, B, Tie), with per-class F1 computed as

$$\text{F1}_c = \frac{2 \cdot \text{Precision}_c \cdot \text{Recall}_c}{\text{Precision}_c + \text{Recall}_c}$$

• Model ranking: The Bradley-Terry model is employed for pairwise strength estimation, with

$$p_{ij} = \frac{e^{\beta_i}}{e^{\beta_i} + e^{\beta_j}}$$

where $\beta_i$ parameterizes the relative “strength” of model $i$.

Statistical robustness is quantified using 100 bootstrap resamplings to report 95% confidence intervals.

4. Impact of Execution Outputs and Multimodality

The inclusion of execution results is found to be crucial. Most tested reward models demonstrably improve in matching human preferences when execution outputs are available. This is attributed to the capacity of execution traces to reveal concrete correctness, UI/UX behavior, and runtime-specific phenomena that are opaque at the code/text level.

A subset of models demonstrates reduced performance when execution results are included, indicating either insufficient multimodal modeling capacity or instability when processing heterogeneous input types. This suggests that effective reward modeling for code generation may require not only strong language understanding but also robust mechanisms for integrating execution context.

5. Human Preference Analysis and Model Comparisons

Empirical findings from BigCodeReward demonstrate:

• Proprietary models such as GPT-5, Claude-Sonnet-4, and Claude-Opus-4 consistently lead in both code generation quality and in capturing human-judged preferences.
• Open-source or mid-tier models show lower alignment, emphasizing persistent capability gaps.
• Evaluation using execution-enhanced preferences reveals underexplored model behaviors across specific domains (language, task, framework), enabling fine-grained diagnostics.

The dataset and benchmark thus provide strong signals for research into model alignment, RLHF for code, and multimodal reward learning.

6. Statistical Evaluation Methods

Reward model and system-level comparisons are supported by sound statistical tools:

• The Bradley-Terry model enables interpretable aggregate model ranking from pairwise comparisons.
• Bootstrap resampling delivers robust confidence intervals for all reported metrics.
• Per-class F1 and macro F1 enable detailed error analysis in multiclass (A/B/Tie) scenarios.

The use of transparent, well-established statistical methods underpins the benchmark’s validity and reproducibility.

7. Significance for RLHF and Reward Model Development

BigCodeReward directly addresses the core requirement of RLHF for code: learning a reward signal that faithfully mirrors human judgments, especially in practical environments where code must run, interact, and achieve non-obvious goals. By centering evaluation on human-voted real-world scenarios enhanced by execution artifacts, the benchmark enables the development and validation of reward models that:

• Are robust to both code/text and multimodal inputs,
• Distinguish subtle differences in functionally correct versus superficially plausible code,
• Provide a testbed for measuring progress toward genuinely aligned, user-oriented code synthesis systems.

A plausible implication is that future reward models aiming for strong alignment in code generation must explicitly incorporate execution-based and multimodal feedback, as static signals alone are insufficient for capturing true human preference in realistic settings.

---

BigCodeReward constitutes a pivotal resource for evaluating reward model alignment with human coding preferences, leveraging a large and carefully constructed set of annotated, execution-backed code generation samples, rigorous statistical metrics, and comprehensive support for multimodal input. Its findings suggest that access to execution results is critical for reward model effectiveness, and that current proprietary LLMs maintain a significant advantage in accurately reflecting human judgment in code generation tasks (Zhuo et al., 9 Oct 2025).