COMPASS: Multi-Dimensional Code-Generation Benchmark

• COMPASS is a multi-dimensional code-generation benchmark that evaluates models on correctness, efficiency, and quality using real-world coding tasks.
• It combines 50 Codility contest problems with 393,150 human submissions to provide a robust, population-relative performance measure.
• The benchmark reveals that high correctness does not guarantee scalable or maintainable code, urging a holistic software evaluation approach.

Searching arXiv for the benchmark paper and closely related code-generation benchmarks so the article can be grounded in published work. COMPASS is a code-generation benchmark introduced to evaluate LLMs on three independent dimensions of programming performance—correctness, efficiency, and quality—rather than on functional correctness alone. Defined as COdility’s Multi-dimensional Programming ASSessment, it consists of 50 competitive programming problems drawn from Codility public contests held between 2011 and 2021 and is paired with authentic human performance data from 393,150 submissions. Its central premise is that pass/fail execution on tests is an incomplete proxy for real software engineering ability, because production-relevant code must also scale on large inputs and remain maintainable under realistic development constraints (Meaden et al., 19 Aug 2025).

1. Scope and rationale

COMPASS was proposed in response to a limitation the authors identify in existing code-generation evaluation: most benchmarks only measure whether generated code passes tests, not whether it is efficient or maintainable (Meaden et al., 19 Aug 2025). The benchmark is motivated by the observation that LLMs are increasingly used in contexts where production readiness matters, so a correctness-only evaluation no longer captures the full operational value of generated code.

The paper situates this critique against widely used benchmarks such as HumanEval, MBPP, APPS, SWE-bench, DS-1000, and HackerRank-ASTRA, which it describes as largely evaluating models via pass/fail test execution or related correctness metrics (Meaden et al., 19 Aug 2025). Within that framing, COMPASS targets two omissions. First, a solution that is correct but has poor time or space complexity can be unusable on large inputs. Second, code that works but is hard to read, deeply nested, poorly named, or otherwise non-idiomatic creates technical debt and maintenance cost. The benchmark therefore makes explicit the tradeoff between whether code works, whether it scales, and whether it is maintainable (Meaden et al., 19 Aug 2025).

A plausible implication is that COMPASS shifts code-generation evaluation closer to software-engineering criteria than to pure program-synthesis criteria. The benchmark’s design suggests that capability claims derived from correctness-only suites may overestimate deployment readiness.

2. Benchmark composition and human reference data

COMPASS contains 50 competitive programming problems taken from Codility public contests held between 2011 and 2021 (Meaden et al., 19 Aug 2025). These tasks are described as authentic, time-constrained algorithmic problems spanning a range of domains and difficulty levels. For the reported study, model outputs were constrained to Python 3, explicitly to reduce variability across programming languages (Meaden et al., 19 Aug 2025).

A defining feature of the benchmark is its human baseline. COMPASS includes 393,150 human submissions, with participants solving problems in 25+ mainstream languages (Meaden et al., 19 Aug 2025). For each task, the benchmark reports human performance distributions including mean, median, standard deviation, skewness, and total submissions. This permits model behavior to be situated relative to real competition performance rather than only against synthetic references.

The human data matter methodologically because they enable percentile-based comparison between model outputs and a large empirical population. This distinguishes COMPASS from benchmarks that report only intrinsic model scores. A plausible implication is that COMPASS supports both absolute evaluation and population-relative evaluation, which can reveal whether a model is merely passing curated tests or operating near the upper tail of human contest performance.

3. Evaluation dimensions and scoring framework

COMPASS evaluates each submission on three dimensions: correctness, efficiency, and quality (Meaden et al., 19 Aug 2025). These axes are scored independently and then combined in an equally weighted composite score.

Correctness is defined as the percentage of passed test cases, with coverage including basic functionality, edge cases, and corner cases. The paper gives the formula

$$C_{\text{corr}} = 100 \cdot \frac{N_{\text{passed}}}{N_{\text{total}}}$$

where $N_{\text{passed}}$ is the number of tests passed and $N_{\text{total}}$ is the total number of tests (Meaden et al., 19 Aug 2025). Correctness is therefore not binary pass/fail, but a percentage score.

Efficiency is evaluated through stress tests intended to expose algorithmic scalability issues. These tests use large inputs near the upper bounds of problem constraints, strict runtime thresholds, and expectations derived from efficient reference solutions with known optimal complexity (Meaden et al., 19 Aug 2025). The efficiency score is the percentage of efficiency test cases passed. This mechanism is specifically designed to separate correct-but-brute-force solutions from correct-and-scalable ones.

Quality is measured with CodeScene, described as an industrial code-quality analysis tool. It evaluates factors including cyclomatic and cognitive complexity, function length, nesting depth, duplication, cohesion, method design, parameter structure, naming conventions, adherence to language idioms, and anti-pattern detection (Meaden et al., 19 Aug 2025). Each solution receives a composite code-quality score from 1 to 10, which is then normalized to a 1–100 scale.

The benchmark also reports a composite score that equally weights correctness, efficiency, and quality (Meaden et al., 19 Aug 2025). This composite is intended to provide a holistic measure rather than privileging any single axis. This suggests that COMPASS treats maintainability as co-equal with functional behavior and scalability, rather than as an auxiliary diagnostic.

4. Experimental protocol and model comparison

The reported experiments evaluate three reasoning-enhanced frontier models: Anthropic Claude Opus 4, Google Gemini 2.5 Pro, and OpenAI O4-Mini-High (Meaden et al., 19 Aug 2025). For each model–problem pair, the study uses 64 independent samples at default temperature $T = 1.0$ (Meaden et al., 19 Aug 2025).

The prompting setup includes four variants: neutral control, correctness-focused, efficiency-focused, and a dual objective combining correctness and efficiency (Meaden et al., 19 Aug 2025). These prompt variants are used to test whether simple framing can change model behavior across the benchmark’s axes.

For human comparison, the paper uses only correctness and efficiency, because human submissions do not have code-quality scores (Meaden et al., 19 Aug 2025). Model scores are converted to human-relative percentiles by first computing a z-score,

$Z = \frac{X - \mu}{\sigma}$

where $X$ is the model score, $\mu$ the human mean, and $\sigma$ the human standard deviation, and then mapping that z-score to a percentile using the CDF of the standard normal distribution (Meaden et al., 19 Aug 2025).

The reported mean percentiles across tasks are 97.840 for O4-Mini-High, 96.280 for Gemini 2.5 Pro, and 76.580 for Claude Opus 4 (Meaden et al., 19 Aug 2025). The paper interprets these results as showing that two models are near or above the top human performance range on the combined correctness-efficiency measure, while Claude is much less consistent.

5. Empirical findings

The central empirical finding is that high correctness does not guarantee high efficiency or high maintainability (Meaden et al., 19 Aug 2025). The benchmark repeatedly shows mismatches in which models produce highly correct but inefficient solutions, or correct solutions that are structurally messy. This is the core phenomenon COMPASS was designed to surface.

The paper reports the following overall mean scores:

Model	Correctness	Efficiency	Quality
Claude Opus 4	72.2	35.4	92.3
Gemini 2.5 Pro	93.8	85.4	93.2
O4-Mini-High	95.6	93.0	89.2

The corresponding composite scores are 66.1 for Claude Opus 4, 90.4 for Gemini 2.5 Pro, and 92.3 for O4-Mini-High (Meaden et al., 19 Aug 2025). The paper identifies O4-Mini-High as the most consistently strong overall and Gemini 2.5 Pro as close behind, with the best quality score. Claude Opus 4 is characterized by strong quality but much weaker and more variable correctness and efficiency (Meaden et al., 19 Aug 2025).

A particularly notable result is that Claude’s correctness can appear decent in some cases while its efficiency is often poor, yielding highly variable and sometimes low composite scores (Meaden et al., 19 Aug 2025). This is presented as an example of the failure mode that correctness-only benchmarks would obscure.

The paper also reports frequencies of recurring code-quality issues across models, including Bumpy Road Ahead, Deep, Nested Complexity, and Complex Method (Meaden et al., 19 Aug 2025). These findings indicate that even strong models often produce functions with excessive logic, nesting, or cyclomatic complexity.

6. Statistical structure, prompt sensitivity, and implications

The paper argues that code quality is largely orthogonal to correctness and efficiency. Correlation analysis across all model submissions found $r = .655$ for correctness versus efficiency, $r = .089$ for correctness versus quality, and $N_{\text{passed}}$0 for efficiency versus quality (Meaden et al., 19 Aug 2025). Thus correctness and efficiency are moderately related, while quality is almost independent of both.

Principal component analysis reinforces this separation. The reported structure is: PC1 loads on correctness and efficiency, PC2 is almost entirely quality, and PC3 reflects a residual correctness–efficiency tradeoff (Meaden et al., 19 Aug 2025). This supports the benchmark’s claim that correctness, efficiency, and quality are genuinely distinct performance dimensions rather than interchangeable proxies.

The study also measures within-task consistency across repeated generations. O4-Mini-High is reported as the most stable and Claude as the least stable (Meaden et al., 19 Aug 2025). Prompt framing has only modest effects overall, although Claude responds more noticeably to prompt direction, while stronger models are more robust to prompt variation (Meaden et al., 19 Aug 2025).

These results bear directly on benchmark design. The broader implication drawn in the paper is that code-generation evaluation should move beyond correctness-only metrics such as pass@$N_{\text{passed}}$1 or test-case success and treat efficiency, maintainability / quality, and stability / consistency as first-class requirements (Meaden et al., 19 Aug 2025). This suggests a shift from asking whether a model can write code that passes tests to asking whether it can produce code that is correct, scalable, readable, maintainable, and reliable under repeated sampling.

7. Position within code-generation benchmark research

Within the broader literature on LLM code evaluation, COMPASS occupies a specific methodological niche. HumanEval was designed around functional correctness for short programming tasks (Chen et al., 2021), MBPP focused on mostly basic Python programming problems (Austin et al., 2021), APPS emphasized broader program-synthesis difficulty with public and hidden test cases (Hendrycks et al., 2021), DS-1000 targeted data-science code generation (Lai et al., 2022), and SWE-bench evaluates software-engineering tasks grounded in real repositories (Jimenez et al., 2023). COMPASS’s contribution is not that it replaces these benchmarks, but that it isolates a distinct failure mode: the collapse of correctness into a stand-in for overall software quality (Meaden et al., 19 Aug 2025).

The benchmark’s reliance on real Codility competition tasks and large-scale human submissions gives it a strong empirical grounding in algorithmic programming performance. At the same time, its quality dimension, based on CodeScene analysis, extends evaluation into maintainability attributes that are typically absent from competitive-programming-style benchmarks (Meaden et al., 19 Aug 2025). This combination makes COMPASS unusually well suited for examining whether frontier models can generate code that is not merely executable, but also scalable and structurally fit for practical use.

A common misconception is that if a model achieves high correctness on benchmark tests, it has therefore demonstrated production readiness. COMPASS directly challenges that assumption. Its evidence suggests that correctness, algorithmic efficiency, and maintainability should be treated as separate axes, with no guarantee that strength on one transfers to the others (Meaden et al., 19 Aug 2025).