---
title: 'COMPASS: Multi-Dimensional Code-Generation Benchmark'
url: https://www.emergentmind.com/topics/compass
type: topic
---

# COMPASS: Multi-Dimensional Code-Generation Benchmark

Searching arXiv for the benchmark paper and closely related code-generation benchmarks so the article can be grounded in published work.
COMPASS is a code-generation benchmark introduced to evaluate large language models on three independent dimensions of programming performance—correctness, efficiency, and quality—rather than on functional correctness alone. Defined as **COdility’s Multi-dimensional Programming ASSessment**, it consists of 50 competitive programming problems drawn from Codility public contests held between 2011 and 2021 and is paired with authentic human performance data from 393,150 submissions. Its central premise is that pass/fail execution on tests is an incomplete proxy for real software engineering ability, because production-relevant code must also scale on large inputs and remain maintainable under realistic development constraints [2508.13757].

## 1. Scope and rationale

COMPASS was proposed in response to a limitation the authors identify in existing code-generation evaluation: most benchmarks only measure whether generated code passes tests, not whether it is efficient or maintainable [2508.13757]. The benchmark is motivated by the observation that large language models are increasingly used in contexts where production readiness matters, so a correctness-only evaluation no longer captures the full operational value of generated code.

The paper situates this critique against widely used benchmarks such as HumanEval, MBPP, APPS, SWE-bench, DS-1000, and HackerRank-ASTRA, which it describes as largely evaluating models via pass/fail test execution or related correctness metrics [2508.13757]. Within that framing, COMPASS targets two omissions. First, a solution that is correct but has poor time or space complexity can be unusable on large inputs. Second, code that works but is hard to read, deeply nested, poorly named, or otherwise non-idiomatic creates technical debt and maintenance cost. The benchmark therefore makes explicit the tradeoff between whether code works, whether it scales, and whether it is maintainable [2508.13757].

A plausible implication is that COMPASS shifts code-generation evaluation closer to software-engineering criteria than to pure program-synthesis criteria. The benchmark’s design suggests that capability claims derived from correctness-only suites may overestimate deployment readiness.

## 2. Benchmark composition and human reference data

COMPASS contains **50 competitive programming problems** taken from **Codility public contests held between 2011 and 2021** [2508.13757]. These tasks are described as authentic, time-constrained algorithmic problems spanning a range of domains and difficulty levels. For the reported study, model outputs were constrained to **Python 3**, explicitly to reduce variability across programming languages [2508.13757].

A defining feature of the benchmark is its human baseline. COMPASS includes **393,150 human submissions**, with participants solving problems in **25+ mainstream languages** [2508.13757]. For each task, the benchmark reports human performance distributions including mean, median, standard deviation, skewness, and total submissions. This permits model behavior to be situated relative to real competition performance rather than only against synthetic references.

The human data matter methodologically because they enable percentile-based comparison between model outputs and a large empirical population. This distinguishes COMPASS from benchmarks that report only intrinsic model scores. A plausible implication is that COMPASS supports both absolute evaluation and population-relative evaluation, which can reveal whether a model is merely passing curated tests or operating near the upper tail of human contest performance.

## 3. Evaluation dimensions and scoring framework

COMPASS evaluates each submission on three dimensions: correctness, efficiency, and quality [2508.13757]. These axes are scored independently and then combined in an equally weighted composite score.

**Correctness** is defined as the percentage of passed test cases, with coverage including basic functionality, edge cases, and corner cases. The paper gives the formula
\[
C_{\text{corr}} = 100 \cdot \frac{N_{\text{passed}}}{N_{\text{total}}}
\]
where \(N_{\text{passed}}\) is the number of tests passed and \(N_{\text{total}}\) is the total number of tests [2508.13757]. Correctness is therefore not binary pass/fail, but a percentage score.

**Efficiency** is evaluated through stress tests intended to expose algorithmic scalability issues. These tests use large inputs near the upper bounds of problem constraints, strict runtime thresholds, and expectations derived from efficient reference solutions with known optimal complexity [2508.13757]. The efficiency score is the percentage of efficiency test cases passed. This mechanism is specifically designed to separate correct-but-brute-force solutions from correct-and-scalable ones.

**Quality** is measured with **CodeScene**, described as an industrial code-quality analysis tool. It evaluates factors including cyclomatic and cognitive complexity, function length, nesting depth, duplication, cohesion, method design, parameter structure, naming conventions, adherence to language idioms, and anti-pattern detection [2508.13757]. Each solution receives a composite code-quality score from 1 to 10, which is then normalized to a 1–100 scale.

The benchmark also reports a **composite score** that equally weights correctness, efficiency, and quality [2508.13757]. This composite is intended to provide a holistic measure rather than privileging any single axis. This suggests that COMPASS treats maintainability as co-equal with functional behavior and scalability, rather than as an auxiliary diagnostic.

## 4. Experimental protocol and model comparison

The reported experiments evaluate three reasoning-enhanced frontier models: **Anthropic Claude Opus 4**, **Google Gemini 2.5 Pro**, and **OpenAI O4-Mini-High** [2508.13757]. For each model–problem pair, the study uses **64 independent samples** at default temperature **\(T = 1.0\)** [2508.13757].

The prompting setup includes four variants: neutral control, correctness-focused, efficiency-focused, and a dual objective combining correctness and efficiency [2508.13757]. These prompt variants are used to test whether simple framing can change model behavior across the benchmark’s axes.

For human comparison, the paper uses only correctness and efficiency, because human submissions do not have code-quality scores [2508.13757]. Model scores are converted to human-relative percentiles by first computing a z-score,
\[
Z = \frac{X - \mu}{\sigma}
\]
where \(X\) is the model score, \(\mu\) the human mean, and \(\sigma\) the human standard deviation, and then mapping that z-score to a percentile using the CDF of the standard normal distribution [2508.13757].

The reported mean percentiles across tasks are **97.840** for **O4-Mini-High**, **96.280** for **Gemini 2.5 Pro**, and **76.580** for **Claude Opus 4** [2508.13757]. The paper interprets these results as showing that two models are near or above the top human performance range on the combined correctness-efficiency measure, while Claude is much less consistent.

## 5. Empirical findings

The central empirical finding is that **high correctness does not guarantee high efficiency or high maintainability** [2508.13757]. The benchmark repeatedly shows mismatches in which models produce highly correct but inefficient solutions, or correct solutions that are structurally messy. This is the core phenomenon COMPASS was designed to surface.

The paper reports the following overall mean scores:

| Model | Correctness | Efficiency | Quality |
|---|---:|---:|---:|
| Claude Opus 4 | 72.2 | 35.4 | 92.3 |
| Gemini 2.5 Pro | 93.8 | 85.4 | 93.2 |
| O4-Mini-High | 95.6 | 93.0 | 89.2 |

The corresponding composite scores are **66.1** for Claude Opus 4, **90.4** for Gemini 2.5 Pro, and **92.3** for O4-Mini-High [2508.13757]. The paper identifies **O4-Mini-High** as the most consistently strong overall and **Gemini 2.5 Pro** as close behind, with the best quality score. **Claude Opus 4** is characterized by strong quality but much weaker and more variable correctness and efficiency [2508.13757].

A particularly notable result is that Claude’s correctness can appear decent in some cases while its efficiency is often poor, yielding highly variable and sometimes low composite scores [2508.13757]. This is presented as an example of the failure mode that correctness-only benchmarks would obscure.

The paper also reports frequencies of recurring code-quality issues across models, including **Bumpy Road Ahead**, **Deep, Nested Complexity**, and **Complex Method** [2508.13757]. These findings indicate that even strong models often produce functions with excessive logic, nesting, or cyclomatic complexity.

## 6. Statistical structure, prompt sensitivity, and implications

The paper argues that code quality is largely orthogonal to correctness and efficiency. Correlation analysis across all model submissions found **\(r = .655\)** for correctness versus efficiency, **\(r = .089\)** for correctness versus quality, and **\(r = .022\)** for efficiency versus quality [2508.13757]. Thus correctness and efficiency are moderately related, while quality is almost independent of both.

Principal component analysis reinforces this separation. The reported structure is: **PC1** loads on correctness and efficiency, **PC2** is almost entirely quality, and **PC3** reflects a residual correctness–efficiency tradeoff [2508.13757]. This supports the benchmark’s claim that correctness, efficiency, and quality are genuinely distinct performance dimensions rather than interchangeable proxies.

The study also measures within-task consistency across repeated generations. **O4-Mini-High** is reported as the most stable and **Claude** as the least stable [2508.13757]. Prompt framing has only modest effects overall, although Claude responds more noticeably to prompt direction, while stronger models are more robust to prompt variation [2508.13757].

These results bear directly on benchmark design. The broader implication drawn in the paper is that code-generation evaluation should move beyond correctness-only metrics such as pass@\(k\) or test-case success and treat **efficiency**, **maintainability / quality**, and **stability / consistency** as first-class requirements [2508.13757]. This suggests a shift from asking whether a model can write code that passes tests to asking whether it can produce code that is correct, scalable, readable, maintainable, and reliable under repeated sampling.

## 7. Position within code-generation benchmark research

Within the broader literature on LLM code evaluation, COMPASS occupies a specific methodological niche. HumanEval was designed around functional correctness for short programming tasks [2107.03374], MBPP focused on mostly basic Python programming problems [2108.07732], APPS emphasized broader program-synthesis difficulty with public and hidden test cases [2105.09938], DS-1000 targeted data-science code generation [2211.11501], and SWE-bench evaluates software-engineering tasks grounded in real repositories [2310.06770]. COMPASS’s contribution is not that it replaces these benchmarks, but that it isolates a distinct failure mode: the collapse of correctness into a stand-in for overall software quality [2508.13757].

The benchmark’s reliance on real Codility competition tasks and large-scale human submissions gives it a strong empirical grounding in algorithmic programming performance. At the same time, its quality dimension, based on CodeScene analysis, extends evaluation into maintainability attributes that are typically absent from competitive-programming-style benchmarks [2508.13757]. This combination makes COMPASS unusually well suited for examining whether frontier models can generate code that is not merely executable, but also scalable and structurally fit for practical use.

A common misconception is that if a model achieves high correctness on benchmark tests, it has therefore demonstrated production readiness. COMPASS directly challenges that assumption. Its evidence suggests that correctness, algorithmic efficiency, and maintainability should be treated as separate axes, with no guarantee that strength on one transfers to the others [2508.13757].

Source: https://www.emergentmind.com/topics/compass