Evaluation Ladder: Progressive Assessment

• Evaluation Ladder is a structured framework that organizes assessments into ordered levels, each with increasing complexity and constraints.
• It moves beyond binary scoring by employing progressive tests—such as stepwise rubric aggregation and hierarchical verification—to reveal subtle performance gaps.
• Its design facilitates practical applications across domains like program synthesis, health agents, and causal discovery, with evolving maintenance practices to sustain diagnostic power.

Searching arXiv for the cited papers to ground the article. to=arxiv_search.search 银航json {"query":"all:(Schnetz, 2012) OR all:(Xia et al., 2024) OR all:(Chandwani et al., 24 Mar 2026) OR all:(Zhang et al., 16 Jun 2026) OR all:(Pezeshki et al., 2015) OR all:(Peyrard et al., 2020) OR all:(Xie et al., 29 May 2025) OR all:(Lim et al., 3 Jan 2025) OR all:(Dubey et al., 2020)","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}【อ่านข้อความเต็มjson {"result":[{"arxiv_id":"(Zhang et al., 16 Jun 2026)","version":"v1","idv":"(Zhang et al., 16 Jun 2026)v1","title":"RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills","categories":"cs.CL cs.AI q-bio.QM","published":"2026-06-16","updated":"2026-06-16","pdf_url":"http://arxiv.org/pdf/([2606.18203](/papers/2606.18203))v1","abs_url":"https://arxiv.org/abs/([2606.18203](/papers/2606.18203))v1"},{"arxiv_id":"([2603.22744](/papers/2603.22744))","version":"v1","idv":"([2603.22744](/papers/2603.22744))v1","title":"Beyond Binary Correctness: Scaling Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks","categories":"cs.LG","published":"2026-03-24","updated":"2026-03-24","pdf_url":"http://arxiv.org/pdf/([2603.22744](/papers/2603.22744))v1","abs_url":"https://arxiv.org/abs/([2603.22744](/papers/2603.22744))v1"},{"arxiv_id":"([2505.24063](/papers/2505.24063))","version":"v1","idv":"([2505.24063](/papers/2505.24063))v1","title":"TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicine","categories":"cs.CL","published":"2025-05-29","updated":"2025-05-29","pdf_url":"http://arxiv.org/pdf/([2505.24063](/papers/2505.24063))v1","abs_url":"https://arxiv.org/abs/([2505.24063](/papers/2505.24063))v1"},{"arxiv_id":"([2501.01603](/papers/2501.01603))","version":"v1","idv":"([2501.01603](/papers/2501.01603))v1","title":"pyBoLaNO: A Python symbolic package for normal ordering involving bosonic ladder operators","categories":"quant-ph","published":"2025-01-03","updated":"2025-01-03","pdf_url":"http://arxiv.org/pdf/([2501.01603](/papers/2501.01603))v1","abs_url":"https://arxiv.org/abs/([2501.01603](/papers/2501.01603))v1"},{"arxiv_id":"([2403.19114](/papers/2403.19114))","version":"v2","idv":"([2403.19114](/papers/2403.19114))v2","title":"Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM","categories":"cs.SE cs.AI cs.LG","published":"2024-03-28","updated":"2024-11-17","pdf_url":"http://arxiv.org/pdf/([2403.19114](/papers/2403.19114))v2","abs_url":"https://arxiv.org/abs/([2403.19114](/papers/2403.19114))v2"},{"arxiv_id":"([2005.02480](/papers/2005.02480))","version":"v2","idv":"([2005.02480](/papers/2005.02480))v2","title":"A Ladder of Causal Distances","categories":"cs.LG cs.AI stat.ML","published":"2020-05-05","updated":"2021-06-17","pdf_url":"http://arxiv.org/pdf/([2005.02480](/papers/2005.02480))v2","abs_url":"https://arxiv.org/abs/([2005.02480](/papers/2005.02480))v2"},{"arxiv_id":"([2001.03896](/papers/2001.03896))","version":"v1","idv":"([2001.03896](/papers/2001.03896))v1","title":"CURE Dataset: Ladder Networks for Audio Event Classification","categories":"cs.SD eess.AS","published":"2020-01-12","updated":"2020-01-12","pdf_url":"http://arxiv.org/pdf/([2001.03896](/papers/2001.03896))v1","abs_url":"https://arxiv.org/abs/([2001.03896](/papers/2001.03896))v1"},{"arxiv_id":"([1511.06430](/papers/1511.06430))","version":"v1","idv":"([1511.06430](/papers/1511.06430))v1","title":"Deconstructing the Ladder Network Architecture","categories":"cs.LG stat.ML","published":"2015-11-19","updated":"2015-11-19","pdf_url":"http://arxiv.org/pdf/([1511.06430](/papers/1511.06430))v1","abs_url":"https://arxiv.org/abs/([1511.06430](/papers/1511.06430))v1"},{"arxiv_id":"([1210.5376](/papers/1210.5376))","version":"v1","idv":"([1210.5376](/papers/1210.5376))v1","title":"Evaluation of the period of a family of triangle and box ladder graphs","categories":"hep-th math-ph","published":"2012-10-19","updated":"2012-10-19","pdf_url":"http://arxiv.org/pdf/([1210.5376](/papers/1210.5376))v1","abs_url":"https://arxiv.org/abs/([1210.5376](/papers/1210.5376))v1"}]} Evaluation Ladder denotes a class of evaluation frameworks that organize assessment into ordered rungs, tiers, or levels, each adding new constraints, stronger notions of correctness, or richer forms of evidence. In recent arXiv literature, this pattern appears in program-synthesis benchmarking, long-horizon enterprise-agent evaluation, personal health-agent assessment, multimodal Traditional Chinese Medicine question answering, and causal discovery. Across these settings, the ladder structure is used to move beyond single scalar scores or binary pass/fail judgments toward staged measurement of robustness, compositionality, artifact quality, personalization, and counterfactual validity (Xia et al., 2024, Chandwani et al., 24 Mar 2026, Zhang et al., 16 Jun 2026, Xie et al., 29 May 2025, Peyrard et al., 2020).

1. Conceptual definition and scope

An evaluation ladder is a progressive evaluation scheme in which increasingly demanding rungs test different failure modes or capability strata. In EvoEval, the ladder begins with baseline syntax and API compliance and ascends through semantic-preserving rewrites, subtle specification changes, difficulty and domain shift, compositionality, decomposition, tool use, and adversarial edge cases (Xia et al., 2024). In LH-Bench, the ladder is explicit in three pillars: expert-grounded rubrics, curated ground-truth artifacts enabling stepwise rewards, and pairwise human preference evaluation for convergent validation (Chandwani et al., 24 Mar 2026). In RubricsTree, the ladder is a directed acyclic graph from macro domains to sub-domains to over 100 atomic, clinically-verifiable Boolean rubrics (Zhang et al., 16 Jun 2026). In causal discovery, the ladder follows Pearl’s three rungs of causation—observational, interventional, and counterfactual—each associated with a distinct distance between causal models (Peyrard et al., 2020). In TCM-Ladder, the progression runs from foundational recall to integrative reasoning, generative precision, multimodal interpretation, and long-form clinical dialogue (Xie et al., 29 May 2025).

The surveyed literature uses the label for several domain-specific schemes rather than for one universal standard. This suggests that “Evaluation Ladder” is best understood as an architectural principle for evaluation design: tasks are decomposed into ordered levels so that a model can be strong on one rung yet weak on a higher one.

2. Structural patterns across domains

The principal implementations differ in domain and scoring machinery, but they share an ordered progression from simpler to more diagnostic tests.

Framework	Ordered structure	Primary target
EvoEval	Tier 0 to Tier 8	Program-synthesis generalization
LH-Bench	Rung A, Rung B, Rung C	Subjective long-horizon enterprise tasks
RubricsTree	Macro domains, sub-domains, atomic leaves	Personal health-agent quality
TCM-Ladder	Foundational to clinical-dialogue levels	Multimodal TCM QA
Causal distances	Observational, interventional, counterfactual	Causal-model fidelity

In EvoEval, the ladder is perturbational and capability-oriented. Tier 0 uses seed tasks akin to HumanEval; Tier 1 tests paraphrase robustness through Verbose and Concise rewrites; Tier 2 introduces Subtle variants with a single inverted or replaced requirement; Tier 3 adds Difficult and Creative transformations; Tier 5 and Tier 6 evaluate composition and decomposition; Tier 7 measures tool use via helper functions; Tier 8 stresses adversarial and edge cases (Xia et al., 2024). The structure is explicitly intended to surface overfitting, phrasing sensitivity, and failures in composition and tool use.

LH-Bench organizes its ladder around evidence sources rather than task perturbations. Rung A supplies expert-authored rubrics with 1–5 anchored scales and weights; Rung B adds curated ground-truth artifacts such as manifest.json, per-frame ground-truth images, and chapter-level source-grounded annotations; Rung C validates induced rankings through side-by-side human preference judgments modeled with Bradley–Terry (Chandwani et al., 24 Mar 2026). The ladder therefore moves from rubric-grounded process inspection to artifact-level verification to stakeholder preference.

RubricsTree replaces free-form holistic judgment with routed atomic verification. Its top level separates Medical skills and Health memory; intermediate nodes refine these into sub-aspects such as Medical Explanation, Health Data interpretation, Advice/Action Plan, and Symptoms; the bottom level contains over 100 binary leaves, each tied to a clinically verifiable item and activated only when contextually relevant (Zhang et al., 16 Jun 2026). TCM-Ladder uses task types as competency levels: single-choice, multiple-choice, fill-in-the-blank, visual comprehension, and diagnostic dialogue (Xie et al., 29 May 2025). The causal ladder is conceptually stricter: observational fit does not imply accurate intervention prediction, and accurate intervention prediction does not imply accurate counterfactuals (Peyrard et al., 2020).

3. Scoring, aggregation, and statistical machinery

A defining feature of evaluation ladders is that they require rung-specific scoring rules rather than a single universal metric. EvoEval uses functional correctness and reports pass@1 with greedy decoding as well as the standard estimator

$$\mathrm{pass@k} = 1 - \frac{\binom{n-m}{k}}{\binom{n}{k}}.$$

Its execution-based oracle uses augmented tests, numeric tolerance $|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$, type-aware recursive comparison, and a timeout policy

$T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$

with $T_{\max} = 1000\ \mathrm{ms}$ and $f = 4$ (Xia et al., 2024).

LH-Bench formalizes stepwise rubric aggregation as

$S = \sum w_i r_i,$

with $\sum w_i = 1$, and normalizes mixed scales via

$\tilde r_i = \frac{r_i - \min}{\max - \min}.$

Its pairwise human-preference layer fits Bradley–Terry latent skill parameters $\beta$ through

$$P(i \text{ beats } j) = \frac{\exp(\beta_i)}{\exp(\beta_i) + \exp(\beta_j)}.$$

Reliability is reported through Cohen’s $|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$0 and weighted $|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$1 for ordinal 1–5 scales (Chandwani et al., 24 Mar 2026).

RubricsTree uses deterministic top-down equal-weight distribution over activated leaves. If a leaf $|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$2 lies on a root-to-leaf path through child sets $|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$3, its weight is

$|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$4

For activated leaves, the score is

$|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$5

with $|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$6. Robustness to degraded context is measured by Detection Rate and Mean Penalty:

$|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$7

$|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$8

This turns open-ended health evaluation into auditable weighted Boolean verification (Zhang et al., 16 Jun 2026).

TCM-Ladder supplements exact-match and generation metrics with Ladder-Score:

$|\mathrm{pred} - \mathrm{gt}| \le 10^{-6}$9

with $T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$0 and $T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$1. TermScore measures the accuracy and completeness of TCM terminology usage, while SemanticScore evaluates logical consistency, semantic accuracy, comprehensiveness of knowledge, and fluency of expression (Xie et al., 29 May 2025).

For causal discovery, each rung is itself a distance between models. The observational distance is

$T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$2

the interventional distance averages observational distances under interventions, and the counterfactual distance averages interventional distances under evidential conditioning. The paper proves the ladder inequalities

$T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$3

$T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$4

These make explicit that higher-rung agreement subsumes lower-rung agreement (Peyrard et al., 2020).

4. Empirical findings and diagnostic power

The practical value of evaluation ladders lies in the way they reorder model rankings and expose hidden failure modes. EvoEval evaluates 51 LLMs on 828 problems across seven datasets and finds a substantial average performance drop of approximately 39.4 points from standard benchmarks to EvoEval, with drops ranging approximately 19.6%–47.7%. The paper reports major ranking reshuffles, about a 24-point drop from seed to Subtle on the same 100 seeds for many models, and composition percentages of 53.8% for GPT-4, 48.1% for GPT-4-Turbo, 43.2% for Claude-3, and 36.9% for ChatGPT on composed problems where the parent tasks were both solved (Xia et al., 2024). The central implication is that top leaderboard performance on small static benchmarks can overestimate general coding proficiency.

LH-Bench supplies analogous evidence for subjective enterprise work. It reports that domain-authored rubrics are more reliable than LLM-authored rubrics, with mean pairwise $T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$5 versus $T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$6. Human preference judgments confirm the same top-tier separation at $T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$7. The benchmark spans two released environments: Figma-to-code with 33 real .fig tasks and Programmatic content with 41 courses comprising 183 individually-evaluated chapters on a course platform serving 30+ daily users (Chandwani et al., 24 Mar 2026). This shows that multi-rung evaluation can scale beyond tasks with unit-test oracles.

RubricsTree extends the same logic to personal health agents. Its taxonomy was shaped by approximately 4,000 real user queries through a 9-member curation panel, and meta-evaluation uses a separate 6-member evaluation panel. It reports overall $T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$8 versus $T = \max(T_{\max}, f \cdot t_{\mathrm{gt}}),$9 for a principle-based baseline, overall $T_{\max} = 1000\ \mathrm{ms}$0 versus $T_{\max} = 1000\ \mathrm{ms}$1, and routing accuracy of 80.61% at 5.6 s average latency for hierarchical traversal, compared with 76.14% at 64.2 s for per-leaf judging and 61.25% at 3.2 s for embedding similarity. When used as structured instructions, text feedback, or training rewards, the framework yields up to approximately 66% relative gains on HealthBench for Gemini, GPT, and Qwen model families (Zhang et al., 16 Jun 2026).

TCM-Ladder demonstrates the ladder design in a multimodal medical benchmark. The dataset contains 52,169 questions, 238,867 answers, 7,455 images, 6,420 audios, and 49 videos, with 80/10/10 train/validation/test splits. It covers single-choice, multiple-choice, fill-in-the-blank, diagnostic dialogue, and visual comprehension across fundamental theory, diagnostics, herbal formulas, internal medicine, surgery, pharmacognosy, and pediatrics. On diagnostic dialogue, Ladder-base attains a Ladder-score of 0.803, Tongyi Qwen attains 0.861, and Bencao attains 0.791; on fill-in-the-blank, Bencao reaches exact match accuracy 0.9034, ahead of Tongyi Qwen at 0.8786 and Ladder-base at 0.8623 (Xie et al., 29 May 2025).

In causal discovery, the ladder yields metric-dependent rankings that differ from graph-only measures. The benchmark uses real Bayesian networks such as Cancer1, Cancer2, Earthquake, Survey, Protein, Child, and Insurance, along with synthetic SCMs such as linGauss, linNGauss, GPAddit, and GP. The paper reports that $T_{\max} = 1000\ \mathrm{ms}$2 and $T_{\max} = 1000\ \mathrm{ms}$3 often disagree away from zero, that multidimensional scaling with $T_{\max} = 1000\ \mathrm{ms}$4 produces coherent branches separating model types and strengths, and that increasing training samples does not clearly improve performance measured by $T_{\max} = 1000\ \mathrm{ms}$5 (Peyrard et al., 2020). A plausible implication is that evaluation ladders are especially useful when structural correctness and decision-relevant behavior diverge.

5. Construction, maintenance, and evolution

A recurring design principle is that ladders are not static. EvoEval explicitly recommends periodic evolution: regenerate a fraction of problems each cycle with targeted transformation prompts, use self-consistency refinement by generating two solutions and refining docstrings until outputs agree on extracted tests, augment tests in an EvalPlus style, and monitor normalized per-tier scores and ranking movement across refreshes (Xia et al., 2024). This treats contamination and benchmark staleness as ongoing engineering problems rather than one-time curation problems.

LH-Bench operationalizes maintenance through versioned repositories, run databases, stored artifacts, bootstrap confidence intervals, and calibration of multiple judges in parallel for trajectory, process, and output scoring. It recommends that rubric weights be adjusted only at rubric definition time, not post hoc on scores, and that 5-point scales be preferred because they reduce ties and improve discriminative power (Chandwani et al., 24 Mar 2026). RubricsTree uses an evolving taxonomy objective: new atomic leaves are added when current leaves cannot deterministically verify what experts deem necessary for safe and useful care. Its router lowers activation thresholds for safety/emergency classes to maximize recall and raises them for narrow factual queries to maximize precision (Zhang et al., 16 Jun 2026). TCM-Ladder similarly emphasizes continual dataset updates through a public website and leaderboard, with automated and manual filtering, practitioner review, and community-facing maintenance infrastructure (Xie et al., 29 May 2025).

These implementations converge on a common operational pattern: human curation defines the ladder, automated procedures scale it, and periodic refresh preserves diagnostic power. This suggests that the ladder metaphor is inseparable from benchmark governance.

6. Misconceptions, limitations, and terminological disambiguation

A common misconception is that a top score on a widely used benchmark is sufficient evidence of broad competence. EvoEval directly challenges this view by showing that high HumanEval performance can coexist with large drops on evolved variants and with substantial ranking changes (Xia et al., 2024). A second misconception is that open-ended evaluation can be reduced to generic LLM judging without expert structure. LH-Bench reports stronger reliability for expert-authored rubrics than for LLM-authored ones, and RubricsTree shows materially stronger expert alignment than a principle-based baseline (Chandwani et al., 24 Mar 2026, Zhang et al., 16 Jun 2026). A third misconception is that lower-rung fidelity automatically guarantees higher-rung fidelity. The causal-distance framework makes the opposite point formal: observational agreement does not entail interventional or counterfactual agreement (Peyrard et al., 2020).

The literature also imposes clear limits. EvoEval does not report formal significance testing or rank-correlation coefficients (Xia et al., 2024). LH-Bench currently reports two environments, and its authors caution that weak per-run $T_{\max} = 1000\ \mathrm{ms}$6 limits fine-grained score interpretation despite aggregate convergence (Chandwani et al., 24 Mar 2026). RubricsTree notes distributional subjectivity, router under-activation risk, and the need for ongoing taxonomy governance (Zhang et al., 16 Jun 2026). TCM-Ladder identifies coverage gaps in pulse, audio, and olfactory modalities and warns against direct clinical use without professional oversight (Xie et al., 29 May 2025).

Finally, “ladder” has unrelated technical meanings that should not be conflated with evaluation ladders. In quantum optics and symbolic algebra, pyBoLaNO concerns bosonic ladder operators and normal ordering (Lim et al., 3 Jan 2025). In semi-supervised learning, the Ladder Network denotes a clean encoder, a corrupted encoder with additive Gaussian noise at every layer, and a decoder with lateral connections and denoising losses; its components were systematically analyzed in "Deconstructing the Ladder Network Architecture" (Pezeshki et al., 2015). That architecture was later applied to audio event classification in the CURE dataset with CNN embeddings, Gaussian noise $T_{\max} = 1000\ \mathrm{ms}$7, and denoising costs $T_{\max} = 1000\ \mathrm{ms}$8 (Dubey et al., 2020). In quantum field theory, “triangle ladders” and “box ladders” denote a family of massless four-point Feynman integrals whose periods evaluate to

$T_{\max} = 1000\ \mathrm{ms}$9

for the even-loop family considered (Schnetz, 2012). These usages share the word “ladder” but not the evaluation concept.

Taken together, the modern evaluation-ladder literature treats assessment as a stratified process rather than a monolithic score. Whether the target is code generation, enterprise automation, personal health advice, multimodal TCM reasoning, or causal discovery, the ladder serves to separate baseline competence from robustness, higher-order reasoning, stakeholder alignment, and decision-relevant validity.