AlphaEval: Multi-Domain Evaluation Framework

• AlphaEval is a term for distinct evaluation frameworks that rigorously assess quantitative models, AI agent performance, and running coupling analyses with domain-specific metrics.
• In quantitative finance, it delivers a backtest-free, parallelizable evaluation of formula alpha mining systems by measuring predictive power, temporal stability, robustness, financial logic, and diversity.
• For AI systems, AlphaEval benchmarks complete agent products using a production-grounded, multi-paradigm approach that integrates rubric-based, execution, and LLM-assisted verifications.

Searching arXiv for papers on “AlphaEval” and closely related evaluation frameworks to ground the article. AlphaEval is used in several technically distinct research contexts. In quantitative finance, it denotes a unified, parallelizable, and backtest-free framework for evaluating formula alpha mining models along five complementary dimensions: predictive power, stability, robustness to market perturbations, financial logic, and diversity (Ding et al., 10 Aug 2025). In AI systems, it names a production-grounded benchmark of 94 tasks from seven companies for evaluating complete agent products across six O*NET domains (Lu et al., 14 Apr 2026). In a separate reconstructed exposition of the running fine-structure constant, “AlphaEval” appears as an Editor’s term for a step-by-step evaluation procedure for $\alpha(E)$ via the Adler function, rather than as the original name of Jegerlehner’s method (0807.4206). This suggests that “AlphaEval” functions less as a single canonical framework than as a recurring label for domain-specific evaluation infrastructures centered on an $\alpha$-quantity or on alpha-producing systems.

1. Nomenclature and scope

The literature represented here assigns the name to multiple non-equivalent objects. The distinctions are methodological, not merely terminological.

Usage	Object evaluated	Defining emphasis
AlphaEval in formula alpha mining	Formula alpha mining models and generated alphas	Backtest-free, five-dimensional evaluation
AlphaEval in agent evaluation	Complete agent products in production-like tasks	Production-grounded, multi-paradigm benchmark
“AlphaEval” (Editor’s term) in running $\alpha(E)$ analysis	Determination of the running fine-structure constant	Adler-function-monitored evaluation of $\alpha(M_Z)$
AlphaEval-style evaluation systems	LLM evaluation workflows	Planning, benchmark resolution, schema normalization, reporting

The first two are explicit paper titles, whereas the third is a retrospective editorial label attached to a specific evaluation recipe for $\Delta\alpha_{\mathrm{had}}^{(5)}(M_Z^2)$ and $\alpha(M_Z)$ (Ding et al., 10 Aug 2025, Lu et al., 14 Apr 2026, 0807.4206, Shen et al., 10 Mar 2026). A common misconception is to treat AlphaEval as a single benchmark family; the available sources instead document separate frameworks with different objects, metrics, and epistemic goals.

2. AlphaEval in quantitative finance

In quantitative equity investment, AlphaEval is a framework for assessing formula alpha mining systems that generate symbolic expressions $\alpha_i(\cdot)$ from panel data $X \in \mathbb{R}^{T \times N \times F}$ and future returns $y \in \mathbb{R}^{T \times N}$. A formula alpha produces scalar scores

$$S^{(i)}_{t,n} = \alpha_i\big(X_{t-L^{(i)}+1:t,n,:}\big),$$

and the central problem is no longer alpha generation alone but model-level evaluation across very large candidate sets (Ding et al., 10 Aug 2025).

The framework is motivated by limitations of two established practices. Backtesting is computationally intensive, inherently sequential, and highly sensitive to portfolio construction rules, rebalancing choices, and transaction-cost assumptions. Correlation-based screening through IC or RankIC is efficient and parallelizable, but it measures predictive power only and neglects temporal stability, robustness, diversity, and interpretability. AlphaEval is proposed as a backtest-free alternative that evaluates alphas directly from score matrices and market data without simulating portfolio NAV.

Its architecture is model-agnostic. A mining model—genetic programming, reinforcement learning, GAN-based generation, or LLM-based generation—produces a set of alphas, optionally followed by an alpha combination model such as linear regression, LightGBM, or XGBoost. AlphaEval then computes five dimensions: predictive power, temporal stability, robustness to market perturbations, financial logic, and diversity. Scores are aggregated across alphas to characterize the generator itself rather than only isolated factors.

The framework is explicitly parallelizable because its computations operate on cross-sectional panels $\alpha$0, $\alpha$1, or on cross-alpha covariance structures, with no dependency from $\alpha$2 to $\alpha$3 as in portfolio backtests. In experiments, using 20 processes, it reduces evaluation time by more than 25% relative to a backtesting-based system while delivering richer diagnostics.

3. Metric system and empirical profile in formula alpha mining

AlphaEval’s predictive component is the Predictive Power Score,

$\alpha$4

with $\alpha$5 in the reported experiments. This combines cross-sectional Pearson correlation with rank-based Spearman correlation, preserving compatibility with established IC-style screening while softening dependence on strictly linear structure (Ding et al., 10 Aug 2025).

Temporal stability is measured by Relative Rank Entropy,

$\alpha$6

where ranks are normalized into probability distributions before KL comparison. Robustness is measured by the Perturbation Fidelity Score, defined from the Spearman correlation between original and perturbed alpha scores under both Gaussian noise and Student’s $\alpha$7 noise with $\alpha$8, with the conservative aggregate

$\alpha$9

Interpretability is supplied by an LLM-based Logic Score using GPT-4o as a financial judge, and diversity is quantified by Diversity Entropy,

$\alpha(E)$0

where $\alpha(E)$1 is the normalized eigenvalue spectrum of the alpha covariance matrix.

The empirical results are explicitly multi-objective. On A-share data, AlphaAgent has the highest Predictive score at 0.041, AlphaGen leads Stability at 0.978, AlphaGen and AlphaQCM lead Robustness at 0.997 and 0.996, AutoAlpha has the highest Diversity at 0.946, and AlphaAgent has the highest Logic score at 70.0. On S&P 500, AlphaAgent again leads Predictive at 0.025 and Logic at 71.5, AlphaQCM leads Stability at 0.995 and Robustness at 0.994, and AlphaForge leads Diversity at 0.994. The reported interpretation is not that one mining paradigm dominates absolutely, but that LLM-based methods exhibit the strongest predictive–logic trade-off while RL-based methods excel in stability and robustness.

The framework is also empirically connected to downstream trading behavior. RRE shows a strong negative linear relationship with annualized turnover, with regression $\alpha(E)$2. High-PFS alphas exhibit significantly lower maximum drawdown, with reported statistical significance around $\alpha(E)$3. In alpha-selection experiments, the integrated AlphaEval score yields better realized portfolios than ranking by PPS, RRE, PFS, or Logic alone. A frequent misunderstanding is therefore that AlphaEval merely re-labels IC screening; in the published formulation, its purpose is precisely to move beyond single-metric predictive evaluation.

4. AlphaEval as a production-grounded benchmark for agents

A second, unrelated AlphaEval is a benchmark for evaluating agents in production. It comprises 94 tasks sourced from seven companies deploying AI agents in their core business and spans six O*NET domains: Human Resources, Finance & Investment, Procurement & Operations, Software Engineering, Healthcare & Life Sciences, and Technology Research (Lu et al., 14 Apr 2026).

Its central methodological claim is that production evaluation should target complete agent products rather than base models. The benchmark therefore evaluates 14 model–scaffold configurations across six frontier LLMs—Claude Opus 4.6, GPT-5.2, Gemini 3 Pro Preview, Kimi K2.5, GLM-5, and MiniMax M2.5—using commercial systems such as Claude Code, Codex, GitHub Copilot, and Cursor. The same model can vary substantially across scaffolds: for Claude Opus 4.6, reported overall scores range from 64.41 via Claude Code to 53.45 via Codex; for GPT-5.2, they range from 54.91 via GitHub Copilot to 39.47 via Claude Code.

The tasks differ sharply from retrospectively curated research benchmarks. Requirements are under-specified, constraints are often implicit, inputs are heterogeneous and fragmented across PDFs, spreadsheets, images, markdown, YAML, and code, and outputs are long-horizon professional deliverables rather than short answers. Examples include resume screening, segment research and pitch critique, BOM cost optimization, full-stack or mini-program app generation, eCRF visit window calculation, insurance policy analysis, and deep technology-industry research.

Scoring is compositional. Per-task scores are defined as

$\alpha(E)$4

domain scores average task scores within a domain and scale by 100, and the overall benchmark score averages the six domain scores equally. This equal-domain weighting prevents large domains from dominating the aggregate.

5. Construction framework, evaluation paradigms, and benchmark results

The production AlphaEval contributes not only a benchmark but a requirement-to-benchmark construction framework with four stages: Partner Engagement, Requirement Elicitation, Task Formalization, and Iterative Validation (Lu et al., 14 Apr 2026). This workflow begins from authentic company requirements rather than archival artifacts. Over roughly one month per company, the process involves workflow discovery, scope negotiation, and ground-truth co-construction. Task formalization uses a standard package schema centered on query.md, task.yaml, raw files/, and .eval/rubric.py, with optional ground_truth.json.

Evaluation is explicitly multi-paradigm. The framework combines reference answer verification, formal logic verification, rubric-based evaluation, execution-based verification, and LLM-as-a-Judge. Each domain usually composes at least two evaluation types per task, with a benchmark-wide average of 2.8 and up to 3.8 for Procurement. The Human Resources domain relies on fuzzy and exact matching; Finance & Investment combines rubrics, semantic evaluation, and structural checks; Procurement uses fuzzy and exact matching, unit tests, mathematical checks, and LLM evaluation; Software Engineering uses environment-state and functional verification; Healthcare & Life Sciences combines rubric-based, LLM, mathematical, code, and matching-based checks; Technology Research uses rubrics, LLM semantic scoring, and factual verification.

The reported best overall configuration is Claude Code plus Claude Opus 4.6 at 64.41, followed by Cursor plus Claude Opus 4.6 at 61.85 and GitHub Copilot plus Claude Opus 4.6 at 61.31. No configuration exceeds 90 in any domain. Human Resources is particularly difficult, with a top score of 38.91, and Healthcare & Life Sciences is constrained by zero-tolerance numerical and protocol tasks, with a best score of 50.06. The benchmark also attaches human-labor estimates to tasks, totaling 2,420 hours and $\alpha(E)$5231K across the 94 tasks, and derives configuration-specific value estimates through domain-weighted aggregation.

The LLM-as-a-Judge component is itself meta-evaluated. Across 1,000 rubric points, agreement between two human experts is 84.7% with Cohen’s $\alpha(E)$6, while agreement between the automated judge and each human is 85.0% and 89.7%, with $\alpha(E)$7 and $\alpha(E)$8 respectively; the three-way Fleiss’ $\alpha(E)$9 is 0.720. The paper interprets this as comparable human–human and human–LLM agreement, while also noting familiar evaluator biases such as mild generosity.

6. Related usage, comparative context, and recurring misconceptions

The term also appears indirectly in adjacent work. One-Eval is an agentic system for automated and traceable LLM evaluation that converts natural-language requests into executable workflows through NL2Bench, BenchResolve, and a metric recommendation and reporting stack. Its discussion repeatedly frames these design choices as relevant to “AlphaEval-style” frameworks, particularly with respect to intent parsing, automatic dataset acquisition, schema normalization, task-aware metric selection, and evidence logging (Shen et al., 10 Mar 2026). This does not make One-Eval an AlphaEval system in title, but it places it in the same design space of end-to-end evaluation orchestration.

A second adjacent case arises in AlphaEvolve. That paper states that it “suggests—implicitly rather than by name—a blueprint for an AlphaEval framework” for coding and discovery agents, emphasizing multi-domain task suites, exact correctness checks, train/test separation, robustness across seeds, and explicit accounting of sample and compute efficiency (Novikov et al., 16 Jun 2025). Here again, AlphaEval is not the paper’s title, but rather a conceptual evaluation pattern induced by the methodology.

Several misconceptions recur across these literatures. First, AlphaEval is not a single benchmark lineage: the finance framework, the production-agent benchmark, and the Editor’s term attached to Jegerlehner’s running-$\alpha(M_Z)$0 evaluation are methodologically unrelated. Second, the finance AlphaEval is not presented as a replacement for all backtesting; it is described as a powerful, cheap filter and diagnostic tool that still leaves room for occasional full strategy validation (Ding et al., 10 Aug 2025). Third, the production AlphaEval does not benchmark bare models; its unit of analysis is the agent product configuration, including scaffold, tools, and environment (Lu et al., 14 Apr 2026). Fourth, in the running-coupling context, “AlphaEval” is editorial shorthand for a stepwise determination of $\alpha(M_Z)$1 and $\alpha(M_Z)$2 via an Adler-function-monitored strategy, not the published name of the original physics analysis (0807.4206).

Taken together, these usages define AlphaEval as a family resemblance term for evaluation frameworks whose common property is methodological rigor under domain-specific constraints. In quant finance, that rigor is backtest-free multidimensional assessment of formula alphas; in agent systems, it is production-grounded, multi-paradigm benchmarking of end-to-end products; and in the Adler-function setting, it is a carefully staged evaluation of the running electromagnetic coupling.