Agentic Quality Judge

• Agentic Quality Judge is a multidimensional evaluation framework that decomposes agent performance into six behavioral dimensions for precise diagnostics.
• It employs an ensemble of three zero-temperature LLMs to score episodes along regime detection, routing, adaptation, risk calibration, strategy coherence, and error recovery axes.
• The framework integrates per-dimension scores into a closed-loop RL reward function, yielding significant improvements in forecasting accuracy and decision consistency.

An Agentic Quality Judge is a multidimensional, system-level evaluation framework that operationalizes LLM ensembles as domain-informed behavioral assessors of complex agentic systems. In the context of autonomous decision-making pipelines—such as agentic stock prediction networks—this architecture decomposes the aggregate success/failure of an agent into semantically distinct quality dimensions, leverages LLMs to provide per-dimension scoring, and structurally integrates this feedback into closed-loop learning protocols for sustained, behaviorally aware performance improvement. Unlike metric-centric or ad hoc evaluation schemes, the Agentic Quality Judge is engineered to yield fine-grained, actionable diagnostics supported by rigorous perturbation validation, reliability analytics, and causal linkage to downstream utilities. The paradigm fundamentally reframes agentic RL and time-series workflows by making system-level behavioral traits directly measurable, interpretable, and optimizable through standardized, reproducible LLM-based judgment (Ridhawi et al., 7 May 2026).

1. Formal Structure and Dimensions of Agentic Quality Judgment

The Agentic Quality Judge framework organizes agent execution traces into five-day, non-overlapping episodes, each logging a high-dimensional tuple per step: $$B_t = \langle m_t, a_t, r_t, u_t, \hat{y}_t, p_t \rangle$$, capturing both raw observables (price, VIX, BERT-derived sentiment) and the full decision/control pathway (autoencoder scores, regime labels, blend weights, RL adjustments, rolling metrics). Each episode $E_t = \{B_t, ..., B_{t+4}\}$ is independently scored by an ensemble of LLM judges along six distinct, domain-specific behavioral axes:

1. Regime Detection (RD): Measures threshold sensitivity and error rates in regime label assignment ($\ell_t$).
2. Routing (RT): Assesses the blend between control pathways, penalizing incoherence near regime thresholds.
3. Adaptation (AD): Scores the proportionality and timeliness of RL-induced action-space adjustments $(\Delta \tau_t, \Delta \alpha_t)$.
4. Risk Calibration (RC): Evaluates alignment of risk thresholds relative to exogenous volatility measures.
5. Strategy Coherence (SC): Checks for logical consistency across whole decision chains, flagging contradictions.
6. Error Recovery (ER): Quantifies agent capacity to self-correct forecasting errors within a fixed temporal window.

Each dimension is rated on a rubric from 1 (“fundamentally flawed”) to 5 (“exemplary”), with explicit definitions and example anchors.

2. LLM Judge Ensemble, Score Aggregation, and Reliability

The judgment ensemble deploys three independent zero-temperature LLMs (GPT-5.4, Claude-4.6 Opus, Gemini-3.1 Pro), each prompted via system description, full episode traces (JSON), dimension-by-dimension rubric descriptions, and a schema specifying per-dimension integer scores, text justifications, and optional failure labels. For each dimension $d$, the consensus score is

$$\bar s_d = \frac{1}{3}(s_{d,1} + s_{d,2} + s_{d,3})$$

with the top-level composite behavioral score defined as

$$S_\textrm{behavioral} = \frac{1}{6}\sum_{d=1}^6 \bar s_d.$$

Empirical reliability is quantified via Krippendorff's $\alpha$ (per-dimension), with $\alpha$ ranging from 0.74 to 0.85 across dimensions (substantial reliability for three of six), demonstrating robust cross-model agreement and substantial reduction of stochastic scoring artifacts.

3. Dimension-Specificity, Perturbation Validation, and Predictive Power

To confirm the judge's ability to isolate failures to targeted behavioral axes, the evaluation is stress-tested using a suite of engineered perturbations, with 60 episodes per perturbation simulating controlled degradations (e.g., regime inversion, pathway blend misrouting, frozen RL controller, disabled volatility scaling, contradiction injection, error-recovery suppression). The judge ensemble exhibits mean score drops of $-$1.6 to $E_t = \{B_t, ..., B_{t+4}\}$0 on the targeted dimension per perturbation and only $E_t = \{B_t, ..., B_{t+4}\}$1 on average across non-target axes—a specificity amplification factor of $E_t = \{B_t, ..., B_{t+4}\}$2–$E_t = \{B_t, ..., B_{t+4}\}$3.

The per-dimension and composite scores show strong predictive validity: across 200 unperturbed validation episodes, Spearman's $E_t = \{B_t, ..., B_{t+4}\}$4 correlations with 20-day performance metrics are $E_t = \{B_t, ..., B_{t+4}\}$5 (composite vs. Sharpe), $E_t = \{B_t, ..., B_{t+4}\}$6 (RD vs. $E_t = \{B_t, ..., B_{t+4}\}$7), $E_t = \{B_t, ..., B_{t+4}\}$8 (RC vs. $E_t = \{B_t, ..., B_{t+4}\}$9), and $\ell_t$0 (SC vs. Return), all significant at $\ell_t$1. These weights are subsequently utilized as impact-sensitive dimension weights $\ell_t$2 in closed-loop RL.

Perturbation	Target Dimension	Δs_target	Δs_off-target	$\ell_t$3
Regime inversion	RD	–2.4	–0.3	0.85
Wrong routing	RT	–2.1	–0.4	0.81
Frozen SAC	AD	–1.9	–0.2	0.79
No vol scaling	RC	–1.7	–0.3	0.74
Contradictory output	SC	–2.2	–0.5	0.82
Disabled recovery	ER	–1.6	–0.2	0.76

4. Closed-Loop RL Fine-Tuning via Deficiency-Penalized Rewards

The Agentic Quality Judge directly modifies the per-step RL reward for the Soft Actor-Critic (SAC) controller to penalize dimension-specific deficiencies:

$\ell_t$4

where $\ell_t$5, $\ell_t$6, and $\ell_t$7 are the Spearman $\ell_t$8-derived weights. Dimension-specific penalties are assigned to the RL subspaces most relevant to the deficient behavior:

• RD, RC: Adjust regime threshold ($\ell_t$9)
• RT: Adjust pathway blend ($(\Delta \tau_t, \Delta \alpha_t)$0)
• AD, SC, ER: Coordinated tuning ($(\Delta \tau_t, \Delta \alpha_t)$1 & $(\Delta \tau_t, \Delta \alpha_t)$2)

Fine-tuning proceeds with short cycles confined to validation data; after three cycles, initially deficient dimensions (RC, RT, AD) are each lifted above the threshold of acceptable performance. SAC weights are frozen after the final cycle, protecting against test-set leakage.

5. Empirical Impact: Backtesting and Behavioral Gains

When evaluated out-of-sample on 2017–2025 held-out data, the closed-loop system driven by the Agentic Quality Judge yields statistically significant improvements:

• 1-day forecast: MAPE reduced from 0.61% to 0.54% (11.5% rel. reduction, $(\Delta \tau_t, \Delta \alpha_t)$3, $(\Delta \tau_t, \Delta \alpha_t)$4)
• Directional accuracy: Increased from 71% to 74% ($(\Delta \tau_t, \Delta \alpha_t)$5)
• Sharpe ratio: Improved by 18% (bootstrap 95% CI [8.2%, 27.4%])
• Longer horizons: 5-day MAPE drops from 1.02% to 0.91%, 20-day MAPE from 1.52% to 1.39%
• Regime stratification: Largest MAPE reductions (17.3%) occur under high volatility ($(\Delta \tau_t, \Delta \alpha_t)$6), indicating corrective behavioral feedback is maximally effective in challenging market conditions

6. Limitations, Best Practices, and Implementation Guidance

While the outlined results are restricted to offline backtesting and do not directly address deployment-specific phenomena, the Agentic Quality Judge demonstrates substantial practical benefits:

• Specificity: Behavioral axes are non-redundant, and dimension-irrelevant perturbations generate minimal off-target score movement.
• Reliability: High inter-judge agreement ($(\Delta \tau_t, \Delta \alpha_t)$7 up to 0.85) validates the ensemble approach.
• Calibrated penalty routing: Dimension-to-action mapping ensures corrective reward signals are focused, not scattered.
• Composite Reporting: The weighted summary score offers a straightforward, interpretable progression metric, highly correlated with risk-adjusted returns.

Adoption of this evaluation framework requires codifying logging standards, constructing domain-anchored rubrics for each axis, scripting robust prompt templates for the LLM judges, and structuring the RL reward function to afford fine-grained credit assignment. Cycle-based fine-tuning protocols should be validated on a rolling-window split to prevent test-set leakage and statistical overfitting.

7. Broader Significance and Impact on Agentic System Evaluation

The Agentic Quality Judge reflects a maturing paradigm in agentic system oversight: outcome metrics are displaced by process-resolving, multidimensional behavioral diagnostics backed by high-capacity LLMs. Instead of treating predictive failure as an undifferentiated deficit, the system surfaces precise loci of weakness with actionable semantics. High-resolution process feedback is credit-assigned directly to RL controllers such that system policies are iteratively refined to maximize not just endpoint accuracy or return but a spectrum of verifiable behavioral traits. This approach serves as a reference architecture for scalable, interpretable, and reliable assessment and optimization of agentic pipelines in high-stakes domains (Ridhawi et al., 7 May 2026).