---
title: Agentic Quality Judge
url: https://www.emergentmind.com/topics/agentic-quality-judge
type: topic
---

# Agentic Quality Judge

An Agentic Quality Judge is a multidimensional, system-level evaluation framework that operationalizes large language model (LLM) ensembles as domain-informed behavioral assessors of complex agentic systems. In the context of autonomous decision-making pipelines—such as agentic stock prediction networks—this architecture decomposes the aggregate success/failure of an agent into semantically distinct quality dimensions, leverages language models to provide per-dimension scoring, and structurally integrates this feedback into closed-loop learning protocols for sustained, behaviorally aware performance improvement. Unlike metric-centric or ad hoc evaluation schemes, the Agentic Quality Judge is engineered to yield fine-grained, actionable diagnostics supported by rigorous perturbation validation, reliability analytics, and causal linkage to downstream utilities. The paradigm fundamentally reframes agentic RL and time-series workflows by making system-level behavioral traits directly measurable, interpretable, and optimizable through standardized, reproducible LLM-based judgment [2605.05739].

## 1. Formal Structure and Dimensions of Agentic Quality Judgment

The Agentic Quality Judge framework organizes agent execution traces into five-day, non-overlapping episodes, each logging a high-dimensional tuple per step: $B_t = \langle m_t, a_t, r_t, u_t, \hat{y}_t, p_t \rangle$, capturing both raw observables (price, VIX, BERT-derived sentiment) and the full decision/control pathway (autoencoder scores, regime labels, blend weights, RL adjustments, rolling metrics). Each episode $E_t = \{B_t, ..., B_{t+4}\}$ is independently scored by an ensemble of LLM judges along six distinct, domain-specific behavioral axes:

1. **Regime Detection (RD)**: Measures threshold sensitivity and error rates in regime label assignment ($\ell_t$).
2. **Routing (RT)**: Assesses the blend between control pathways, penalizing incoherence near regime thresholds.
3. **Adaptation (AD)**: Scores the proportionality and timeliness of RL-induced action-space adjustments $(\Delta \tau_t, \Delta \alpha_t)$.
4. **Risk Calibration (RC)**: Evaluates alignment of risk thresholds relative to exogenous volatility measures.
5. **Strategy Coherence (SC)**: Checks for logical consistency across whole decision chains, flagging contradictions.
6. **Error Recovery (ER)**: Quantifies agent capacity to self-correct forecasting errors within a fixed temporal window.

Each dimension is rated on a rubric from 1 (“fundamentally flawed”) to 5 (“exemplary”), with explicit definitions and example anchors.

## 2. LLM Judge Ensemble, Score Aggregation, and Reliability

The judgment ensemble deploys three independent zero-temperature LLMs (GPT-5.4, Claude-4.6 Opus, Gemini-3.1 Pro), each prompted via system description, full episode traces (JSON), dimension-by-dimension rubric descriptions, and a schema specifying per-dimension integer scores, text justifications, and optional failure labels. For each dimension $d$, the consensus score is

\[
\bar s_d = \frac{1}{3}(s_{d,1} + s_{d,2} + s_{d,3})
\]

with the top-level composite behavioral score defined as

\[
S_\textrm{behavioral} = \frac{1}{6}\sum_{d=1}^6 \bar s_d.
\]

Empirical reliability is quantified via Krippendorff's $\alpha$ (per-dimension), with $\alpha$ ranging from 0.74 to 0.85 across dimensions (substantial reliability for three of six), demonstrating robust cross-model agreement and substantial reduction of stochastic scoring artifacts.

## 3. Dimension-Specificity, Perturbation Validation, and Predictive Power

To confirm the judge's ability to isolate failures to targeted behavioral axes, the evaluation is stress-tested using a suite of engineered perturbations, with 60 episodes per perturbation simulating controlled degradations (e.g., regime inversion, pathway blend misrouting, frozen RL controller, disabled volatility scaling, contradiction injection, error-recovery suppression). The judge ensemble exhibits mean score drops of $-$1.6 to $-2.4$ on the targeted dimension per perturbation and only $-0.32$ on average across non-target axes—a specificity amplification factor of $5$–$7\times$.

The per-dimension and composite scores show strong predictive validity: across 200 unperturbed validation episodes, Spearman's $\rho$ correlations with 20-day performance metrics are $\rho=0.72$ (composite vs. Sharpe), $\rho=0.69$ (RD vs. $1/\textrm{MAPE}$), $\rho=0.64$ (RC vs. $1/\textrm{MaxDrawdown}$), and $\rho=0.57$ (SC vs. Return), all significant at $p<0.01$. These weights are subsequently utilized as impact-sensitive dimension weights $w_d$ in closed-loop RL.

| Perturbation           | Target Dimension | Δs_target | Δs_off-target | $\alpha_K$ |
|------------------------|-----------------|-----------|---------------|------------|
| Regime inversion       | RD              | –2.4      | –0.3          | 0.85       |
| Wrong routing          | RT              | –2.1      | –0.4          | 0.81       |
| Frozen SAC             | AD              | –1.9      | –0.2          | 0.79       |
| No vol scaling         | RC              | –1.7      | –0.3          | 0.74       |
| Contradictory output   | SC              | –2.2      | –0.5          | 0.82       |
| Disabled recovery      | ER              | –1.6      | –0.2          | 0.76       |

## 4. Closed-Loop RL Fine-Tuning via Deficiency-Penalized Rewards

The Agentic Quality Judge directly modifies the per-step RL reward for the Soft Actor-Critic (SAC) controller to penalize dimension-specific deficiencies:

\[
R'_t = R_t - \lambda \sum_{d=1}^6 \max(0, \theta - \bar s_d)w_d
\]

where $\lambda=0.15$, $\theta=3$, and $w_d$ are the Spearman $\rho$-derived weights. Dimension-specific penalties are assigned to the RL subspaces most relevant to the deficient behavior:

- **RD, RC**: Adjust regime threshold ($\Delta \tau$)
- **RT**: Adjust pathway blend ($\Delta \alpha$)
- **AD, SC, ER**: Coordinated tuning ($\Delta \tau$ & $\Delta \alpha$)

Fine-tuning proceeds with short cycles confined to validation data; after three cycles, initially deficient dimensions (RC, RT, AD) are each lifted above the threshold of acceptable performance. SAC weights are frozen after the final cycle, protecting against test-set leakage.

## 5. Empirical Impact: Backtesting and Behavioral Gains

When evaluated out-of-sample on 2017–2025 held-out data, the closed-loop system driven by the Agentic Quality Judge yields statistically significant improvements:

- **1-day forecast**: MAPE reduced from 0.61% to 0.54% (11.5% rel. reduction, $p<0.001$, $d=0.31$)
- **Directional accuracy**: Increased from 71% to 74% ($p=0.002$)
- **Sharpe ratio**: Improved by 18% (bootstrap 95% CI [8.2%, 27.4%])
- **Longer horizons**: 5-day MAPE drops from 1.02% to 0.91%, 20-day MAPE from 1.52% to 1.39%
- **Regime stratification**: Largest MAPE reductions (17.3%) occur under high volatility ($\textrm{VIX} \geq 25$), indicating corrective behavioral feedback is maximally effective in challenging market conditions

## 6. Limitations, Best Practices, and Implementation Guidance

While the outlined results are restricted to offline backtesting and do not directly address deployment-specific phenomena, the Agentic Quality Judge demonstrates substantial practical benefits:

- **Specificity**: Behavioral axes are non-redundant, and dimension-irrelevant perturbations generate minimal off-target score movement.
- **Reliability**: High inter-judge agreement ($\alpha_K$ up to 0.85) validates the ensemble approach.
- **Calibrated penalty routing**: Dimension-to-action mapping ensures corrective reward signals are focused, not scattered.
- **Composite Reporting**: The weighted summary score offers a straightforward, interpretable progression metric, highly correlated with risk-adjusted returns.

Adoption of this evaluation framework requires codifying logging standards, constructing domain-anchored rubrics for each axis, scripting robust prompt templates for the LLM judges, and structuring the RL reward function to afford fine-grained credit assignment. Cycle-based fine-tuning protocols should be validated on a rolling-window split to prevent test-set leakage and statistical overfitting.

## 7. Broader Significance and Impact on Agentic System Evaluation

The Agentic Quality Judge reflects a maturing paradigm in agentic system oversight: outcome metrics are displaced by process-resolving, multidimensional behavioral diagnostics backed by high-capacity language models. Instead of treating predictive failure as an undifferentiated deficit, the system surfaces precise loci of weakness with actionable semantics. High-resolution process feedback is credit-assigned directly to RL controllers such that system policies are iteratively refined to maximize not just endpoint accuracy or return but a spectrum of verifiable behavioral traits. This approach serves as a reference architecture for scalable, interpretable, and reliable assessment and optimization of agentic pipelines in high-stakes domains [2605.05739].

Source: https://www.emergentmind.com/topics/agentic-quality-judge