A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents

Abstract: Measurements of proprietary LLM evaluators can become invalid within weeks -- we document one case and provide the diagnostic framework to detect it. We introduce EPC -- comprising the Multimodal Preference Collapse Index (MPCI), evaluator-indexed coupling matrix, and Jensen-Shannon divergence (JSD) -- and apply it across eight experimental conditions (N=112 main + N=10 ablation = 122 unique repetitions, all reported). Coupling coefficients range from 0.00 to 1.18 across per-condition means (CV approx 0.9, n=8 conditions). Four conditions show strong coupling (N=36; GPT-4o May, GPT-4o-mini, Qwen3.7-plus, DashScope 30r); four collapse to near-zero (N=76; GPT-4o June, qwen-plus N=30, symmetric LR, DeepSeek self-eval). The May-to-June GPT-4o drift -- an N=8 re-replication inverting the study's conclusion -- is the most informative measurement: a diagnostic instrument detecting its own instability demonstrates the fragility it was designed to measure. Self-evaluation (97% zero, JSD=0.003) consistently collapses, though floor effects are possible. Output-format confound analysis finds per-strategy aggregate rho=0.89 but per-instance rho=0.219 (p=0.093); PCI reported as preference-convergence metric. We release EPC with all data. The finding is not any single coupling magnitude but the pattern of version-conditional instability that makes single-snapshot evaluator studies unreliable.

• The paper introduces the EPC framework to quantify evaluator-driven preference collapse in self-adapting LLM agents.
• It employs a multi-evaluator, multi-protocol empirical audit (N=122 experiments) demonstrating significant version-conditional instability.
• The study highlights practical recommendations for calibrating evaluator performance and emphasizes routine multi-version audits.

A Diagnostic Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents

Introduction

This paper introduces the Evaluator Preference Collapse (EPC) framework—a metric system and auditing protocol explicitly targeting evaluator-driven dynamics in closed-loop, self-adapting LLM agents. The central insight is rigorous: evaluator-induced agent adaptation is neither stable nor evaluator-invariant but instead highly conditional on both model version and feedback protocol. Through a multi-evaluator, multi-protocol empirical audit ($N=122$ experiments), the study demonstrates that coupling phenomena—agents adapting their behavioral distribution to evaluator preferences—are strongly evaluator-conditional and subject to complete reversal via unheralded version drift. The EPC framework enables the detection, quantification, and systematic study of these coupling and collapse phenomena.

The EPC Diagnostic Framework

EPC quantifies preference-propagation using several metrics:

• Preference Collapse Index (PCI): Measures the relative concentration of the agent’s strategy weight vector, invariant to the number of strategies.
• Multimodal PCI (MPCI): Aggregates PCI across modalities and quantifies domain transfer via the cross-modal PCI (CPCI) component.
• Jensen-Shannon Divergence (JSD): The primary bounded metric for comparing strategy distributions.
• Evaluator-indexed coupling matrix ($\Gamma^{(\mathcal{J})}$): Captures adaptation/coupling between strategy spaces under different modality-evaluator pairs.
• Coupling coefficient ($\gamma_{A \to B}$): Quantifies the deviation after adaptation from source (A) to target (B) modality relative to the baseline.
• ECE and Brier: Measures of calibration between evaluator preference and ground-truth accuracy.

The agent uses parameter-free, test-time reinforcement learning (TTRL) in a stochastic bandit setting, updating 11-dimensional strategy distributions over rounds. Empirical coupling is induced through repeated adaptation against evaluators including GPT-4o (OpenAI), Qwen3.7-plus, and DeepSeek self-evaluation.

Figure 1: Strategy weights from an $N=1$ exploratory run; visual-strategy allocation demonstrates non-trivial coupling, with 9.1% to visual.

Experimental Audit and Findings

Eight principal experimental conditions were audited, covering multiple evaluators, input modalities, and strategy adaptation protocols. A central empirical finding is version-conditional instability: when OpenAI's GPT-4o evaluator was re-run just four weeks after the initial study, the coupling metric $\gamma$ collapsed from strong values (~1.2) to zero, inverting the main claim. Under the May 2026 snapshot, all GPT-4o cross-modal adaptation runs produced high coupling magnitudes, while the June 2026 snapshot identically repeated produced none.

This version-conditional pattern is recapitulated in the cross-evaluator results: four of eight conditions show strong coupling (notably GPT-4o May, GPT-4o-mini, Qwen3.7-plus, DashScope), while four demonstrate collapse (GPT-4o June, qwen-plus, symmetric LR, DeepSeek self-eval).

Figure 2: PCI comparison; GPT-4o cross-model exhibits high preference-convergence (1.464), vastly exceeding self-evaluation (0.461) and random baseline (0.716).

Crucially, the high variability of coupling magnitudes (coefficient of variation ≈ 0.9 across the eight conditions) demonstrates that treating any individual evaluator's behavior as indicative or stable is unwarranted. The per-experiment sample size ($N$) influences these outcomes: small batches (8–10 runs) are prone to showing substantial coupling, while larger batches collapse, suggesting that power analyses must be subsumed by the measurement protocol's sensitivity to both evaluator drift and sample size.

Analysis of Format Effects and Calibration

A dedicated analysis of the output-format confound revealed only weak evidence for output length driving strategy preference at the per-instance level (Spearman $\rho$ = 0.219, $p=0.093$), even though aggregate association is high ($\rho \approx 0.89$) due to the dominance of strategies like step_by_step. This supports the interpretation that evaluator preferences propagate in more ways than just verbosity selection, and that PCI captures a real, not merely format-induced, preference dynamic.

Self-evaluation experiments demonstrate another critical limitation: DeepSeek-chat self-eval shows concentrated strategy allocation (PCI ≈ 0.74), yet evaluator preferences are effectively uncalibrated with respect to actual accuracy (ECE ≈ 0.31). The 97% zero-coupling rate thus likely reflects a floor effect—i.e., evaluator incapacity—rather than real evaluation stability.

Theoretical and Practical Implications

The theoretical ramification is clear: evaluator-as-judge studies are fragile measurement instruments. Any finding about behavior adaptation is conditional on the current snapshot of proprietary black-box models, whose implicit utility functions are subject to unannounced drift. This undermines the reliability of downstream agent training and aligns with concurrent literature on the fragility and instability of LLM-based evaluation heuristics. Furthermore, it demonstrates that claims about reward overoptimization, bias propagation, or "optimal" prompt engineering cannot be decoupled from the versioning history of the evaluator.

From a practical perspective, the EPC framework provides a minimal, transparent, and reproducible dependency for regular auditing. The authors make specific recommendations: always report evaluator identity and version, baseline against multiple evaluator families, monitor for version drift, and explicitly calibrate self-evaluation metrics with external ground-truth benchmarks.

Limitations and Future Work

The study is limited by its primary focus on text-proxied visual tasks, a fixed executor model (DeepSeek-chat), and a narrow window of evaluation (May–June 2026). The approach does not establish causal attribution for drift (whether executor or evaluator), and the cognitive definition of agent strategies is not independently validated. Proposed future experiments include evaluating open-weight baselines (Llama), N-controlled replication, and robustness to length normalization and input modality.

Conclusion

This paper establishes that measurements of LLM evaluator-driven agent preference adaptation are frequently invalidated within weeks due to unheralded evaluator version drift. The EPC framework empirically grounds the measurement of coupling and collapse phenomena, supporting the claim that the future development of reliable evaluation pipelines for LLM-agent systems requires routine, multi-evaluator, version-aware audits and careful calibration protocols.