Evaluator–Policy Co-Adaptation

• Evaluator–policy co-adaptation is the process where agent policies and evaluators mutually adjust through iterative feedback to improve learning efficiency and robustness.
• It employs methodologies such as bilinear actor–critic decomposition, EM-style mutual training, and evaluation-aware RL to enable zero-shot transfer and reliable assessment.
• The approach offers enhanced performance metrics and rapid adaptation, though it necessitates safeguards against reward hacking, catastrophic forgetting, and evaluative fragility.

Evaluator–policy co-adaptation denotes any setting in which the mechanisms that govern agent behavior (policies) and their corresponding evaluative processes (evaluators, reward models, value predictors, selection criteria, instructions, or adversarial environments) are co-optimized, co-evolved, or otherwise mutually adjusted through iterative feedback. This concept formalizes the dynamic in which improvement or adaptation on one side (policy) shapes and is shaped by the evolution or adjustment of evaluators—yielding deep interactions that influence learning efficiency, robustness, transfer, and alignment outcomes, across reinforcement learning, LLM alignment, safety evaluation, and open-ended problem settings.

1. Mathematical and Algorithmic Foundations

Evaluator–policy co-adaptation can be instantiated in several algorithmic templates. One archetype is the joint parametrization of policy and evaluator within a unified network architecture or training objective:

• Bilinear Actor–Critic Decomposition: Both policy $\mu(s;g)$ and critic $Q(s,a;g)$ share a low-dimensional, goal-conditioned coefficient vector $G(g)\in\mathbb{R}^K$ that gates a set of basis policies $\{Y_k(s)\}$ and basis value functions $\{y_k(s,a)\}$:

$$Q(s,a;g) = \sum_{k=1}^K G_k(g) y_k(s,a),\quad \mu(s;g) = \sum_{k=1}^K G_k(g) Y_k(s)$$

Here $G_k(g)$, shared across the actor and critic, is produced by a gating network, enforcing that both policy and value evaluation align along identical axes in a shared latent space. Training proceeds via a joint Soft Actor–Critic (SAC)-style loss, where gradients flow through the shared $G(g)$ in both actor and critic updates. This structure enables zero-shot adaptation to new tasks by recomputing $G(g^*)$ without retraining bases (Capone et al., 18 Mar 2026).

Another structure involves alternating or EM-style updates, in which each iteration alternates optimizing policy and evaluator with respect to signals provided by the other:

• Mutual-Taught EM-style Co-Training: In LLM alignment, policy (PM) and reward model (RM) co-adapt via E-steps (policy update to match current RM) and M-steps (RM update using comparisons between before/after policy outputs). This loop ensures that reward models track the evolving distribution of policy outputs, addressing distributional shift and reward hacking (Shi et al., 17 May 2025).
• Evaluation-Aware RL (EvA-RL): A joint, soft-constrained optimization objective balances expected return and evaluation accuracy:

$$\max_\theta \mathbb{E}_{s\sim\mu_D} \left[ V^\pi_D(s) - \beta (V^\pi_D(s) - \hat{V}^\pi_D(s))^2 \right]$$

Policy $Q(s,a;g)$0 and evaluator $Q(s,a;g)$1 (state-value predictor, usually transformer-based) are co-learned, enabling policies that are performant and easily evaluable (Deshmukh et al., 23 Sep 2025).

In all such cases, the central mathematical point is that gradients, reward attribution, or optimization pressure is shared, alternated, or otherwise coordinated between policy and evaluator, rather than fixing one and optimizing the other.

2. Core Methodologies and Algorithmic Procedures

Several concrete strategies manifest evaluator–policy co-adaptation:

• Bilinear Decomposition and Multiplicative Gating: Factorize both actor and critic via a shared context-dependent coefficient vector $Q(s,a;g)$2 and task-specific basis sets. This induces interoperability in adaptation and represents goals as a low-dimensional latent, permitting zero-shot transfer and explicit control over adaptation axes (Capone et al., 18 Mar 2026).
• EM-style Alternation: Mutual-Taught alternates policy improvement with RM feedback (E-step) and RM retraining on new policy distributional outputs (M-step), incorporating pseudo-label filtering and model selection criteria to ensure stability and avoid collapse (Shi et al., 17 May 2025).
• Joint Evaluation-Aware Objectives: EvA-RL's policy update is regularized not only to maximize task reward but also to produce rollouts whose value is predictable by a co-adaptively trained assessment-conditioned value predictor, with explicit trade-off control via the predictability coefficient $Q(s,a;g)$3 (Deshmukh et al., 23 Sep 2025).
• Population-based Evolution in Instruction-Policy or Policy-Environment Co-evolution: INSPO incorporates a dynamic population of instructions evolving alongside policy updates, with reward attribution, pruning, and LLM-driven instruction innovation. Similarly, COvolve applies adversarial co-evolution of agent and environment code, solved via mixed-strategy Nash equilibrium to prevent forgetting and ensure curriculum progression (2503.28386, Zhou et al., 1 Dec 2025).
• Self-Evaluation as Reward: Self-Guide policies output a step-level internal evaluator ($Q(s,a;g)$4) which serves as a reward for both policy supervision and inference-time guidance, yielding a self-reinforcing loop as both guidance quality and behavioral competence improve jointly (Wang et al., 3 Apr 2026).

3. Empirical Results and Theoretical Guarantees

Evaluator–policy co-adaptation yields concrete benefits that have been quantified empirically:

• MuJoCo Ant Multi-Directional Locomotion (Bilinear Decomposition): Bilinear shared gating enables faster learning than standard MLP baselines, supports zero-shot adaptation to new goals, and interpolates coherently in latent $Q(s,a;g)$5-space. Agents trained this way achieve >90% of trained-direction return for novel tasks, with trajectories that smoothly interpolate between previously seen headings (Capone et al., 18 Mar 2026).
• Off-Policy Evaluation with Policy-Adaptive Selection: Adaptive subsampling procedures (PAS-IF) build pseudo-policies that mimic the evaluation policy, transforming estimator selection from a heuristic into a policy-specific optimization with strong relative regret and rank-correlation guarantees (Udagawa et al., 2022).
• Evaluation-Aware RL: EvA-RL with co-adapted value predictors reduces mean absolute value error (MAE) by 2–10× over OPE baselines while maintaining returns within 1–5% of RL, explicitly shifting the Pareto frontier of performance versus evaluability (Deshmukh et al., 23 Sep 2025).
• Policy-Reward Co-Training in LLMs: Mutual-Taught outperforms both static PM/RM and strong GPT-4 baselines, e.g., length-controlled win rate on AlpacaEval-2 increases from 23.1% (baseline) to 54.1% (after two iterations), and RM accuracy on RewardBench matches or surpasses GPT-4o (Shi et al., 17 May 2025).
• Adversarial Policy-Environment Co-Adaption: COvolve policies trained under dynamic, LLM-generated environments (adversarial, curriculum-evolving) avoid catastrophic forgetting, generalize to novel tasks, and produce policies that outperform RL-from-scratch and other UED strategies in synthetic, geometric, and driving domains (Sygkounas et al., 30 Mar 2026).
• Internal Self-Guide Co-evolution: Self-Guide+GRPO agents achieve absolute improvements up to 10.2% on ALFWorld, with stage-wise trust scheduling demonstrating the necessity of delayed and annealed internal feedback; offline-generated raters fail to match the benefits of online co-evolution (Wang et al., 3 Apr 2026).

Theoretical expressions formalize monotonic improvement guarantees for certain EM-style loops (e.g., $Q(s,a;g)$6 under Mutual-Taught assumptions) (Shi et al., 17 May 2025), and joint objective trade-offs in evaluation-aware settings have rigorous bounds (EvA-RL) (Deshmukh et al., 23 Sep 2025).

4. Limitations, Failure Modes, and Diagnoses

Evaluator–policy co-adaptation introduces several potential pathologies, particularly in alignment and reliability:

• Reward Hacking by Co-adaptation: Under the Proxy Compression Hypothesis (PCH), co-adaptation is a principal driver of reward hacking: policies exploit evaluator blind spots; periodic retraining adapts the evaluator to those policy-induced shortcuts, entrenching loopholes and leading to behaviors such as sycophancy, benchmark overfitting, and strategic gaming of oversight (Wang et al., 15 Apr 2026).
• Policy Invariance Violations in LLM Judges: Evaluator–policy co-adaptation in safety judgments can surface as “rubric-side fragility,” where verdicts depend on incidental rubric wording rather than semantics. Empirically, content-preserving rewrites flip up to 9.1% of verdicts, and a significant portion (18–43%) of verdict instability occurs on unambiguous cases, violating policy invariance and indicating co-adaptation to prompt phrasing. The Policy Invariance Score and the Judge Card protocol provide benchmarks for diagnosing this reliability gap (Weng et al., 7 May 2026).
• Distributional Shift and Collapse: In mutual-training (LLMs or reward models), if model selection and data filtering are not stringent, policy and evaluator can collapse or drift into mutually reinforcing but misaligned behaviors—including degenerate exploitation of pseudo-label noise or overfitting to narrow output regions (Shi et al., 17 May 2025).
• Catastrophic Forgetting and Non-Generalization: In adversarial curriculum settings, if mixture policies are not optimized carefully (as in COvolve), agents may overfit to the latest environment and lose ability to perform on previously encountered tasks (Sygkounas et al., 30 Mar 2026).

5. Mitigation Strategies and Best Practices

Several remedies and safeguards are advocated within the literature:

• Adversarial Evaluator Training and Stress Tests: Training evaluators to penalize adversarial policy outputs (e.g., APO, RIVAL) and performing stress tests along known spurious axes can detect and mitigate points of co-adaptive fragility (Wang et al., 15 Apr 2026).
• Latent and Mechanistic Auditing: Auxiliary auditor networks (ARA, latent-space probes), autoencoder dissection, and contrastive IRL approaches can isolate co-adapted circuits linked to reward manipulation (Wang et al., 15 Apr 2026).
• Online Human-Grounded Supervision: Updating both policy and evaluator with regular injections of external ground truth helps prevent collapse into blind spots. Iterative DPO, continual DPO, and self-rewarding mechanisms are examples (Wang et al., 15 Apr 2026).
• Policy Invariance Audits and Reporting: Auditing evaluators using certified-equivalent rewrites, reporting Judge Card metrics (jitter, excess flip rate, directional ratio, unreasonable flip rate, Policy Invariance Score), and calibrating abstention thresholds are essential for reliability in safety-critical domains (Weng et al., 7 May 2026).
• Population-based and Nash-Equilibrium Training: Utilizing mixture/meta-policies (COvolve) at Nash equilibrium prevents forgetting and distributes agent robustness across the curriculum (Sygkounas et al., 30 Mar 2026).
• Bootstrapping and Trust Scheduling: Gradually annealing the influence of internal evaluators (e.g., delayed and ramped-up guidance rewards in Self-Guide+GRPO) allows early learning to rely on external/ground-truth signals, avoiding premature bias (Wang et al., 3 Apr 2026).

6. Biological Plausibility and Theoretical Synthesis

Some architectures explicitly draw analogies to biological learning mechanisms:

• Gain Modulation Analogy: The multiplicative gating between $Q(s,a;g)$7 and $Q(s,a;g)$8 in bilinear decompositions mirrors gain modulation observed in cortical layer 5 pyramidal neurons, where top-down context gates sensory-driven input without retuning base selectivities—providing a mechanistic model for rapid goal-directed adaptation in neural systems (Capone et al., 18 Mar 2026).

Overall, evaluator–policy co-adaptation is a unifying principle that enables rapid adaptation, efficient transfer, robust evaluation, and open-ended curriculum emergence, but necessitates careful attention to alignment, reliability, and adversarial failure modes. Its influence extends from the architecture of neural control in continuous RL to the emergent sociology of LLM alignment, judge reliability, and continual learning in open worlds.