LWE: Adaptive Evaluation Framework

• LWE is a dynamic framework that adapts evaluation prompts through self-generated feedback to enhance language model judgment consistency.
• It employs a sequential update mechanism that accumulates evaluation heuristics and corrects biases without relying on external training data.
• Selective LWE targets inconsistent cases to reduce computational loads while maintaining or improving evaluation accuracy.

Learning While Evaluating (LWE) is a framework for automatic evaluation in LLM alignment and reasoning tasks that enables evaluators—typically LLMs—to sequentially improve their evaluation process at test time by accumulating domain- or task-specific experience. Unlike classical approaches that employ fixed prompts and treat each evaluation instance independently, LWE utilizes an evolving meta-prompt mechanism and self-generated feedback to refine evaluation criteria dynamically, resulting in enhanced judgment consistency and adaptability, all achieved without external training or validation data. A selective-update variant, Selective LWE, focuses computational resources on cases identified as difficult by self-inconsistency, thereby economizing inference cost while sustaining or surpassing full-update performance (Jwa et al., 7 Dec 2025).

1. Motivation and LLM-as-Evaluator Scenario

LLMs are widely used as automated evaluators for MMLU, open-ended text, and multimodal data. Given a test set $\mathcal{D} = \{x_1, ..., x_T\}$, each evaluation instance $x_t$ (e.g., a pair of candidate responses with context) is judged with a prompt-constructed template $P_t$ from a base meta-prompt $M_0$. Traditional approaches leave $M_0$ unchanged throughout test-time deployment: $y_t \leftarrow \text{Judge}(P_t, x_t)$ This static protocol fails to exploit evaluators' repeated deployment on correlated instances, missing opportunities to (1) learn persistent patterns of failure, (2) generate context-sensitive evaluation instructions, or (3) self-correct positional or spurious biases. LWE addresses these gaps by actively refining a meta-prompt $M_t$ to encode accreted heuristics and sample-specific judging strategies (Jwa et al., 7 Dec 2025).

2. The LWE Framework: Structure and Update Rule

The core LWE mechanism replaces static prompting with a feedback-driven, evolving meta-prompt:

1. At evaluation time $t$, LWE uses $M_{t-1}$ to instantiate a task- and sample-specific prompt:

$P_t = \text{BuildEvalPrompt}(M_{t-1}, x_t)$

2. This prompt is used to judge $x_t$:

$y_t = \text{Judge}(P_t, x_t)$

3. After each judgment, LWE elicits self-feedback assessing the adherence of $y_t$ to the evaluation criteria and extracting "learned tips" for future decisions:

$f_t = \text{Feedback}(M_{t-1}, P_t, x_t, y_t)$

Feedback $f_t$ aggregates a numerical score $s_t \in \{1, ..., 5\}$, a binary confidence label, and 1–3 optimization tips as text.

4. Periodically (after a batch of $b$ cases), LWE refines the meta-prompt:

$$M_t = \text{RefineMetaPrompt}(M_{t-1}, \{f_{t-b+1}, ..., f_t\})$$

In practice, $\text{RefineMetaPrompt}$ is a single LLM call that summarizes feedback into updated instructions, yielding $M_t = M_{t-1} + \Delta_t$ for learned heuristics $\Delta_t$ synthesized from feedback.

This protocol enables the LLM to develop evaluation criteria that adapt over time, leveraging prior failures for prompt construction and bias correction.

3. Selective LWE: Efficient Sequential Adaptation

Full LWE applies the above update rule to every instance, which is computationally intensive. Selective LWE introduces a screening process to limit updates to "difficult" cases:

• For each $x \in \mathcal{D}$, compute judgments on both original and swapped candidate order:

$$y^{(AB)} = \text{Judge}(P, x), \quad y^{(BA)} = \text{Judge}(P, \text{swap}(x))$$

• If $y^{(AB)} = y^{(BA)}$, the case is deemed consistent and requires no update.
• If $y^{(AB)} \neq y^{(BA)}$, the case is flagged as inconsistent and included in the set $I$ for subsequent full LWE updates.

This policy is formalized as: $$\boxed{ \begin{aligned} I &= \{\} \ S &= \{\} \ \text{for } x \in \mathcal{D}: \ \qquad y^{(AB)} &= \text{Judge}(P, x),\quad y^{(BA)} = \text{Judge}(P, \text{swap}(x)) \ \qquad \text{if } y^{(AB)} = y^{(BA)}: &\quad S \cup \{(x, y^{(AB)})\} \ \qquad \text{else:} &\quad I \cup \{x\} \ (S', M) &= \text{LWE}(I, M_0, b) \ S &\leftarrow S \cup S' \ \text{return } (S, M) \end{aligned} }$$ As only $|I| \ll T$ typically, selective updates reduce the overhead associated with meta-prompt refinement. Empirically, Selective LWE achieves $\approx 3.9\times$ vanilla cost versus $\approx 10.9\times$ for full LWE, and yields consistency and pair-accuracy scores that match or surpass those of full LWE (Jwa et al., 7 Dec 2025).

4. Empirical Evaluation and Comparative Results

Experimental validation on the VLRewardBench and Multimodal RewardBench benchmarks demonstrates LWE’s advantages in both accuracy and statistical consistency:

Method	VLRewardBench Acc./Cons./PairAcc.	MMRewardBench Acc./Cons./PairAcc.	Relative Cost (vanilla = 1.0×)
Vanilla	0.629 / 0.801 / 0.529	0.808 / 0.863 / 0.747	1.0×
TextGrad*	0.730 / 0.749 / 0.615	0.821 / 0.836 / 0.741	4.4×
Dynamic Cheatsheet	0.698 / 0.868 / 0.629	0.811 / 0.901 / 0.764	12.9×
Sample-Specific	0.661 / 0.727 / 0.529	0.815 / 0.865 / 0.742	2.5×
LWE (full)	0.745 / 0.805 / 0.646	0.799 / 0.846 / 0.727	10.9×
Selective LWE	0.676 / 0.940 / 0.648	0.836 / 0.947 / 0.808	3.9×

Selective LWE attains the highest consistency and pair accuracy (fraction where swap judgments agree and both match ground-truth) on both datasets, while incurring only a moderate computational burden. Notably, the method is robust to batch size settings and input order, and its utility grows as the fraction of inconsistent (difficult) cases increases (Jwa et al., 7 Dec 2025).

5. Limitations, Challenges, and Potential Extensions

Several limitations constrain LWE's applicability:

• Reliance on a reliable and high-capacity base LLM; weak models may fail to generate meaningful meta-prompts or feedback.
• Inability to correct "consistent but wrong" judgments, since ground-truth labels are unavailable at test time.
• Need for extra inference passes for inconsistency checks (esp. in Selective LWE).
• Potential growth of meta-prompt size, necessitating periodic summarization for tractability.

Proposed extensions include multi-turn update loops per sample, alternative feedback signals (e.g., log-prob gaps, multi-critic agreement), and generalization to open-ended scoring or rubric-based grading tasks. Adaptive batch sizing and external symbolic memory integration are natural directions to address scaling and factuality (Jwa et al., 7 Dec 2025).

6. Significance and Comparative Position

The LWE paradigm operationalizes test-time evaluator learning for LLM-as-a-judge settings, providing a simple, prompt-based mechanism for real-time adaptation. It generalizes previous approaches (such as dynamic cheatsheets and sample-specific static prompting) by enabling continuous accumulation of reusable evaluation heuristics and focusing learning effort where most informative—cases with internal disagreement. Its empirical advantages are most pronounced in tasks with nuanced evaluation criteria, high rates of input variation, or susceptibility to position-related biases.

A plausible implication is that LWE's protocol may serve as the foundation for broader meta-evaluation frameworks that harness large model deployment traces for self-directed improvement, especially where full labels or retraining data are inaccessible (Jwa et al., 7 Dec 2025).