LLM-as-a-Judge: Implicit Aggregation

• LLM-as-a-Judge systems use implicit aggregation to combine multiple evaluative criteria into a single score without explicit weighting, enabling streamlined decision-making.
• Various aggregation methods, such as mean, mode, reliability-weighted voting, programmatic synthesis, and probabilistic models, address different biases and calibration challenges.
• Mitigation strategies including dynamic jury assembly and distribution-sensitive techniques improve consistency and reduce errors in multi-criteria LLM evaluations.

LLMs are increasingly deployed as automated evaluators for AI generation tasks, a paradigm known as "LLM-as-a-Judge." Central to the reliability and interpretability of this paradigm is the concept of "implicit aggregation": the unobserved, often algorithmically or architecturally encoded combination of multiple evidence sources, criteria, or judgment signals into a unified evaluation output. This article provides a comprehensive examination of implicit aggregation in LLM-as-a-judge systems, integrating formal definitions, major aggregation frameworks, known pathologies, mitigation approaches, and advanced methodologies for reliability-aware and dependence-aware assessment.

1. Formalization of Implicit Aggregation in LLM-as-a-Judge

Implicit aggregation refers to the process by which multiple evaluation criteria, rubric dimensions, judge votes, or probability distributions are synthesized into a single decision or score, often without explicit weightings or transparency. In the canonical architecture, for a given evaluation item $x$ and set of candidate judges or criteria $J_j$, the final score is computed as: $$S_i(x) \;=\; \frac{1}{m}\sum_{j=1}^m J_j\bigl(A_i(x)\bigr)$$ where $A_i(x)$ is the answer from candidate model $i$, and $J_j$ denotes the (possibly identical) LLM-judge applied across criteria or agents (Karp et al., 6 Nov 2025). In most practical LLM-based evaluation systems, this aggregation is implicit—no explicit assignment of weights $w_j$ or rationale for combination is provided by the model or the orchestration pipeline.

The logic of implicit aggregation surfaces both at the individual judge level (as an internal collapse of multi-faceted textual reasoning into a numerical output) and at the ensemble level (average, majority vote, regression, or more complex learned rules over multiple judges or criteria).

2. Aggregation Architectures and Algorithms

2.1 Mean and Mode Aggregation

Most LLM-as-a-judge systems begin with elementary aggregation mechanisms:

• Single Judge Averaging: Where the same LLM is prompted multiple times or across criteria, aggregation reduces to mean or majority over normalized individual scores (Karp et al., 6 Nov 2025).
• Probability Distributional Aggregation: Rather than taking the mode score label (greedy decoding), it is often preferable to compute the mean of the model's judgment distribution, thus leveraging the full uncertainty signal:

$$\hat{y}_\mathrm{mean}(x) = \sum_{y=1}^K y \cdot p(y|x)$$

Continuous aggregation (mean, risk-averse mean) consistently yields higher calibration and accuracy compared to discrete mode (Wang et al., 4 Mar 2025).

2.2 Aggregation Across Multiple Judges

The implicit ensemble is a common strategy to improve robustness:

• Majority Voting / Weighted Voting: For a set of judges $\{y_i\}$, aggregate via

$$\hat{y} = \operatorname*{arg\,max}_c \sum_{i=1}^M w_i\,\mathbf{1}(y_i = c)$$

where weights $w_i$ can reflect historical reliability or consistency (Fu et al., 18 May 2025).

• Reliability-Aware Weighted Aggregation: Each judge's score is weighted by an instance-specific reliability predictor $r_i$, learned from metafeatures of the input item:

$$y^* = \sum_{i \in J} w_i\,y_i, \quad w_i = \frac{r_i}{\sum_j r_j}$$

This dynamic jury assembly is a cornerstone of Jury-on-Demand architectures (Li et al., 1 Dec 2025).

2.3 Probabilistic and Programmatic Aggregation

• Program Synthesis for Judging: Instead of black-box model calls, programmatic judges (e.g., PAJAMA) synthesize explicit Python routines for each rubric, enabling local execution and Snorkel-style weak supervision aggregation. Bias and inconsistency are thereby reduced by 15–25% compared to the LLM-as-a-judge baseline (Huang et al., 12 Jun 2025).
• Crowdsourced and Bayesian Label Models: For scenarios with multiple, possibly dependent LLM judges, aggregation can follow probabilistic graphical frameworks, such as judge-aware Bradley-Terry models or Ising Markov random fields, which capture reliability structure and conditional dependence (Qian et al., 18 Feb 2026, Balasubramanian et al., 29 Jan 2026).

2.4 Specialized Aggregation for Ambiguity

• Soft Response Aggregation: When human or LLM labels are ambiguous (multiple plausible options), implicit aggregation via soft-response-set or multi-label vectors avoids committing to deterministic gold labels:

$\Omega_i = \Lambda \theta^*_i \in [0,1]^o$

allowing the evaluation performance to be measured against the entire plausible distribution of responses (Guerdan et al., 7 Mar 2025).

3. Failure Modes and Pathologies of Implicit Aggregation

A range of problematic behaviors arises from naive or opaque aggregation in LLM-as-a-judge:

3.1 Hallucination and Conflation Errors

LLM judges often hallucinate legal provisions, confuse statutory elements, or generate disjointed logical reasoning, which is compounded when aggregation compresses all criteria into high marks, masking deficiencies (Karp et al., 6 Nov 2025).

3.2 Inconsistency and Transitivity Violations

Score-comparison and pairwise transitivity inconsistencies (e.g., A>B>C>A or A=B=C≠A) emerge when discrete modes discard uncertainty or when positional/prompt biases accumulate without compensatory mechanisms. TrustJudge formalizes these inconsistencies and introduces probabilistic aggregation to reduce error rates by up to 11 percentage points (Wang et al., 25 Sep 2025).

3.3 Hidden Shortcut Reliance and Explanation Gaps

LLM judges are sensitive to irrelevant contextual cues (e.g., source, recency, education), as revealed by large verdict shift rates (VSRs) with negligible cue acknowledgment rates (CAR), producing a significant explanation gap not detectable by superficial rubric audit (Marioriyad et al., 8 Feb 2026).

3.4 Multilingual and Demographic Instability

Implicit aggregation fails under multilingual judgments, with average Fleiss' Kappa of 0.3 and near-zero cross-language pairwise agreement in low-resource contexts, unless ensembles are used to buffer inconsistency (Fu et al., 18 May 2025).

4. Mitigation Strategies and Robust Aggregation Methods

4.1 Reliability-Weighted and Learned Aggregation

Jury-on-Demand approaches train instance-wise reliability predictors for each judge based on input features and prior agreement with human scores, dynamically assembling weighted juries that significantly outperform static and single-judge baselines on evaluation correlation (Li et al., 1 Dec 2025).

Judge-aware extensions to classical aggregation models, such as BT-sigma (Bradley-Terry with per-judge temperature scaling), enable fully unsupervised, reliability-calibrated item rankings without human-labeled supervision, outperforming average-probability or vanilla majority voting aggregation both in terms of Spearman rank and cycle consistency (Qian et al., 18 Feb 2026).

4.2 Programmatic and Distribution-Sensitive Methods

PAJAMA synthesizes explicit, auditable program judges and weakly supervised label models for aggregation, which reduces cost by three orders of magnitude and improves consistency and debiasing relative to pure LLM scoring (Huang et al., 12 Jun 2025).

TrustJudge replaces single discrete scores with continuous probabilistic expectations over fine-grained scales and uses bidirectional likelihood aggregation for pairwise judgments, reducing transitivity and score-comparison inconsistency with minimal loss of accuracy (Wang et al., 25 Sep 2025).

4.3 Persona and Rubric Diversity for Robustness

Learning non-linear aggregators (GAMs, MLPs) over diverse, rubric-conditioned judges allows the ensemble to adapt to human preference heterogeneity, de-bias rubric sensitivity, and maintain calibration under systematic or random contamination (Sprejer et al., 29 Oct 2025).

4.4 Dependence-Aware Inference in Judge Ensembles

Classical label aggregation assumes conditional independence, but LLM judges often exhibit dependence due to shared failure modes. Dependence-aware models, such as class-dependent Ising graphical models, learn both per-judge reliability and inter-judge correlation structure: $$P(J|Y=y) \propto \exp\left(\sum_{j} h_j^{(y)}J_j + \tfrac{1}{2}\sum_{j\neq k} W^{(y)}_{jk}J_j J_k\right)$$ yielding quadratic or correlation-adjusted linear voting rules that outperform Dawid-Skene and majority-vote even as $K\to\infty$ (Balasubramanian et al., 29 Jan 2026).

4.5 Validation without Gold Labels

When consensus gold is unavailable, implicit aggregation over multi-label or soft human distributions, combined with evaluation via MSE or symmetric JS divergence, robustly identifies optimal judge models and avoids failures inherent to categorical gold labeling (Guerdan et al., 7 Mar 2025).

5. Applications and Empirical Outcomes

Table: Representative Implicit Aggregation Algorithms and Outcomes

Aggregation Method	Core Mechanism	Quantitative Effect
Mean/Mode Distributional	Average over criteria or model's score tokens	+2–8% acc, but prompt/position/tie bias (mode) (Wang et al., 4 Mar 2025)
PAJAMA	Program synthesis + weak label sharing	+15% consistency, −23% bias, major inference speedup (Huang et al., 12 Jun 2025)
BT-sigma	Reliability-weighted pairwise, unsupervised	+3–5 Spearman ρ, improved transitivity (Qian et al., 18 Feb 2026)
Jury-on-Demand	Dynamic instance-weighted jury via metafeatures	+4–12 pp Kendall's τ over static, robust across domains (Li et al., 1 Dec 2025)
TrustJudge	Distributional expectation, bidirectional pairs	−8%–11% inconsistency, higher win rates (Wang et al., 25 Sep 2025)
Persona-learned GAM/MLP	Non-linear judge calibration, persona mixing	+5–12% R² vs. single or naive mean (Sprejer et al., 29 Oct 2025)
Ising Model Aggregation	Directly models judge dependencies	+5–10% accuracy vs CI-Dawid–Skene (Balasubramanian et al., 29 Jan 2026)
Soft/Multilabel Human Aggregation	No-gold full-label aggregation for validation	Prevents up to 34% judge selection error (Guerdan et al., 7 Mar 2025)

Implicit aggregation strategies are critical in scaling human-aligned, reliable LLM evaluation for diverse modalities: legal judgment simulation (Karp et al., 6 Nov 2025), multi-lingual assessment (Fu et al., 18 May 2025), boundary mining in dataset creation (Ma et al., 14 Jan 2026), and preference reward model construction for RLHF (Sprejer et al., 29 Oct 2025).

6. Limitations, Open Problems, and Future Directions

Despite these advances, multiple limitations persist:

• Implicit aggregation can mask or exacerbate shortcut reliance and explanation gaps, requiring adversarial testing and transparency metrics (e.g., VSR/CAR) for audit (Marioriyad et al., 8 Feb 2026).
• No implicit aggregation method obviates the need for domain-expert oversight when cardinal errors (e.g., legal misjudgments) entail high risk (Karp et al., 6 Nov 2025).
• Fine-grained judge calibration (e.g., dynamic thresholding, context-aware reliability scoring) is still subject to data limitations and nonconvex optimization, especially in the presence of class-dependent dependencies (Balasubramanian et al., 29 Jan 2026, Qian et al., 18 Feb 2026).
• Extensions to interactive, multi-turn, or structured outputs remain open challenges for implicit aggregation architectures.

Emerging best practices converge on multimodal, reliability-aware, and explicitly auditable aggregation mechanisms, often combining learned instance-level weighting, program synthesis, and stochastic or distributional inference, sometimes with human-in-the-loop validation for critical edge cases. Adaptive frameworks—dynamic juries, learned aggregators, and probabilistic calibration—are poised to underpin scalable, robust, and fair LLM evaluation pipelines across domains.

---

Key references: (Karp et al., 6 Nov 2025, Huang et al., 12 Jun 2025, Qian et al., 18 Feb 2026, Wang et al., 4 Mar 2025, Fu et al., 18 May 2025, Li et al., 1 Dec 2025, Sprejer et al., 29 Oct 2025, Wang et al., 25 Sep 2025, Guerdan et al., 7 Mar 2025, Balasubramanian et al., 29 Jan 2026, Marioriyad et al., 8 Feb 2026, Ma et al., 14 Jan 2026)