GrandJury: Pluralistic Model Evaluation

• GrandJury is a collaborative evaluation protocol that replaces fixed benchmarks with dynamic, context-dependent human judgments.
• It employs explicit rubrics, time-decayed aggregation, and multi-rater voting to capture the evolving criteria of quality in open-ended tasks.
• The protocol ensures complete traceability and ambiguity detection, promoting transparency in the assessment of generative models.

Searching arXiv for the primary GrandJury paper and closely related work on jury/panel-based evaluation and legal-jury analogues. GrandJury is a collaborative machine-learning model evaluation protocol for settings in which outputs lack a single absolute ground truth and quality is context-dependent, plural, and time-varying. It was proposed for generative-model assessment in domains such as creative writing, summarization, multi-hop reasoning, and context-aware dialogue, where static benchmark-style evaluation is argued to be misaligned with evolving user needs and contested notions of quality. Rather than defining a new benchmark or scalar metric, GrandJury is framed as governance infrastructure built from dynamic and transparent task rubric attribution, multi-rater human judgment, time-decayed aggregation, complete traceability, and explicit ambiguity detection (Cho, 4 Aug 2025).

1. Conceptual scope and problem setting

GrandJury addresses evaluation regimes in which the target is neither fixed nor uniquely correct. The motivating claim is that standard evaluation for open-ended generation typically presumes a static target, a single correct answer, and a fixed rubric, assumptions that are poorly matched to tasks such as humor, creativity, dialogue, summarization, and open-ended reasoning. In such settings, human disagreement is common, correctness depends on context and audience, and quality criteria evolve over time (Cho, 4 Aug 2025).

Within that framing, GrandJury treats evaluation as a structured social process rather than as a one-shot benchmark computation. Its central object is not a reference answer but a publicly specified “voter prompt” or rubric under which multiple human jurors cast judgments. This makes the protocol explicitly pluralistic: consensus is measured, but disagreement is preserved rather than collapsed into an apparently objective score. The paper therefore positions GrandJury as “not a benchmark, metric, or product” but as “a protocol and community infrastructure” (Cho, 4 Aug 2025).

This positioning distinguishes GrandJury from several adjacent paradigms. Static benchmark metrics such as BLEU and ROUGE are criticized for incentivizing optimization toward leaderboard performance rather than real utility. Proxy-based approaches are also treated as insufficient: RAGAS is described as reference-free in form but still shaped by benchmarked human labels; embedding similarity metrics are described as too shallow to capture pragmatics, style, or narrative quality; and LLM-as-judge systems are described as scalable but prone to bias, prompt sensitivity, position bias, and instability (Cho, 4 Aug 2025). A plausible implication is that GrandJury is intended not to eliminate subjectivity, but to formalize its provenance, temporal evolution, and contestability.

2. Core protocol architecture

GrandJury is organized around four elements: a public voter prompt or task rubric, time-decayed aggregation, complete traceability, and multi-rater human voting with ambiguity detection (Cho, 4 Aug 2025). Together, these specify the semantic target of evaluation, the mechanism for combining judgments, the audit trail for each judgment, and the conditions under which disagreement becomes operationally salient.

The protocol begins with a publicly shared rubric that specifies what to evaluate, how to evaluate it, and why. The rubric is not latent inside a model prompt or hidden metric definition; it is an explicit protocol object associated with each vote. The paper describes this as dynamic and transparent task rubric attribution. “Dynamic” means that the rubric can change as community needs change. “Transparent” means that any score is attributable to the rubric in force at the time of judgment (Cho, 4 Aug 2025). This makes evaluation situated rather than timeless.

The multi-rater component is explicitly human. GrandJury combines multiple human votes and allows optional reputation weighting. The paper’s notation is binary, with $1=$ accept and $0=$ reject, but its worked example uses fractional scores, indicating that the implementation can ingest graded judgments (Cho, 4 Aug 2025). The system is therefore neither purely ordinal nor restricted to dichotomous verdicts.

The protocol’s components can be summarized as follows.

Component	Function	Formalization
Public voter prompt / rubric	Defines evaluation criteria and context	Associated with each vote
Time-decayed aggregation	Updates score using recency-weighted evidence	$$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$
Complete traceability	Preserves audit trail for every judgment	Timestamp, juror ID, inference ID, prompt ID
Ambiguity detection	Flags high-disagreement items	$\sigma_t^2 = \operatorname{Var}(v_{t,*})$

This architecture makes the score only one output of the protocol. Traceability, freshness, vote distributions, and ambiguity signals are co-equal artifacts. That design choice matters because it prevents a single aggregate from erasing the conditions under which it was produced.

3. Temporal aggregation, freshness, and disagreement signaling

GrandJury’s scoring model is explicitly temporal. Instead of taking an undifferentiated mean over all historical votes, it applies exponential decay so that recent evidence carries greater weight. The update rule is

$$S_t = \alpha S_{t-1} + (1 - \alpha)\langle v_t \rangle,$$

where $S_t$ is the updated score, $S_{t-1}$ is the prior cumulative score, $\langle v_t \rangle$ is the reputation-weighted mean vote in the current batch, and $\alpha=e^{-\lambda \Delta t}$ is the decay factor determined by elapsed time $\Delta t$ and decay constant $0=$0 (Cho, 4 Aug 2025).

The batch mean is defined as

$0=$1

with vote $0=$2 and optional reputation weight $0=$3, defaulting to $0=$4 when no weighting is used (Cho, 4 Aug 2025). The use of optional reputations introduces asymmetry across raters, but the paper does not specify a formal reputation-learning procedure. A plausible implication is that GrandJury separates the existence of weighted aggregation from any specific theory of reputational authority.

A second temporal observable is “freshness,” defined as

$0=$5

Freshness measures how much of the current score is attributable to the newest batch of evidence (Cho, 4 Aug 2025). In the paper’s worked example, with $0=$6, $0=$7 days, and $0=$8, one obtains $0=$9. For votes $$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$0, the batch mean is $$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$1, the updated score is $$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$2, and freshness is $$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$3 (Cho, 4 Aug 2025). The intended interpretation is that the score remains stable yet responsive.

GrandJury also treats disagreement as an informative output. Variance in the current batch is computed as

$$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$4

and an item is flagged as ambiguous if

$$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$5

The paper gives $$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$6 as an example threshold (Cho, 4 Aug 2025). For $$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$7, variance is $$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$8, so no flag is raised; for $$S_t = \alpha S_{t-1} + (1-\alpha)\langle v_t\rangle$$9, variance is $\sigma_t^2 = \operatorname{Var}(v_{t,*})$0, so the item is flagged for curator review or another voting round (Cho, 4 Aug 2025). This operationalizes a central premise of the protocol: disagreement is surfaced, not erased.

4. Workflow, provenance, and software realization

GrandJury is described as a two-phase workflow. The first phase is human evaluation collection: download the inference dataset, define voter prompts or rubrics, collect human votes on model outputs, and store those votes with metadata including voter ID, timestamps, inference ID, and voter prompt ID. The second phase is vote analysis: load the vote data with the grandjury Python client, apply time-decay scoring and consensus analysis, and generate reports covering traceability, freshness, vote distributions, and ambiguity signals (Cho, 4 Aug 2025).

The traceability layer is unusually explicit. Every vote is preserved as a structured record timestamped in ISO 8601 format, linked to a pseudonymous juror ID, and associated with its voter prompt context (Cho, 4 Aug 2025). This enables audit trails, meta-analysis, and temporal reconstruction of how judgments were formed. In practical terms, the protocol can answer who voted, when they voted, under what rubric, and on which inference output. That design places GrandJury closer to an evaluative ledger than to a conventional annotation file.

The paper also provides an open-source implementation through the grandjury PyPI package. The package exposes a GrandJuryClient and analysis methods including vote_histogram, vote_completeness, population_confidence, votes_distribution, and evaluate_model. Supported data formats include pandas DataFrames, Polars, CSV, Parquet, and JSON. The architecture is described as client-side data formatting and API access together with server-side scoring and time-decay computation (Cho, 4 Aug 2025). This division suggests a protocol boundary between local data handling and centralized scoring consistency.

The software layer is consequential because it operationalizes the protocol’s institutional claims. GrandJury is not presented merely as an abstract scoring rule; it is designed to support repeated evaluation sessions with stable provenance and cross-session temporal updates.

5. Demonstration corpus and intended application domains

To illustrate the protocol’s motivation, the paper releases a public inference dataset called “Jokes on Us,” hosted on Hugging Face. The dataset contains 149 joke responses from 31 distinct models across 7 major AI platforms, all queried with the prompt “tell me a joke,” using temperature $\sigma_t^2 = \operatorname{Var}(v_{t,*})$1, and collected within a 6-minute window (Cho, 4 Aug 2025). The dataset includes model ID, provider platform, timestamps, prompt and generated output, and platform-specific configuration (Cho, 4 Aug 2025).

Humor is a deliberately chosen demonstration domain because it lacks a single gold reference and tolerates substantial legitimate disagreement. The dataset therefore functions as an existence proof for the type of task GrandJury targets: outputs are diverse, there is no single correct answer, and exact-match or reference-based metrics are structurally inadequate (Cho, 4 Aug 2025). This suggests that GrandJury is especially relevant for evaluation contexts where semantic adequacy, style, audience fit, and other pragmatic dimensions are not reducible to lexical overlap.

More broadly, the paper situates the protocol in the assessment of generative systems used for creative writing, summarization, multi-hop reasoning, context-aware dialogue, workflow automation, and autonomous decision-making (Cho, 4 Aug 2025). The common feature across these domains is the absence of stable absolute ground truth. GrandJury’s answer is to record evolving human judgment under explicit rubrics rather than to infer an objective target from a static test set.

6. Relation to adjacent “jury” paradigms and major limitations

Although its name invokes a jury, GrandJury is not a legal-procedure model and is not concerned with indictment, verdict prediction, or juror selection in the ordinary judicial sense. That distinction is important because the surrounding literature uses “jury” in multiple technical ways. In legal NLP, for example, JUSTICE models Supreme Court judgment prediction from case facts and frames an ensemble as “voting like the Supreme Court’s 9 justices,” but it is a benchmark for judicial outcome prediction rather than a literal grand-jury dataset (Alali et al., 2021). In empirical legal analytics, work on racial bias in jury selection studies strike decisions in Flowers v. Mississippi using Optimal Feature Selection and Optimal Classification Trees, addressing discriminatory exclusion rather than collaborative model evaluation (Dunn et al., 2021). GrandJury differs from both: its “jury” is a panel of human evaluators scoring model outputs under public rubrics.

The protocol is also distinct from LLM-judge panel methods. RoPoLL formalizes a Panel of LLM Evaluators under the Huber contamination model and argues that plain mean aggregation yields unbounded bias under biased contamination, motivating geometric-median aggregation for a robust panel of LLM judges (Acharya et al., 29 Jun 2026). GrandJury, by contrast, critiques LLM-as-judge systems for bias, prompt sensitivity, position bias, and instability, and centers multi-rater human judgment instead (Cho, 4 Aug 2025). A plausible implication is that the two lines of work are complementary: RoPoLL studies robust aggregation when judges are models, whereas GrandJury specifies auditable governance when judges are humans.

The paper also identifies several limitations. It does not provide a large-scale empirical validation showing that time-decayed scoring correlates better with downstream utility. Reputation weighting is introduced but not fully specified or learned. Thresholds such as $\sigma_t^2 = \operatorname{Var}(v_{t,*})$2 are configurable rather than theoretically justified. The protocol depends on human labor and may therefore be neither cheap nor fast at scale. It also does not resolve how rubrics should be chosen when stakeholders fundamentally disagree (Cho, 4 Aug 2025). These constraints are substantive rather than incidental: GrandJury improves accountability and transparency, but it does not abolish subjectivity. It formalizes it.

In that sense, GrandJury’s core contribution is not a new notion of correctness, but a protocol for maintaining an auditable, evolving record of plural human judgment. Its main claim is that, for open-ended generative systems, evaluation should be renewable rather than frozen, attributable rather than opaque, and disagreement-aware rather than benchmark-saturated (Cho, 4 Aug 2025).