MultEval: Supporting Collaborative Alignment for LLM-as-a-Judge Evaluation Criteria

Abstract: LLM-as-a-judge approaches have emerged as a scalable solution for evaluating model behaviors, yet they rely on evaluation criteria often created by a single individual, embedding that person's assumptions, priorities, and interpretive lens. In practice, defining such criteria is a collaborative and contested process involving multiple stakeholders with different values, interpretations, and priorities; an aspect largely unsupported by existing tools. To examine this problem in depth, we present a formative study examining how stakeholders collaboratively create, negotiate, and refine evaluation criteria for LLM-as-a-judge systems. Our findings reveal challenges in human oversight, including difficulties in establishing shared understanding, aligning values across stakeholders with different expertise and priorities, and translating nuanced human judgments into criteria that are interpretable and actionable for LLM judges. Based on these insights, we developed MultEval, a system that supports collaborative criteria by enabling multiple evaluators to surface and diagnose disagreements using consensus-building theory, iteratively revise criteria with attached examples and proposal history, and maintain transparency over how judgments are encoded into an automated evaluator. We further report a case study in which a team of domain experts used MultEval to collaboratively author criteria, illustrating how coordination and collaborative consensus-making shape criteria evolution.

• The paper introduces MultEval, a system that transforms evaluation criteria into dynamic, collaboratively negotiated artifacts using consensus-building theory.
• It employs user interviews and role-aware system design to address multi-stakeholder alignment challenges and resolve conflicts in LLM-as-a-judge evaluations.
• The study demonstrates enhanced transparency and accountability through structured proposal workflows, disagreement diagnosis, and iterative refinement of evaluation criteria.

Collaborative Alignment for LLM-as-a-Judge Evaluation: An Expert Summary of "MultEval" (2604.26679)

Motivation and Problem Formulation

The transition to LLM-as-a-judge evaluation in NLP reflects the need for scalable, consistent assessment of generative model outputs. However, the prevailing paradigm assumes criteria are authored and iterated by a single evaluator, obscuring the challenges encountered in real-world, collaborative settings. Criteria specification is not merely technical; it is a negotiated artifact, shaped by interdisciplinary teams with heterogeneous expertise, priorities, and values. This introduces multi-layered alignment requirements: human-AI alignment merges with human-human-AI alignment, demanding mechanisms for surfacing, diagnosing, and resolving disagreement among stakeholders, while encoding nuanced judgments into criteria actionable by LLM judges.

MultEval addresses these coordination, alignment, and criteria translation challenges via a formative empirical study and system design grounded in consensus-building theory. The authors show that evaluation criteria exist within an ecosystem of social negotiation, role stratification, and iterative refinement, demanding explicit workflow support for collaborative alignment.

Formative Study: Coordination and Misalignment

Through semi-structured interviews with industry and academic evaluators, MultEval exposes several coordination friction points:

• Role Asymmetries: Developers are gatekeepers of system-level implementation, with domain experts often excluded from daily iteration, resulting in loss of critical knowledge and misaligned criteria. Domain experts perceive technical aspects as inaccessible, while developers are burdened with translating implicit requirements.
• Hierarchical Decision Making: Criteria inclusion typically follows a dominant stakeholder's preferences, streamlining decisions but limiting diversity and potentially distorting alignment.
• Desire for Peer Consultation: Experts express need for collaborative deliberation and reflection, with current workflows inhibiting group sense-making and leading to criteria drift.
• Human-Human-AI Alignment: Decision boundaries on goal articulation and output assessment are split; stakeholders must align with each other to meaningfully encode requirements into LLM-judge criteria.

These findings inform MultEval’s design goals: support for multi-stakeholder alignment, asynchronous co-creation while respecting hierarchical roles, and mechanisms for highlighting nuanced criteria and evaluator confidence.

System Design: Consensus-Building Theory Operationalized

MultEval implements a comprehensive, multi-user web system under Briggs’ Consensus Building Theory. Its workflow scaffolds proposal, review, conflict diagnosis, and revision, explicitly supporting cycles of divergence and convergence (commitment, conflict, resolution). Key features include:

• Global criteria set and evaluation screen: Immediate visibility of criteria and dataset with color-coded markers for pass/failure.

Figure 1: Criteria, assertions, and dataset visualization; colored dots indicate evaluation status per criterion.

• Private Sandbox: Individual iterative testing, editing, and context attachment for proposed changes optimizes reflection before group deliberation.

Figure 2: Evaluators privately explore, edit, and test criteria against specific data.

• Proposal Tab and Admin Review: Structured proposal submission, voting, commentary, and administrative acceptance/rejection enable tracking of iterative refinement and authority.

Figure 3: Administrators manage proposals; non-admins participate via voting and commentary.

• Disagreement Diagnosis: Tagging and classification of disagreement types (meaning, mental model, information, goals, taste) helps teams directly surface and address conflict.

Figure 4: Disagreement types are explicitly tagged with definitions accessible via UI.

• Version History and Role-Cognizant Analysis: Chronological timelines annotate contributor roles, enabling transparency of expertise and decision rights in criteria evolution.

Figure 5: Visualized timeline of criterion versions, proposals, and authorship.

• Diverse Data Curation: Farthest-neighbors algorithm (cosine similarity, text embeddings) surfaces structurally/semantically diverse examples to maximize disagreement visibility and reduce redundancy.

Case Study and Results

MultEval was deployed with a team of pedagogy experts evaluating a teaching assistant chatbot’s outputs. The collaborative session and subsequent interviews revealed crucial dynamics:

• Initial Grounding: Teams pooled individual criteria, established shared vocabulary, reducing friction in subsequent coordinated authoring.
• Struggle with Specificity: Negotiation of criteria granularity was central; participants debated between broad intent and precise, executable requirements, recognizing the tradeoff between flexible generalization and overconstraint.
• Multiple Perspectives: Expert consensus emphasized evidence grounding and discipline-specific context, advocating for role-aware personas and group work over single-author approaches.
• Negotiating Hierarchies: Reluctance to immediately designate administrative authority was supplanted by pragmatic assignment, revealing the need for graduated, context-sensitive authority and explicit accountability.
• System Usability: Interface considered generally intuitive; however, minor discoverability and terminology issues surfaced, particularly in comment and proposal workflows.

Automated Disagreement Annotation Reliability

A corpus of criterion-change proposals was annotated for disagreement taxonomy using both human coders and LLM pipeline (OpenAI gpt-5.4-mini). Krippendorff's α agreement scores demonstrated that worked examples substantially improved annotation reliability; interpretive categories (e.g., mental model, information) showed moderate agreement, while meaning and taste lagged. This validates MultEval’s auto-annotation as a deliberative scaffolding rather than a definitive classifier.

Practical and Theoretical Implications

MultEval reframes LLM-as-a-judge evaluation workflows as fundamentally social and iterative. Criteria, as alignment artifacts, demand mechanisms for explicit negotiation, rationale preservation, asynchronous role management, and legible behavioral impact on LLM-judge performance. The system provides infrastructure for scalable, distributed alignment—making rationale, disagreement, and role dynamics persistent and visible. Scaling collaborative alignment tools is increasingly vital as LLM-judge systems are deployed across domains with conflicting requirements, heterogeneous teams, and complex regulatory environments.

Figure 6: System overview, depicting global criteria, dataset, evaluation process, proposal mechanics, analytics, role-swap, and role-based authority.

Future Directions

MultEval's limitations point to necessary next steps: longitudinal deployment in more heterogeneous, larger teams; deeper studies of asynchronous collaboration; improvement in feature discoverability and semantic clarity of social signals; and tighter coupling of system features to moments of breakdown and decision points. Improving behavioral legibility and sensitivity analysis of criteria wording impact on LLM judges is critical for epistemic confidence in collaborative evaluation.

Conclusion

MultEval establishes a robust framework for collaborative alignment in LLM-as-a-judge workflows, transitioning evaluation criteria from static technical artifacts to dynamic, negotiated constructs. By operationalizing consensus-building theory, structuring proposal processes, surfacing disagreement, and persisting rationale, MultEval enables transparency, accountability, and trustworthy evaluation. Delivered as an open system, it addresses both the practical bottlenecks and theoretical imperatives of human-human-AI alignment, paving the way for more equitable, domain-representative model oversight.