MD-Judge: Multi-Dimensional Evaluation

• MD-Judge is a multi-dimensional model-based evaluation framework that uses LLMs and MLLMs to assess outputs on multiple rubrics such as fluency, factuality, and relevance.
• It employs multi-rubric, iterative, and multi-agent judging techniques to mimic human evaluation and mitigate biases across various tasks.
• The framework is applied in diverse domains including scientific process assessment, automated code evaluation, and multimodal content verification.

MD-Judge denotes a multi-dimensional, model-based or agent-based judge—a system or framework, often instantiated by LLMs or multimodal LLMs (MLLMs), designed to deliver nuanced, robust, and scalable annotation, scoring, or selection of candidate outputs in natural language, code, or multimodal generative tasks. Modern MD-Judge systems are characterized by their capacity to evaluate across multiple rubrics (dimensions), adapt to domain specificity, mitigate bias, and—when properly engineered—deliver outputs and rationales matched or benchmarked against human judgment across complex settings.

1. Foundations and Emergence

MD-Judge frameworks are rooted in the evolution of automatic evaluation for generative models. Early methodologies utilized string-based metrics (e.g., BLEU, ROUGE), but these proved insufficient for nuanced, open-ended, or multi-modal outputs. The LLM-as-a-Judge paradigm arose from the recognition that LLMs (and later MLLMs) possess sufficient semantic and contextual modeling capacity to simulate human evaluators, providing scalable, cost-effective, and context-adaptive judgments (Gu et al., 23 Nov 2024). This approach is further extended by MD-Judge systems, which operate not only over text but also across multiple output dimensions—such as fluency, factuality, relevance, faithfulness, and, in multimodal or scientific contexts, process- and domain-specific rubrics (Ai et al., 9 Mar 2025, Liu et al., 26 Feb 2025, Pu et al., 21 Mar 2025, Ding et al., 29 Aug 2025).

MD-Judge is not tied to a single architectural instantiation; it encompasses:

• Single-LLM multi-rubric judges with explicit reasoning and rationales (Trivedi et al., 7 Oct 2024, Yu et al., 17 Feb 2025)
• Multimodal or process-based judges for complex input/output (e.g., text-image, image-audio, code step validation) (Pu et al., 21 Mar 2025, Ai et al., 9 Mar 2025, Ding et al., 29 Aug 2025)
• Multi-agent dynamic judges for personalization and debate (Cao et al., 1 Apr 2025, Yu, 5 Aug 2025)

2. Core Methodologies and System Architectures

Multi-Rubric and Consistency Frameworks

A distinguishing aspect of MD-Judge systems is simultaneous, hybrid, or sequenced evaluation across orthogonal dimensions—such as accuracy, coherence, faithfulness, creativity, and comprehensiveness (Liu et al., 26 Feb 2025, Ding et al., 29 Aug 2025). Judge-Consistency (ConsJudge) evaluates models' outputs by prompting multiple aspect-rubric judgments and measuring intra-judge consistency to identify stable, reliable decisions. Consistency is quantified by calculating mean cosine similarity among embedding representations of judgment rationales [(Liu et al., 26 Feb 2025), Eqn. 1].

Process-level judges (e.g., ProJudge) require stepwise annotation, capturing not only the correctness but also the error type and rationale at each solution step of complex scientific problems (Ai et al., 9 Mar 2025).

Iterative and Multi-Agent Judging

Dynamic agent-based designs enable iterative prompt refinement and personalized evaluation. The MD-Judge system automatically samples representative examples, applies an LLM judge under current rubrics, evaluates score alignment with human reference labels, and uses an agent to rewrite the prompt until satisfactory performance is attained. Algorithmically, this is formalized as an iterative process—partitioning data, evaluating, updating prompts based on feedback, and terminating upon meeting a preset threshold (Cao et al., 1 Apr 2025).

Multi-agent debate (agent-as-a-judge) systems assemble committees of agents adopting conflicting or specialized roles (e.g., "Prosecutor," "Defense," "Expert"), debating the merits of candidate responses and aggregating judgments (Yu, 5 Aug 2025). This hedges against single-LLM biases.

Training and Fine-tuning Paradigms

Modern MD-Judge models leverage staged training:

• Supervised Fine-tuning (SFT) with demonstrations covering diverse judgment styles and stepwise rationales (Yu et al., 17 Feb 2025, Trivedi et al., 7 Oct 2024)
• Direct Preference Optimization (DPO): using preference pairs (chosen/rejected) often derived from either their own (rationalized) outputs or self-consistency checks (Trivedi et al., 7 Oct 2024, Yu et al., 17 Feb 2025, Liu et al., 26 Feb 2025)
• Data synthesis and augmentation, including automatic rewriting of judging instructions and injection of diverse semantically rich errors (Yu et al., 17 Feb 2025, Ai et al., 9 Mar 2025)

3. Evaluation Metrics, Bias, and Reliability

Performance and Calibration

MD-Judge system performance is measured via:

• Agreement with human labels (percentage or correlation metrics: Pearson, Spearman, kappa)
• Robustness to prompt or order variation (e.g., position consistency)
• Internal calibration: whether the confidence provided by the judge matches accuracy, as assessed by ECE, Brier score, Wrong@High-Conf, and risk-coverage area under the risk-coverage curve (AURC) (Kim et al., 23 Aug 2025).

Bias and Fairness

Systematic studies show position bias (favoring one candidate due to input order), verbosity bias, self-enhancement bias (judging outputs like one's own as superior), and rubric mismatch. Quantification uses metrics such as position consistency (PC), preference fairness (PF), and repetition stability (RC) (Shi et al., 12 Jun 2024). Bias is influenced by both model-level (e.g., family, prompt length) and task-level (e.g., quality gap between candidates) factors. Mitigation includes prompt shuffling, majority voting across agents or judgments, and ensemble evaluation (Shi et al., 12 Jun 2024, Gu et al., 23 Nov 2024).

MD-Judge systems that record and analyze inter-judge disagreement help flag ambiguous or difficult cases, which are often those with smallest quality-gap between candidates.

4. Domain-Specific Extensions

Science, Law, Medicine, and Multimodality

MD-Judge frameworks have been instantiated for domain- and modality-specific evaluation:

• In scientific process judgment, MD-Judge models are trained and scored for correctness and error classification against expert-annotated benchmarks such as ProJudgeBench (Ai et al., 9 Mar 2025). Instruction tuning with ProJudge-173k enhances open-source model process evaluation capacity.
• In medicine, Med-RewardBench enables evaluation of judges ("reward models") over accuracy, relevance, comprehensiveness, and additional clinical rubrics. Performance gaps with human experts and between models are explicitly measured (Ding et al., 29 Aug 2025).
• For retrieval-augmented generation, ConsJudge's multidimensional rubric and intra-judge self-consistency calibration generate higher-quality supervision signals (Liu et al., 26 Feb 2025).
• In multimodal content (text-image, audio, video), MLLMs as judges assessed via JudgeAnything and OmniArena platform perform pair-comparison and score evaluation against human reference judgments across any-to-any modality categories (Pu et al., 21 Mar 2025).

5. Limitations and Future Research Directions

Despite their versatility, MD-Judge systems have limitations:

• Overconfidence: Many state-of-the-art models exhibit high confidence even as accuracy drops under adversarial or complex scenarios (e.g., multi-turn jailbreaks) (Kim et al., 23 Aug 2025). Selective prediction and abstention are recommended for operational safety.
• Domain coverage and scaling: Performance degrades on rare modalities, unseen answer styles, or domains with fine-grained subtleties (e.g., ophthalmology in medicine, high-level math, or certain legal criteria) (Ding et al., 29 Aug 2025, Ai et al., 9 Mar 2025).
• Iterative processes and multi-agent debate induce higher computational cost, requiring optimization for latency and resource usage (Cao et al., 1 Apr 2025, Yu, 5 Aug 2025).
• Calibration and prompt design remain areas of ongoing research—dynamic or learned prompt generation, alignment of rationales, and meta-evaluation frameworks are active targets.
• There remains a gap between the open-source community and leading proprietary models, driving the need for large, diverse training datasets and publicly available benchmarks and tools.

6. Practical Applications and Impact

MD-Judge systems are leveraged in:

• Automated model selection, ranking, and reward feedback for RLHF/RLAIF pipelines (Yu et al., 17 Feb 2025, Liu et al., 26 Feb 2025)
• Quality assurance and moderation for retrieval-augmented search, customer-facing QA, and dialog systems (Dey et al., 7 May 2025)
• Automated code evaluation, process tracking, and error explanation in code generation or educational settings (Wang et al., 18 Feb 2025, Ai et al., 9 Mar 2025)
• Multimodal and cross-modal leaderboard platforms (e.g., OmniArena) for the dynamic ranking and evaluation of omni-models (Pu et al., 21 Mar 2025)

These systems enable scalable, explainable, and more human-aligned evaluation in place of expensive manual annotation, and they inform refinements to both base generation models and reward/selection models in practice.

---

MD-Judge frameworks represent an overview of methodological advances in LLM/MLLM-based evaluation, integrating multi-rubric, multi-agent, and domain-tailored architectures to achieve nuanced and robust judgments. Benchmarks such as ProJudgeBench, Med-RewardBench, JudgeAnything, and ConsJudge lay the empirical foundation for future developments and meta-evaluation in this rapidly evolving field.