Human-Grounded Metrics

• Human-grounded metrics are evaluation criteria that integrate human judgments, behaviors, and cognitive processes to assess AI performance beyond traditional algorithmic measures.
• They employ methodologies like direct human ratings, behavioral measurements, and LLM-calibrated evaluations to capture usability, interpretability, and robustness.
• Applications span dialogue, summarization, explainable AI, and adversarial robustness, ensuring models meet practical, human-centric standards.

Human-grounded metrics are evaluation criteria and protocols designed to measure the performance, interpretability, robustness, and utility of AI systems in ways that are empirically anchored to human intuitions, behaviors, or judgments. In contrast to purely automatic metrics that quantify overlap or distance in feature space, human-grounded metrics explicitly incorporate human preferences, cognitive capacities, annotation, or decision processes—either as direct measurement targets or as data sources for the definition and validation of metrics. These metrics are critical for advancing user-facing models in domains such as dialogue, explanation, summarization, document question answering, adversarial robustness, social reasoning, and more, where alignment with real-world user needs, usability, and interpretability cannot be captured by reference-based or algorithmic metrics alone.

1. Principles and Motivation for Human-Grounded Metrics

Human-grounded metrics are motivated by several central observations:

• Inadequacy of purely automatic metrics: Metrics such as BLEU, ROUGE, or embedding-based similarity often diverge from human judgments of quality, salience, or usefulness, particularly in open-ended tasks or when system improvements saturate traditional scores (Liu et al., 2022, Ryan et al., 19 Dec 2025).
• Capturing usability and interpretability: Many application settings (e.g., explainable ML, dialogue, VQA, summarization) require not only correct outputs but also rationale, engagement, or actionable insight that are meaningful to human users (Gyevnar et al., 31 Jan 2025, Giorgi et al., 2023, Sung et al., 2024).
• Evaluating collaboration and robustness: Tasks such as human–AI teaming, interactive grounding, or adversarial QA require measurement of human–system interplay, resilience to ambiguous or adversarial inputs, and calibration to human capabilities rather than to simplistic test suites (Imai et al., 4 Sep 2025, Sung et al., 2024, Gyevnar et al., 31 Jan 2025).
• Ground-truth ambiguity and diversity: Human-grounded evaluation can account for legitimate one-to-many mappings, divergent preferences, or alternate explanations, moving beyond single-reference assessments (Wakaki et al., 2024, Ryan et al., 19 Dec 2025).

2. Core Methodologies and Protocols

Human-grounded metrics take several distinct forms, depending on the domain and research question:

Anchored Protocols

• Direct human rating: Ratings or rankings are collected from human assessors on dimensions such as fluency, coherence, informativeness, (engagingness), etc. (Wakaki et al., 2024, Liu et al., 2022, Shuster et al., 2018).
• Behavioral measurement: Quantification of observable human behaviors—such as labeling accuracy, task-completion rates, response times, or intervention rates—when interacting with or using the model (Gyevnar et al., 31 Jan 2025, Bhan et al., 2023, Weerts et al., 2019).
• Agreement with human rationales: Comparison of system explanations (e.g., attention heatmaps or feature attributions) with human-annotated rationales, feature importances, or attention (Bhan et al., 2023, Weerts et al., 2019).

Automatic Metrics Calibrated to Human Judgments

• LLM-as-a-Judge/LLM Rubric metrics: Using LLMs, prompted and tuned to emulate detailed human evaluation, sometimes with regression over diverse automatic and human-inspired base metrics (Ryan et al., 19 Dec 2025, Wakaki et al., 2024).
• Regression-based composite metrics: Aggregating and weighting base metrics (reference-based and reference-free) to maximize correlation (e.g., in Kendall's $\tau$) with collected human judgments (AutoMetrics) (Ryan et al., 19 Dec 2025).

Human-Anchored Dataset Design

• Multi-reference and multi-response evaluation: Benchmarks such as ComperDial enable robust assessment by providing multiple human-generated responses/annotations, reflecting response diversity (Wakaki et al., 2024).
• Fine-grained annotation protocols: The ACU (Atomic Content Unit) protocol in summarization decomposes evaluation into binary inclusion judgments for each minimal “fact,” enhancing objectivity and reliability (Liu et al., 2022).

Item Response Theory and Adversarialness

• IRT-based discrimination: Calculation of difficulty and discriminability of individual samples, jointly modeling humans and models. The AdvScore metric quantifies the gap between human and model performance, penalizing trivial/impossible or non-discriminative items (Sung et al., 2024).

3. Domains of Application

Dialogue and Conversational Agents

• Engagingness and persona-consistency: Direct human A/B tests, multi-aspect scoring (fluency, coherence, humanness), and psycholinguistic metrics (emotion entropy, style matching, empathy, agreeableness) (Wakaki et al., 2024, Giorgi et al., 2023, Shuster et al., 2018).
• Turn- and dialogue-level evaluation: Chain-of-thought rubrics applied at both individual turn and holistic dialogue level, capturing both local quality and longer-term interactional patterns (Wakaki et al., 2024).

Summarization

• Salience via ACUs: Fine-grained atomic content units provide binary judgments that increase inter-annotator agreement and reduce subjectivity compared to Likert and holistic scales (Liu et al., 2022).
• Robustness to human “priors”: Analysis shows that simple Likert or unconstrained protocols are susceptible to annotator bias and summary length effects, motivating reference-bound, fact-level human-grounded approaches (Liu et al., 2022).

Explainable AI

• Actionability and effectiveness: Human-grounded metrics in explainable reinforcement learning include next-action, goal, and sub-goal prediction accuracies, time to answer, and the utility of counterfactual explanations, measured directly in controlled environments (Gyevnar et al., 31 Jan 2025).
• Interpretability efficiency: Reaction-time and accuracy improvements under system-generated explanations versus random or baseline methods characterize the practical interpretability for real users (Bhan et al., 2023, Weerts et al., 2019).

Multimodal and Social Reasoning

• Semantic and structural trace overlap: Social Genome metrics evaluate the match of model-generated reasoning traces to human traces at step-level, sequence-level, emotional cue, and modality granularity (Mathur et al., 21 Feb 2025).
• Interactive grounding: In VLM eval, metrics such as grounding efficiency, content alignment, lexical adaptation, and human-likeness are constructed to mirror psycholinguistics literature on incremental grounding and dialogue (Imai et al., 4 Sep 2025).

Adversarial Robustness

• Adversarialness by human–model margin: AdvScore detects benchmarks where even the strongest models fail but humans succeed, correcting for triviality and providing continuous diagnostics of benchmark validity as models and human pools evolve (Sung et al., 2024).

4. Metric Formalization and Implementation

Human-grounded metrics are rigorously operationalized, often by direct mathematical formulae. Examples include:

• ACU score: $$f(s,A) = | \{ a \in A : a \text{ matched in } s \} |$$ (summary-level recall over ACUs) (Liu et al., 2022).
• AdvScore: $\mathrm{ADVSCORE}(A) = p^A \times (K^A + d^A)$, where $p^A$ is the human–model margin, $K^A$ the discriminability, $d^A$ the human-difficulty spread (Sung et al., 2024).
• SMuDGE score: $s_i = \alpha m_i + (1-\alpha) g_i$, combining type-aware match and spatial alignment (Nourbakhsh et al., 24 Mar 2025).
• Psychological metrics: Entropy, style matching, and empathy scores via lexicon aggregation, functional word proportion matching, regression over topic features, or emotion NER overlap (Giorgi et al., 2023, Mathur et al., 21 Feb 2025).
• Composed metric regression: Composite metric $R^{(w)}(x) = \sum_{i} w_i M_i(x)$, with weights $w$ fit to match human rank orderings or class labels (Ryan et al., 19 Dec 2025).

These metrics typically require carefully curated human-generated gold data or direct behavioral measurement to ensure reference to human ground truth.

5. Validation, Benchmarking, and Empirical Findings

Validation of human-grounded metrics is conducted through:

• Correlation with human preference and task outcomes: Metrics are benchmarked via rank and score correlations with human judgments or behavioral scores (Kendall’s $\tau$, Spearman ρ, Pearson r), with large-scale experiments showing that composite or LLM-judge metrics outperform prior baselines on correlation to human labels (Ryan et al., 19 Dec 2025, Wakaki et al., 2024).
• Power and significance analysis: Sample size sensitivity, statistical significance, and power analyses are conducted to ensure that metrics can reliably distinguish system differences under realistic evaluation budgets (Liu et al., 2022).
• Multi-benchmark and capability coverage: Human-grounded capabilities and metric gaps are systematically assessed for extensiveness and alignment with real-world usage, with documented deficiencies motivating new benchmark constructs (Miller et al., 13 May 2025, Wakaki et al., 2024).
• Robustness, calibration, and diagnostic utility: Metrics such as SMuDGE or AdvScore are shown to better identify robust, well-calibrated, or consistently high-performing models compared to standard measures (Nourbakhsh et al., 24 Mar 2025, Sung et al., 2024).

6. Limitations, Open Challenges, and Best Practices

Despite progress, significant challenges remain:

• Data and labor intensity: High-fidelity human annotation (e.g., ACUs, persona-level dialogue scores) is labor- and expertise-intensive; best practices include detailed annotation protocols, inter-rater reliability reporting, and open data release to maximize reproducibility (Liu et al., 2022, Wakaki et al., 2024).
• Generalization and pool specificity: Metrics grounded in particular user pools or demographics may not transfer; results can be sensitive to task representativeness and user experience (Sung et al., 2024, Gyevnar et al., 31 Jan 2025).
• Overfitting, bias, and protocol alignment: LLM-judge metrics or human ratings can overfit to superficial cues (length, fluency) if protocols are unconstrained; mechanisms to align metric theme with protocol, enforce diversity, and avoid spurious correlations are necessary (Liu et al., 2022, Ryan et al., 19 Dec 2025).
• Interpretability versus cost trade-offs: Chain-of-thought LLM evaluator metrics are interpretable but computationally expensive; smaller or more efficient surrogates are an ongoing research direction (Wakaki et al., 2024).
• Stability under rapid model improvement: Adversarial benchmarks become obsolete as models progress; human-grounded metrics such as AdvScore enable timely re-evaluation and ongoing benchmark curation (Sung et al., 2024).

Prominent recommendations are to release code, data, and metric definitions, open-source testbeds, collect multi-level (task/behavioral/subjective) measures, and ensure protocol–metric alignment to maintain robust, human-relevant evaluation standards across the field.