Functionally-Grounded Evaluation

• Functionally-grounded evaluation is defined as assessing systems against explicit functional contracts rather than relying solely on human labels or free-form criteria.
• It spans various domains including XAI, multimodal generation, and agent tasks, using measurable criteria like evidence paths, state transitions, and rubric-based scoring.
• Structured scoring methods such as IRT models and deterministic checkers are employed to diagnose failure modes and ensure reproducible, objective performance evaluation.

Functionally-grounded evaluation is a family of evaluation regimes in which model behavior is assessed against explicit functional criteria rather than only against fixed labels, free-form preferences, or end-user studies. In the XAI and interpretability literature, it denotes evaluation by computational or functional criteria without human participants; in recent LLM, multimodal, and agent work, it also denotes evaluation on realistic, grounded tasks whose difficulty is anchored in the function the system must perform, the evidence it may use, or the state it may change (Velmurugan et al., 2020, Velmurugan et al., 2024, Patel et al., 18 May 2026, Flynt, 22 Jun 2026). Across these uses, the common thread is that success must be tied to an auditable mechanism: a known computation, a constrained evidence path, a grounded state transition, a rubric linked to domain functions, or a representation-aware support relation.

1. Taxonomic meaning and conceptual scope

A recurrent starting point is the three-level taxonomy associated with Doshi-Velez and Kim. In that taxonomy, application-grounded evaluation uses domain experts in the full end-use task, human-grounded evaluation uses humans on simplified or proxy tasks, and functionally-grounded evaluation uses objective, automatable criteria without human participants (Velmurugan et al., 2020, Suglia et al., 2020, Velmurugan et al., 2024, Elshabrawy et al., 11 Jun 2026). In predictive process analytics, this framing is used to assess explanation quality through instance-level metrics such as stability and internal fidelity rather than through user studies (Velmurugan et al., 2020). In grounded language learning, it motivates proxy tasks that probe whether hidden states encode attribute composition and zero-shot generalization (Suglia et al., 2020). In procedural reasoning dataset construction, it appears as representation-aware validation against closed-set evidence units extracted from Task–Method–Knowledge models (Elshabrawy et al., 11 Jun 2026). In NeuroCognition, it denotes controlled proxy tasks that operationalize cognitive functions such as abstract relational reasoning, working memory maintenance, and cognitive flexibility (Haznitrama et al., 3 Mar 2026).

Recent LLM and agent benchmarks broaden the emphasis. GIM defines functionally-grounded evaluation as measuring models on realistic tasks whose difficulty arises from the integration of multiple cognitive operations rather than from obscure facts or purely abstract puzzles (Patel et al., 18 May 2026). GroundEval treats evidence-path validity as a deterministic, machine-checkable contract over state, access, time, and causal structure (Flynt, 22 Jun 2026). LH-Bench scores long-horizon agents by how well they achieve domain-specific functional objectives through multi-step workflows rather than by a single pass/fail endpoint (Chandwani et al., 24 Mar 2026). GIE-Bench evaluates whether a text-guided image edit happened in the grounded region where it should occur, while also verifying preservation outside that region (Qian et al., 16 May 2025).

This suggests that the term has acquired two closely related emphases. One emphasis is taxonomic: functionally-grounded evaluation is evaluation without humans-in-the-loop. The other is operational: evaluation is grounded when the task function, evidence, state, or rubric is explicit enough that performance can be checked mechanically or against realistic grounded conditions.

2. Grounding targets and what is being verified

Recent work grounds evaluation to different objects: cognitive operations, explanation properties, evidence paths, workflow artifacts, image regions, multilingual constraints, or canonical environment state.

Setting	What is functionally verified	Grounding mechanism
GIM	Integrated reasoning on realistic problems	Original expert-authored tasks, rubric decomposition, IRT calibration
GroundEval	Validity of the evidence path	State, access, time, and causal contracts over traces
LH-Bench	Functional success over long workflows	Expert-grounded rubrics and curated intermediate artifacts
FAGER	Factual correctness of generated images	Verified factual rubrics and visual QA
GIE-Bench	Intended edit and non-target preservation	MCQs plus object masks and masked metrics
Incognita	Consequential actions in social environments	Deterministic canonical state and offline reward
CL-IFEval / CL-GSM Symbolic	Instruction adherence and exact computations	Template variables, distractors, and deterministic checkers

GIM is explicit in opposing two benchmark design strategies. Knowledge-escalation benchmarks such as GPQA and HLE raise difficulty by moving into specialized, obscure expert domains, thereby conflating training-data coverage and memorization with reasoning capability. Abstraction-axis benchmarks such as ARC-AGI strip away world knowledge to isolate fluid intelligence, but strong performance there has not been shown to predict practical utility for tasks such as analyzing a contract, planning logistics, or auditing a technical claim. GIM instead centers difficulty in integration density and practical grounding, using broadly accessible knowledge and tasks that require constraint satisfaction, state tracking, epistemic vigilance, audience calibration, temporal reasoning, and procedural fidelity (Patel et al., 18 May 2026).

GroundEval grounds evaluation at the level of evidence use. It defines an evidence universe $E$, an access policy $P$, and a time window $[t_0,t_1]$, and evaluates whether the agent’s trace retrieved, cited, and relied on evidence that was actually available to the relevant actor at the relevant time (Flynt, 22 Jun 2026). Incognita grounds evaluation at the level of executable operations on a deterministic canonical state, but places those operations behind role-isolated entities, so that communication becomes exploration over socially partitioned knowledge and grounded execution becomes exploitation over state (Hsu et al., 3 Jul 2026).

In multimodal generation, FAGER grounds text-to-image evaluation in visually verifiable facts that are explicit in the prompt, implied by external knowledge, or identity-defining for the entity being depicted (Lim et al., 18 May 2026). GIE-Bench grounds image-edit evaluation to concrete target regions and non-target regions, using object-aware masks rather than global image–text similarity (Qian et al., 16 May 2025). A plausible implication is that “grounding” increasingly refers not only to real-world content, but also to an explicit binding between what should count as success and where, when, or how it must be realized.

3. Measurement architectures and scoring formalisms

Functionally-grounded evaluation typically couples a structured task design to a structured scoring rule. GIM routes scoring by rubric presence. For 528 of 820 problems, each rubric criterion is judged independently and returns a score $s_i \in [0,1]$ and confidence $c_i \in [0,1]$; the per-sample score is the confidence-weighted mean,

$\frac{1}{n}\sum_{i=1}^n s_i c_i.$

The benchmark then calibrates a continuous-response 2-parameter logistic IRT model across more than 200k prompt–response pairs, using a linear 2PL in logit space and a closed-form scorer for any subset of items (Patel et al., 18 May 2026). This makes ability estimates robust to missing data, modality filtering, and inference failures.

GroundEval combines answer correctness with trajectory validity. Its adjusted score is

$$S_{\text{adj}} = [w_a \cdot S_{\text{answer}} + w_t \cdot S_{\text{traj}}] \cdot (1-v)^2,$$

where $v$ is the observed violation rate over actor-gate, subsystem, and horizon violations (Flynt, 22 Jun 2026). The framework therefore scores both what the agent concluded and whether the conclusion was reached through a valid evidence path.

FAGER constructs a verified factual rubric

$R = \{(f_i, v_i, w_i, \ell_i, \gamma_i)\},$

then converts each fact into an atomic yes/no question whose intended answer is “yes.” The evaluation model answers each question with yes, no, or unknown; $unknown$ receives partial credit $P$0, and the final score is the weighted average over all questions. The system applies a coarse-to-fine gate, using $P$1 and $P$2 to decide whether to regenerate, edit, or keep an image (Lim et al., 18 May 2026).

GIE-Bench uses two separate axes. Functional correctness is measured by MCQ accuracy over automatically generated VQA-style multiple-choice questions that can only be answered correctly if the intended edit appears in the edited image. Preservation is measured over the complement of the target mask after geometric alignment, yielding masked MSE, masked SSIM, masked PSNR, and masked CLIP similarity (Qian et al., 16 May 2025).

In XAI, functionally-grounded scoring often takes the form of stability and fidelity metrics. The predictive process analytics study defines stability by subset, stability by weight, and internal fidelity as a MAPE-like change in prediction probability under perturbations of features selected by the explanation (Velmurugan et al., 2020). The tabular-XAI guidelines expand this into a taxonomy of 20 criteria spanning fidelity to the model, explanation invariance, fidelity to ground truth, XAI complexity, explanation safety, and application utility (Velmurugan et al., 2024).

In procedural QA generation, the validation framework measures groundedness, self-containedness, usability, and multi-hop coverage against a closed evidence set extracted from TMK models (Elshabrawy et al., 11 Jun 2026). In multilingual functional evaluation, CL-GSM Symbolic uses exact numeric matching against a known function $P$3 of template variables, while CL-IFEval uses deterministic constraint checkers for length, keyword frequency, case, and formatting (Ojewale et al., 25 Jun 2025).

4. Benchmark embodiments across domains

The breadth of applications is unusually large. In GIM, functionally-grounded evaluation is embodied as a benchmark of 820 original, expert-authored problems, with 615 public items and 205 private items, spanning text-only and multimodal tasks and designed to avoid obscure-fact gating (Patel et al., 18 May 2026). In GroundEval, it appears as a judge-free framework for stateful agents operating over enterprise state, where questions and trajectories are scored against explicit contracts derived from event logs, artifact corpora, access policies, and causal or silence specifications (Flynt, 22 Jun 2026). In LH-Bench, it takes the form of expert-grounded rubrics, curated ground-truth artifacts, and pairwise human preference validation for long-horizon subjective enterprise work such as Figma-to-code and programmatic content generation (Chandwani et al., 24 Mar 2026).

In multimodal generation, FAGER uses reference-guided fact extraction, LLM-based fact proposal and verification, and VLM-based question answering to evaluate whether generated images respect visually verifiable facts in science, history, products, culture, and knowledge-intensive concepts (Lim et al., 18 May 2026). GIE-Bench evaluates text-guided image editing across 1,080 curated pairs covering 9 edit types and 20 categories, with the evaluation grounded both to automatically generated MCQs and to human-validated masks (Qian et al., 16 May 2025).

In learning and cognition, NeuroCognition adapts Raven’s Progressive Matrices, Spatial Working Memory, and the Wisconsin Card Sorting Test into text and image formats with verifiable, process-aware metrics (Haznitrama et al., 3 Mar 2026). CompGuessWhat?! extends GuessWhat?! with a semantic attribute layer and defines goal-oriented evaluation, object attribute prediction via diagnostic classifiers, and zero-shot evaluation as a functionally-grounded suite for grounded language learning (Suglia et al., 2020). The TMK procedural work uses functionally-grounded validation not to score models directly, but to score the quality of generated question–answer datasets against a structured instructional representation (Elshabrawy et al., 11 Jun 2026).

This domain spread is significant. The same general idea is being used to evaluate explanation methods, dataset construction strategies, LLM reasoning, multilingual robustness, image editing, text-to-image factuality, long-horizon enterprise agents, and socially distributed action-taking. The unifying principle is that each setting specifies an observable functional contract before evaluation begins.

5. Empirical findings and recurrent failure modes

A central empirical pattern is that functionally-grounded evaluation often reveals failures hidden by conventional scoring. GroundEval reports a case in which two frontier LLM judges scored a plausible agent response 0.90 and 0.85, yet the agent had never retrieved the artifact its answer depended on; GroundEval scored the trajectory 0.000 because the evidence path violated the configured contract (Flynt, 22 Jun 2026). GIM shows that raw mean can misorder configurations when failures are treated as zeros: GPT 5.4 X-High had a lower raw mean than High because its failure rate was 2.3% versus 0.7%, yet the IRT estimate correctly ordered X-High above High, with $P$4 versus $P$5 (Patel et al., 18 May 2026).

The same pattern appears in multimodal editing. GIE-Bench shows that GPT-Image-1 leads in instruction-following accuracy at 85.00% overall, yet preservation metrics indicate substantial collateral modification of non-target regions, with masked SSIM 0.5704 and masked PSNR 13.36, whereas OneDiffusion preserves non-target content much better, with masked SSIM 0.9585 and masked PSNR 26.24 (Qian et al., 16 May 2025). FAGER’s factual A/B test similarly indicates that conventional alignment metrics miss implicit and identity-defining factual constraints; FAGER scores 0.73 on I-HallA-Science, 0.83 on I-HallA-History, 0.82 on ABO, 0.97 on Culture, and 0.87 on T2I-FactualBench-SKCM, outperforming FineGRAIN and VQAScore on each dataset (Lim et al., 18 May 2026).

Several works show that functionally-grounded evaluation changes conclusions about comparative model ability. LH-Bench reports that expert-authored rubrics yield higher inter-judge agreement than LLM-authored rubrics, with $P$6 versus $P$7, and that human preference judgments confirm the same top-tier separation in Figma-to-code (Chandwani et al., 24 Mar 2026). The TMK procedural study finds that strict TMK generation achieves 96.5% grounded questions and 92.6% usable questions, while transcript-first generation is more learner-like but more context-dependent and weakly grounded (Elshabrawy et al., 11 Jun 2026). The multilingual functional benchmarks report performance drops between static and functional evaluation, including decreases of 24%, 17%, and 18% between M-GSM and CL-GSM Symbolic in English, French, and Spanish, and a 15–24% drop across languages between Belebele and CL-IFEval (Ojewale et al., 25 Jun 2025).

Functionally-grounded evaluation also exposes process failures before reliable task success. In Incognita-Retail, success rises from 0% to 8.9% and 17.2% across three evaluated generative agent models, while premature finalization falls from 100% to 87% and 58% (Hsu et al., 3 Jul 2026). In NeuroCognition, strong performance on standard benchmarks does not eliminate failures on simple tasks for humans, and complex reasoning is not universally beneficial (Haznitrama et al., 3 Mar 2026). In CompGuessWhat?!, models reach solid in-domain gameplay accuracy yet show average attribute-prediction F1 around 44.27 and poor zero-shot robustness on novel scenes or objects (Suglia et al., 2020).

6. Limitations, controversies, and open directions

The literature repeatedly notes that functionally-grounded evaluation is powerful but not complete. In the XAI taxonomy, it is explicitly presented as a foundational step toward human-grounded and application-grounded evaluation rather than a replacement for them (Velmurugan et al., 2020). The TMK procedural work states that dataset-quality proxy measures do not translate directly to learner outcomes or system performance (Elshabrawy et al., 11 Jun 2026). The tabular-XAI guidelines emphasize that evaluation outcomes can be dominated by perturbation strategy, neighborhood construction, and other protocol choices, especially in the presence of mixed feature types, missingness, domain constraints, and feature interactions (Velmurugan et al., 2024).

Many contemporary benchmarks also depend on expensive or proprietary components. GIM notes a single proprietary judge dependency and per-criterion judging cost that currently blocks panel-of-judges scale-ups, although the authors plan to release a re-calibrated bank under an open-weight judge (Patel et al., 18 May 2026). GroundEval requires front-loaded authoring of domain configurations, including event types, causal links, silence pair specifications, and access policies, and context mode cannot verify reliance on uncited artifacts in an injected window (Flynt, 22 Jun 2026). LH-Bench identifies harness coupling to model families and residual judge biases, even with multi-judge scoring and human validation (Chandwani et al., 24 Mar 2026).

Open directions are correspondingly concrete. GIM proposes extensions to audio, video, and agentic multi-step processes while retaining rubric authoring, IRT calibration, and a private split for contamination checks (Patel et al., 18 May 2026). GroundEval points to multi-agent and longer-horizon interactions, richer mechanism vocabularies, and structural causal models for counterfactual rollouts (Flynt, 22 Jun 2026). FAGER suggests threshold calibration and broader multimodal extensions such as text-to-video (Lim et al., 18 May 2026). GIE-Bench identifies multi-step editing, interactive refinement, and stronger mask quality control as future work (Qian et al., 16 May 2025). The multilingual functional benchmarks call for professionally translated templates, broader language coverage, and expansion to code and multimodal tasks (Ojewale et al., 25 Jun 2025).

Taken together, these works indicate that functionally-grounded evaluation is less a single method than a design principle. It requires that the evaluated system’s target function be externalized into a contract: a rubric, a checker, a state transition system, a support relation, a mask, a fact bank, or a closed evidence set. Where that contract is precise, evaluation becomes more reproducible, more diagnostic, and often more conservative than static accuracy or free-form judging. Where the contract is incomplete, expensive, or brittle, human-grounded and application-grounded evaluation remain necessary complements.