Functionally-Grounded Metrics

• Functionally-grounded metrics are quantitative evaluation methods that rely solely on formal properties without human intervention.
• They operationalize key criteria across domains like machine translation, explainable AI, finance, and software engineering through explicit algorithms and loss functions.
• These metrics provide objective benchmarks for model fidelity, stability, risk, and interpretability while also highlighting limitations in assumptions and calibration.

Functionally-grounded metrics are quantitative evaluation criteria and protocols that assess the intrinsic quality or utility of a model, system, or explanation method based solely on formally defined properties—without the involvement of human raters or task-oriented end goals. This approach is distinct from human-grounded or application-grounded metrics, as it relies exclusively on computational or model-centric tests that can be executed automatically, resulting in reproducible and interpretable scores. Functionally-grounded metrics are fundamental in diverse domains such as machine translation, explainable AI, financial modeling, and software engineering, where they enable systematic comparison of models or explanations on the basis of attributes like fidelity, stability, risk, or interpretability (Kocmi et al., 2024, Velmurugan et al., 2020, Velmurugan et al., 2024, Castro et al., 2018, Khubiev et al., 4 Sep 2025).

1. Foundational Concepts and Scope

Functionally-grounded metrics operationalize theoretical desiderata or practical criteria by defining explicit algorithms, formulas, or loss functions that can be computed on data without reference to human judgment. In machine translation, metrics such as BLEU, COMET, and BLEURT are used to score generated texts; in explainable AI (XAI), criteria like stability, fidelity, and parsimony are instantiated through concrete mathematical formulas; in finance, established risk-return statistics such as Sharpe Ratio and Maximum Drawdown are used as performance metrics. The primary motivation is to provide standardized, objective, and automatable means for comparing systems and for aligning model optimization with formalized expectations regarding utility, fairness, or safety (Kocmi et al., 2024, Velmurugan et al., 2024, Khubiev et al., 4 Sep 2025).

2. Methodologies for Designing and Computing Functionally-Grounded Metrics

The derivation of functionally-grounded metrics varies substantially by domain:

• Metric–Accuracy Calibration: For machine translation, Kocmi et al. map metric score deltas (Δ) to human noticeability by fitting systemic accuracy curves to empirical distributions, using parametric sigmoids ($f(\Delta) = \phi_1 / (1 + \exp(-\phi_2 \Delta))$) for interpretability and invertibility, resulting in closed-form formulas for the threshold δ needed to reach a desired accuracy (e.g., 80% agreement with human rankings) (Kocmi et al., 2024).
• Feature Attribution Evaluation: In XAI, functionally-grounded metrics quantify aspects such as:
  • Subset stability: $φ_{subset}$
  • Attribution weight stability: $φ_{weight}$
  • Internal fidelity via mean absolute percentage change: $\mathcal{F}(x)$ (Velmurugan et al., 2020)
  • Fidelity, invariance, safety, and utility through a taxonomy of 20 criteria with explicit protocols and (often) LaTeX formalization (Velmurugan et al., 2024).
• Finance: Loss functions are derived from domain-specific risk/reward statistics—PnL, Sharpe, Maximum Drawdown, and Turnover—yielding differentiable objectives ($L_{Sharpe}$, $L_{RiskAdj}$, $L_{MDD}$, $L_{total}$) targeting trader-relevant performance for optimization in reinforcement learning or deep learning (Khubiev et al., 4 Sep 2025).
• Software Metrics: “Functional Elements” (EF) provides a continuous, regression-based size metric grounded in software function-point analysis, overcoming known weaknesses of ordinal measurement through strictly quantitative, continuous aggregation formulas (Castro et al., 2018).

3. Key Criteria and Formal Taxonomies in Explainable AI

A major contribution in the XAI literature is the formalization of 20 functionally-grounded criteria, grouped into six categories (Velmurugan et al., 2024):

Dimension	Example Metrics	Evaluation Method
Fidelity to the Model	Internal Correctness, Completeness, Descriptive Accuracy	Perturbation, glass-box, surrogate modeling
Explanation Invariance	Similarity, Sensitivity, Synonymity	Multiple runs, cluster analysis
Fidelity to Ground Truth	Coverage, Realism, Sufficiency	Domain knowledge, synthetic benchmarks
Complexity	Computation time, Parsimony	Runtime, entropy, feature count
Explanation Safety	Adversarial robustness, Subgroup fairness	Robust perturbations, subgroup comparison
Application Utility	Suitability, Degree of Explainability	Task-based scoring, language-model heuristics

Each criterion is defined mathematically, accompanied by a standard evaluation protocol, and contextualized with domain-specific guidelines. A central focus is to ensure that scores are meaningful and comparable despite differences in feature space (tabular data, mixed types, high-dimensionality) or explainer architecture.

4. Empirical Evaluation Protocols and Data-Driven Thresholding

Functional grounding is not purely formalistic—large-scale empirical evaluations enable the mapping of metric values or deltas to interpretable, task-relevant utility. In translation, this occurs via binning system pairs by Δ and empirically estimating the accuracy with respect to human judgment. The result is a direct link between metric improvement and desired confidence in human-observable gains. In XAI, local instance-level metrics are averaged over cases and used to benchmark methods like LIME and SHAP for stability ($φ_{subset}$, $φ_{weight}$) and direct effect-measurement (fidelity). In finance, extensive backtesting links loss minimization with improved portfolio returns, reduced drawdown, and Sharpe ratio improvement (Kocmi et al., 2024, Velmurugan et al., 2020, Khubiev et al., 4 Sep 2025).

5. Applications Across Domains

Machine Translation: Functionally-grounded metrics allow practitioners to select system improvements that are reliably observable by humans. For example, a +2.34 BLEU increase is required for 80% human-agreement in system ranking, while a +0.56 COMET '22 improvement suffices—enabling cross-metric calibration and stable reporting irrespective of corpus size or statistical power artifacts (Kocmi et al., 2024).

Explainable AI: Functionally-grounded metrics are pivotal in evaluating the trustworthiness of model explanations in predictive process analytics and tabular contexts. They reveal systematic instabilities of explainers (e.g., LIME vs SHAP), enable rigorous benchmarking via fidelity and invariance, and highlight key shortcomings such as the generally moderate internal fidelity of local explanations ($φ_{subset}$0) (Velmurugan et al., 2020, Velmurugan et al., 2024).

Finance: Optimizing predictive models directly on finance-grounded metrics such as Sharpe ratio and Maximum Drawdown leads to substantive improvements in real-world trading outcomes over generic losses (e.g., MSE). This functional alignment is realized through custom, differentiable objective functions used in model training (Khubiev et al., 4 Sep 2025).

Software Engineering: The EF metric, a continuous functionally-grounded alternative to IFPUG FP, exhibits higher empirical correlation with development effort, supports incremental sizing, and integrates into IT governance dashboards and contracts, highlighting the governance utility of functionally-grounded metrics (Castro et al., 2018).

6. Limitations, Theoretical Assumptions, and Open Challenges

Several challenges are endemic to functionally-grounded evaluation:

• Some criteria assume local linearity, additive feature contributions, or plausible background distributions for perturbations—assumptions often violated in high-dimensional or correlated tabular data (Velmurugan et al., 2024).
• Instabilities in explanation methods (e.g., LIME) are exacerbated by feature dimensionality, while high-fidelity surrogate explainer assumptions may not hold in practice.
• Empirically derived thresholds (e.g., MT metrics) may not generalize perfectly across domains, language pairs, or system similarity regimes (Kocmi et al., 2024).
• The absence of standardized distance metrics, benchmark datasets, and theoretical understanding of neighborhoods hampers full reproducibility and comparability (Velmurugan et al., 2024).
• For rapidly evolving domains (e.g., finance), functional metrics must be continually revalidated vis-à-vis market structure and model drift (Khubiev et al., 4 Sep 2025).

Emerging safety, fairness, and utility metrics (e.g., adversarial robustness, subgroup fairness, DoX) remain underdeveloped, with open questions regarding their standardization and interpretation in deployment contexts (Velmurugan et al., 2024).

7. Practitioner Guidance and Future Directions

Best practices synthesized from the literature recommend:

• Reporting, alongside any system or model delta ($φ_{subset}$1), the estimated system-level accuracy or task-relevant effect, using functionally-grounded calibration (e.g., $φ_{subset}$2 in MT) (Kocmi et al., 2024).
• Selecting primary metrics with demonstrated superior accuracy at lower deltas (e.g., COMET '22 in MT, SHAP in XAI) and supplementing with orthogonal secondary metrics to guard against blind spots.
• Avoiding reliance on statistical significance (e.g., $φ_{subset}$3-values) divorced from functional effect size, as large data can make trivial deltas appear significant (Kocmi et al., 2024).
• Tailoring criteria and evaluation protocols to domain-specific data characteristics (e.g., feature types, sparsity, intercorrelation) and adopting rigorous cross-domain/instance benchmarking (Velmurugan et al., 2024, Velmurugan et al., 2020).
• Recognizing and quantifying the tradeoff between interpretability, parsimony, and fidelity in explanation methods (Velmurugan et al., 2024).

Future research is required to close gaps in metric standardization, support functional evaluation in mixed and temporally-evolving tabular data, and establish robust protocols for emerging safety and utility-related metrics (Velmurugan et al., 2024). There is also a need to further bridge high-level desiderata taxonomies with explicit, reproducible, functionally-grounded protocols across application domains.