Decisive Feature Fidelity (DFF)

• Decisive Feature Fidelity (DFF) is a SUT-specific metric that measures causal mechanism parity by comparing decisive features from synthetic and real data.
• It leverages counterfactual-XAI to generate decisive maps that pinpoint regions responsible for model outputs under domain variations.
• DFF identifies hidden sim-to-real gaps overlooked by pixel-level measures and guides calibration to improve simulator and generator alignment.

Decisive Feature Fidelity (DFF) is a system-under-test (SUT)-specific metric that quantitatively measures the “mechanism parity” between synthetic and real imagery by comparing the decisive features—the regions or attributes causally responsible for the SUT’s outputs—across matched data pairs. Unlike traditional fidelity measures that focus on pixel-level similarity or output-value consistency, DFF is grounded in the actual decision mechanisms of the SUT, leveraging explainable-AI (XAI) techniques to interrogate and align the model’s causal attributions under domain variation. DFF enables identification and remediation of hidden sim-to-real gaps that are invisible to output- or input-level metrics, making it a pivotal tool for safety-critical validation in domains such as autonomous vehicle virtual testing (Safaei et al., 18 Dec 2025).

1. Formal Definition and Conceptual Foundations

Let $F: \mathbb{R}^{d_0} \to \mathbb{R}^{d_L}$ denote the SUT (e.g., end-to-end driving policy, perception network). For inputs $x_r$ (real) and $x_s$ (synthetic) generated under a matched scenario description $SD$, and an explainability map generator $\mathcal{H}$, DFF is defined over the explanation space via a distance function $\mathrm{Dist}(e_1,e_2)$ (e.g., mean-squared error between heatmaps). DFF-fidelity is attained if all three of the following hold:

1. Input-value fidelity: $$\mathrm{D}_{\mathrm{in}}(x_s, x_r) \leq \varepsilon_{\mathrm{in}}$$
2. Output-value fidelity: $$\mathrm{D}_{\mathrm{out}}(F(x_s), F(x_r)) \leq \varepsilon_{\mathrm{out}}$$
3. Decisive-feature proximity: $$\mathrm{Dist}(\mathcal{H}(F(x_s)), \mathcal{H}(F(x_r))) \leq \varepsilon_{\mathrm{dff}}$$

where $$\varepsilon_{\mathrm{in}}, \varepsilon_{\mathrm{out}}, \varepsilon_{\mathrm{dff}}$$ are explicit user-specified tolerances. The proportion of pairings satisfying (3) at threshold $\varepsilon_{\mathrm{dff}}$ yields the DFF pass-rate: $$\mathrm{PassRate} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\left[\mathrm{Dist}(\mathcal{H}(F(x_{s,i})), \mathcal{H}(F(x_{r,i}))) \leq \varepsilon_{\mathrm{dff}}\right]$$ DFF explicitly expands the fidelity spectrum to include mechanism parity, i.e., agreement in causal evidence underlying SUT decisions across domains (Safaei et al., 18 Dec 2025).

2. Identification of Decisive Features via Counterfactual Explanations

The core of DFF is the identification of decisive features that influence the SUT’s output for an input image. DFF operationalizes this via a counterfactual-XAI (CF–XAI) explainer. For a given input $x$, the CF–XAI method seeks a sparse binary or soft mask $m^\star(x)$ where removing or infilling the masked pixels suffices to flip the SUT’s prediction. Averaging across $K_{\mathrm{cf}}$ random seeds mitigates stochasticity: $$\mathcal{H}(F(x)) = \frac{1}{K_{\mathrm{cf}}} \sum_{k=1}^{K_{\mathrm{cf}}} m^\star(x; \zeta_k)$$ where $\zeta_k$ denotes seed for the infilling prior or mask optimization. This map is interpreted as a “decisive map” highlighting image regions causally responsible for the specific output.

3. Practical Estimator and Algorithmic Workflow

Assessment of DFF for a matched data pair proceeds as follows:

1. Compute decisive maps $\mathcal{H}(F(x_r))$ and $\mathcal{H}(F(x_s))$ using the CF–XAI method with mask averaging.
2. Pool the decisive maps to a common spatial resolution (e.g., $16 \times 16$ grid).
3. Compute explanation distance via mean-squared error or analogous metric.
4. Threshold at $\varepsilon_{\mathrm{dff}}$ for pass/fail.

For calibration or model guidance, decisive-feature distances can be incorporated as a loss term, enabling parameter updates to synthetic generators with respect to DFF-based objectives.

4. DFF-Guided Calibration for Simulator/Generator Alignment

Beyond passive assessment, DFF supports active correction of sim-to-real mechanism gaps. By introducing a calibrator network $C_\eta$ predicting adjustments to a synthetic generator’s parameters ($\Theta^* = C_\eta(x_s^{\mathrm{init}}, SD)$), calibration seeks to minimize DFF distance while maintaining output-value performance: $$L(\eta) = \sum_i \Big[ L_{\text{recon}}(x_{s,i}^*, x_{r,i}) + \beta\,L_{\text{OV}}(F(x_{s,i}^*), F(x_{r,i})) + \lambda_{\text{dff}}\,\mathrm{Dist}(\mathcal{H}(F(x_{s,i}^*)), \mathcal{H}(F(x_{r,i}))) \Big]$$ Backpropagation flows through $C_\eta$ and generator $G_\Theta$ (continuous or evolutionary strategies), but not through the fixed SUT $F$. The effect is direct closing of the mechanism gap—minimization of divergences in the causal features exploited by the SUT—subject to output non-inferiority (Safaei et al., 18 Dec 2025).

5. Experimental Findings and Metric Interpretations

Empirical validation on 2,126 real-synthetic frame pairs from KITTI and VirtualKITTI2, across PilotNet-style steering regressors and YOLOP segmentation heads, demonstrates that DFF reveals mechanism gaps overlooked by conventional metrics. Output-value distances (e.g., steering error, mask IoU) may cluster tightly even as DFF distances vary widely (Spearman correlation near zero), confirming that output agreement does not guarantee concurrent mechanism parity.

DFF-guided calibration reduces DFF and input-value divergence without sacrificing output-value fidelity. Specifically:

• DFF-calibrated variants show negative shifts in $\Delta\mathrm{DFF}$ (mechanism alignment) while $\Delta\mathrm{OV}$ remains stable and $\Delta\mathrm{IV}$ is improved.
• Qualitative inspection reveals that DFF calibration targets texture, structure, and visual features in decisive regions, often ignored by output-focused optimization. Output-only calibration may fix task performance while neglecting true causal evidence.

Thresholds for DFF pass are derived empirically (e.g., at the 90th or 95th percentile of the calibration distribution).

Metric	Realization	Lower/Better	Output Value (OV) Example
Input Fidelity	LPIPS	Yes	-
Output Fidelity	$\exp(-5| \theta_r-\theta_s |)$ (steering), Mask IoU (YOLOP)	-	Scalar, mask
DFF	MSE on $16\times16$ maps	Yes	-

6. Advantages, Limitations, and Prospective Extensions

Advantages

• SUT-specific, enabling behavior-grounded fidelity assessment.
• Captures nontrivial mechanism gaps missed by pixel- and output-distance.
• Enables feedback-driven calibration of simulators or generators.

Limitations

• Dependent on the reliability of the chosen XAI method, with variance mitigated by multi-seed averaging.
• Currently evaluated only for camera image inputs; generalization to other modalities (e.g., LiDAR) necessitates suitable explainers.
• Computational overhead due to repeated, expensive counterfactual optimization.
• Focused on single-frame analysis; does not address closed-loop or temporal consistency.

Future Directions

• Extension to sequence-level or multi-modal explanations (e.g., fusing LiDAR and camera).
• Theoretical analysis to characterize when low DFF guarantees safety-aware fidelity or robust sim-to-real transfer.
• Exploration of alternative explanation paradigms (concept-based, attention-based).
• Closing the sample-complexity gap for practical DFF estimation in large-scale scenarios.
• Deeper integration of DFF into the design of complex simulators, addressing broader scene parameters including traffic, weather, and lighting.

7. Relation to Broader Fidelity Assessment and Context in Simulation

DFF generalizes the traditional fidelity spectrum by shifting the evaluative focus from superficial appearance and pure task outcome to the causal, evidence-based mechanism of decision-making within the SUT. This mechanism-centric lens is particularly crucial in safety- and policy-critical virtual testing regimes, where unacknowledged shifts in decisive feature utilization can undermine real-world validity despite ostensibly high output-value concordance (Safaei et al., 18 Dec 2025). A plausible implication is that for robust deployment of learning-based perception and control stacks, mechanism parity (as measured by DFF) should be treated as a primary target for alignment and verification, potentially informing regulatory standards for virtual scenario validation.