Behavior of confidence scores under distribution shifts in multimedia forensics

Characterize the behavior of confidence scores produced by multimedia forensic detectors when the test distribution differs from the training distribution, to ascertain how these scores respond and potentially miscalibrate under distribution shifts.

Background

The paper advocates for AI forensic agents that orchestrate multiple detectors and explicitly quantify uncertainty. A key requirement is that these systems not only provide predictions but also calibrated confidence scores and the ability to abstain when evidence is insufficient.

The authors note that current multimedia forensics often treats uncertainty superficially. Despite detectors producing various scores (detection, similarity, anomaly), confidence calibration is uncommon, and, crucially, how these confidence scores behave under distribution shifts remains unknown. This gap impedes reliable fusion and decision-making in real-world settings where data distributions frequently change.

References

In current multimedia forensics practice, uncertainty is often treated superficially: confidence scores are rarely calibrated, and their behavior under distribution shifts is largely unknown.

— Don't Guess, Escalate: Towards Explainable Uncertainty-Calibrated AI Forensic Agents  (2512.16614 - Boato et al., 18 Dec 2025) in Section: TRUSTWORTHY FORENSIC AGENTS

Calibration from an unfiltered deployment stream remains to be evaluated.

— SPARC: Subspace Position-Aware Robust Few-Shot Calibration for Distribution-Shifted Industrial Anomaly Detection  (2608.18585 - Han et al., 19 Aug 2026) in Appendix, Section Robustness and Sensitivity, subsection Calibration-Set Contamination (Section \ref{sec:supp-ab4})

Mirroring the device mix of web collections, the corpus probes the first neural-ISP generation ($2018$--$2020$); current flagships are essentially absent, leaving the modern-ISP regime as the open gap.

— LAION-Mobile: Evaluating Deepfake Detectors On One Million Smartphone Photos  (2609.11134 - Stryk et al., 10 Sep 2026) in Abstract; Section 3, Figure 3 caption; Section 4.3; Section 7

At the same time, this analysis is limited to the five-split LOIO benchmark considered here, so whether calibration remains equally reliable under more extreme acquisition shifts, such as substantially different noise statistics or sampling regimes, remains an open question.

— SpikeSSL: A Universal Spike Inference Framework with Dynamics-Informed State-Space Layers  (2610.11456 - Yue et al., 8 Oct 2026) in Section Discussion, subsection “Uncertainty Calibration”