Diffusion-Based Artifact Localization

• Diffusion-based artifact localization is a set of techniques that leverage temporal dynamics and model-internal representations to isolate and correct artifact-prone regions in generated images.
• These methods integrate unsupervised score dynamics, supervised segmentation, and explainable AI to pinpoint stage-specific anomalies during image synthesis and enhancement tasks.
• The approaches enhance diagnostic precision and enable on-the-fly corrective mechanisms while preserving essential image details and overall fidelity.

Diffusion-based artifact localization refers to a class of techniques designed to precisely identify and, in many cases, correct visual artifacts that arise during generative processes in diffusion models. Unlike global image-level scoring or naive spatial uncertainty estimation, these methods leverage the intrinsic temporal dynamics and model-internal representations of diffusion, combined in some cases with supervised or explainable vision models, to isolate artifact-prone regions at fine spatial resolution. Applications span image synthesis, super-resolution, inpainting, and forgery detection, with both diagnostic and corrective roles.

1. Mechanisms of Artifact Formation in Diffusion Models

Artifacts in diffusion models are not spatially random but result from complex, stage-specific phenomena during the sampling trajectory. Temporal score analysis delineates three distinct process regimes (Cao et al., 20 Mar 2025):

• Profiling (Early Steps): The model synthesizes coarse, low-frequency structure. Score magnitudes are globally smooth and slowly varying.
• Mutation (Middle Steps): Local variations and textures are stochastically introduced. Here, certain pixels undergo abrupt score accelerations and decelerations—so-called "score traps"—that decouple them from the generative context. These are precursors to visible artifacts.
• Refinement (Late Steps): The model regularizes local detail into globally consistent patterns. Most local anomalies are attenuated unless score traps have already formed, in which case persistent artifacts are evident.

This temporal decomposition, supported by activations of the learned score network $s_\theta(x_t, t)$ and its weighted finite-difference increments, establishes that artifact localization requires dynamic multi-timestep analysis rather than static spatial heuristics (Cao et al., 20 Mar 2025). Similar temporal or stage-resolved phenomena underlie practical artifact correction strategies in super-resolution (e.g., SARGD (Zheng et al., 2024)) and in face forgery localization (e.g., via DSSIM evolution in DiffusionFF (Peng et al., 3 Aug 2025)).

2. Algorithmic Architectures for Artifact Localization

Diffusion-based artifact localization algorithms typically fall into the following categories:

• Score-Dynamics Based Unsupervised Localization: ASCED (Cao et al., 20 Mar 2025) computes weighted score increments at each pixel and timestep, then identifies anomalies where these exceed a data-adaptive threshold (e.g., MAD or running mean). Detected regions are accumulated over a detection window, resulting in masks $\Omega^a = \bigcup_t \Omega^a_t$.
• Explainable-AI (XAI) Driven Guidance: Self-Refining Diffusion (Lee et al., 9 Dec 2025) employs a Grad-CAM-based flaw activation map (FAM), derived from a binary real-vs-fake VGG-16 classifier, averaged batchwise to form a mean flaw map $\bar M$. These FAMs are injected as saliency priors both for the forward diffusion (modulating noise injection) and reverse denoising (modulating attention).
• Supervised Pixel-Level Detectors: DiffDoctor (Wang et al., 21 Jan 2025) uses a per-pixel segmentation network (SegFormer-b5 backbone) regression-trained on human-annotated artifact masks and large-scale semi-supervised pseudo-labels, outputting dense confidence maps. SARGD (Zheng et al., 2024) employs PAL artifact detectors to yield binary artifact masks per decoding step.
• Diffusion-Conditioned Quality Map Synthesis: DiffusionFF (Peng et al., 3 Aug 2025) applies a diffusion model to iteratively predict DSSIM maps that specify where, structurally, an image deviates from reference patterns. These DSSIM maps are spatially sharp and can be fused with semantic detector features for localization.

Each approach combines spatial and temporal cues, with varying reliance on supervised labels or auxiliary models.

3. Mathematical Formulations

The following summarizes core mathematical concepts underpinning prominent methods:

Method	Core Formula / Detection Metric	Notation
ASCED	$$\Omega^a_t = \{(i,j) : |\Delta[w(t) s_\theta(x^{i,j}_t, t)]| > \tau\}$$	Score increments, threshold $\tau$
Self-Refining	$\bar{M} = \frac{1}{B}\sum_b M^{(b)}$	FAMs by Grad-CAM, averaged over batch
DiffDoctor	$$L_\text{AD}(\theta) = \frac{1}{N}\sum_i \| \hat H_\theta(x_i) - C(x_i) \|^2_2$$	Segmentation regression
SARGD	$E_A^{(t)} = \mathcal{A}(I^{(t)})$	Detector mask at step $t$
DiffusionFF	$$y_t = \sqrt{\bar\alpha_t} y_0 + \sqrt{1-\bar\alpha_t}\epsilon$$	Diffused DSSIM map

• ASCED's Anomaly Score: $$L_{\rm anomaly}(x_t^{i,j}) = |\Delta[w(t) s_\theta(x_t^{i,j}, t)]|$$, exceeding $\tau$ flags an artifact pixel (Cao et al., 20 Mar 2025).
• Self-Refining Diffusion: $M$ is computed via Grad-CAM. During forward diffusion, noise is amplified in $\bar M$-highlighted regions: $\epsilon_\text{sr} = \epsilon + \lambda \bar{M}$; attention weights in the reverse process are modulated by embedding $\bar{M}$ (Lee et al., 9 Dec 2025).
• DiffDoctor: The network predicts per-pixel artifact probabilities, trained via MSE against binary segmentation masks from manual and semi-supervised sources (Wang et al., 21 Jan 2025).
• SARGD: Localizes artifacts by binary masking in each decoding step, then replaces latent channels with those from a fixed, realistic "guide" latent vector (Zheng et al., 2024).
• DiffusionFF: The diffusion model reconstructs the DSSIM map, which pinpoints spatial regions of structural deviation due to manipulation (Peng et al., 3 Aug 2025).

4. Corrective and Feedback Mechanisms

Detection masks and localization cues obtained from the aforementioned algorithms directly inform artifact mitigation. Several approaches to correction are documented:

• On-the-Fly Stochastic Correction: ASCED introduces targeted noise into identified artifact regions at a correction time $T_c$—trajectory-aware targeted correction (TTC)—re-coupling anomaly pixels with the generative context. This is achieved without replaying earlier states or naïve post-hoc smoothing (Cao et al., 20 Mar 2025).
• Attention and Noise Modulation: Self-Refining Diffusion uses FAMs to both amplify randomness during forward steps in flawed areas and increase denoiser attention towards them. This dual strategy leverages the complementary action of forward and reverse correction (Lee et al., 9 Dec 2025).
• Masked Latent Replacement: SARGD replaces artifact-labeled latent spatial elements with the corresponding components from a realistic reference latent, iteratively updating this reference based on post-correction reality scores—thereby maintaining sharpness while suppressing artifacts (Zheng et al., 2024).
• Gradient Feedback and Fine-Tuning: DiffDoctor propagates pixel-level feedback loss through the last steps of the diffusion process, updating parameters (often only LoRA adapters) to disincentivize the generation of artifacts mapped by the detector (Wang et al., 21 Jan 2025).

A notable observation is that unsupervised dynamic correction (e.g., ASCED) can match or surpass the best supervised post-hoc strategies in both localization and fidelity, without extra training data or substantial increases in computational budget (Cao et al., 20 Mar 2025).

5. Quantitative Benchmarks and Qualitative Analyses

Empirical results are available for several benchmark datasets, summarizing the effectiveness of diffusion-based artifact localization and correction methods:

Dataset	Method	Detection/Loc. Metric	FID↓	Notable Findings
FFHQ	ASCED	Accuracy 56.7%	Best FID	Unsupervised ASCED close to sup. SARGD/PAL (Cao et al., 20 Mar 2025)
ImageNet	ASCED	Accuracy 67.7%	High Recall	Outperforms unsup. baselines, close to sup. (Cao et al., 20 Mar 2025)
CelebA-HQ	SelfRef	FID 8.369 (vs 8.985 DDPM)	PSNR↑, SSIM↑	FAMs outperform edge/center maps (Lee et al., 9 Dec 2025)
DiffusionFF	-	Cross-dataset AUC 97.24% (CDF2)	FID 43.09	Superior localization vs. direct regression (Peng et al., 3 Aug 2025)
SARGD	-	PSNR/SSIM/LPIPS better than StableSR/LDM	2X speedup	Artifact locality clear in visual overlays (Zheng et al., 2024)
DiffDoctor	-	MSE on held-out: 0.34 (with 1M pseudo)	-	Dense detector generalizes across domains (Wang et al., 21 Jan 2025)

Qualitative visualizations confirm that localized score increments, FAM maps, and DSSIM reconstructions all exhibit strong correspondence to genuinely perceived artifacts (blur, disjunctions, unnatural features) and outperform global or low-resolution baselines in both spatial precision and perceptual alignment (e.g., 87% overlap with human-drawn flaws for FAMs (Lee et al., 9 Dec 2025)).

6. Applications and Broader Implications

• Image Synthesis and Inpainting: Artifact localization enables in-process correction, boosting overall fidelity and perception-based metrics without sacrificing diversity (Cao et al., 20 Mar 2025, Lee et al., 9 Dec 2025).
• Super-Resolution: Localized latent guidance and correction strategies, as in SARGD, yield high-frequency detail and suppress artifacts even at reduced sample counts (Zheng et al., 2024).
• Face Forgery Detection: DiffusionFF shows that artifacts indicative of manipulation can be spatially localized using diffusion-driven DSSIM maps, significantly improving detection generalization and explanation (Peng et al., 3 Aug 2025).
• Model Debugging and Forensics: DiffDoctor demonstrates that systematic artifact localization can directly inform robust model fine-tuning via pixel-level feedback, reducing failure rates across challenging semantic categories (Wang et al., 21 Jan 2025).

A plausible implication is that temporal or stage-aware artifact localization within the diffusion trajectory, potentially in conjunction with supervised or XAI-driven spatial priors, will become standard in next-generation generative model evaluation and deployment pipelines. This could result in a shift from purely global, image-level assessment to process-aware, dynamical robustness strategies.

7. Limitations and Open Problems

• Detection/Correction Tradeoff: Some approaches (e.g., aggressive artifact masking in SARGD or DiffDoctor) risk over-smoothing or excessive suppression of image detail, particularly in low-capacity or heavily penalized regimes (Zheng et al., 2024, Wang et al., 21 Jan 2025).
• Class-Imbalance and Semantic Generalization: Dense supervised detectors must balance artifact-prone and artifact-free categories, requiring large, balanced annotation and careful negative sampling to avoid bias (Wang et al., 21 Jan 2025).
• Unsupervised-Label Efficacy: While methods like ASCED achieve strong performance with no human annotation, they depend critically on accurate temporal weighting and anomaly criteria (Cao et al., 20 Mar 2025).
• Explainability vs. Corrective Power: Grad-CAM-based FAMs align well with human intuition and perceptual judgments but require strong pretrained backbones and may miss model-specific artifact patterns (Lee et al., 9 Dec 2025).

Continued research is required to unify these localization paradigms, further reduce computational burden, and close the gap between automated detection and subjective perceptual robustness.