In-Generation Detection (IGD) in Diffusion Models

• In-Generation Detection (IGD) is a safety-monitoring paradigm that analyzes intermediate denoising stages in diffusion models to identify unsafe content before full image synthesis.
• IGD leverages model-internal signals such as predicted noise and decoded latent representations, enabling early detection of NSFW intent even against adversarial prompts.
• Empirical results show IGD methods achieve high detection accuracy and efficiency, though challenges remain in subtype discrimination and cross-model consistency.

Searching arXiv for papers on in-generation detection in generative models. I’ll look up recent arXiv papers related to in-generation detection, especially diffusion-model safety detection. In-Generation Detection (IGD) denotes a safety-monitoring paradigm for generative models in which detection is performed during the generation process rather than before generation from prompts or after generation from completed outputs. In the recent literature represented here, IGD is formulated for diffusion-based text-to-image models, where the detector inspects model-internal states at intermediate denoising steps to identify Not-Safe-for-Work (NSFW) intent or content early enough to permit intervention, including aborting the generation before completion (Yang et al., 5 Aug 2025, Yang et al., 9 Apr 2026).

1. Conceptual scope and placement in the safety pipeline

IGD was introduced as a third operating point in the moderation pipeline alongside pre-detection and post-detection. Pre-detection filters prompts before image generation; post-detection analyzes output images after generation; IGD instead monitors generation-time internals during denoising itself (Yang et al., 5 Aug 2025). This distinction is central because the in-generation phase of diffusion models was described as largely unexplored for NSFW detection, despite diffusion models’ iterative structure providing internal signals before a final image is synthesized (Yang et al., 5 Aug 2025).

The motivation is partly operational and partly epistemic. Operationally, early detection can save computation by terminating unsafe generations before the full denoising trajectory finishes. Epistemically, prompt-only moderation faces a gap between prompt safety and image safety, while final-image moderation cannot act until the unsafe image already exists (Yang et al., 9 Apr 2026). IGD therefore treats the denoising trajectory itself as the object of analysis.

A common misconception is to equate IGD with prompt classification. The cited work explicitly separates the two. In these papers, the critical observation is that internal denoising signals may retain the semantic intent of unsafe generation even when prompts are adversarially rewritten to evade text-based filters (Yang et al., 5 Aug 2025). This suggests that safety-relevant evidence can be more stable in model-internal trajectories than in surface text alone.

2. Diffusion-model basis and internal observables

The mathematical basis for IGD in diffusion models follows the standard forward and reverse processes. In the 2025 IGD formulation, if $x^0$ is the original image latent, $x^t$ the noisy latent at timestep $t$, and $c$ the prompt encoding, then the forward process is

$$x^t = \sqrt{\bar{\alpha}^t} x^0 + \sqrt{1 - \bar{\alpha}^t} \, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I),$$

and the reverse process predicts noise via

$\epsilon_t = \epsilon_\theta(x^{t+1}, t, c).$

IGD leverages the predicted noise $\epsilon_t$ at a particular early timestep as a semantic feature for NSFW/SFW discrimination (Yang et al., 5 Aug 2025).

The core empirical intuition is that predicted noise is conditioned on the text prompt and reflects the intended semantics of the prompt even before the actual image is synthesized. The paper reports that predicted noise vectors for NSFW prompts, including adversarial prompts, form coherent, separable clusters in feature space, whereas adversarial prompts can fool prompt embeddings but not the predicted noise patterns, which remain consistent with their unsafe intent (Yang et al., 5 Aug 2025).

FlowGuard addresses a different internal observable. Rather than classifying predicted noise directly, it inspects intermediate denoising steps in latent diffusion models by approximately decoding intermediate latents into image-like reconstructions (Yang et al., 9 Apr 2026). This is motivated by a specific difficulty: in latent diffusion, early-stage noise obscures visual signals, and raw latents are heterogeneous across models, making universal raw-latent classifiers difficult (Yang et al., 9 Apr 2026).

3. Predicted-noise IGD

The IGD method introduced in “Seeing It Before It Happens: In-Generation NSFW Detection for Diffusion-Based Text-to-Image Models” uses a binary classifier $f_\phi$ trained on predicted noise:

$y = f_\phi(\epsilon_t), \qquad y \in \{0, 1\},$

where $y=1$ denotes NSFW (Yang et al., 5 Aug 2025). The classifier is a 5-layer MLP. Training uses pairs of prompt and label, with $x^t$0 extracted from naive NSFW and SFW prompts; at inference, the detector monitors predicted noise at early timesteps and triggers early intervention if NSFW intent is detected (Yang et al., 5 Aug 2025).

Its pipeline is described in four stages: prompt encoding, diffusion generation, classification, and intervention. A text encoder produces the prompt embedding $x^t$1; during denoising, the model extracts the predicted noise vector at a preselected timestep, such as step 5; the MLP classifies the noise as NSFW or SFW; and generation is aborted or allowed based on the classifier output (Yang et al., 5 Aug 2025). The detector is described as lightweight and as introducing negligible overhead because it uses early, partial denoising steps.

The same study reports that IGD can be extended from binary NSFW/SFW detection to multi-class detection across seven NSFW types plus SFW, and that it maintains similar high performance across Stable Diffusion v1.4, v1.5, and v2.1 (Yang et al., 5 Aug 2025). It therefore frames IGD not merely as a binary safety gate but as a general internal-state classification mechanism over diffusion trajectories.

4. FlowGuard and linear latent decoding

FlowGuard generalizes the in-generation safety-detection idea to a cross-model setting by separating model-specific latent decoding from model-agnostic safety classification (Yang et al., 9 Apr 2026). Its first component is a linear latent decoder:

$x^t$2

learned from latent-image pairs by minimizing

$x^t$3

The linear decoder is trained independently for each backbone (Yang et al., 9 Apr 2026).

To suppress noise-dominated detail in these reconstructions, FlowGuard applies a 2D Fourier low-pass filter:

$x^t$4

The paper reports that low-pass filtering, especially with $x^t$5, improves detection accuracy and F1 on noisy steps (Yang et al., 9 Apr 2026).

The third component is curriculum learning. Rather than training immediately on early noisy states, FlowGuard starts from late-step, nearly clean images and gradually moves to earlier timesteps. It adds a temporal consistency loss

$x^t$6

and optimizes

$x^t$7

At inference, it evaluates a small set of early steps, such as steps 20–30, and exits early if any predicted NSFW probability exceeds a threshold $x^t$8 (Yang et al., 9 Apr 2026).

This architecture is explicitly cross-model. The same NSFW classifier is reused across many text-to-image backbones, while only the lightweight decoder is retrained for a new model. The implementation summary specifies a linear decoder per model, a ViT-B/16 backbone for classification, and a custom cross-model dataset with approximately 4000 diverse prompts and thousands of prompt-latent-image-label tuples across nine backbones (Yang et al., 9 Apr 2026).

5. Empirical performance, adversarial robustness, and efficiency

The two principal IGD-style systems differ in signal choice but converge on the same claim: intermediate denoising states support effective NSFW detection substantially earlier than final-image moderation.

System	Internal signal	Reported results
IGD	Predicted noise $x^t$9	91.32% average detection accuracy in the abstract; 92.45% accuracy, 96.78% AUROC, 16.78% FPR@TPR95 for overall binary detection
FlowGuard	Linearly decoded intermediate latents with LPF	Over 30% F1 improvement over existing methods; peak GPU memory reduced by over 97%; projection time reduced from 8.1 s to 0.2 s

For the predicted-noise IGD method, experiments were conducted on seven NSFW categories—sexual, violence, self-harm, harassment, hate, shocking, and illegal activity—using I2P for NSFW prompts and MSCOCO for clean prompts (Yang et al., 5 Aug 2025). The paper reports 89.5% accuracy for sexual, 89.0% for violence, 89.5% for self-harm, 90.5% for harassment, 95.7% for hate, 90.0% for shocking, and 89.0% for illegal activity, with a naive-prompt average of 90.46%. For adversarial prompts generated by I2P-sexual, MMA-Diffusion, Ring-A-Bell, SneakyPrompt, P4D, and DiffZOO, it reports 93.77% accuracy and 98% AUROC. The same study states that IGD outperforms seven baseline methods, including NSFW-text-classifier, Detoxify, OpenAI Moderation API, Aliyun/Azure APIs, Latent Guard, and GuardT2I, and reports attack success rates of 9.8% on naive prompts and 3.9% on adversarial prompts when compared with concept-erasing defenses (Yang et al., 5 Aug 2025).

FlowGuard reports in-generation detection performance at step 20 of 50 with F1 in the range 0.86–0.90 for in-distribution models and 0.73–0.88 for out-of-distribution models, whereas the best listed baselines—LlavaGuard, Falconsai, and Qwen3-VL—are reported at 0.66–0.73 in-distribution and much lower out-of-distribution (Yang et al., 9 Apr 2026). Its efficiency claims are equally prominent: compared with standard VAE decoding, it reports slashing peak GPU memory demand by over 97% and reducing projection time from 8.1 seconds to 0.2 seconds (Yang et al., 9 Apr 2026). These measurements support the practical premise of IGD: safety screening can be inserted into denoising with lower marginal cost than full post-generation moderation.

6. Limitations, unresolved questions, and acronym ambiguity

The current IGD literature also states clear limitations. The predicted-noise method notes that discrimination among semantically close NSFW subtypes may require improvement, that the classifier must be trained on predicted noise from the actual generation model or a related family, and that it is unknown whether early IGD will always suffice if an adversary manipulates the denoising trajectory at later steps (Yang et al., 5 Aug 2025). FlowGuard identifies strong Gaussian noise in intermediate latents, latent space heterogeneity across models, and limited cross-model latent datasets as core obstacles for in-generation detection in diffusion systems (Yang et al., 9 Apr 2026).

A second source of confusion is terminological rather than methodological: the acronym “IGD” is heavily overloaded in the broader technical literature. In recommendation, “IGD” denotes “Information Gain-based Decisiveness-aware Token handling” for LLMs (Lin et al., 16 Jun 2025). In one-class classification, it denotes the “Interpolated Gaussian Descriptor” (Chen et al., 2021). In graphene transport, “IGD” denotes “individual group delay” (Song et al., 2013). None of these is synonymous with In-Generation Detection. In current generative-model safety work, the term refers specifically to generation-time moderation of intermediate states, especially in diffusion-based text-to-image models (Yang et al., 5 Aug 2025, Yang et al., 9 Apr 2026).

Taken together, these studies define IGD as a model-internal, temporally localized detection regime. One line of work uses predicted noise directly; another reconstructs intermediate latents into filtered image-like representations. Both reject the assumption that safety must be inferred only from prompts or final outputs. The immediate research direction suggested by these results is not a replacement of pre- or post-detection, but a reallocation of moderation effort toward the denoising trajectory itself, where semantic intent may already be measurable before image synthesis completes.