SafeDiffusion-R1: Online Reward Steering for Safe Diffusion Post-Training

Abstract: Diffusion models have been widely studied for removing unsafe content learned during pre-training. Existing methods require expensive supervised data, either unsafe-text paired with safe-image groundtruth or negative/positive image pairs, making them impractical to scale. Furthermore, offline reinforcement learning and supervised fine-tuning approaches that generate synthetic data offline suffer from catastrophic forgetting, degrading generation quality. We propose a novel online reinforcement learning framework that addresses both data scarcity and model degradation through post-training with Group Relative Policy Optimization (GRPO) on both negative and positive text prompts. To eliminate the need for fine-tuning specialized safe/unsafe reward models, we introduce a \textit{steering reward mechanism} that exploits an inherent property of CLIP embeddings: steering text representations toward positive safety directions and away from negative ones in the embedding space. Our online-policy approach enables the model to learn from diverse prompts, including explicit unsafe content, without catastrophic forgetting. Extensive experiments demonstrate that our method reduces inappropriate content to 18.07\% (vs. 48.9\% for SD v1.4) and nudity detections to 15 (vs. 646 baseline) while improving compositional generation quality from 42.08\% to 47.83\% on GenEval. Remarkably, these safety gains generalize to out-of-domain unsafe prompts across seven harm categories, achieving state-of-the-art performance without supervised paired data or reward tuning. Github: https://github.com/MAXNORM8650/SafeDiffusion-R1.

• The paper introduces online group relative policy optimization to update diffusion models dynamically, reducing unsafe content without catastrophic forgetting.
• It leverages CLIP-based geometric reward steering by shifting unsafe prompt embeddings toward safe anchors, eliminating the need for costly paired data.
• Empirical results demonstrate marked safety improvements, with nudity detections dropping from 646 to as few as 15, while maintaining semantic fidelity and compositional quality.

SafeDiffusion-R1: Online Reward Steering for Safe Diffusion Post-Training

Motivation and Problem Formulation

Text-to-image (T2I) diffusion models pretrained on internet-scale datasets reliably propagate unsafe and explicit visual concepts—nudity, violence, harassment—whenever activatable via user prompts. Existing safety interventions for T2I models have major operational limitations: dataset filtering is computationally intractable and incomplete, post-hoc output filtering does not change the underlying generation distribution, and post-training methods (supervised fine-tuning, offline RL) require costly labeled data and are prone to catastrophic forgetting, harming utility on benign prompts. Critically, most prior works use static (offline) training signals or reward models, decoupled from the model’s evolving on-policy behavior, resulting in poor adaptability and limited generalization when confronted with novel (OOD) unsafe categories.

Method: Online Policy Optimization with Geometric Reward Steering

SafeDiffusion-R1 addresses these challenges with two major contributions:

1. Online Group Relative Policy Optimization (GRPO): Rather than relying on synthetic or fixed datasets, SafeDiffusion-R1 employs GRPO, updating the diffusion model based on current generations in response to both safe and unsafe prompts. Group-based advantage normalization in GRPO ensures stable credit assignment despite high-variance, prompt-dependent reward magnitudes, avoiding overcorrection and catastrophic forgetting.
2. CLIP-based Geometric Reward Steering: Instead of training dedicated safe/unsafe reward models, the method encodes “safety” as a direction in CLIP embedding space—obtained by contrasting mean text embeddings of curated safe and unsafe anchors. For an unsafe prompt, its embedding is steered towards the safe direction before reward computation. Thus, the model is not rewarded for explicit content but for producing images aligned with steered, semantically safer versions of unsafe prompts. This reward manipulation requires only anchor texts and a vision-LLM, not annotated image pairs.

   Figure 1: GRPO-based reward steering: unsafe prompt embeddings are steered towards safety anchors before reward computation, aligning generation with safe semantic attributes.

This framework enables safe RL post-training with minimal supervision and robust generalization to unseen unsafe prompt classes.

Empirical Results

SafeDiffusion-R1 delivers strong numerical improvements and makes bold claims regarding OOD safety generalization and utility preservation (as detailed below):

Quantitative Safety Evaluations

• On the I2P benchmark (4,703 nudity-focused prompts), SD v1.4 baseline yields 646 nudity detections. SafeDiffusion-R1 attains only 15 detections with unsafe-only anchors or 31 with anchor-based reward scaling, outperforming virtually all concept erasure, supervised alignment, and prior RL approaches.
• On OOD harm categories, SafeDiffusion-R1 achieves an overall inappropriate rate of 18.07% (down from 48.9% for SD v1.4), beating prior SoTA erasure, RL, and inference-time filter methods. Subscores for the Sexual category drop to 11.60%, Violence to 17.33%, and Self-harm to 15.86%—without explicit training on these prompt types.

  Figure 3: Out-of-domain generalization curves—despite training with nudity prompts, SafeDiffusion-R1 achieves monotonic safety improvement across all Q16 harm categories.

Utility and Compositional Generalization

• On GenEval compositional benchmarks, SafeDiffusion-R1 increases accuracy from 42.08% (SD v1.4) and 38.36% (RECE) to 47.83% when using GenEval+Nudity prompts, and maintains 44.12% even when trained on nudity-only data.
• On COCO-3K, SafeDiffusion-R1 maintains CLIP-T scores near baseline (0.311 vs 0.313) and shows high-fidelity image generations, with an observed tradeoff of increased FID likely resulting from distribution shift inherent in RL-based post-training.

  Figure 5: Qualitative comparison on compositional tasks (GenEval): SafeDiffusion-R1 maintains or improves semantic fidelity while suppressing unsafe content.

Qualitative Assessment

Qualitative evaluation confirms that SafeDiffusion-R1 not only removes unsafe content but also consistently preserves or improves semantic fidelity and image realism compared to prior methods, which often suffer from over-smoothing, structure collapse, or severe utility loss.

Figure 6: On challenging unsafe prompts, SafeDiffusion-R1 robustly suppresses explicit content while retaining compositional coherence, outperforming leading concept erasure and recent RL safety alignment methods.

Figure 8: Nudity-focused I2P prompt outputs: SafeDiffusion-R1 exhibits both effective suppression and higher visual quality than EraseDiff, ESD-x, and adversarial unlearning baselines.

Ablations and Geometric Analysis

SafeDiffusion-R1 includes detailed ablations:

• Reward design ablations: Pure negative-reward training collapses utility and fails to generalize, while geometric anchor scaling with safe/unsafe anchors and moderate steering strength achieves the best Pareto point on safety–utility tradeoff.
• Schedule robustness: Safety enhancement remains stable and scheduler-agnostic, converging to low unsafe scores regardless of inference sampler.
• Embedding space analyses: UMAP projections confirm that steering increases safety scores monotonically for unsafe prompts without collapsing semantic boundaries between safe and unsafe regions.

  Figure 9: UMAP and embedding analyses: Anchor-based steering shifts unsafe prompt clusters into the safe region without collapsing structure, and scores improve with steering strength.

  

  Figure 2: Safety steering geometry: unsafe text embeddings are rotated towards the safe region, reshaping the reward signal for RL without altering model inputs.

Implications and Future Directions

Practically, SafeDiffusion-R1 eliminates expensive requirements for paired data or reward model finetuning, is compatible with publicly released models (e.g., SD v1.4), and supports continuous online safety tuning, making deployment feasible at scale for open platforms.

Theoretically, this work shifts the paradigm from static, external constraint-based alignment to geometric, reward-driven post-training, promising more adaptive, scalable, and generalizable safety alignment in generative models. The embedding-based abstraction of safety could be extended to other modalities and prompt classes, assuming rich enough pre-trained cross-modal models (e.g., recent advances in alignment benchmarking (Alam et al., 24 Nov 2025), interpretability (Wang et al., 5 Feb 2026), and control (Tater et al., 26 Mar 2025)).

Open directions include combinatorial geometric steering for multi-attribute safety, reward model co-adaptation, extending to large-scale vision-language diffusion backbones, and integration with real-time or user feedback driven RLHF pipelines.

Conclusion

SafeDiffusion-R1 demonstrates that online RL with anchor-based geometric reward steering achieves robust safety alignment in T2I diffusion models, delivering both state-of-the-art inappropriate content reduction (down to 18.07% OOD) and improved compositional utility (GenEval +6%). It establishes a scalable post-training template for safety alignment that avoids catastrophic forgetting, requires minimal supervision, and generalizes strongly across harm categories—suggesting a path forward for safe open-domain generative AI deployment (2605.18719).