DiffSign: Multimodal Diffusion Models

• DiffSign is a suite of frameworks that harness diffusion models and multimodal conditioning to synthesize high-fidelity sign language videos and generate adversarial traffic sign attacks.
• It integrates SMPL-X based pose retargeting with cross-attention visual adapters to control appearance and ensure temporal consistency using tailored loss functions.
• Performance evaluations show enhanced SSIM and FID metrics for video synthesis and an over 80% attack success rate for physically robust traffic sign adversarial attacks.

DiffSign refers to multiple advanced frameworks in recent literature, each leveraging generative diffusion models and multimodal conditioning for domain-specific synthesis and manipulation tasks. The two primary contexts are: (1) the AI-assisted generation of customizable, high-fidelity sign language videos for accessibility and content personalization (Krishnamurthy et al., 2024); and (2) T2I-based (text-to-image) synthesis for robust physical-world adversarial attacks on traffic sign recognition (TSR) systems in autonomous vehicles (Ma et al., 17 Nov 2025). Across both domains, DiffSign distinguishes itself through the integration of parametric modeling, diffusion-based generative backbones, multimodal guidance, and tailored optimization objectives, resulting in state-of-the-art controllability, realism, or adversarial effectiveness.

1. Parametric Modeling and Pose Retargeting in Sign Language Video Synthesis

DiffSign for sign language video generation employs the SMPL-X parametric human body model to accurately retarget detected 2D sign language keypoints onto a 3D human avatar. This process optimizes the pose (θ), shape (α), and facial expression (ϕ) parameters so that the projected 3D joints closely reconstruct the original 2D keypoints, subject to Gaussian or VAE-style priors. The core optimization objective is:

$$(\theta^*, \alpha^*, \phi^*) = \arg\min_{\theta, \alpha, \phi} [w_J L_{reproj}(\theta, \alpha, \phi) + L_{prior}(\theta, \alpha, \phi)]$$

where $L_{reproj}$ penalizes 2D projection error and $L_{prior}$ encodes priors over pose, shape, and expression. Once optimized frame-wise, high-fidelity 3D avatar renders are produced (e.g., in Blender), achieving realistic base pose tracks as direct conditioning signals for subsequent generative modeling (Krishnamurthy et al., 2024).

2. Diffusion-Based Generative Frameworks and Architecture

Both instantiations of DiffSign employ diffusion models as the generative mechanism, although with distinct purposes and pipelines:

• Sign Language Video Synthesis: Utilizes a pretrained Stable Diffusion v1.5 backbone with a ControlNet branch for pose conditioning and an IP-Adapter–style visual adapter for appearance control. The reverse denoising model operates as:

$$p_\theta(x_{t-1} | x_t, c_p, c_v) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t, c_p, c_v), \sigma_t^2 I)$$

where $c_p$ encodes SMPL avatar pose (canny edges, keypoints), and $c_v$ encodes the appearance image prompt via CLIP embeddings. Temporal consistency and realism outperform text-prompt-only baselines by large margins in SSIM and FID metrics (Krishnamurthy et al., 2024).

• Physical-World Adversarial Attack Generation: Constructs adversarial traffic sign images using a T2I diffusion model (Stable Diffusion), integrating text and optional image prompt conditioning, and employs an iterative adversarial optimization loop. Conditional and unconditional embeddings are manipulated during DDIM inversion/sampling, with gradient-based losses promoting misclassification in TSR models and semantic misalignment (via CLIP loss) (Ma et al., 17 Nov 2025).

3. Multimodal Appearance and Style Control

DiffSign incorporates multimodal control mechanisms at multiple levels:

• Visual Adapter (Sign Language): The IP-Adapter module injects CLIP image prompt features into every U-Net attention block by cross-attention, ensuring consistent signer appearance (facial characteristics, clothing, illumination) throughout the sequence. Total attention is computed as:

$$\text{Attn}_{total} = \text{Attn}_{text} + \text{Attn}_{img}$$

This mechanism is further extended for text+image prompts by concatenating keys and values, supporting fine-grained signer customization (e.g., specified attire, accessories) (Krishnamurthy et al., 2024).

• Masked Prompt Mechanism (Adversarial T2I): By masking benign sub-prompts and focusing classifier guidance via CLIP to the adversarial prompt content, the attack pipeline localizes semantic perturbations while retaining critical shape and color attributes required for physical plausibility (Ma et al., 17 Nov 2025).

4. Optimization Objectives and Robustness Procedures

• Sign Language Synthesis: The diffusion process is performed frame-by-frame, leveraging frozen backbone weights. Loss functions include the standard $L_{diff}$ (denoising) objective, with evaluation via SSIM, Directional Similarity (DS) in CLIP space, and FID (Krishnamurthy et al., 2024).
• Adversarial Traffic Sign Generation: The total loss combines detection and CLIP similarity terms:

$L = L_s + \lambda L_d$

where $L_d$ penalizes detection confidence for the true class in the TSR model output, and $L_s$ enforces semantic dissimilarity with the benign class using CLIP (Ma et al., 17 Nov 2025). Adversarial robustness is further supported by Expectation-over-Transformation (EoT), simulating varied backgrounds, scales, orientations, and lighting conditions.

5. Quantitative Evaluation and Comparative Performance

Sign Language Video Synthesis

DiffSign achieves superior realism and temporal consistency metrics in high-resolution signer synthesis:

Approach	SSIM	DS	FID
Avatar (parametric + smoothing)	0.812	0.185	176.035
Pretrained SD (text prompt only)	0.553	0.183	158.525
Pretrained SD + IP-Adapter (image prompt)	0.769	0.194	146.490
Fine-tuned SD (DreamBooth)	0.668	0.169	130.896

Higher SSIM and DS indicate improved temporal smoothness and edit alignment, while lower FID signals more realistic imagery (Krishnamurthy et al., 2024).

Physical Adversarial Attack Generation

DiffSign substantially outperforms prior baselines in physical attack success rate (ASR), transferability, and stealth:

Method	AASR (8 detectors)
NDD	15.3%
NDD*	45.8%
SIB	17.0%
DiffSign	82.7%

Physical ASR reaches 83.3% overall for printed signs under real-world protocols. Stealthiness, measured by the fraction of human subjects rating attacks as “likely in daily life,” is 41% for DiffSign (Ma et al., 17 Nov 2025).

6. Applications, Limitations, and Open Problems

Applications of the DiffSign family span:

• Accessible Media: Generation of expressive, personalized sign language interpreter videos for diverse DHH audiences, including support for customizable attributes (age, gender, region, appearance), zero-shot anonymization, and rapid style transfer (Krishnamurthy et al., 2024).
• Adversarial Testing: Generation of physically robust, model-transferable, and visually inconspicuous adversarial attacks for benchmarking and auditing the safety of autonomous driving systems (Ma et al., 17 Nov 2025).

Limitations include frame-by-frame generation latency, propagation of pose estimation errors, limited explicit modeling of sign-language-specific motion priors, and the need for further study on robustness against defenses and generalization to more sign types or longer video durations. Potential research directions involve integration of end-to-end video diffusion with temporal attention, fusion with sign-language-specific motion models, and defense mechanisms incorporating semantic reasoning.

7. Relationship to Related Work

For sign language synthesis, DiffSign advances on previous works such as SignDiff (“Diffusion Model for American Sign Language Production” (Fang et al., 2023)) by introducing parametric 3D avatar models for pose retargeting, stronger multimodal (image+text) control via adapters, and improved realism metrics. While SignDiff provides notable architectural innovations for skeleton-to-image translation and frame reinforcement, DiffSign’s pipeline achieves higher expressivity through fusion of diffusion backbones and parametric avatars.

In the context of adversarial T2I attacks, DiffSign improves upon both pixel-level Expectation-over-Transformation methods and previous T2I-based pipelines by producing more transferable, robust, and stealthy attacks and introducing masked CLIP-guided losses and new style customization mechanisms.

Both applications exemplify the growing trend towards highly controllable, multimodal, and semantically or adversarially guided diffusion pipelines, with architecture and optimization tailored to their distinct generative or security-oriented objectives.