Text2Sign: A Single-GPU Diffusion Baseline for Text-to-Sign Language Video Generation

Abstract: Sign language is a primary communication channel for millions of Deaf and hard-of-hearing people, yet text-to-signer video generation remains costly because video diffusion models are expensive to train and evaluate. This paper presents Text2Sign, a text-conditioned diffusion model for short sign-language clips that runs on a single NVIDIA L4 GPU. It combines a frozen vision-language text encoder with a 3D encoder-decoder and factorized spatiotemporal attention to reduce the cost of full-video attention while preserving motion coherence. We compare convolution-only and transformer-style backbones, frozen pretrained and task-specific text encoders, and factorized versus full attention. On a signer-disjoint How2Sign split, the best short-run ablation reaches a validation loss of 0.0648, while a longer-run checkpoint reaches 0.00999. On a compact evaluation slice, the latter achieves an SSIM of $0.2403 \pm 0.0238$, a PSNR of $15.11 \pm 0.42$ dB, and temporal consistency of $1.0000 \pm 0.0000$ using 8-step DDIM sampling with a guidance scale of 5.0. It generates a 32-frame, $64 \times 64$ clip in 12.60 seconds, or 2.54 frames per second, with peak inference memory of 3.12 GB. A held-out denoising audit shows only weak prompt sensitivity: removing text increases late-timestep loss from 0.9875 to 0.9891, while shuffled prompts perform similarly to correct prompts. Frozen text conditioning therefore improves short-budget validation loss, but prompt-specific separation remains limited. The system is restricted to low-resolution, short clips and lacks expert linguistic evaluation, so it should be viewed as a single-GPU research baseline rather than a complete sign-language production system. Code is available at https://github.com/xiaruize0911/text2sign.

• The paper presents a novel single-GPU diffusion model that generates sign language videos using text-conditioned frozen CLIP encoding and transformer-based DiT blocks.
• The methodology employs factorized spatiotemporal attention to reduce computational complexity, achieving a 19.5% validation loss improvement over convolution-only baselines.
• The approach establishes a reproducible and efficient baseline for text-to-sign language video synthesis on accessible hardware, paving the way for real-time applications.

Text2Sign: A Diffusion Baseline for Text-to-Sign Language Video Generation

Introduction

Text2Sign introduces a single-GPU diffusion architecture for text-conditioned sign language video generation, targeting practical generation of short sign-language clips from textual prompts on accessible hardware. Motivated by the limitations of prior generative SLP methods—particularly GANs and rule-driven avatar approaches—the system leverages a three-dimensional encoder–decoder backbone augmented by factorized spatial and temporal attention, while conditioning generation on frozen vision–language encoder outputs (CLIP). The methodology specifically addresses the computational bottlenecks of spatiotemporal attention in video diffusion, investigates the impact of transformer blocks versus convolution-only baselines, and explores text encoder strategies via systematic ablation on the How2Sign dataset.

Methodology

Dataset and Preprocessing

The model is trained and validated on processed $64 \times 64$ RGB clips sampled at 32 frames per sequence from the How2Sign corpus, employing a signer-disjoint protocol with 231 metadata-derived identities in training and 24 in validation. This split constrains memorization and encourages generalization of text-conditioned sign motion.

Diffusion Architecture

The generative backbone consists of a 3D UNet with DiT transformer blocks that operate at multiple spatial scales. Factorized attention decomposes 3D attention into sequential spatial and temporal passes, reducing computational complexity from $O((THW)^2)$ to $O(T(HW)^2 + HW \cdot T^2)$. Text conditioning is achieved via cross-attention with a frozen CLIP text encoder, injecting rich semantic features without expanding trainable parameter count or introducing encoder instability. Timestep embeddings are integrated via Adaptive Layer Normalization (AdaLN), following DiT conventions.

Figure 1: Memory–quality Pareto trade-off visualizing architectural efficiency; the factorized attention approach achieves lowest validation loss despite higher memory usage.

Training and Sampling

Training employs DDPM noise-prediction objectives, AdamW-based optimization, AMP, gradient checkpointing, and an EMA for inference stability. Sampling is conducted with DDIM (typically 8–15 steps), paired with classifier-free guidance at scale 5.0 to promote text adherence. Quantitative metrics include validation denoising loss, SSIM, PSNR, temporal consistency, motion magnitude, FVD-proxy, and frame-level gradients.

Ablation and Empirical Results

Architectural Ablation

Three axes are evaluated: DiT block presence, text encoder strategy, and attention factorization. Four variants—Full (DiT + frozen CLIP), No DiT (convolutional only), Custom TextEnc (domain-specific text encoder), Full 3D Attention (non-factorized transformer)—are compared under uniform training budgets.

• DiT block effect: Transformer-based global attention reduces validation loss by 19.5% relative to convolution-only (0.0648 vs. 0.0805).
• Text encoder effect: A frozen CLIP encoder outperforms a custom encoder by 11.0% on validation loss (0.0648 vs. 0.0728) with fewer trainable parameters.
• Attention factorization effect: Factorized attention slightly outperforms full 3D attention (0.0648 vs. 0.0664) and scales better theoretically.

Figure 2: Generalization gap analysis; the full model exhibits superior regularization and generalization characteristics.

Computational Efficiency

The factorized DiT variant achieves the lowest loss within practical memory bounds (19.0 GB peak on L4), with only modest overhead relative to convolutional baselines. No DiT is fastest yet sacrifices generation quality.

Figure 3: Training-step time and peak memory profile for each ablation variant; all variants remain within the L4 budget.

Generation Quality

The full model delivers the most controlled motion dynamics—lowest motion magnitude and spatial gradient—indicative of natural signing. However, prompt-specific separation remains weak in held-out audits. FVD-proxy and physical consistency are competitive with fully attentional designs, but hand/facial articulation is coarse due to resolution bottlenecks.

Figure 4: Metric comparison across ablation variants; the full model excels in motion control and spatial smoothness.

Figure 5: Radar chart of multi-metric normalization; the full model demonstrates a balanced performance profile.

Qualitative Inspection

Generated clips (under 8-step DDIM, CFG=5.0) are visually stable and exhibit clear body motion. Nevertheless, fine linguistic details (handshape, facial grammar) are not recoverable at $64 \times 64$ resolution, precluding strong claims of sign intelligibility.

Figure 6: Qualitative generation for "Hello"; frames demonstrate global temporal smoothness, though linguistic fidelity is low.

Figure 7: Prompt variation ("Hello", "Thank you"); the model adapts coarse pose but lacks gesture specificity.

Extended Results and Audit

A longer 100-epoch run achieves validation loss 0.00578 (epoch 84), SSIM 0.2403, PSNR 15.11 dB, and high temporal consistency (1.0000), maintaining efficiency (12.60 s/clip, 3.12 GB inference). Conditioning audits indicate measurable text benefit in denoising loss (+0.00161 penalty when removed), but shuffled prompts remain nearly indistinct, highlighting weak prompt-specific alignment.

Figure 8: Epoch-wise loss curve for extended run; loss plateau achieved by epoch 84.

Figure 9: Train–validation gap stabilizes, evidencing robust generalization.

Figure 10: Extended checkpoint evaluation, underscoring temporal consistency but insufficient signing fidelity.

Discussion

Numerical Outcomes

• Validation loss: Full DiT + frozen CLIP (0.0648), convolution-only (0.0805), custom encoder (0.0728), full 3D attention (0.0664).
• 100-epoch checkpoint: Validation loss 0.00578, SSIM 0.2403, PSNR 15.11 dB, temporal consistency 1.0000.
• Efficiency: 12.60 s for 32 frames, 3.12 GB inference, all within L4 constraints.

Limitations

Low-resolution output constrains hand articulation; prompt-specific differentiation is weak. Training remains limited by dataset scale and computational budget. Metric suite cannot directly assess linguistic quality.

Practical and Theoretical Implications

Text2Sign establishes an architectural baseline for SLP under constrained resources, revealing the necessity of transformer-based global attention and pretrained encoder priors for sign video generation. Factorized spatiotemporal attention is not only computationally scalable but, within the tested regime, provides competitive or superior quality. Efficient architectures such as this pave the way for future real-time sign-generation systems deployable in accessibility pipelines. However, higher fidelity, longer temporal horizons, and direct linguistic evaluation are essential for clinical or educational adoption.

Figure 11: Architectural impact summary; DiT blocks yield the most substantial improvement.

Conclusion

Text2Sign delivers an efficient diffusion pipeline for text-conditioned sign video synthesis, quantifying trade-offs in transformer-based attention, frozen text encoding, and attention factorization. The model achieves strong numerical improvements in validation loss and qualitative stability over prior baselines in a signer-disjoint regime, but remains limited to low-res, short clips with weak prompt specificity. The approach clarifies computational and architectural constraints for SLP generation on accessible hardware and establishes a reproducible research foundation for advancing both practical deployment and theoretical modeling of multimodal sign generation.