FlashSign: Pose-Free Guidance for Efficient Sign Language Video Generation

Abstract: Sign language plays a crucial role in bridging communication gaps between the deaf and hard-of-hearing communities. However, existing sign language video generation models often rely on complex intermediate representations, which limits their flexibility and efficiency. In this work, we propose a novel pose-free framework for real-time sign language video generation. Our method eliminates the need for intermediate pose representations by directly mapping natural language text to sign language videos using a diffusion-based approach. We introduce two key innovations: (1) a pose-free generative model based on the a state-of-the-art diffusion backbone, which learns implicit text-to-gesture alignments without pose estimation, and (2) a Trainable Sliding Tile Attention (T-STA) mechanism that accelerates inference by exploiting spatio-temporal locality patterns. Unlike previous training-free sparsity approaches, T-STA integrates trainable sparsity into both training and inference, ensuring consistency and eliminating the train-test gap. This approach significantly reduces computational overhead while maintaining high generation quality, making real-time deployment feasible. Our method increases video generation speed by 3.07x without compromising video quality. Our contributions open new avenues for real-time, high-quality, pose-free sign language synthesis, with potential applications in inclusive communication tools for diverse communities. Code: https://github.com/AIGeeksGroup/FlashSign.

• The paper introduces FlashSign, a pose-free text-to-video diffusion framework with Trainable Sliding Tile Attention (T-STA) that jointly learns timestep-adaptive sparsity during training and inference.
• On WLASL isolated-sign generation, FlashSign achieves FVD 453, SSIM 0.83, and LPIPS 0.07, while improving perceptual metrics over pose-guided and sparse-attention baselines despite lower PSNR than SignGen.
• The method reduces generation latency by 2.32× for a 1.3B model and 3.07× for a 14B model, but sentence-level signing, other sign languages, continuous co-articulation, and human evaluation remain open challenges.

Motivation and problem setting

Existing sign language video generation systems almost universally follow a two-stage, pose-guided pipeline: text is first mapped to a skeleton pose sequence, which is then rendered into video. This decomposition introduces cascading errors from pose estimation, supervision bottlenecks, and additional inference cost, and it fails in scenarios where pose information is absent, such as zero-shot or deaf-blind communication. At the same time, diffusion-based video generators, while producing high-fidelity output, are too slow for interactive use. The paper observes that sign language videos are structurally favorable for acceleration: backgrounds are largely static, upper-body posture is relatively fixed, and semantic content is concentrated in the hand regions, creating substantial spatio-temporal redundancy in the attention maps.

Method

FlashSign is built on the Wan2.1 video diffusion transformer and consists of two components: a pose-free text-to-video formulation and a Trainable Sliding Tile Attention (T-STA) mechanism.

Pose-free generation. Instead of the two-stage mapping $\mathbf{P} = f_{\text{text2pose}}(\mathbf{T})$ followed by $\mathbf{V} = f_{\text{pose2vid}}(\mathbf{P})$, the model learns a direct conditional mapping $\mathbf{V} = f_{\text{text2vid}}(\mathbf{T})$. Videos are encoded by a 3D VAE into a compact latent space, a pre-trained UmT5 text encoder provides the semantic conditioning signal, and a video DiT performs denoising with standard $\epsilon$-prediction loss conditioned on both the noised latent and the text embedding. The authors argue this removes error propagation from pose estimation, simplifies deployment, and improves generalization in low-resource and zero-shot settings.

T-STA. Standard Sliding Tile Attention (STA) applies a fixed local attention window over spatio-temporal tiles. T-STA extends this in three ways. First, the latent sequence is partitioned into tiles along temporal, height, and width dimensions. Second, the window size is piecewise and timestep-dependent: larger windows are used at high noise levels to capture coarse dependencies, shrinking at later denoising steps to focus on fine detail. Third, and most distinctively, the window size parameters $\{T_w^{(s)}, H_w^{(s)}, W_w^{(s)}\}$ are treated as trainable variables optimized jointly with the denoising objective. This is the paper's central claim of novelty relative to training-free sparsity methods (SageAttention, SpargeAttn, SparseVideoGen, STA): because sparsity is enforced consistently during both training and inference, the train–test gap in attention patterns that degrades heuristic sparse methods is eliminated. A cross-modal mask governs image-to-image, image-to-text, and text-to-image/text attention, preserving linguistic guidance while restricting visual attention to localized neighborhoods.

Experimental results

Experiments use the WLASL dataset (21,083 videos, 2,000 ASL glosses, 119 signers, ~14 hours). Quality is evaluated with PSNR, SSIM, LPIPS, and FVD; latency is measured on a 4×NVIDIA H20 tensor-parallel setup generating 81-frame, 480p, 5-second videos with 50 sampling steps.

Quality comparison. Against SageAttention, SpargeAttn, SparseVideoGen, SignGen, and STA, the method achieves the best FVD (453), an 8.1% improvement over the pose-guided SignGen (493) and 8–24% over other baselines, along with the best LPIPS (0.07) and SSIM (0.83). Notably, SignGen retains a higher PSNR (27.72 vs. 25.74); the authors dismiss this on the grounds that PSNR has limited reference value for perceptual quality in dynamic sign video — a claim that is asserted rather than demonstrated with human evaluation, and which readers should weigh accordingly.

Latency. The speedups are the headline result:

Model	Method	Latency (s)	Speedup
WanX2.1-1.3B	baseline	271	1.00×
	SageAttention	141	1.91×
	STA	143	1.89×
	Ours	117	2.32×
WanX2.1-14B	baseline	1301	1.00×
	STA	548	2.37×
	SparseVideoGen	613	2.12×
	Ours	423	3.07×

The 3.07× speedup on the 14B model, cited throughout the paper, is the largest reported gain and suggests the advantage of trainable sparsity grows with model scale. The gap over the strongest baseline (STA, 2.37×) is, however, more modest at roughly 1.3×.

Ablations. A three-stage window configuration — $(18,24,32) \to (12,16,24) \to (12,16,16)$ — yields more stable training curves than two-stage alternatives, with the intermediate depth-window size (24) bridging the receptive-field gap between the first and last stages. Tile size ablations show monotonic improvement with larger tiles: the largest configuration ($24\times32\times32$) attains PSNR 25.74, SSIM 0.83, LPIPS 0.07, at the cost of higher memory. Gains diminish between $(6,8,8)$ and $(12,16,16)$, which the authors note is useful for resource-constrained deployments.

Limitations and open questions

The paper is explicit that its results cover isolated sign generation only. The absence of pose supervision limits fine-grained spatio-temporal consistency over longer utterances, and the authors anticipate that sentence-level synthesis will require reintroducing structured guidance — sparse pose priors or hierarchical constraints — in a hybrid design. This is a significant concession, since continuous signing with co-articulation is where pose-based methods retain their clearest justification. Evaluation is confined to WLASL (ASL, isolated glosses); generalization to continuous signing, other sign languages, and non-manual signals (facial expression) is untested. The claim that trainable sparsity eliminates the generalization gap of training-free methods is supported by aggregate metric comparisons but not by a direct analysis of train/inference attention-pattern divergence. Finally, the dismissal of the PSNR deficit relative to SignGen would be strengthened by human or deaf-community evaluation, which the paper does not report.

Conclusion

FlashSign combines a pose-free latent diffusion formulation for sign language video with T-STA, a trainable, timestep-adaptive sliding tile attention mechanism applied uniformly during training and inference. It reports state-of-the-art FVD, SSIM, and LPIPS on WLASL and latency reductions of 2.32× (1.3B) and 3.07× (14B) over the Wan2.1 baselines, outperforming existing sparse attention methods on both quality and speed. The framework's scope remains isolated-sign generation, and extending the pose-free paradigm to sentence-level, continuous signing without reintroducing full pose dependence is the central open problem the paper leaves unresolved.