Stable Signer: End-to-End SLP Model

• Stable Signer is a hierarchical generative model for sign language production that integrates text-to-gloss and pose-to-video synthesis in an end-to-end framework.
• The model leverages a Sign Language Understanding Linker and Mixture-of-Experts block to minimize error propagation and enhance semantic accuracy.
• Empirical evaluations show substantial improvements in semantic precision and video realism over traditional multi-stage SLP pipelines.

Stable Signer is a hierarchical generative model for sign language production (SLP) that reframes the SLP task as an end-to-end, two-stage pipeline. Instead of the canonical multi-step process—typically decomposed into Prompt2Gloss, Gloss2Pose, Pose2Avatar, and Pose2Vid—Stable Signer jointly models text-to-gloss conversion and pose-to-video synthesis, thus minimizing stagewise error propagation and leveraging semantic information more directly in generating high-fidelity, multi-style sign language videos. The model architecture introduces the Sign Language Understanding Linker (SLUL) and a Mixture-of-Experts (SLP-MoE) block, paired with specialized objectives such as the Semantic-Aware Gloss Masking Loss (SAGM). This approach yields substantial empirical gains over prior methods in both semantic accuracy and video quality, as measured on standard SLP benchmarks (Fang et al., 3 Dec 2025).

1. Hierarchical End-to-End Architecture

Stable Signer reconceptualizes SLP as a two-stage, jointly trained hierarchy:

1. Text Understanding: Input text or prompt (with language identification) is translated to a gloss sequence using SLUL. This step subsumes Prompt2Gloss and Text2Gloss sub-tasks within a unified Transformer-based sequence transduction framework.
2. Pose-to-Video Synthesis: Predicted glosses are mapped—via a gated Mixture-of-Experts (SLP-MoE) pose retrieval and temporal stabilization subsystem—directly to stabilized pose sequences, which are rendered as sign language video through a diffusion-based generator.

This design omits separate Gloss2Pose and Avatar modeling stages (in contrast to multi-stage SLP pipelines), supporting direct semantic-to-pose-to-video mappings and minimizing cumulative aliasing and error injection. Figure 1 of (Fang et al., 3 Dec 2025) illustrates this workflow, showing the bypassing of redundant or error-prone intermediaries in favor of an end-to-end trainable path.

2. Sign Language Understanding Linker (SLUL)

The SLUL module is a Transformer encoder-decoder (based on T5-Base). Given any prompt $x$ and language tag $\ell$, it encodes and autoregressively decodes a sequence of gloss tokens $g = (g_1,...,g_T)$:

• Encoder: $$H = f_{\theta_E}([\ell; x]) \in \mathbb{R}^{L \times d}$$
• Decoder: Predicts each $g_t$ conditioned on $(g_{<t}, H)$.

The loss objective comprises cross-entropy over gloss tokens: $$L_\text{SLUL} = -\sum_{t=1}^T \log p_\theta(g_t | g_{<t}, H)$$

GLUL enables prompt-to-gloss and text-to-gloss in a single module, with glosses serving as the mediating semantic representation for sign language gesture generation.

3. Semantic-Aware Gloss Masking Loss (SAGM)

To improve semantic robustness and prevent overfitting, Stable Signer employs the SAGM auxiliary loss during SLUL training:

• Random gloss masking: For each $t$, $\tilde{g}_t = [\text{MASK}]$ with probability $\rho$, else $\ell$0.
• Masked cross-entropy: Computed only at masked positions,

$\ell$1

• KL divergence posterior stability:

$\ell$2

• Prompt–gloss alignment (contrastive):

$\ell$3

The total objective: $\ell$4 This joint objective enforces semantic coverage in the learned gloss representations, induces robustness to missing information, and aligns prompts and generated glosses in embedding space (Fang et al., 3 Dec 2025).

4. SLP-MoE: Pose Retrieval, Gating, and Temporal Stabilization

After gloss prediction, SLP-MoE serves as the expert block for pose retrieval and stabilization:

• Mixture-of-Experts (MoE): Given semantic embedding $\ell$5, pose experts $\ell$6, and gloss $\ell$7, expert weights $\ell$8 are computed as:

$\ell$9

• Final pose sequence:

$g = (g_1,...,g_T)$0

• Gating and diversity losses:

$g = (g_1,...,g_T)$1

$g = (g_1,...,g_T)$2

• Temporal stabilization: Smoothing the output pose stream with:

$g = (g_1,...,g_T)$3

Hand fidelity and velocity losses further refine kinematic realism.

• Diffusion-based renderer: The stabilized $g = (g_1,...,g_T)$4 is rendered by a ControlNeXt-derived video diffusion model, yielding photorealistic and temporally coherent sign language video frames.

This module ensures the generated hand gestures are not only lexically faithful but also physically plausible and visually continuous.

5. Training Protocols and Hyperparameters

• Datasets: Training leverages Prompt2Sign (~30K ASL videos with LLM-paraphrased prompts, text, and glosses), ASL-WLASL (~10K clips), and OpenVid-1M for renderer pretraining.
• Keypoint extraction & preprocessing: 2D OpenPose keypoints, normalization, and segmentation into gloss–pose pairs.
• Model configuration: SLUL on T5-Base; SLP-MoE with $g = (g_1,...,g_T)$5 pose experts; ControlNeXt-style diffusion renderer.
• Optimization: AdamW optimizer ($g = (g_1,...,g_T)$6 for SLUL, $g = (g_1,...,g_T)$7 for Pose2Vid), batch size 32, 50 epochs, PyTorch mixed-precision.
• Loss weights: Mask rate $g = (g_1,...,g_T)$8; $g = (g_1,...,g_T)$9, $$H = f_{\theta_E}([\ell; x]) \in \mathbb{R}^{L \times d}$$0, $$H = f_{\theta_E}([\ell; x]) \in \mathbb{R}^{L \times d}$$1, $$H = f_{\theta_E}([\ell; x]) \in \mathbb{R}^{L \times d}$$2, $$H = f_{\theta_E}([\ell; x]) \in \mathbb{R}^{L \times d}$$3, $$H = f_{\theta_E}([\ell; x]) \in \mathbb{R}^{L \times d}$$4, $$H = f_{\theta_E}([\ell; x]) \in \mathbb{R}^{L \times d}$$5, $$H = f_{\theta_E}([\ell; x]) \in \mathbb{R}^{L \times d}$$6 (Fang et al., 3 Dec 2025).

These settings enable Stable Signer’s modules to be trained end-to-end, supporting both semantic and pose-level supervision.

6. Quantitative and Qualitative Benchmarking

Stable Signer is evaluated extensively on SLP benchmarks:

• Back-translation (BLEU-n, ROUGE): Surpasses prior models (SignLLM, Fast-SLP) with relative gains of 40–60% BLEU-4 and ROUGE on the How2Sign ASL challenge.
• Video realism (SSIM, Hand-SSIM, FID, Pose Error): Achieves SSIM of 0.892, Hand-SSIM of 0.732, PoseErr of 17.68, FID of 21.04, outperforming SignDiff and other baselines.
• Ablations: Table 4 shows all core modules (SLUL+SAGM, MoE, stabilization, diffusion rendering) contribute positively to both semantic accuracy and video quality.

Empirical results support both quantitative and qualitative advancements: user studies report that Stable Signer outputs appear more natural, legible, and style-diverse, with minimal jitter and high signer-style fidelity (Fang et al., 3 Dec 2025).

Method	BLEU4 (TEST)	ROUGE (TEST)	SSIM	FID
Fast-SLP (SOTA)	12.85	46.89	—	—
Stable Signer (ours)	21.03	65.26	0.892	21.04

7. Context and Significance

Stable Signer’s streamlined conceptualization of SLP stands in contrast to prior pipelines that propagate semantic and generative errors across four or more loosely coupled modules, a key challenge identified as hampering progress in SLP. By encapsulating semantic parsing and pose rendering within end-to-end differentiable structures (SLUL, SAGM, SLP-MoE, diffusion video synthesizer), the model realizes significantly higher back-translation accuracy and perceptually realistic signing. This approach demonstrates the practical viability and advantage of jointly modeling semantic interpretation and gesture realization for sign language video generation (Fang et al., 3 Dec 2025).

A plausible implication is that end-to-end architectures with explicit expert gating and stability-aware rendering objectives may generalize to other gesture- or multimodal video synthesis problems beyond sign language, especially where semantically grounded, temporally stable motion is critical.