Sign Language Production Challenge

• Sign Language Production Challenge is a benchmarking initiative that translates spoken language into sign video using text-to-pose systems and standardized skeleton representations.
• It addresses data scarcity, inconsistent evaluation, and challenges in modeling both manual and non-manual articulators in continuous sign language production.
• The initiative has spurred advances through diverse approaches including retrieval-based, diffusion-based, and latent sequence modeling methods, setting new evaluation standards.

Searching arXiv for papers on the Sign Language Production Challenge and closely related methods. The Sign Language Production Challenge is a benchmarking initiative for Sign Language Production (SLP), the task of generating sign language video from spoken language inputs. The first Sign Language Production Challenge, held as part of the third SLRTP Workshop at CVPR 2025, evaluates Text-to-Pose (T2P) systems that translate spoken language sentences into sequences of skeleton poses over a range of metrics, using the RWTH-PHOENIX-Weather-2014T dataset and a custom hidden test set from a similar domain of discourse (Walsh et al., 9 Aug 2025). Its emergence reflects longstanding constraints in SLP research, including scarce large-scale parallel data, inconsistent evaluation, regression-to-the-mean effects in direct pose generation, and the difficulty of modeling manual and non-manual articulators in continuous signing (Rastgoo et al., 2021, Rastgoo et al., 2022).

1. Historical and technical background

The challenge belongs to a research trajectory in which SLP has progressively moved from avatar pipelines and gloss-mediated systems toward continuous, end-to-end generation. Earlier neural work established Progressive Transformers for direct text-to-pose generation and introduced counter decoding for variable-length continuous sequence generation, together with a back-translation evaluation mechanism on PHOENIX14T (Saunders et al., 2020). This line was extended to continuous 3D multi-channel SLP with adversarial training and a Mixture Density Network formulation, explicitly targeting realistic and expressive sign pose sequences and reporting a user evaluation with Deaf participants (Saunders et al., 2021).

Subsequent work diversified the design space. "Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives" separates translation and animation, using gloss supervision without treating gloss as a bottleneck and reporting an 11% improvement over competing results on PHOENIX14T (Saunders et al., 2021). "A Data-Driven Representation for Sign Language Production" reformulates continuous pose generation as discrete sequence generation via Vector Quantisation, eliminating dependence on costly annotation and increasing the BLEU-1 back translation score by up to 72% (Walsh et al., 2024). "Select and Reorder: A Novel Approach for Neural Sign Language Production" decomposes Text-to-Gloss translation into Gloss Selection and Gloss Reordering with Non-AutoRegressive decoding, achieving a 37.88% BLEU-1 improvement on mDGS (Walsh et al., 2024). Diffusion-based SLP also became prominent: "Neural Sign Actors" generates 3D SMPL-X avatar motion from raw text, and later challenge systems adopted text-conditioned diffusion directly in the T2P setting (Baltatzis et al., 2023, He et al., 20 Mar 2025).

This background is essential to understanding the challenge’s role. It does not merely rank systems; it operationalizes comparisons across families that differ in their dependence on gloss supervision, their use of retrieval versus generation, and their treatment of realism, expressiveness, and temporal alignment (Walsh et al., 9 Aug 2025).

2. Dataset design and pose representation

The challenge uses RWTH-PHOENIX-Weather-2014T, a German Sign Language weather-broadcast dataset, with 7,096 training videos, 519 development videos, and 642 test videos, each paired with continuous sign language video, gloss sequences, and spoken language subtitles (Walsh et al., 9 Aug 2025). In addition, the organizers curated a custom hidden test set from additional Phoenix broadcast material collected via the EASIER project, selecting weather-related sentences and manually verifying 500 sampled sentences to form an unseen evaluation set (Walsh et al., 9 Aug 2025).

A central contribution of the challenge is the standardization of skeleton representation. Initial 2D keypoints are extracted with Mediapipe holistic keypoints, then uplifted to 3D via neural optimization, producing signer-invariant skeletons with consistent bone lengths (Walsh et al., 9 Aug 2025). The resulting representation has 178 keypoints: 21 per hand, 128 for face, and 8 for body, normalized with the neck at the origin and the body fixed in the $xy$-plane (Walsh et al., 9 Aug 2025). This standardized geometry is intended to remove a major source of historical incomparability across SLP papers, namely divergent keypoint schemes and normalization protocols (Walsh et al., 9 Aug 2025).

The PHOENIX14T corpus also remains the substrate for several challenge-related methods. The hfut-lmc team’s Text-driven Diffusion Model (TDM), for example, was trained on PHOENIX14T, described there as featuring 8,257 full facial and gesture sequences in German Sign Language (He et al., 20 Mar 2025). The persistence of PHOENIX14T across multiple methodological generations has made it the de facto reference dataset for T2P benchmarking, but the challenge’s hidden test set was introduced precisely to test generalization beyond the standard public split (Walsh et al., 9 Aug 2025).

3. Evaluation protocol and metric ecology

Challenge evaluation combines text-based and pose-based measures. Text-based scores are produced via a standardized back-translation network, specifically a Sign Language Transformer with 3 layers, 8 heads, and encoder/decoder embedding sizes of 256 and 512, which converts generated skeletons back into spoken language for computing BLEU, CHRF, ROUGE, and WER (Walsh et al., 9 Aug 2025). Pose-based metrics include DTW-MJE, which measures average error between temporally aligned joints in predicted and ground-truth sequences, and Total Distance, defined as the ratio of predicted total hand movement distance to ground-truth total hand movement distance, with a score of 1 as optimal (Walsh et al., 9 Aug 2025).

This protocol formalizes ideas introduced earlier in Progressive Transformer work, where back translation was proposed because direct comparison of continuous poses does not reliably capture whether a generated sequence communicates the source sentence (Saunders et al., 2020). The challenge generalizes that idea by fixing the evaluation network, so that all submissions are mapped back to text identically (Walsh et al., 9 Aug 2025). This makes leaderboard scores more reproducible than prior cross-paper comparisons.

At the same time, the challenge literature explicitly identifies metric tension. Pose-based and text-based metrics can disagree, DTW-MJE can favor less articulated and longer sequences, and reliance on a fixed pose-to-text model introduces a back-translation bottleneck (Walsh et al., 9 Aug 2025). The organizers state that there is no perfect automatic metric for SLP yet, and that historical divergence in skeleton representation, normalization, and evaluation made comparison difficult before the release of the standardized pipeline (Walsh et al., 9 Aug 2025). A plausible implication is that the challenge is as much an intervention in evaluation methodology as in model ranking.

4. Competitive methods and leaderboard structure

The challenge attracted 33 participants who submitted 231 solutions, and solutions were ranked using a multi-metric Pareto dominance approach so that no single metric dominated the evaluation (Walsh et al., 9 Aug 2025). The top-performing systems illustrate three distinct design philosophies: gloss-grounded retrieval, gloss-free diffusion, and gloss-free latent sequence modeling (Walsh et al., 9 Aug 2025).

Team	Method	Reported hidden-set results
USTC-MoE	Retrieval-based pipeline with fine-tuned XLM-R and a gloss-pose dictionary	BLEU-1 31.40, DTW-MJE 0.0574, Total Distance 1.185
hfut-lmc	Text-Driven Conditional Diffusion Model	BLEU-1 30.54, DTW-MJE 0.0492, Total Distance 0.971
Hacettepe	Gloss-free transformer with disentangled latent pose autoencoder	BLEU-1 27.51, DTW-MJE 0.0531, Total Distance 0.761

USTC-MoE, the winning system, uses a retrieval-based pipeline grounded in gloss. It translates text to gloss with fine-tuned XLM-R, segments training pose sequences by gloss using continuous sign language recognition, builds a gloss-pose dictionary, and retrieves and concatenates pose segments at inference time (Walsh et al., 9 Aug 2025). The organizers attribute the strength of this approach to its use of real sign motion for each gloss, which sidesteps generative pose artifacts and yields highly expressive outputs (Walsh et al., 9 Aug 2025). This places it in conceptual proximity to dictionary-based and sign-stitching approaches that also seek to avoid regression-to-the-mean by assembling motion from real exemplars rather than regressing dense trajectories directly (Walsh et al., 2024).

The second-place hfut-lmc system is fully generative and gloss-free. It conditions a diffusion process directly on text and optimizes both joint position and bone orientation constraints (He et al., 20 Mar 2025, Walsh et al., 9 Aug 2025). The third-place Hacettepe system is also gloss-free, but uses an autoencoder that decomposes skeletons into face, body, and hands, learns an 80-dimensional latent pose space, and translates German text embeddings into that latent space with a seq2seq transformer (Walsh et al., 9 Aug 2025). Together these systems made the challenge a comparison not only of architectures, but of supervision regimes and motion priors.

5. The hfut-lmc Text-driven Diffusion Model

The hfut-lmc team’s solution, "Text-Driven Diffusion Model for Sign Language Production," targets semantically aligned sign language pose generation directly from text, bypassing the intermediate gloss representation commonly used in previous pipelines (He et al., 20 Mar 2025). During training, TDM uses an encoder to encode text sequences and incorporates them into the diffusion model as conditional input to generate sign pose sequences (He et al., 20 Mar 2025). The text encoder is a Progressive Transformer that embeds the textual sequence into a latent representation capturing global semantic information, and its output is combined with the time step, processed by an MLP, to form a relevant condition $g$ for denoising (He et al., 20 Mar 2025).

The denoiser $\mathcal{D}$ receives the noisy pose and the condition $g$, projects the input pose through a linear embedding layer, adds sinusoidal positional encoding for temporal context, and applies both multi-head self-attention and cross-attention to model spatial and temporal dependencies (He et al., 20 Mar 2025). The forward diffusion process is

$p_t = \gamma_t p_0 + \sigma_t \epsilon,$

where $\epsilon$ is sampled from a normal distribution and $\gamma_t^2 + \sigma_t^2 = 1$; in the reverse process, the denoiser predicts clean pose sequences directly,

$p_0' = \mathcal{D}(p_t, g).$

Unlike classic diffusion models, the denoiser is trained to predict the clean data rather than just the noise (He et al., 20 Mar 2025).

To improve pose quality, the model uses two losses. The joint loss,

$$\mathcal{L}_{joint} = \frac{1}{J} \sum_{j=1}^J |p_j - p_j'|,$$

measures differences between predicted and ground-truth joint positions, while the bone orientation loss,

$$\mathcal{L}_{bone} = \frac{1}{B} \sum_{b=1}^{B} (q_b - q_b')^2,$$

penalizes deviations in bone vectors computed from parent-child relationships (He et al., 20 Mar 2025). The total loss is

$g$0

with $g$1 (He et al., 20 Mar 2025). Training uses 4 layers, 8 heads, and embedding size 1024 for both encoder and denoiser, diffusion steps $g$2 with a cosine scheduler, inference iterations $g$3, Adam with learning rate $g$4, PyTorch, and an NVIDIA RTX 4090 GPU (He et al., 20 Mar 2025).

In the team’s reported challenge comparison, TDM achieves BLEU-1 $g$5, BLEU-4 $g$6, ROUGE $g$7, WER $g$8, and the lowest DTW $g$9, placing second in the challenge table shown in that paper (He et al., 20 Mar 2025). The authors interpret the lowest DTW as evidence of strong temporal alignment quality, while the overall result supports the viability of gloss-free, text-conditioned diffusion for SLP (He et al., 20 Mar 2025).

6. Significance, limitations, and future trajectories

The challenge’s principal significance lies in standardization. The organizers released a public evaluation pipeline, including skeleton extraction, normalization, pose-based metrics, and a fixed back-translation network, explicitly to establish a consistent baseline for future SLP comparisons (Walsh et al., 9 Aug 2025). This addresses a problem already emphasized in review work: SLP evaluation has historically lacked standardized, comprehensive measures that jointly assess intelligibility, realism, and linguistic correctness (Rastgoo et al., 2021, Rastgoo et al., 2022).

The results also sharpen a central methodological trade-off. Retrieval-based systems grounded in gloss can yield highly expressive and accurate motion when a high-quality gloss-pose dictionary can be built, but gloss dependency does not scale well to open domains and to languages lacking full gloss annotation (Walsh et al., 9 Aug 2025). Gloss-free generative methods, including diffusion and latent autoencoding, reduce annotation dependence and remain competitive, but current models still struggle with prosody, sequence length, and subtle temporal rhythm (Walsh et al., 9 Aug 2025). This suggests that the challenge formalized a live controversy in SLP: whether the field should prioritize direct generative modeling or preserve symbolic intermediates wherever gloss resources exist.

The challenge also sits within a broader shift from 2D skeleton benchmarks toward richer motion representations and broader downstream uses. "SignAvatars" introduced a large-scale 3D multi-prompt benchmark with 70,000 videos, 8.34 million frames, 153 signers, and biomechanically valid body, hand, and face annotations, explicitly supporting 3D SLP from text scripts, individual words, and HamNoSys notation (Yu et al., 2023). "Using Sign Language Production as Data Augmentation to enhance Sign Language Translation" showed that skeleton-based production, sign stitching, SignGAN, and SignSplat can augment existing datasets and improve translation performance by up to 19% (Walsh et al., 11 Jun 2025). "Beyond Words: AuralLLM and SignMST-C for Sign Language Production and Bidirectional Accessibility" introduced CNText2Sign and CNSign as a unified environment for Chinese sign language, with direct pose assessment rather than sole reliance on back-translation (Li et al., 1 Jan 2025). More recently, "SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning" proposed sparse keyframe learning, Conditional Flow Matching, and 3D Gaussian Splatting across four sign languages, explicitly positioning itself against both regression-to-the-mean and robotic dictionary retrieval (Low et al., 11 Mar 2026).

Future work identified in the challenge report is correspondingly broad: reducing gloss reliance, improving fluency and co-articulation, devising better automatic or human-in-the-loop metrics, releasing complete preprocessing and extraction pipelines, leveraging LLMs and multimodal pretraining, and extending evaluation from pose generation to photorealistic signer or avatar video (Walsh et al., 9 Aug 2025). In that sense, the Sign Language Production Challenge is both a competition and a reference point: it consolidates a benchmark for T2P translation while exposing the unresolved questions that continue to define SLP research.