TEAM-Sign: Direct Text-to-Sign Generation

• TEAM-Sign is a novel sign language production framework that leverages LLMs and VQ-VAE to generate discrete sign pose sequences directly from text without requiring gloss annotations.
• It employs a three-step prompting method to describe sign syntax, reorder text, and estimate temporal weights, effectively bridging linguistic and modality gaps.
• Empirical results demonstrate improvements in DTW-MJE and BLEU scores on How2Sign and Phoenix14T benchmarks over traditional baselines, indicating enhanced cross-modal accuracy.

TEAch Me Sign (TEAM-Sign) is a sign language production (SLP) framework that treats sign language as a natural language, leveraging LLMs via stepwise prompting and a VQ-VAE-based pose quantization pipeline. The system is designed to generate sign pose sequences—discrete keypoint tokens representing sign motion—directly from spoken language text, using the LLM’s own embedded knowledge and a structured prompting strategy to bridge linguistic and modality gaps, and does so without requiring manual gloss annotations (An et al., 15 Jul 2025).

1. Motivation and Linguistic Challenges

Sign language production differs fundamentally from spoken or written language tasks due to its multi-channel modality (hand shapes, motion, body posture, facial expression), non-linear grammar, and data scarcity. Traditional methods often use two-stage pipelines—text to gloss to pose/video—introducing error propagation and relying on gloss annotations, which are labor-intensive and inconsistent across languages. Direct text-to-sign modeling avoids the bottleneck of glosses, but is hindered by weak, implicit cross-modal alignment and the complexity of sign syntax, which often diverges syntactically and semantically from the source text. The lack of large-scale, high-fidelity training data further complicates direct generative attempts. TEAM-Sign’s approach is premised on the hypothesis that the reasoning and linguistic competence of modern LLMs can be internalized and exploited for cross-modal sign sequence generation if sign poses can be mapped into a tokenized form amenable to language modeling (An et al., 15 Jul 2025).

2. Model Architecture and Stepwise Prompting System

TEAM-Sign’s architecture consists of two subsystems: sign language encoding via VQ-VAE and relation learning with stepwise prompting. The key architectural elements are:

• Sign Encoding: A VQ-VAE encoder projects pose frames $p$ into latent vectors $z_e(p)$, which are quantized against a codebook $\{e_k\}$. Each video is thus represented as a sequence of discrete indices $\{k_1, \ldots, k_n\}$.
• LLM Backbone: Fine-tuned LLaMA 3-8B and Qwen2-7B serve as the generative backbone. During LLM fine-tuning, only LoRA adapters are trained; all VQ-VAE components remain frozen.
• Stepwise Prompting: The central novelty is a three-step prompting protocol that elicits latent sign language knowledge from the LLM:
  1. The LLM is prompted to describe the grammatical ordering of the target sign language (e.g., ASL/DGS).
  2. Given a source-language sentence, the LLM is prompted to reorder it in sign-language syntax.
  3. The LLM is prompted to estimate the temporal weight (relative motion duration) of each content word, producing an “assistance sequence” encoding key semantic elements and their durations.

Assistance sequences are generated with GPT-4o and incorporated into downstream LLM prompts as auxiliary planning information during sign-token sequence prediction. The LLM models $P(k_{1:T}\mid x, a)$ autoregressively, where $x$ is text and $a$ is the assistance sequence.

The VQ-VAE training loss combines reconstruction and penalizes repeated code usage to preserve fine-grained motion. The full objective includes $L_{Recon}$ and a repetition term $L_{Repet}$ to discourage code collapse, resulting in $L_{VQVAE} = L_{Recon} + L_{Repet}$.

3. Dataset Processing and Training Regimen

TEAM-Sign is evaluated on How2Sign (American Sign Language, ~35k sentences, 79 hours) and Phoenix14T (German, 8,257 sentences, 10.5 hours). Videos are pose-extracted and compressed to discrete token sequences via VQ-VAE. The resulting training triplets are $z_e(p)$0. Unlike previous works, gloss supervision is omitted; the vocabulary for generation is the token space of codebook indices.

For sequence modeling, the LLM is trained using cross-entropy over discrete code sequences, conditioned on both text and generated assistance. No explicit gloss pipeline or specialized semantic annotations are required. The pose encoder, decoder, and codebook remain frozen during LLM training, with LoRA providing an efficient adaptation pathway. Hyperparameters such as learning rate, batch size, and LoRA rank are not disclosed in the available data.

4. Evaluation Methodology and Comparative Baselines

Performance is measured by:

• DTW-MJE (Dynamic Time Warping Mean Joint Error): Minimizes pose alignment discrepancies after time warping.

• BLEU-1 to BLEU-4: Computed in a back-translation pipeline whereby generated poses are translated to text using a retrained translation model, and output is compared to ground truth reference text.

TEAM-Sign is benchmarked against ProgressiveTransformer (PT)—with and without data augmentation—as well as ablations disabling stepwise assistance. Ground-truth pose extraction is itself imprecise due to noise and resolution constraints; thus, all evaluations are subject to some level of uncertainty regarding fine hand and facial motions (An et al., 15 Jul 2025).

Model/Baseline	Phoenix14T DTW-MJE	Phoenix14T BLEU-4	How2Sign DTW-MJE	How2Sign BLEU-4
ProgressiveTransformer	0.1276	2.077	0.1733	0.628
TEAM-Sign - LLaMA3	0.1056	3.151	0.1371	1.116
TEAM-Sign - Qwen2	0.1038	3.107	0.1397	1.140
TEAM-Sign - LLaMA3 (no assist)	0.1204	2.636	0.1406	1.058

TEAM-Sign achieves ~18.7% DTW-MJE improvement on Phoenix14T and ~20.9% on How2Sign over PT.

5. Empirical Results and Ablation Findings

Quantitative analyses show that TEAM-Sign outperforms the PT baseline in both pose fidelity (lower DTW-MJE) and back-translation accuracy (higher BLEU). Ablation experiments substantiate the contribution of stepwise assistance: removing it yields higher error and lower BLEU, particularly on lower-resource Phoenix14T. Qualitative assessment reveals that TEAM-Sign captured coarse body and arm motion trends and signing rhythm more accurately than baselines, but still underperformed at resolving fine handshape articulations and facial cues—a limitation attributed both to the codebook design and the pose extraction noise inherent in existing datasets.

6. Current Limitations and Future Directions

The central strengths of TEAM-Sign are its elimination of the gloss annotation bottleneck, exploitation of the LLM’s own sign knowledge through structured prompting, and improved quantitative performance. Limitations remain in modeling fine hand/finger motion and dynamic facial expressions, as well as reliance on the VQ-VAE codebook for pose discretization. Back-translation BLEU is an indirect evaluation measure and may not fully capture non-linguistic quality. Assistance from the LLM is not guaranteed to be fully accurate or universally valid across sign language variants.

Prospective extensions include:

• Enhanced motion decoders for precise hand/finger/face articulation.
• More structured prompt engineering or grammar decomposition to improve sign-specific reasoning.
• Multilingual expansion leveraging the LLM paradigm.
• Hybrid systems merging LLM planning with specialized physical motion generation.
• Development of more granular perceptual or motion-specific metrics, possibly surpassing BLEU for embodied tasks.

A plausible implication is that TEAM-Sign's framing of SLP as language modeling with explicit reasoning steps could generalize to other embodied AI domains requiring linguistic structure to be mapped to sequenced physical outputs (An et al., 15 Jul 2025).