Sign Language Instruction Generation

• Sign Language Instruction Generation (SLIG) is a multimodal task that synthesizes instructional sign content from various inputs such as text, speech, images, and live signing.
• It leverages diverse representational choices—ranging from gloss embeddings, pose tokens, to latent diffusion models—to coordinate manual (hand shape, movement) and non-manual (facial expression, prosody) channels.
• SLIG research drives advancements in sign synthesis for education, accessibility, and embodied control while addressing challenges like data scarcity and the accurate translation of linguistic structure into visual signs.

Sign Language Instruction Generation (SLIG) is a family of generation tasks centered on producing instructional sign-language content from multimodal inputs. Recent work suggests two closely related operationalizations. In one formulation, SLIG produces step-by-step textual instructions that enable non-signers to imitate and learn sign gestures (Tariquzzaman et al., 22 Aug 2025). In a broader research usage, the same area also encompasses generation of sign images, 3D motions, pose sequences, photorealistic videos, and sign-conditioned downstream instructions from text, speech, glosses, images, or live signing (Yadav et al., 13 Aug 2025, Dong et al., 2024, Bai et al., 18 Jun 2026). Across these formulations, the technical problem is not merely lexical substitution: it requires coordinated synthesis or description of manual and non-manual channels, including handshape, movement, location, palm orientation, facial expression, and body movement (Viegas et al., 2022, Zhang et al., 8 Feb 2025).

1. Scope, lineage, and disciplinary setting

The modern framing of SLIG inherits the broader sign language generation agenda described in the interdisciplinary workshop synthesis on sign language recognition, generation, and translation (Bragg et al., 2019). That work argued that successful systems require expertise spanning computer vision, computer graphics, natural language processing, human-computer interaction, linguistics, and Deaf culture, and it emphasized that research had often proceeded in disciplinary silos. It also stated that there was no fully automated, high-quality SLIG system, with human intervention still needed for smooth transitions, modulation, and linguistically complex constructions such as classifiers, adverbs, and adjectival modifications (Bragg et al., 2019).

The survey on sign language production organized the field by input modality, datasets, applications, and model families, and described a standard deep-learning pipeline in which spoken or written input is mapped to gloss, then to skeleton or pose, and then to image or video output (Rastgoo et al., 2022). The same survey also noted end-to-end variants that skip the gloss step. Its taxonomy makes clear that SLIG is not reducible to ordinary sequence transduction: sign languages are visual-gestural, multi-channel, and grammatically distinct from spoken languages, while systems must also address photo-realism, real-time constraints, data scarcity, and signer acceptability (Rastgoo et al., 2022).

A common misconception is that SLIG is a narrow subproblem of text-to-avatar animation. Recent work indicates a much wider design space: instructional systems may output pedagogical text, static sign images, continuous pose, photorealistic video, or even robot manipulation commands derived from live signing (Tariquzzaman et al., 22 Aug 2025, Yadav et al., 13 Aug 2025, Bai et al., 18 Jun 2026).

2. Representational choices and conditioning signals

A persistent design question in SLIG is the choice of intermediate representation. Earlier pipelines emphasized gloss and symbolic notation, including HamNoSys and SiGML, because they provide a structured bridge between spoken language and sign synthesis (Rastgoo et al., 2022). More recent work has diversified this space into gloss-aligned embeddings, pose tokens, vector-quantized latent codes, explicit sign parameters, and sentence-level or multimodal latent spaces.

GLOS exemplifies a strong gloss-centered formulation. It replaces a sentence-level condition with gloss-level conditions, defined as sequences of gloss embeddings temporally aligned with the motion sequence, and introduces Temporal Alignment Conditioning (TAC) so that each motion timestep receives the corresponding gloss semantics rather than indiscriminate cross-attention over the entire sentence (Lee et al., 9 Jun 2025). Its central fusion mechanism is

$$\mathbf{X}^\text{part}_\text{TAC} = \text{Conv1D}\left( \text{LN}(\mathbf{X}^\text{part}_\text{IPA}) \otimes (1 + u) \oplus v \right),$$

where $u$ and $v$ are temporal modulation parameters produced from the gloss condition (Lee et al., 9 Jun 2025). This design yielded lower DTW-JPE and substantially better back-translation scores than sentence-level alternatives on CSL-Daily and PHOENIX-2014T, including WER $62.24$, BLEU-4 $13.27$, and ROUGE $35.83$ on CSL-Daily (Lee et al., 9 Jun 2025).

At the opposite end, gloss-free methods aim to remove the annotation bottleneck. Text2SignDiff formulates sign language production as a gloss-free latent diffusion model over SignVAE latents, conditioned by a cross-modal signing aligner that places text and pose into a shared latent space (Feng et al., 13 Sep 2025). Its total timestep loss combines diffusion denoising and time-weighted semantic supervision,

$$\mathcal{L} = \mathcal{L}_d + \left(1-\frac{t}{T}\right)\mathcal{L}_s,$$

so that early denoising steps receive stronger semantic pressure (Feng et al., 13 Sep 2025). This architecture achieved state-of-the-art performance on PHOENIX14T and How2Sign (Feng et al., 13 Sep 2025).

Instruction-text generation introduces yet another representational regime. BdSLIG structures prompts around standard sign parameters—handshape, movement type, location, palm orientation, spatial interaction, temporal dynamics, and facial cues—and injects them directly into the prompt through Sign Parameter-Infused (SPI) prompting (Tariquzzaman et al., 22 Aug 2025). The conditional form is

$$a_i = \arg\max \mathbb{P}(a \mid s(\mathcal{V}), f(\mathcal{P}, \mathcal{S}), a_{<i}),$$

where $s(\cdot)$ is the frame-sampling function, $\mathcal{P}$ the base prompt, and $u$0 the structured parameter set (Tariquzzaman et al., 22 Aug 2025). This formulation makes the instructions more structured and reproducible than free-form natural text from vanilla prompting (Tariquzzaman et al., 22 Aug 2025).

Large-language-model approaches further expand the representational space by treating sign outputs as discrete linguistic tokens. TEAM-Sign uses VQ-VAE to encode sign videos into token sequences and adds stepwise prompting that asks an LLM to explain sign-language sentence structure, produce a gloss-like reformulation, and estimate relative durations before generating pose tokens (An et al., 15 Jul 2025). SOKE similarly treats signs as tokens, but decouples upper body, left hand, and right hand into separate VQ-VAE streams and inserts language- and part-specific special tokens into a pretrained multilingual LM (Zuo et al., 2024). Taken together, these systems suggest that representation choice is now one of the main axes along which SLIG methods differ.

3. Generative architectures and synthesis pipelines

Image generation, pose generation, avatar synthesis, and token-based language modeling all appear in current SLIG research. One line of work focuses on high-quality static sign imagery. The Indian Sign Language image generator combines ProGAN and SAGAN into an attention-based, progressively growing, class-conditional GAN. The generator begins at $u$1 resolution, doubles up to $u$2, inserts self-attention at $u$3 and $u$4, uses WGAN-GP loss, weight standardization, pixel-wise feature normalization, and a fade-in mechanism, and achieves improvements of 3.2 IS and 30.12 FID over traditional ProGAN on the authors’ dataset (Yadav et al., 13 Aug 2025). On the Gunji et al. dataset, the maximum reported improvements are 2.47 IS and 32.12 FID, while attention ablation shows FID values of 29.7 at $u$5, 28.8 at $u$6, and 23.3 when both attention layers are present (Yadav et al., 13 Aug 2025).

Video-oriented systems often decompose the problem into explicit stages. SignAligner operates in three stages: text-driven pose modalities co-generation, online collaborative correction (OCC), and realistic sign video synthesis (Wang et al., 13 Jun 2025). It jointly generates Pose, Hamer, and Smplerx modalities, refines them via cross-modal attention and dynamically weighted OCC loss, and renders them through a retrained RealisDance model. On PHOENIX14T+, it reports BLEU-1 20.56, BLEU-4 8.17, ROUGE 20.88, SSIM 0.73, PSNR 15.32, and FID 26.26, outperforming PTSLP+RealisDance and CogVideoX (Wang et al., 13 Jun 2025).

3D avatar-centric generation adopts a different decomposition. SignAvatar uses a transformer-based conditional variational autoencoder with CLIP embeddings for text and image conditioning, Hand4Whole to recover SMPL-X parameters, and Blender rendering of the decoded motion (Dong et al., 2024). Its ASL3DWord dataset contains 103 gloss words and 1547 videos, with train/test split 1208/339, and the model supports both word-level reconstruction from video and generation from text or image prompts (Dong et al., 2024). Curriculum learning is implemented through masked motion modeling with a mask ratio increasing by

$u$7

which improves robustness and generalization (Dong et al., 2024).

Token-based and diffusion-LLMs push SLIG toward general-purpose sequence modeling. MaDiS introduces a masked diffusion LLM with tri-level cross-modal pretraining over token, latent, and 3D physical space, an unmasking with temporal checkpoints strategy that reduces combinatorial complexity by over $u$8 times, and a mixture-of-parts embedding layer for gated fusion of part-wise sign tokens (Zuo et al., 27 Jan 2026). It reduces inference latency by nearly 30\% and introduces SiBLEU and SiCLIP as sign-specific evaluation metrics (Zuo et al., 27 Jan 2026). SOKE, by contrast, remains autoregressive but improves efficiency through multi-head decoding that predicts part-wise token streams simultaneously and augments generation with external sign dictionaries (Zuo et al., 2024). A plausible implication is that SLIG has moved from monolithic pose regression toward compositional latent or token-based generation with explicit part structure.

4. Non-manual markers, prosody, and linguistic fidelity

A central result across the literature is that SLIG quality depends on non-manual signals. The Dual Encoder Transformer for sign generation was the first to employ facial action units in sign language generation (Viegas et al., 2022). It jointly models text and gloss through separate Transformer encoders, fuses them via Hadamard multiplication, and autoregressively predicts upper-body and hand joints, 68 facial landmark coordinates, and a 17-dimensional vector of continuous AU intensities (Viegas et al., 2022). On the test set, the dual-encoder model achieved BLEU-4 8.19 with manual features only and BLEU-4 5.76 when manual features, facial landmarks, and AUs were included, outperforming G2S and T2S in both settings (Viegas et al., 2022). The underlying claim is not simply that faces add realism; rather, facial muscle activity carries affective, grammatical, and semantic information that gloss alone omits (Viegas et al., 2022).

Prosody is also not reducible to generic smoothness. The computational study of intensification in sign language generation introduced four annotation schemes—End-Marking, Delayed Release, Suffixation, and Reiteration—to encode intensity modifiers in glosses, annotated 12.8K gloss-token-to-sentence pairs on PHOENIX-14T, and trained an intensity tagger where M-BERT achieved 75.3\% F1 on the test set (İnan et al., 2022). In human evaluation, enhanced-gloss models were preferred in 50\% of cases versus 26\% for the baseline, with $u$9 (İnan et al., 2022). This suggests that prosodic control variables are not optional refinements but affect overall understanding.

The modular AI-driven ASL video generator with non-manual markers extends this argument into photorealistic synthesis. It uses GPT-4o with few-shot prompting to produce ASL gloss strings, zero-shot prompting to detect whether an English sentence invokes specific non-manual markers, a motion-matching system over a library of >$v$0 signed pose sequences across 3,915 glosses, and a signer-conditioned U-Net-like video model trained with frame $v$1, hand-region $v$2, and LPIPS losses (Zhang et al., 8 Feb 2025). The system reports BLEU-4 0.276 for English-to-ASL-gloss translation, non-manual prediction precision/recall up to 0.98/0.93 for yes/no questions and overall P=0.91, R=0.97, and a user study with 30 DHH participants in which the full AI model was rated at least acceptable in 53.8\% of cases and translation quality acceptable or better in 65.3\% (Zhang et al., 8 Feb 2025). A reasonable reading of these results is that non-manual markers function as structural constraints on intelligibility, not as stylistic decoration.

5. Data resources and evaluation regimes

Current SLIG research draws on heterogeneous data regimes: low-resource instruction text, multi-pose video corpora, 3D motion datasets, image datasets, and speech-aligned sign collections. Representative resources include the following.

Resource	Properties	Role
BdSLIG	First Bengali SLIG dataset; sourced from BdSLW60; 3 human annotators; two-stage validation	Stepwise textual instruction generation
PHOENIX14T+	Extension of RWTH-PHOENIX-Weather 2014T with Pose, Hamer, Smplerx	Multimodal pose co-generation and video synthesis
ASL3DWord	103 gloss words, 1547 videos, 3D joint rotation data	Word-level 3D reconstruction and generation
ISL image dataset	247,500 images, 165 classes, 3×1024×1024, >1,500 samples per class	Class-conditional image generation
ISL speech dataset	~9,092 recordings, ~18 hours, 5 professional signers, ~10,000-word English vocabulary	Direct speech-to-sign generation

These resources are reported in recent work on Bengali SLIG, SignAligner, SignAvatar, Indian Sign Language image generation, and speech-to-sign generation (Tariquzzaman et al., 22 Aug 2025, Wang et al., 13 Jun 2025, Dong et al., 2024, Yadav et al., 13 Aug 2025, Kapoor et al., 2021).

Evaluation is correspondingly fragmented. Textual-instruction generation typically uses ROUGE-1/2/L, BLEU, METEOR, and BERTScore, as in BdSLIG, where SPI prompting improves most models and GPT-4.1 with SPI reaches ROUGE-1 0.553, ROUGE-2 0.271, ROUGE-L 0.384, BLEU 0.196, METEOR 0.492, and BERTScore 0.441 (Tariquzzaman et al., 22 Aug 2025). Pose and motion generation rely heavily on back-translation and sequence alignment metrics such as DTW, DTW-JPE, WER, BLEU-4, and ROUGE (Lee et al., 9 Jun 2025, Kapoor et al., 2021). Image and video synthesis additionally report IS, FID, SSIM, PSNR, LPIPS, FID-VID, and FVD (Yadav et al., 13 Aug 2025, Zhang et al., 8 Feb 2025). MaDiS adds SiBLEU and SiCLIP to directly evaluate sign token overlap and sign-text semantic alignment (Zuo et al., 27 Jan 2026).

At the same time, the literature repeatedly cautions against metric reification. BdSLIG explicitly notes that only human, sign-literate evaluators can truly assess instruction quality, because automated metrics may penalize semantically equivalent but lexically divergent instructions (Tariquzzaman et al., 22 Aug 2025). This caution is consistent with the earlier interdisciplinary argument that comprehension and perception studies with Deaf signers remain primary evaluation methods for sign-generation systems (Bragg et al., 2019).

6. Deployment settings and applications

The application space of SLIG already extends beyond laboratory text-to-pose benchmarks. One deployed setting is large-scale dialogue. The goal-oriented conversational system with ASL instructions integrates LLM-based text-to-gloss translation, rule-based alternatives, token-level video retrieval from a sign dictionary, and a touch-based interface on Amazon Alexa Echo Show devices (İnan et al., 2024). It was designed with community input, supports recipes and wikiHow-style instructions, and reports BERTScore F1 0.80 between LLM and rule-based glosses, high retrieval coverage, and user feedback ratings on par with the non-signing system variant (İnan et al., 2024).

Another deployment class is sign-conditioned embodied control. SignVLA presents a modular framework in which live sign video is converted into text instructions for a downstream Vision-Language-Action policy (Bai et al., 18 Jun 2026). It extracts 544-dim MediaPipe hand-landmark features, models temporal dynamics with an attention-enhanced LSTM, stabilizes predictions by requiring the same class to be the argmax for at least 3 consecutive frames with an 8-frame cooldown, and supports both alphabet-level and command-level input over 33 common ASL command signs (Bai et al., 18 Jun 2026). The system achieves Top-1 accuracy 88.9\%, Top-5 accuracy 96.5\%, and >97\% success on various LIBERO manipulation tasks (Bai et al., 18 Jun 2026). In this configuration, SLIG functions as an accessibility layer for embodied intelligence rather than as a standalone signing avatar.

Speech- and multimodal-input systems extend the same logic to continuous spoken content. The first direct speech-to-sign model generates continuous pose sequences from speech segments using a multi-task Transformer with an auxiliary speech-to-text decoder and a cross-modal discriminator, achieving DTW 14.05 and PCK 53.33 on its Indian Sign Language benchmark (Kapoor et al., 2021). MS2SL later generalized continuous sign production from either text or speech through a sequence diffusion model, a joint embedding space for text, audio, and sign, and embedding-consistency learning, reporting the strongest BLEU and ROUGE values among compared systems on How2Sign and PHOENIX14T, including text-to-sign BLEU-1 14.67 and ROUGE 16.38 on How2Sign (Ma et al., 2024).

Educational applications remain prominent. The Indian Sign Language image generator explicitly describes automated glossaries, dynamic text-to-sign illustration, and the possibility of combining output images for animation or textbook-like sequential illustration (Yadav et al., 13 Aug 2025). In the textual-instruction setting, BdSLIG is explicitly framed as producing pedagogically useful stepwise descriptions for sign imitation (Tariquzzaman et al., 22 Aug 2025). The field therefore spans both instructional media for learners and accessibility interfaces for sign-first users.

7. Open problems and research directions

Low-resource and unpaired-data settings remain central obstacles. USLNet addresses this by learning from abundant single-modality text and video without parallel sign data, using single-modality reconstruction, cross-modality back-translation, and a sliding window aligner to manage sequence-length and feature-dimension mismatch (Guo et al., 2024). It is presented as the first unsupervised sign language translation and generation model capable of generating both natural language text and sign language video in a unified manner (Guo et al., 2024). MS2SL similarly minimizes reliance on sign triplets and supports continuous refinement even when audio is missing (Ma et al., 2024). These systems indicate that data efficiency is no longer restricted to standard pretraining; it is now an architectural objective.

Several problems identified in the interdisciplinary and survey literature remain active. These include data scarcity, lack of standardized annotation and representation, difficulties with depiction and classifier constructions, the need for smooth transitions and natural non-manual synthesis, the absence of robust quantitative comprehension metrics, and persistent uncanny valley concerns in avatar-mediated delivery (Bragg et al., 2019, Rastgoo et al., 2022). Recent work adds further nuances: gloss dependence limits flexibility and generalization when gloss annotations are unavailable (Feng et al., 13 Sep 2025), while even strong automated text metrics may fail to capture actual instruction utility (Tariquzzaman et al., 22 Aug 2025).

A plausible implication is that future SLIG systems will combine several trends already visible in the literature: explicit linguistic control when fine-grained sign parameters or gloss timing are available, gloss-free conditioning when annotation is scarce, discrete or latent motion tokenization for scalable sequence modeling, stronger treatment of non-manual markers and prosody, and evaluation protocols that include Deaf and Hard-of-Hearing participants rather than relying only on back-translation or lexical overlap (Lee et al., 9 Jun 2025, Feng et al., 13 Sep 2025, Zhang et al., 8 Feb 2025, İnan et al., 2024). Under that view, SLIG is best understood not as a single benchmark task, but as a multimodal generation field whose central challenge is preserving linguistic structure, perceptual quality, and instructional usefulness simultaneously.