Advances in Sign Language Translation

Updated 30 August 2025

Sign Language Translation is the computational task of converting visual-gestural sign language into spoken text, bridging modality and linguistic gaps.
Modern SLT research leverages multimodal learning by integrating spatio-temporal feature extraction, gloss-free methods, and large language models for improved accuracy.
Emerging approaches employ end-to-end architectures and hierarchical alignment strategies to overcome data scarcity and complex non-monotonic translations.

Sign Language Translation (SLT) is the computational task of mapping signed language, typically captured as video sequences, to spoken language text. Unlike spoken languages, sign languages are visual-gestural, high-dimensional, and richly multi-channel, encoding information spatially and temporally via hand shapes, locations, orientations, movements, non-manual markers, and facial expressions. SLT systems address the challenging problem of bridging a substantial modality gap and deep linguistic divergence, with approaches historically reliant on intermediate representations (notably glosses), but now increasingly realizing direct gloss-free or end-to-end methods. Research in SLT encompasses multi-modal machine learning, computer vision, natural language processing, and linguistic annotation theory.

1. Traditional Frameworks and the Role of Glosses

Conventional SLT architectures have been built on a two-stage pipeline: a Sign Language Recognition (SLR) component maps video frames to glosses—written tokens or labels transcribed by experts to approximate the signed message—and a downstream Neural Machine Translation (NMT) model translates gloss sequences to the target spoken language. Pipeline variants include RNN-based and, more recently, Transformer-based models.

However, it has been demonstrated that this gloss-centric paradigm imposes a representational bottleneck. Specifically, translation from predicted glosses can outperform that from ground-truth glosses, revealing that glosses do not upper bound SLT performance and may be an inefficient, lossy encoding of the underlying multi-channel sign signal (Yin et al., 2020). Studies also show that glosses compress nuanced spatial, temporal, and non-manual visual information into a one-dimensional, sequential form, discarding critical cues for accurate translation.

This recognition has motivated research into end-to-end architectures where joint training of recognition and translation modules, or alternate richer annotation schemes, enables intermediate representations not tied to glosses, but optimized for the translation objective itself. There is increasing evidence that direct or alternative representations can exploit the full capacity of the visual-linguistic correspondence, thereby driving current progress.

2. Spatio-Temporal Feature Learning and Temporal Modeling

Addressing the intrinsic spatio-temporal nature of sign languages, modern SLT systems emphasize expressive visual feature extraction and temporal modeling.

Notable advances include multi-cue decomposition, where spatial features are separated per-channel (e.g., face, hands, global frame, pose) and then temporally aggregated. For example, the STMC-Transformer framework extracts and models both intra-channel and inter-channel temporal correlations, using stacked BiLSTM layers combined with CTC loss for local recognition, followed by a compact Transformer-based translation module (Yin et al., 2020).

Hierarchical feature learning using multi-scale temporal segments further alleviates the need for explicit boundary annotation and enhances discriminability. TSPNet implements a pyramid of segmentations (e.g., sliding windows of 8, 12, 16 frames), aggregating local and non-local semantics through inter-scale and intra-scale attention modules. This hierarchical modeling results in substantial BLEU and ROUGE improvements on challenging datasets (Li et al., 2020).

Spatial preservation is also central in architectures employing 2D convolutional feature maps and 2D pixel-wise self-attention (e.g., (Ruiz et al., 4 Feb 2025)), rather than flattening frames. Two-dimensional positional encodings ensure spatial information is not lost, enabling the model to maintain both local and global context over gesture events.

In graph-based approaches, hierarchical spatio-temporal graphs represent relationships across body regions at multiple scales—high-level graphs for major regions (hands, face), and fine-level graphs for joints and facial keypoints. Graph convolutions and transformer-like self-attention allow simultaneous modeling of local articulation and holistic context (Kan et al., 2021).

3. Gloss-Free and Weakly-Supervised Approaches

Given the scarcity and cost of manual gloss annotation, recent work pivots toward gloss-free or resource-efficient SLT. Key innovations include:

Direct keypoint sequence modeling: Extracting body/hand keypoints, applying customized normalization per body part, and learning translation via Seq2Seq (e.g., GRU with attention models) allows direct mapping from skeleton dynamics to text. Stochastic Augmentation and Skip Sampling (SASS) further addresses variable sequence length and ensures salient motion is preserved (Kim et al., 2022).
Data augmentation from monolingual text: Rule-based heuristics (lemmatization, content-word filtering, syntactic reordering, and language-specific reordering) transform abundant monolingual textual corpora into pseudo gloss-text pairs, enabling large-scale pre-training and overcoming low-resource bottlenecks. Compared to traditional back-translation, these heuristics better exploit gloss-specific lexical overlap and syntactic divergence (Moryossef et al., 2021, Peng et al., 2023).
Pseudo gloss generation via LLMs: In-context learning—prompting an LLM with a small set of text–gloss pairs—enables draft pseudo annotation, substantially reducing reliance on expert glossing. Weakly supervised temporal reordering, using visual alignment and CTC loss, further bridges mismatches between LLM-generated gloss and actual sign sequence (Guo et al., 21 May 2025).
Conditional variational autoencoding and latent alignment: Direct cross-modal alignment, bypassing glosses, is realized by training conditional VAEs with dual KL divergences for encoder (visual-text latent regularization) and decoder (self-distillation between prior and posterior outputs), enhanced by attention-based residual Gaussian distribution (Zhao et al., 2023).
Diffusion models for diversity: Non-autoregressive latent diffusion transforms random noise to target text, conditioned on multi-scale visual features (frame and video-level) fused for semantic guidance, and can be further enriched by pseudo-glosses, offering both accuracy and diversity in SLT (Moon et al., 26 Nov 2024).

4. Scaling, Multilinguality, and Contextual Modeling

Efforts have increasingly focused on scaling SLT models to broad domains, multilingual settings, and incorporating discourse context:

Large-scale pretraining: Unifying pretraining across noisy sign video data, synthetic SLT “augmented” via MT-labeled captions, and large-scale machine translation corpora. Prompt-driven encoder–decoder models initialized from pretrained LLMs (e.g., mT5, ByT5) allow unified text-to-text learning, facilitating zero-shot SLT capabilities (ASL-to-X) by exploiting shared decoding representations and control tokens for task delineation (Zhang et al., 16 Jul 2024).
Multilingual and many-to-many mapping: Dual-CTC objectives supervise both token-level sign language identification and spoken text generation directly from sign video. This design (Sign2[LID+Text]) allows universal models supporting ten sign languages (SP-10, PHOENIX14T, CSL-Daily) and multiple spoken targets, with token-level CTC for fine-grained language cues and text-oriented CTC for alignment. Such approaches mitigate language conflicts and alignment noise inherent to gloss-free many-to-many SLT (Tan et al., 30 May 2025).
Context-aware architectures: Multi-modal Transformer models employ complementary encoders (video, sign spotting, and context) with a decoder attending over all modalities, thus leveraging prior sequence context and confident sign predictions to disambiguate ambiguous visual cues. Contextual modeling nearly doubles prior BLEU scores on large datasets and is critical in scaling to open domains (Sincan et al., 2023).

5. Integration with LLMs and Efficient Visual-Text Bridging

Recent trends emphasize the integration of LLMs (both text-only and multimodal) for translation generation, moving beyond domain-specific models:

Tokenization and hierarchical linguistic regularization: Visual encoders extract video features which are then vector-quantized into discrete character-level tokens. These tokens are grouped into word-level units via optimal transport, producing a language-like “sign sentence” suitable for frozen LLM prompting. Sign-text alignment is enforced using a Maximum Mean Discrepancy (MMD) loss (Gong et al., 1 Apr 2024).
Multimodal LLMs and sign descriptions: Off-the-shelf MLLMs (such as LLaVA-OneVision) are used to automatically generate fine-grained textual descriptions of each frame (e.g., hand shape, facial cues), which are then fused with visual features for alignment in the spoken sentence space. To reduce computational costs at inference, learned mappers predict description embeddings from vision (Kim et al., 25 Nov 2024). This approach eliminates the need for glosses entirely while providing a synergistic text–vision representation that is highly compatible with downstream LLM decoding.
Spatio-temporal efficiency with off-the-shelf encoders: The SpaMo framework demonstrates that off-the-shelf visual encoders (ViT, VideoMAE) can extract spatial and motion dynamics, which are fused, aligned to the LLM embedding space using contrastive pretraining, and then fed (with a task-specific language prompt) to a fixed LLM for translation. This method achieves state-of-the-art results and reduces reliance on expensive domain-specific visual pretraining (Hwang et al., 20 Aug 2024).

6. Alignment, Decoding, and Training Strategies

Effective mapping from complex, variable-length sign video to spoken text requires both alignment and efficient decoding:

Joint CTC/Attention alignment and hierarchical encoding: Hierarchical encoders are introduced, with early stages trained with CTC to align sign video representations to intermediate gloss or token sequences (handling monotonic alignment), while a text-oriented encoder plus text-based CTC loss provides reordering capabilities for final translation output. Joint decoding—scoring via a fusion of attention and CTC log probabilities—mitigates exposure bias and allows the model to manage both monotonic and non-monotonic alignments (Tan et al., 12 Dec 2024).
Iterative refinement and distillation: The IP-SLT framework iteratively refines an initial prototype semantic representation of the video, using a combination of self-attention and cross-attention across iterations, with a distillation loss aligning intermediate and final predictions. This recurrent “rereading” approach mimics human translation strategies, boosting final accuracy (Yao et al., 2023).
Cross-modal information fusion via graphs: Multimodal graph encoders fuse visual and textual/gloss features into a unified latent space, updating dynamic graphs as recognition module (e.g., CSLR) performance improves, and supporting concurrent training of both recognition (e.g., CTC loss for glosses) and translation (e.g., Transformer-based decoding) (Zheng et al., 2022).

7. Datasets, Metrics, and Benchmarks

Benchmark datasets are fundamental to SLT research:

PHOENIX-Weather 2014T comprises German sign videos, glosses, and parallel text, and is the most widely used benchmark for SLT, supporting evaluations with metrics such as BLEU-n, ROUGE-L, and Word Error Rate (WER) (Yin et al., 2020).
ASLG-PC12, a synthetic ASL-English dataset, is commonly employed for gloss-to-text.
CSL-Daily, How2Sign, FLEURS-ASL#0, and LSA-T add diversity with different sign/spoken language pairs, domains (controlled vs. real-world), and levels of annotation complexity (Bianco et al., 2022, Zhang et al., 16 Jul 2024).
Ablation studies, human evaluations, and alternative metrics (such as WER-N for rare tokens, diversity indices for diffusion models) are widely adopted.

Performance on these benchmarks is principally measured using BLEU-4 for n-gram precision, ROUGE-L for sequence overlap, and BERTScore for semantic fidelity. Notably, even incremental BLEU gains (typically 1–2 points) are viewed as significant due to the complexity of the task.

8. Current Limitations and Prospects

Despite progress, SLT faces several open challenges:

Modality gap: Effective mapping from high-dimensional, temporally redundant visual signals to discrete language sequences (with very different word order, density, and syntax) remains a central challenge, worsened by data scarcity and the lack of large annotated corpora.
Alignment: Non-monotonic and variable-length alignments between sign video and text complicate model supervision and evaluation, motivating continued research into hierarchical and flexible alignment approaches.
Diversity, ambiguity, and robustness: Generative diffusion models aspire to capture the lexical and syntactic diversity inherent in real sign language expression, and serve to address ambiguity missing in deterministic models (Moon et al., 26 Nov 2024).
Scaling and transferability: Universal models for open-domain, multilingual, and cross-lingual SLT are in early stages, with multitask pretraining and token-level identification emerging as promising strategies (Tan et al., 30 May 2025, Zhang et al., 16 Jul 2024).

Progress in gloss-free and end-to-end methods, integration of visual–text representations, and large backbone models has established SLT as a rapidly evolving field at the intersection of vision and language. Future directions are expected to further explore semi-supervised learning, richer multimodal representations, more effective scaling and adaptation, and practical deployment for communication accessibility.