SinLlama: Hierarchical Sign Language Translation Model

• SinLlama is a modeling strategy for sign language translation that integrates hierarchical visual encoding with prompt-based LLM integration.
• It decomposes sign language videos via a frozen DINOv2 ViT backbone and multi-level encoding with local attention and contrastive losses.
• The approach achieves near-parity with gloss-based methods by leveraging modular visual-language connectors and extensive pretraining on diverse datasets.

SinLlama refers to a modeling strategy for sign language translation that integrates hierarchical visual encoding with LLMs through a modular, prompt-based architecture. This technical paradigm is exemplified by the LLaVA-SLT framework, which forgoes costly gloss annotation and achieves near-parity with gloss-based methods on competitive sign language translation (SLT) benchmarks via a composite recipe of multimodal pretraining, hierarchical attention, and minimal trainable bridging (Liang et al., 2024).

1. Model Architecture

SinLlama employs a modular, three-block architecture that maps sign language videos to spoken language texts:

1. Hierarchical Visual Encoder: Videos are spatially and temporally decomposed using a frozen DINOv2 ViT-B/14-distilled backbone (patch size 14, embedding dimension 768, with LoRA finetuning at rank γ=8, α=16 for domain adaptation).
   • Frame-level: Extracts [cls] and patch-level features per frame.
   • Word-level: Local-attention Transformer (depth=4, with RoFormer rotary positional embeddings) aggregates patch embeddings over windows corresponding to putative sign "words." Outputs are downsampled (nearest-neighbor, step=4), resulting in M word-level tokens, each in ℝ^768. Inner contrastive loss (SignCL) is applied to regularize and decorrelate these embeddings.
   • Sentence-level: An 8-layer Transformer encodes word-level tokens to produce a single [cls] sentence embedding.
2. Vision–Language Connector: A two-layer MLP maps each word-level embedding (ℝ^768) to the LLM’s token embedding space (e.g., ℝ^768 or ℝ^1024), using GELU activations and LayerNorm after each layer. Only this connector is trainable during visual language tuning.
3. LLM Backbone and Visual Injection: The base models are decoder-only Transformers (e.g., Qwen-2.5-Instruct at 3B, 7B, or 14B parameters). Transformed visual embeddings (ĕ_v) replace <video_placeholder> positions in the prompt, concatenated with standard text token embeddings, enabling downstream translation within the LLM's autoregressive framework.

2. Training Procedure and Objectives

SinLlama-style training proceeds through three sequential phases:

• I. Linguistic Continued Pretraining (CPT): The LLM is adapted to the sign language domain using datasets such as gloss–text pairs, sign language books, and web pages (~737K tokens in total, across varied vocabularies). The objective is next-token autoregressive prediction,

$$L_{AR}(\theta) = -\sum_{i=1}^{|c|} \log P(c_i \mid c_{i-k,\ldots,i-1}; \theta)$$

Data order is permuted to encourage flexibility in mapping sign grammar and natural language. Hyperparameters include batch size 64, AdamW optimizer, peak LR $2 \times 10^{-4}$, LoRA adaptation, and ZeRO offloading on 8×A100 GPUs.

• II. Visual Contrastive Pretraining: The visual encoder is aligned with a frozen mBART text encoder using 264K unannotated sign language videos (~401 h) with captions. Key objectives are:
  • Outer CLIP-style contrastive loss ($L_{CLIP}$): matches video [cls] and text [cls] representations.
  • Inner SignCL loss: decorrelates M word-level embeddings within each clip.
  • $$L_{CL} = L_{CLIP} + \lambda L_{SignCL},\ \lambda=10^{-2}$$
  • Hyperparameters: 200 epochs, batch 128, AdamW (weight decay $10^{-5}$), image resolution 224² or 336².
• III. Visual Language Tuning (VLT): Only the MLP connector is trainable; both the visual encoder and LLM are frozen. Visual features corresponding to the user's <video_placeholder> are injected at the prompt level. The end-task objective is cross-entropy loss over the reference spoken-language translation:

$$L_{CE} = -\sum_{t} \log P(y_t \mid y_{<t}, [\hat{e}_v; \text{text prompt}])$$

Optional: fine-tune LoRA adapters within visual encoder and LLM at low LR ($10^{-5}$) for 1-2 epochs.

3. Data Regimes and Preprocessing

SinLlama’s reference implementation employs:

• CSL-Daily: 18,401 training, 1,077 validation, 1,176 test samples (Chinese Sign Language, gloss-annotated).
• Phoenix-2014T: 7,096 train, 519 val, 642 test (German sign language).
• CSL-400h: 264,461 unannotated videos, 401 hours, 50,785 captions.

Processing involves uniform frame sampling per clip (e.g., T=16), spatial resizing/cropping (224² or 336²), and consistent data augmentations across frames (color jitter, horizontal flips, small rotation).

4. Evaluation Protocols and Results

Performance is reported on canonical SLT benchmarks via BLEU4 and ROUGE:

Dataset	Method	BLEU4	ROUGE
CSL-Daily	Gloss-based SOTA	25.79	55.72
	GFSLT-VLP-SignCL	16.16	48.92
	LLaVA-SLT (no extra)	20.42	51.26
	LLaVA-SLT (+CSL-400h)	25.23	56.21
Phoenix-2014T	Gloss-based SOTA	28.95	—
	GFSLT-VLP-SignCL	22.74	49.04
	LLaVA-SLT	23.43	50.44

LLaVA-SLT approaches or surpasses gloss-based SOTA when leveraging large-scale annotation-free corpora (e.g., CSL-400h), and consistently exceeds previous gloss-free baselines.

5. Ablation Analyses

Key architectural and procedural components are systematically ablated:

• Removing local attention in the word-level encoder or using smaller ViT backbones notably decreases BLEU4 (e.g., 14.87 without local-attn).
• Reducing the resolution or omitting additional pretraining data yields significant performance drops.
• Alternate connector designs (linear vs. 2-layer MLP), prompt structure removals, and absence of full-tuning (unfreezing LoRA) are all suboptimal relative to the full recipe.

Scaling the LLM backbone from 3B to 14B parameters further improves performance in the CPT phase (BLEU4: 16.69 → 20.92).

6. Implementation and Computational Considerations

Total training involves ≈2000 A100 GPU-hours distributed over:

• CPT: ~12 hours (1 epoch, 8×A100)
• Visual contrastive pretraining: 2–3 days (200 epochs)
• Visual language tuning: 4–6 hours (5–10 epochs)

Augmentation is meticulously configured and replicated across frames to preserve temporal consistency.

7. Key Insights, Limitations, and Future Directions

Salient findings include:

• Hierarchical visual encoding at word granularity is crucial for efficient transfer of sign language video to LLM-acceptable embeddings.
• Contrastive pretraining tightly aligns video and text, reducing representation density and improving downstream SLT.
• Lightweight, MLP-based connectors suffice for effective visual-to-LLM bridging, minimizing trainable parameter count and preventing catastrophic forgetfulness in the frozen LLM.

Limitations concern the absence of multilingual or non-CSL benchmarks, lack of explicit modeling for sign language “gloss” structure in visual pretraining, and modest transparency in hyperparameters for full replicability.

Proposed future work targets coverage of less-represented sign languages, scaling with additional unannotated datasets, advancing token injection strategies, and extended multimodal tasks such as captioning or cross-lingual transfer (Liang et al., 2024).