---
title: 'SinLlama: Hierarchical Sign Language Translation Model'
url: https://www.emergentmind.com/topics/sinllama
type: topic
---

# SinLlama: Hierarchical Sign Language Translation Model

SinLlama refers to a modeling strategy for sign language translation that integrates hierarchical visual encoding with large language models (LLMs) through a modular, prompt-based architecture. This technical paradigm is exemplified by the LLaVA-SLT framework, which forgoes costly gloss annotation and achieves near-parity with gloss-based methods on competitive sign language translation (SLT) benchmarks via a composite recipe of multimodal pretraining, hierarchical attention, and minimal trainable bridging [2412.16524].

## 1. Model Architecture

SinLlama employs a modular, three-block architecture that maps sign language videos to spoken language texts:

1. **Hierarchical Visual Encoder**: Videos are spatially and temporally decomposed using a frozen DINOv2 ViT-B/14-distilled backbone (patch size 14, embedding dimension 768, with LoRA finetuning at rank γ=8, α=16 for domain adaptation). 
   - *Frame-level*: Extracts [cls] and patch-level features per frame.
   - *Word-level*: Local-attention Transformer (depth=4, with RoFormer rotary positional embeddings) aggregates patch embeddings over windows corresponding to putative sign "words." Outputs are downsampled (nearest-neighbor, step=4), resulting in M word-level tokens, each in ℝ^768. Inner contrastive loss (SignCL) is applied to regularize and decorrelate these embeddings.
   - *Sentence-level*: An 8-layer Transformer encodes word-level tokens to produce a single [cls] sentence embedding.

2. **Vision–Language Connector**: A two-layer MLP maps each word-level embedding (ℝ^768) to the LLM’s token embedding space (e.g., ℝ^768 or ℝ^1024), using GELU activations and LayerNorm after each layer. Only this connector is trainable during visual language tuning.

3. **LLM Backbone and Visual Injection**: The base models are decoder-only Transformers (e.g., Qwen-2.5-Instruct at 3B, 7B, or 14B parameters). Transformed visual embeddings (ĕ_v) replace <video_placeholder> positions in the prompt, concatenated with standard text token embeddings, enabling downstream translation within the LLM's autoregressive framework.

## 2. Training Procedure and Objectives

SinLlama-style training proceeds through three sequential phases:

- **I. Linguistic Continued Pretraining (CPT)**: The LLM is adapted to the sign language domain using datasets such as gloss–text pairs, sign language books, and web pages (~737K tokens in total, across varied vocabularies). The objective is next-token autoregressive prediction,
  $$
  L_{AR}(\theta) = -\sum_{i=1}^{|c|} \log P(c_i \mid c_{i-k,\ldots,i-1}; \theta)
  $$
  Data order is permuted to encourage flexibility in mapping sign grammar and natural language. Hyperparameters include batch size 64, AdamW optimizer, peak LR $2 \times 10^{-4}$, LoRA adaptation, and ZeRO offloading on 8×A100 GPUs.

- **II. Visual Contrastive Pretraining**: The visual encoder is aligned with a frozen mBART text encoder using 264K unannotated sign language videos (~401 h) with captions. Key objectives are:
   - *Outer CLIP-style contrastive loss* ($L_{CLIP}$): matches video [cls] and text [cls] representations.
   - *Inner SignCL loss*: decorrelates M word-level embeddings within each clip.
   - $$
     L_{CL} = L_{CLIP} + \lambda L_{SignCL},\ \lambda=10^{-2}
     $$
   Hyperparameters: 200 epochs, batch 128, AdamW (weight decay $10^{-5}$), image resolution 224² or 336².

- **III. Visual Language Tuning (VLT)**: Only the MLP connector is trainable; both the visual encoder and LLM are frozen. Visual features corresponding to the user's \<video_placeholder\> are injected at the prompt level. The end-task objective is cross-entropy loss over the reference spoken-language translation:
  $$
  L_{CE} = -\sum_{t} \log P(y_t \mid y_{<t}, [\hat{e}_v; \text{text prompt}])
  $$
  Optional: fine-tune LoRA adapters within visual encoder and LLM at low LR ($10^{-5}$) for 1-2 epochs.

## 3. Data Regimes and Preprocessing

SinLlama’s reference implementation employs:
- **CSL-Daily**: 18,401 training, 1,077 validation, 1,176 test samples (Chinese Sign Language, gloss-annotated).
- **Phoenix-2014T**: 7,096 train, 519 val, 642 test (German sign language).
- **CSL-400h**: 264,461 unannotated videos, 401 hours, 50,785 captions.

Processing involves uniform frame sampling per clip (e.g., T=16), spatial resizing/cropping (224² or 336²), and consistent data augmentations across frames (color jitter, horizontal flips, small rotation).

## 4. Evaluation Protocols and Results

Performance is reported on canonical SLT benchmarks via BLEU4 and ROUGE:

| Dataset         | Method                   | BLEU4 | ROUGE   |
|-----------------|-------------------------|-------|---------|
| CSL-Daily       | Gloss-based SOTA        | 25.79 | 55.72   |
|                 | GFSLT-VLP-SignCL        | 16.16 | 48.92   |
|                 | LLaVA-SLT (no extra)    | 20.42 | 51.26   |
|                 | LLaVA-SLT (+CSL-400h)   | 25.23 | 56.21   |
| Phoenix-2014T   | Gloss-based SOTA        | 28.95 | —       |
|                 | GFSLT-VLP-SignCL        | 22.74 | 49.04   |
|                 | LLaVA-SLT               | 23.43 | 50.44   |

LLaVA-SLT approaches or surpasses gloss-based SOTA when leveraging large-scale annotation-free corpora (e.g., CSL-400h), and consistently exceeds previous gloss-free baselines.

## 5. Ablation Analyses

Key architectural and procedural components are systematically ablated:

- Removing local attention in the word-level encoder or using smaller ViT backbones notably decreases BLEU4 (e.g., 14.87 without local-attn).
- Reducing the resolution or omitting additional pretraining data yields significant performance drops.
- Alternate connector designs (linear vs. 2-layer MLP), prompt structure removals, and absence of full-tuning (unfreezing LoRA) are all suboptimal relative to the full recipe.

Scaling the LLM backbone from 3B to 14B parameters further improves performance in the CPT phase (BLEU4: 16.69 → 20.92).

## 6. Implementation and Computational Considerations

Total training involves ≈2000 A100 GPU-hours distributed over:
- CPT: ~12 hours (1 epoch, 8×A100)
- Visual contrastive pretraining: 2–3 days (200 epochs)
- Visual language tuning: 4–6 hours (5–10 epochs)

Augmentation is meticulously configured and replicated across frames to preserve temporal consistency.

## 7. Key Insights, Limitations, and Future Directions

Salient findings include:
- Hierarchical visual encoding at word granularity is crucial for efficient transfer of sign language video to LLM-acceptable embeddings.
- Contrastive pretraining tightly aligns video and text, reducing representation density and improving downstream SLT.
- Lightweight, MLP-based connectors suffice for effective visual-to-LLM bridging, minimizing trainable parameter count and preventing catastrophic forgetfulness in the frozen LLM.

Limitations concern the absence of multilingual or non-CSL benchmarks, lack of explicit modeling for sign language “gloss” structure in visual pretraining, and modest transparency in hyperparameters for full replicability.

Proposed future work targets coverage of less-represented sign languages, scaling with additional unannotated datasets, advancing token injection strategies, and extended multimodal tasks such as captioning or cross-lingual transfer [2412.16524].

Source: https://www.emergentmind.com/topics/sinllama