VTaMo: Video-Text Alignment Model for Sign Language Translation

Abstract: Sign language translation (SLT) converts continuous sign videos into spoken language text. Gloss-free approaches leverage pre-trained visual encoders and LLMs but rely on implicit cross-modal alignment from translation supervision alone. We present VTaMo, a framework that introduces explicit multi-granularity alignment at three levels: (1) local alignment via entropy-regularized optimal transport with a learnable null token for fine-grained frame-to-token correspondences; (2) global alignment via a learnable orthogonal transformation that calibrates embedding space geometry through Earth Mover's Distance; and (3) position-aligned contrastive learning for discriminative token-level representations. Experiments on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL demonstrate consistent state-of-the-art performance, with ablations confirming the complementary contributions of each component. Code is available at https://github.com/junyi2005/vtamo.

• The paper introduces VTaMo, which combines local Sinkhorn optimal transport with a null token, global orthogonal alignment, and position-aware contrastive learning to explicitly align sign videos with pseudo-gloss tokens before decoding.
• VTaMo achieves state-of-the-art gloss-free results across Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL, including 28.86 BLEU-4 on Phoenix-2014T and gains of up to 8.36 BLEU-4 over SpaMo.
• Ablations show that contrastive, local, and global alignment objectives are complementary, while robustness tests indicate that performance changes by no more than 0.42 BLEU-4 when backgrounds or signer appearance are altered.

VTaMo addresses a central weakness of gloss-free sign language translation (SLT): prior systems decode text directly from visual features using pre-trained LLMs, but leave cross-modal alignment implicit, relying on decoder attention to simultaneously learn translation and discover a latent correspondence between signing order and target word order. The paper argues that this is ill-posed because sign languages frequently do not follow spoken-language word order, and because many frames carry no lexical content. VTaMo makes alignment explicit at three granularities and reorders visual features before decoding, achieving state-of-the-art gloss-free results on four benchmarks.

Problem formulation

Given a sign video, frame-level features are extracted with a frozen CLIP-ViT-Large backbone (with S$^2$-Wrapper multi-scale extraction), temporally downsampled $4\times$ by an attention-based temporal encoder, and projected into the Flan-T5-XL embedding space. A key design choice is that both alignment targets and decoder outputs are a pseudo-gloss rather than the raw sentence: a frozen spaCy POS tagger retains content tokens (NOUN, VERB, ADJ, ADV, NUM, PRON, PROPN) and drops function words (DET, ADP, AUX, PART, PUNCT, CCONJ). This makes strict frame-to-token alignment well-posed. At inference, a lightweight text-only recovery model—trained purely on permuted pseudo-gloss sentences reconstructed into fluent text—restores word order and re-inserts function words; it never sees sign video.

Method: three alignment objectives

Local alignment via optimal transport. Entropy-regularized OT (Sinkhorn, log-domain, 10 iterations) aligns temporal visual segments to pseudo-gloss token embeddings from the frozen LM embedding layer. A single learnable null token, prepended to the token sequence with a learnable bias $b_\varnothing$, absorbs transitional gestures and co-articulation so that non-lexical frames are never forced onto content tokens. The local loss combines transport cost, a total-variation regularizer penalizing adjacent-frame token switching, and a null-cost penalty that keeps mean null affinity below a target ratio $\rho_\text{target}=0.2$. A multi-phase $\varepsilon$ annealing schedule (0.12 → 0.10 → 0.03) moves plans from soft exploration to sharp assignments.

Global alignment. Sentence-level embeddings from attention pooling are related by a learnable orthogonal transformation $\mathbf{T}$ (initialized near identity, penalized by $\|\mathbf{T}^\top\mathbf{T}-\mathbf{I}\|_F^2$), which corrects orientation mismatch between modality-specific embedding spaces without distorting cosine geometry. An Earth Mover's Distance loss over a 256-entry FIFO memory queue diversifies sentence pairs beyond small batches; activation is scheduled after local alignment stabilizes.

Position-aligned contrastive learning. After window-based reordering of visual features into target token order, an InfoNCE objective binds each reordered visual representation to its corresponding token embedding through projection heads, with gradients blocked on the text branch to preserve LM representations.

Window-based reordering partitions frames into $U+2$ overlapping windows, assigns each window to its highest-mass real token, and concatenates segments in token order while preserving within-window temporal continuity. Reordering is training-only; at inference the decoder reads features in signing order and emits pseudo-gloss in visual order, relying on the contrastive grounding for per-token lexical correctness. Training proceeds in two phases: alignment-only with frozen LM, then joint optimization with LoRA-adapted Flan-T5-XL ($r=16$, $\alpha=32$).

Results

On Phoenix-2014T, VTaMo reaches 28.86 BLEU-4 / 60.24 ROUGE-L, surpassing the strongest gloss-free baseline SpaMo by 4.54 BLEU-4 and coming within 0.09 of the best gloss-based result (TS-SLT, 28.95)—notable given no gloss supervision and no visual fine-tuning. On CSL-Daily it attains 27.16 BLEU-4 (+6.61 over SpaMo, +0.80 over Uni-Sign despite Uni-Sign's pose+RGB input and large-scale pretraining).

Benchmark	BLEU-1	BLEU-4	ROUGE-L	BLEURT
How2Sign	46.67	18.47	37.14	50.26
OpenASL	53.58	25.94	46.85	62.48

On How2Sign, gains over SpaMo reach +8.36 BLEU-4, and VTaMo exceeds SHuBERT and Uni-Sign on BLEU-4 and BLEURT while ranking second on ROUGE-L behind SSVP-SLT. On OpenASL (>10k vocabulary), it outperforms Uni-Sign and SHuBERT by 2.80 and 2.74 BLEU-4 respectively. The pattern of larger gains on larger-vocabulary datasets supports the claim that explicit alignment helps more as task complexity increases.

Ablations on Phoenix-2014T show all three losses are necessary and non-substitutable: removing the contrastive loss costs −6.30 BLEU-4, the local OT loss −5.09, and the global EMD loss −2.65. A backbone ablation following SpaMo's protocol shows consistent gains of +2.49 to +4.54 BLEU-4 across five LMs (Flan-T5-large, mBART-50, mT0-XL, Llama-2, Flan-T5-XL), isolating the improvement to the alignment block rather than backbone choice; notably, the 7B Llama-2 underperforms 3B Flan-T5-XL in this limited-data regime. Transport-plan metrics confirm the full model produces the sharpest alignments (peak assignment 0.76, entropy 0.18, change rate 0.12). The memory queue is the largest contributor to global-alignment convergence speed (peak epoch 36 vs. 68 without it). Supplementary comparisons show generic clip-level video–text objectives (CLIP4Clip, VideoCLIP, X-CLIP) reach only 9.47–11.13 BLEU-4 in the same pipeline versus 28.86, indicating that near-monotonic clip-to-caption alignment cannot substitute for token-level, partial, non-monotonic matching.

A noise-sensitivity study on the full How2Sign test set, using Stable Diffusion + ControlNet pose-conditioned synthesis to replace backgrounds or signer appearance while preserving kinematics, shows degradation of at most 0.42 BLEU-4 across four conditions, suggesting the model attends to gesture semantics rather than scene context or signer identity. Qualitative transport plans exhibit block-diagonal structure with largely monotonic ordering and local reorderings where signing diverges from target word order.

Limitations and open questions

The evaluation targets offline accuracy only; real-time or on-device deployment is not addressed. The recovery of spoken word order at inference depends entirely on a text-only model trained on synthetic permutations of pseudo-gloss, so translation quality inherits any distribution mismatch between synthetic shuffles and genuine signing-order output—an assumption the paper does not empirically stress-test. The pseudo-gloss filter itself is a fixed rule-based POS heuristic applied uniformly across languages, and its sensitivity to tagger errors or language-specific morphology is left unexamined. Whether the learned transport plans generalize to sign languages with substantially different grammatical structures than German, Chinese, and English SLs remains open.

Conclusion

VTaMo demonstrates that explicit multi-granularity vision–text alignment—local Sinkhorn OT with a null token, orthogonal global calibration, and position-aligned contrastive learning—yields consistent state-of-the-art gloss-free SLT across four benchmarks in three languages, at negligible solver overhead (<3.2% of per-step time), without gloss annotations, pose inputs, or large-scale sign pretraining.