SigLIP-VQ: Gloss-Free Sign Language Synthesis

• The paper presents a novel gloss-free sign language production model that bypasses intermediate gloss annotations using discrete latent variables.
• It employs a spatio-temporal VQ-VAE and an autoregressive Transformer to encode pose segments and predict sign tokens from text.
• Empirical results on benchmarks like PHOENIX14T and How2Sign show improved BLEU-4 and Fréchet Gesture Distance, validating its performance.

SigLIP-VQ, referred to as “SignVQNet” in foundational work, is a state-of-the-art model for gloss-free sign language production (SLP). It translates spoken or written sentences directly into continuous 3D skeleton-based sign pose sequences, entirely bypassing intermediate gloss annotations. At its core, the approach introduces discrete latent variables via vector quantization, utilizes a spatio-temporal graph-based variational autoencoder for pose encoding and decoding, and predicts these discrete codes autoregressively from text using a Transformer architecture. This combination enables accurate linguistic-to-gesture synthesis and supports advanced search strategies in decoding, with state-of-the-art results on major SLP benchmarks (Hwang et al., 2023).

1. Problem Formulation and Discrete Representation

SigLIP-VQ addresses gloss-free SLP, where the objective is to map an input sequence $x = \{x_u\}_{u=1}^U$ (text or speech) to a continuous sequence $y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$, with $V$ joints and $C$ feature dimensions. The model introduces an intermediate sequence of discrete representations $z = \{z_i\}_{i=1}^M$, corresponding to short sign segments. The probabilistic formulation is:

$p(y\mid x) = \sum_{z} q(y\mid z) p(z \mid x),$

where $p(z \mid x)$ is modeled autoregressively, and $q(y \mid z)$ reconstructs poses from discrete tokens.

Significance

This architecture enables decoupling of the mapping from language to gesture into a tractable sequence prediction task, leveraging the benefits of discrete representations for both modeling efficiency and inference using standard NLP techniques such as beam search.

2. Spatio-Temporal VQ-VAE Architecture

A spatio-temporal vector-quantized variational autoencoder (VQ-VAE) forms the backbone for learning meaningful discrete codes from sign pose segments.

• Encoder (STGP-dVAE): Operates on fixed-length pose segments $y^{(\ell)} \in \mathbb{R}^{L \times V \times C}$ (with $L$ typically 32), using stacked Spatio-Temporal Graph Pyramid (STGP) blocks. Each block combines spatial graph convolutions across the skeletal structure, temporal convolutions for sequence modeling, and residual connections to encode multi-scale motion features.
• Quantizer: Utilizes a Gumbel-Softmax bottleneck with a codebook $y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$0 ($y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$1 learnable embeddings). The encoder output is quantized via soft one-hot assignments:

$y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$2

The quantized latent is $y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$3.

• Decoder (STGP-dVAE): Mirrors the encoder with upsampling, reconstructing full 3D segment poses $y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$4 from quantized vectors.

Loss Functions

• Reconstruction (Pose) Loss:

$y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$5

• Diversity Loss: Encourages uniform code usage.

$y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$6

• Total VQ-VAE Loss:

$y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$7

with $y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$8.

After VQ-VAE training, the encoder, quantizer, and decoder are frozen for subsequent stages.

3. Autoregressive Text-to-Token Model

The mapping from text to sign tokens is handled by an autoregressive Transformer that predicts discrete segment indices from language.

• Tokenization: The full sequence $y = \{y_t\}_{t=1}^T \in \mathbb{R}^{V \times C}$9 is segmented and each chunk encoded as a token index $V$0, with special $V$1 / $V$2 tokens appended.
• Text Encoder: Sentence tokenization via byte-pair encoding (BPE) with 3k/10k vocabularies, word embedding, and optional BERT initialization.
• Transformer Decoder: 4 layers, hidden size 768, 8 heads. Predicts each $V$3 autoregressively conditioning on prior tokens and text context.
• Losses:
  • Cross-Entropy:

  $V$4 - Latent Alignment:

  $V$5 - Total AR Loss:

  $V$6

  with $V$7.

Latent alignment enforces correspondence between learned decoder states and pose encoder features, aiding coherence.

4. Inference and Decoding Procedures

• Autoregressive Decoding: Inference begins with the text input; the decoder is initialized and generates token sequences using beam search. At each step, the top-$V$8 partial hypotheses are retained. Decoding terminates at the $V$9 token or after reaching the maximum token length.

• De-quantization: The final sequence of tokens $C$0 is passed through the frozen STGP decoder, segment by segment, to reconstruct continuous 3D skeleton poses, which are concatenated to yield the complete synthesized sign sequence.

Significance

True autoregression ensures that at inference, only the input text is required, with no ground-truth pose supervision.

5. Evaluation Protocols

SigLIP-VQ employs rigorous evaluation metrics targeting both linguistic fidelity and motion realism:

Metric	Description	Reference use
Back-Translation (BT)	BLEU-4 between a sign-to-text model’s output (given $C$1) and original text	SLP fidelity
Fréchet Gesture Distance	FGD between distributions of layer-wise pose features from real vs. generated	Distributional match [Yoon et al. 2020]
DTW-MJE	Dynamic Time Warping + Mean Joint Error; shown as less reliable	For comparison

Evaluation on PHOENIX14T and How2Sign benchmarks demonstrated superior performance in both FGD and BLEU-4 scores relative to Progressive Transformers (PT) and NSLP-G, e.g., PHOENIX14T test FGD: 92.64 for SignVQNet vs. 150.28 (NSLP-G) and 360.62 (PT); BLEU-4: 6.85% vs. 5.56%.

6. Empirical Results and Ablation Studies

Benchmarked on German (PHOENIX14T) and American (How2Sign) sign languages, SigLIP-VQ establishes state-of-the-art results:

• Ablation Findings:
  • Optimal window size $C$2 frames.
  • Codebook size $C$3 achieves a balance between representational power and overfitting.
  • Combining cross-entropy and latent alignment losses yields better results than either alone.
  • Beam search decoding consistently improves FGD and BLEU-4.

7. Model Properties and Strengths

• True Autoregressive Generation: Eliminates the requirement for ground-truth poses during inference.
• Discrete Token Space: Enables integration of NLP-standard decoding (e.g., beam search), enhancing flexibility.
• Latent-Level Alignment: Couples linguistic and gestural representations, particularly when leveraging strong text encoders such as BERT.
• Benchmark Performance: Validated as the leading approach on robust SLP metrics, emphasizing accuracy and the naturalness of synthesized sign sequences.

In summary, SigLIP-VQ (SignVQNet) advances gloss-free SLP by introducing a spatio-temporal discrete bottleneck via vector quantization, combining this with an autoregressive Transformer for text-to-token prediction and a highly expressive frozen pose decoder. The architecture’s discrete representations, informed by both LLM features and detailed pose encoding, deliver robust and faithful sign language synthesis (Hwang et al., 2023).