Semantic-Aware Gloss Masking Loss

• The paper introduces SAGM Loss as a denoising objective that compels the decoder to predict masked gloss tokens using semantic inputs instead of relying solely on local context.
• SAGM Loss is implemented by randomly masking gloss tokens during training, forcing the model to leverage the encoder’s semantic representations for accurate recovery.
• Empirical results indicate that while SAGM Loss improves semantic robustness in development evaluations, it presents a trade-off by slightly reducing BLEU and ROUGE scores on held-out tests.

Semantic-Aware Gloss Masking Loss (SAGM Loss) is a denoising-style regularization objective introduced in the context of sign language production (SLP) to enhance the robustness and semantic grounding of gloss sequence generation. Applied within the hierarchical Stable Signer framework, SAGM Loss selectively masks gloss tokens in the target sequence during training, compelling the generative decoder to recover masked elements by leveraging semantic representations from the input text or prompt, rather than relying solely on local sequence context or copying behavior (Fang et al., 3 Dec 2025).

1. Mathematical Specification

Let the target gloss sequence be $\mathbf{g} = (g_{1}, ..., g_{T})$. At each time step $t$, a Bernoulli mask $m_t \sim \mathrm{Bernoulli}(\rho)$ is sampled, with default mask probability $\rho=0.15$. The masked sequence $\tilde{\mathbf{g}}$ is defined by: $$\tilde g_t = \begin{cases} [\mathrm{MASK}], & m_t = 1 \ g_t, & m_t = 0 \end{cases}$$ The Semantic-Aware Gloss Masking Loss is the sum of the cross-entropy losses over only masked positions: $$\boxed{ \mathcal{L}_{\mathrm{SAGM}} = -\sum_{t=1}^T \mathbf{1}[m_t=1] \log p_\theta \left( g_t \mid \tilde{\mathbf{g}}, H \right) }$$ where $H = f_{\theta_E}([\ell;\mathbf{x}])$ denotes encoder states derived from input text $\mathbf{x}$ and language tag $\ell$ (Fang et al., 3 Dec 2025).

2. Component Analysis and Theoretical Motivation

SAGM Loss operates solely over gloss positions randomly replaced by the special $t$0 token. The model’s decoder is tasked with predicting the true gloss $t$1 at masked positions, conditioned both on the partially masked gloss sequence $t$2 and the semantic encoder representation $t$3. By restricting the loss computation to the masked indices ($t$4), the formulation parallels denoising autoencoding schemes.

The theoretical motivation arises from the limitations of maximum-likelihood training, in which the decoder can achieve low loss by simply copying glosses from local preceding context, without leveraging the semantics encoded in $t$5. Masking enforces a reliance on underlying text or prompt representations, combating overfitting to surface patterns and promoting robust recovery from incomplete or ambiguous gloss input.

A plausible implication is an improved ability for the model to generalize to conditions where glosses are missing, noisy, or poorly aligned with the source input, as SAGM encourages semantic instead of purely lexical reconstructions.

3. Mask Construction and Training Integration

At each training iteration, every gloss sequence undergoes independent token-wise masking, with each position masked independently with probability $t$6. The $t$7 token replaces the ground truth gloss token at masked indices; all others are left unaltered. The partially masked sequence, together with the encoder output $t$8, is passed to the decoder to estimate gloss token distributions.

Loss computation aggregates the negative log-probabilities only for the masked indices, leaving non-masked positions unaffected by SAGM. SAGM is typically added to the overall training objective alongside standard cross-entropy loss on the unmasked (full) gloss target (termed SLUL loss), a Kullback-Leibler consistency penalty (KL term), and a prompt–gloss contrastive alignment term: $t$9 where default weights are $m_t \sim \mathrm{Bernoulli}(\rho)$0, $m_t \sim \mathrm{Bernoulli}(\rho)$1, and $m_t \sim \mathrm{Bernoulli}(\rho)$2.

4. Implementation and Optimization Details

Implementation involves:

• Mask rate $m_t \sim \mathrm{Bernoulli}(\rho)$3: Default 0.15, tuned within $m_t \sim \mathrm{Bernoulli}(\rho)$4.
• Batch size: 32 sequences.
• Optimizer: AdamW, with learning rate $m_t \sim \mathrm{Bernoulli}(\rho)$5 and weight decay $m_t \sim \mathrm{Bernoulli}(\rho)$6.
• The loss computation is performed in parallel per batch, sampling the mask for each gloss position.

Pseudocode for core loss computation follows: $\rho=0.15$3 Optimization is end-to-end, adjusting encoder, decoder, and any additional contrastive modules.

5. Empirical Performance and Effect

Ablation results on the How2Sign ASL dataset demonstrate the impact of integrating SAGM Loss. On the development set, incorporating SAGM into SLUL training increases BLEU-4 from 22.62 (SLUL only) to 23.24, and yields a slight decrease in ROUGE from 72.21 to 70.68 (suggesting a trade-off in lexical versus semantic evaluation metrics). On the test set, BLEU-4 decreases from 17.09 (SLUL only) to 15.67 with SAGM, highlighting that SAGM’s principal benefit is enhanced semantic robustness in development evaluation scenarios rather than improved n-gram overlap in final held-out testing (Fang et al., 3 Dec 2025).

The primary benefit of SAGM Loss is in improving semantic recall and gloss prediction accuracy in contexts where gloss sequences are incomplete, noisy, or ambiguous. In combination with KL and contrastive losses, SAGM regularizes the decoder to reference encoder semantics and discourages rote sequential copying.

6. Relationship to Related Approaches and Broader Implications

SAGM Loss implements a masking-based denoising training objective structurally analogous to masked language modeling and sequence denoising autoencoders. It is specifically tailored to the gloss-level prediction regime in sign language generation, where the interplay between surface gloss order and underlying semantic input is critical.

The introduction and adoption of SAGM Loss in Stable Signer reduces error accumulation across stages in sign language generation pipelines by strengthening grounding in the input text or prompt. A plausible implication is that such masking-based regularizers can be beneficial in other sequence transduction tasks involving partial or ambiguous labelings, or domains with high alignment noise between semantic and surface forms.

7. Summary Table: Core SAGM Loss Parameters and Effects

Setting	Default / Range	Purpose / Effect
Mask probability $m_t \sim \mathrm{Bernoulli}(\rho)$7	$m_t \sim \mathrm{Bernoulli}(\rho)$8 (tuned $m_t \sim \mathrm{Bernoulli}(\rho)$9)	Proportion of gloss tokens masked during training
$\rho=0.15$0	$\rho=0.15$1	Weight of SAGM loss in total objective
Evaluated metric (dev BLEU-4)	$\rho=0.15$2	Shows gain from SLUL-only to SLUL+SAGM

This configuration enforces a training regime where models are encouraged to predict glosses by semantic understanding rather than rote memorization, leading to systems that are resilient to gloss sequence degradation and are more robust in downstream sign-video generation (Fang et al., 3 Dec 2025).