SiLVERScore: Multimodal Sign Language Metric

• SiLVERScore is a semantically-aware metric that directly assesses sign language generation by aligning sign videos with their textual references in a shared embedding space.
• It utilizes a joint video–text framework with contrastive learning and transformer architectures to capture both global semantics and local signing nuances such as facial expressions and prosody.
• Empirical results on PHOENIX-14T and CSL-Daily demonstrate near-perfect discrimination with a ROC AUC of 0.99, underscoring its robustness to semantic variations and expressive differences.

Searching arXiv for the SiLVERScore paper and closely related retrieval/embedding work to ground the article. SiLVERScore is a semantically-aware embedding-based evaluation metric for sign language generation that compares a generated sign video and its paired text in a shared, multimodal embedding space, thereby avoiding the standard back-translation pipeline in which generated signs are first recognized back to text and then scored with text-based metrics such as BLEU or ROUGE (Imai et al., 4 Sep 2025). It was introduced to address three recurrent problems in sign language generation evaluation: ambiguity in error origin, multimodal loss, and insensitivity to meaning shifts. The method is built on a joint video–text embedding framework derived from CiCo, and the reported experiments on PHOENIX-14T and CSL-Daily show near-perfect discrimination between correct and random pairs, with ROC AUC $= 0.99$ and overlap below $7\%$ on PHOENIX-14T and only slightly above $7\%$ on CSL-Daily (Imai et al., 4 Sep 2025).

1. Motivation and problem setting

Evaluation in sign language generation is often performed through back-translation: generated signs are converted back to text and then compared to a reference using text-based metrics such as BLEU and ROUGE (Imai et al., 4 Sep 2025). In the SiLVERScore formulation, this dominant pipeline is treated as structurally inadequate for multimodal assessment because it introduces ambiguity about whether an observed evaluation error should be attributed to the sign generation model or to the translation system used as the evaluator. The same pipeline also discards sign-specific information, including facial expressions, spatial grammar, and prosody, because the final comparison is conducted over text rather than over signing behavior itself (Imai et al., 4 Sep 2025).

A central critique is that text-surface agreement can fail to reflect semantic fidelity. The paper gives the example of a meaning reversal—such as swapping “John gave Mary a book” with “Mary gave John a book”—that can still receive a perfect back-translation score if the textual output happens to match by accident (Imai et al., 4 Sep 2025). This establishes the conceptual basis for a direct video–text metric: the evaluation target is not n-back textual reconstruction, but semantic alignment between the signed realization and its intended linguistic content.

Within this framing, SiLVERScore is positioned as a direct multimodal comparison metric. It operates on the generated sign video and its reference text in a shared embedding space, eliminating the additional recognition step and its associated error propagation (Imai et al., 4 Sep 2025). A plausible implication is that the metric is best understood not merely as an alternative scorer, but as a redefinition of what constitutes evidence of correctness in sign language generation: semantic and prosodic compatibility in a joint representation rather than lexical overlap after transcription.

2. Joint embedding space and model construction

The semantically-aware joint embedding space underlying SiLVERScore is built on CiCo, described in the source material as “Domain-Aware Cross-Lingual Contrastive Learning” (Imai et al., 4 Sep 2025). In this setup, video and text encoders are trained jointly in a shared space using contrastive learning with InfoNCE loss. For each video–text pair, the encoders produce vectors, and the optimization objective encourages matched pairs to lie close together while pushing mismatched pairs farther apart (Imai et al., 4 Sep 2025).

The sign video encoder uses a sliding window mechanism to extract local video segments. These segments are represented through fused features from two I3D streams: a domain-agnostic pretrained I3D network trained on large generic datasets, and a domain-aware I3D network fine-tuned on sign datasets such as PHOENIX-14T and CSL-Daily (Imai et al., 4 Sep 2025). The resulting embeddings are then processed through a 12-layer Transformer and a ViT-B encoder from CLIP (Imai et al., 4 Sep 2025). The text side is lowercased, byte-pair-encoded, and translated to English so that the text representations align with CLIP’s pretraining regime (Imai et al., 4 Sep 2025).

The learned alignment is both global and local. Global alignment refers to sentence/video-level compatibility, while local alignment refers to segment/token-level compatibility (Imai et al., 4 Sep 2025). This duality is important because sign language semantics are distributed across spatiotemporal patterns rather than being reducible to isolated glosses. The paper further states that facial expressions and prosody are encoded via the visual features, that spatial grammar is captured through the spatiotemporal arrangement of the signing signal, and that sentence-level and local prosodic or expressive features are represented in the embedding (Imai et al., 4 Sep 2025).

This design also explains the metric’s stated robustness to semantic/pragmatic equivalence: because the embeddings favor semantic content rather than strict surface form, they are reported to be robust to word order and other surface-level changes (Imai et al., 4 Sep 2025). This suggests that the embedding space is intended to model equivalence classes of signed realizations that preserve meaning even when local formal properties differ.

3. Score definition and mathematical formulation

SiLVERScore is computed from fine-grained similarities between segmented video clips and text tokens. For a video segmented into $M$ clips and a text segmented into $L$ tokens, the method defines a similarity matrix

$E(i, j) = S_i \cdot W_j^T$

where $S_i$ is the embedding for video clip $i$ and $W_j$ is the embedding for word or token $j$ (Imai et al., 4 Sep 2025).

The raw similarity matrix is then re-weighted with a Softmax operation to emphasize salient similarities:

$7\%$0

This produces a fine-grained similarity structure in which stronger correspondences contribute more heavily to the final score (Imai et al., 4 Sep 2025).

Aggregation proceeds in two directional variants. For video-to-text similarity, the rows are summed so that each video window aggregates its similarity to all tokens, and the resulting values are averaged to obtain $7\%$1. For text-to-video similarity, the columns are summed so that each token aggregates its similarity to all windows, and those values are averaged to obtain $7\%$2 (Imai et al., 4 Sep 2025). Because the loss is symmetric, either direction can be used in practice; the reported SiLVERScore uses $7\%$3 (Imai et al., 4 Sep 2025).

For interpretability, the output similarity is multiplied by $7\%$4 and scaled to the $7\%$5 range (Imai et al., 4 Sep 2025). The source material explicitly states that this was done to match BLEU/ROUGE scales. The scoring procedure can therefore be summarized as a local-to-global semantic similarity computation over a shared video–text embedding geometry, followed by scalar calibration for human-readable comparison with established text metrics.

4. Evaluation protocol and reported empirical results

The principal empirical evaluation studies the ability of SiLVERScore to discriminate aligned video–text pairs from random, misaligned pairs on two datasets: PHOENIX-14T, described as German Sign Language weather forecasts with parallel video and text/gloss, and CSL-Daily, described as Chinese Sign Language daily-life content (Imai et al., 4 Sep 2025). Two discrimination metrics are reported. The first is overlap percentage, defined as the area where the score distributions of correct and random pairs intersect, with lower values indicating better separation. The second is ROC AUC, defined as the probability that a randomly chosen correct pair scores higher than a random misaligned pair, with higher values indicating better discrimination and $7\%$6 as the maximum (Imai et al., 4 Sep 2025).

The reported quantitative results place SiLVERScore well ahead of the other listed metrics in overall separation quality on PHOENIX-14T, and at the top tier on CSL-Daily (Imai et al., 4 Sep 2025).

Metric	PHOENIX-14T / CSL-Daily overlap	PHOENIX-14T / CSL-Daily ROC-AUC
BLEU-1	19.78% / 6.04%	0.95 / 0.99
BLEU-2	24.30% / 5.27%	0.90 / 0.98
BLEU-3	38.63% / 23.89%	0.81 / 0.88
ROUGE	19.94% / 6.12%	0.95 / 0.99
BERTScore	14.17% / 9.27%	0.97 / 0.98
BLEURT	21.65% / 11.90%	0.95 / 0.98
SiLVERScore	6.85% / 7.40%	0.99 / 0.99

On PHOENIX-14T, SiLVERScore attains overlap $7\%$7 and ROC AUC $7\%$8; on CSL-Daily, it attains overlap $7\%$9 and ROC AUC $7\%$0 (Imai et al., 4 Sep 2025). The paper also reports kernel density plots in which the score distributions for correct and random pairs are “cleanly, sharply separated” under SiLVERScore (Imai et al., 4 Sep 2025). The abstract characterizes this performance as “near-perfect discrimination between correct and random pairs” and emphasizes that the overlap is below $7\%$1, with traditional metrics substantially worse overall (Imai et al., 4 Sep 2025).

These results are significant because they target a property more fundamental than correlation with back-translation output: whether a metric can reliably separate semantically aligned multimodal pairs from arbitrary mismatches. In that sense, the reported gains indicate improved discriminative validity rather than only improved agreement with prior text-based evaluators.

5. Robustness to semantic and prosodic variation

The paper includes two targeted robustness analyses: semantic variation under word reordering and prosodic variation under human-annotated intensity levels (Imai et al., 4 Sep 2025). In the semantic-variation experiment, sentence words are reordered using GPT-4o so that meaning is preserved while order changes. Under this manipulation, rule-based metrics such as BLEU and ROUGE show sharp score drops, indicating high sensitivity to word order. SiLVERScore, together with BERTScore and BLEURT, remains stable and high, with high overlap in score between original and reordered cases and low AUC, which the source interprets as robustness to rephrasing and paraphrasing (Imai et al., 4 Sep 2025).

This behavior is consistent with the metric’s embedding-based design. Because the learned representation is intended to favor semantic content over surface order, meaning-preserving reordering does not substantially alter the score. The result also directly supports the paper’s claim that SiLVERScore is less sensitive to surface variants that do not change meaning (Imai et al., 4 Sep 2025).

The prosody experiment is conducted on PHOENIX-14T using human-annotated prosody levels with token-level and sentence-level intensity annotations. Sentences are categorized into No, Low, and High intensity prosody (Imai et al., 4 Sep 2025). BLEU and ROUGE scores decrease as prosody intensity increases, which the paper interprets as evidence that back-translation and word-overlap methods struggle with expressive signs. By contrast, SiLVERScore remains stable and shows no significant correlation with prosody intensity, with reported correlation $7\%$2 and $7\%$3 (Imai et al., 4 Sep 2025).

The accompanying interpretation is that prosody and facial expressions are encoded in the embeddings, allowing the metric to evaluate intended meaning without penalizing visual or expressive richness (Imai et al., 4 Sep 2025). It is important, however, that the paper also notes a limitation: SiLVERScore is not explicitly modeling prosody as a separate feature. Its robustness arises from the properties of the learned representation rather than from a dedicated prosodic submodule (Imai et al., 4 Sep 2025). This distinguishes robustness to prosodic variation from explicit prosodic analysis.

6. Generalization, domain adaptation, and limitations

A major limitation identified in the work concerns cross-dataset generalization. The paper states that embedding-based models such as SignCLIP and CiCo trained on one dataset do not generalize well to others out of the box: text-to-video retrieval performance drops sharply when moving between datasets, and additional data, augmentation, or specialized contrastive objectives such as SignCL only marginally improve this behavior (Imai et al., 4 Sep 2025). The problem is attributed to data scarcity and domain mismatch across sign language datasets (Imai et al., 4 Sep 2025).

SiLVERScore addresses this issue pragmatically through domain-adaptive training. Each evaluation is tailored by fine-tuning or using models adapted to the target dataset, including PHOENIX-14T and CSL-Daily (Imai et al., 4 Sep 2025). This improves semantic alignment within the target domain, but it limits plug-and-play generalization. The paper therefore presents the metric not as a universal cross-domain evaluator, but as a high-fidelity evaluation method contingent on appropriate domain adaptation (Imai et al., 4 Sep 2025).

This domain dependence is one of the central trade-offs in the framework. On the one hand, it enables strong performance in highly variable sign language domains; on the other, it leaves open the problem of general-purpose evaluation across sign languages and datasets (Imai et al., 4 Sep 2025). A plausible implication is that future progress in sign language generation evaluation may depend not only on better scoring functions, but also on broader multilingual and cross-domain representation learning for signing video. The source material stops short of claiming a solution to that broader problem.

7. Position within sign language generation evaluation

SiLVERScore is presented as an alternative to back-translation-based evaluation rather than as a minor refinement of text metrics (Imai et al., 4 Sep 2025). Its principal advantages, as stated in the source material, are direct multimodal comparison, the ability to capture linguistically relevant features such as facial expressions, spatial action, and prosody, reduced sensitivity to surface variants, direct penalization of signing errors without translation-model noise, and superior discrimination between aligned and unaligned pairs (Imai et al., 4 Sep 2025).

Methodologically, the work also sits within a broader family of contrastive video–text embedding approaches for sign language, specifically those derived from or compared against CiCo and SignCLIP (Imai et al., 4 Sep 2025). In that sense, SiLVERScore is both an evaluation metric and an application of multimodal retrieval-style representation learning to the evaluation problem. Rather than generating text and scoring strings, it evaluates whether video and text occupy a consistent region of a semantically structured joint space.

A common misconception would be to equate its robustness to prosodic and surface variation with explicit linguistic modeling of all nonmanual and discourse-level phenomena. The paper does not make that claim. It reports robustness to semantic and prosodic variations and attributes this to the embedding space, while explicitly noting that prosody is not modeled as a separate feature (Imai et al., 4 Sep 2025). Another misconception would be to interpret the reported high ROC AUC as evidence of universal transfer across sign languages or domains; the paper explicitly reports the opposite, namely that generalization remains challenging and currently requires dataset/domain-adaptive fine-tuning (Imai et al., 4 Sep 2025).

Taken together, SiLVERScore marks a shift from text-reconstruction evaluation toward semantically-aware multimodal evaluation in sign language generation. Its contribution lies in formalizing a score derived from joint video–text embeddings, demonstrating strong discrimination on PHOENIX-14T and CSL-Daily, and showing empirical robustness to meaning-preserving reordering and expressive variation while acknowledging unresolved cross-domain generalization challenges (Imai et al., 4 Sep 2025).