CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment (2303.05725v4)

Abstract: Sign language recognition (SLR) is a weakly supervised task that annotates sign videos as textual glosses. Recent studies show that insufficient training caused by the lack of large-scale available sign datasets becomes the main bottleneck for SLR. Most SLR works thereby adopt pretrained visual modules and develop two mainstream solutions. The multi-stream architectures extend multi-cue visual features, yielding the current SOTA performances but requiring complex designs and might introduce potential noise. Alternatively, the advanced single-cue SLR frameworks using explicit cross-modal alignment between visual and textual modalities are simple and effective, potentially competitive with the multi-cue framework. In this work, we propose a novel contrastive visual-textual transformation for SLR, CVT-SLR, to fully explore the pretrained knowledge of both the visual and language modalities. Based on the single-cue cross-modal alignment framework, we propose a variational autoencoder (VAE) for pretrained contextual knowledge while introducing the complete pretrained language module. The VAE implicitly aligns visual and textual modalities while benefiting from pretrained contextual knowledge as the traditional contextual module. Meanwhile, a contrastive cross-modal alignment algorithm is designed to explicitly enhance the consistency constraints. Extensive experiments on public datasets (PHOENIX-2014 and PHOENIX-2014T) demonstrate that our proposed CVT-SLR consistently outperforms existing single-cue methods and even outperforms SOTA multi-cue methods.

• The paper introduces the CVT-SLR framework that leverages pretrained visual and textual modules with a novel contrastive alignment strategy for improved SLR accuracy.
• It employs a Variational Autoencoder and a simple MLP-based video-gloss adapter to effectively align and integrate cross-modal features.
• Experimental results on PHOENIX-2014 datasets show significant reductions in word error rates, outperforming current state-of-the-art single- and multi-cue methods.

Contrastive Visual-Textual Transformation for Sign Language Recognition

The paper "CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment" focuses on improving the performance of sign language recognition (SLR) systems by addressing the issue of insufficient training data and leveraging pretrained modules. The authors propose a novel framework, CVT-SLR, which integrates contrastive visual-textual transformation techniques and employs variational alignment mechanisms to enhance SLR accuracy.

Methodological Innovations

The CVT-SLR framework introduces several methodological innovations aimed at improving the efficiency of SLR systems, particularly in resource-constrained settings:

1. Pretrained Modules Utilization: The framework utilizes pretrained models in both visual and textual domains, capitalizing on existing large-scale datasets. The visual module uses a ResNet18 based feature extractor trained on general human action datasets, while the textual module employs a Variational Autoencoder (VAE) pretrained in a pseudo-translation task called Gloss2Gloss.
2. VAE for Contextual Knowledge: Unlike traditional contextual modules like RNNs or Transformers, CVT-SLR employs a VAE to implicitly align visual and textual modalities. This approach not only takes advantage of pretrained contextual linguistic knowledge but also maintains input-output consistency, inherently supporting cross-modal alignment.
3. Contrastive Cross-Modal Alignment: Inspired by contrastive learning paradigms, the authors introduce a contrastive alignment algorithm which accentuates both positive and negative sample pairs within a batch. This method aims to explicitly enhance cross-modal consistency constraints by optimizing for better alignment between visual and textual features.
4. Video-Gloss Adapter: To harmonize the different modalities, a simple MLP-based video-gloss adapter is introduced. It acts as an intermediary, preserving pretrained parameters while effectively linking visual outputs with the VAE's input requirements.

Experimental Validation

Extensive experiments conducted on public datasets, namely PHOENIX-2014 and PHOENIX-2014T, validate the effectiveness of the proposed methods. The CVT-SLR framework consistently outperforms current state-of-the-art (SOTA) single-cue methods in SLR, and even surpasses multi-cue approaches, which include additional inputs such as hand shapes, facial expressions, and keypoints. The reported improvements highlight the advantages of integrating pretrained models and leveraging innovative cross-modal alignment strategies.

Numerical Results and Implications

The results demonstrate a significant decrease in word error rates (WER), with the CVT-SLR framework achieving 19.8% on the PHOENIX-14 development set and 20.1% on the test set, marking a notable improvement over existing approaches like SMKD and $\text{C}^2\text{SLR}$. These strong outcomes suggest that capitalizing on pretrained knowledge from both visual and language domains, in conjunction with robust cross-modal alignment, can greatly enhance performance even with limited training data.

Future Directions and Implications

The introduction of CVT-SLR lays the groundwork for further exploration in the intersection of pretrained models and SLR. Future research could explore the integration of larger pretrained LLMs to further guide learning in the textual module, or focus on real-time application scenarios to enhance the accessibility of SLR technologies. The approach may also be extended to other domains of cross-modal learning, offering a generalizable framework for incorporating pretrained modules with efficient alignment strategies.

By effectively leveraging pretrained knowledge and fostering robust alignment mechanisms, CVT-SLR provides a significant advancement in SLR methodologies, emphasizing the potential impact of these strategies across various artificial intelligence applications.