- The paper introduces a curated parallel corpus of 1500 video samples capturing unique lexical items for Syrian Arabic Sign Language.
- It benchmarks modified MotionCLIP, T2M-GPT-ARABIC, and AraSignCLIP, highlighting their performance under low-resource constraints.
- The study emphasizes dataset expansion, inclusion of non-manual cues, and standardized evaluation to enhance future sign language translation.
SyriSign: A Parallel Corpus for Arabic Text to Syrian Arabic Sign Language Translation
Motivation and Dataset Construction
The SyriSign project addresses the acute underrepresentation of low-resource sign languages in computational sign language processing, focusing specifically on Syrian Arabic Sign Language (SyArSL). In Syria, a significant proportion of the Deaf and Hard-of-Hearing (DHH) population lacks access to vital news and public information due to the dominance of spoken and written Arabic in traditional media—modalities that are either not accessible or only partially accessible to this audience. Existing Arabic Sign Language (ArSL) resources do not capture the unique grammatical and lexical characteristics of SyArSL, given the regionally distinct nature of its dialect and a critical shortage of certified interpreters.
SyriSign fills this gap by providing a curated parallel dataset of 1500 video samples representing 150 unique lexical items, sampled from news and public announcement contexts and recorded from two native signers. Videos are captured at 60 FPS in RGB format, with each sign performed five times, yielding approximately three hours of annotated signing data. Data preprocessing leverages the MediaPipe Holistic framework for extracting full-body, hand, and facial landmarks, enabling both efficient downstream modeling and the retention of salient sign language features.
Methodological Contributions and Model Architectures
The study benchmarks three contemporary architectures for text-to-sign translation, emphasizing their applicability under low-resource conditions and the trade-offs they present between retrieval-based and generative paradigms.
Modified MotionCLIP
MotionCLIP forms the backbone of the retrieval-based approach. It aligns motion embeddings with CLIP-derived semantic text and image representations through a transformer encoder-decoder, optimizing a composite loss function comprising motion reconstruction, text alignment, and image similarity. For adaptation to SyArSL, the pipeline incorporates a CLIP text encoder with an Arabic translation layer and custom data preprocessing for skeletal consistency. Additional regularization terms—velocity smoothness and latent norm—are incorporated for temporal coherence and stability. This architecture achieved a validation loss as low as 0.5285 on sequence reconstruction, demonstrating robust retrieval capacity in the constrained-data regime.
T2M-GPT Adaptation
T2M-GPT is deployed as a generative baseline, repurposed here as T2M-GPT-ARABIC. Motion sequences are vector-quantized via a VQ-VAE, reducing the continuous pose space to discrete motion tokens. A GPT-based autoregressive decoder then generates these tokens conditioned on AraT5-encoded Arabic text embeddings, optimizing a loss balancing positional accuracy and interframe smoothness. The removal of facial landmarks for tractability, while simplifying learning dynamics, sacrifices non-manual cues that are critical for sign linguistic richness. Despite achieving a reconstruction test loss of 0.1205 and a cross-entropy of 0.0242, performance remains fundamentally bottlenecked by data scale, with generative outputs struggling for diversity and generalization.
SignCLIP (AraSignCLIP) Fine-Tuning
SignCLIP (specifically, AraSignCLIP) leverages a CLIP-style contrastive learning objective to align pose and text representations in a shared embedding space, using InfoNCE loss for robust coupling. Preprocessing reduces the original MediaPipe landmark set, focusing on 193 linguistically salient points (full hands, reduced facial cues, upper body). The text encoder is adapted to Arabic via machine translation and ISO-coded prompts. The model delivered an R@5 of 0.5526, indicating competitive retrieval performance but ultimately reflecting the limitations in sample complexity and signer variety.
Experimental Analysis and Results
The comparative analysis confirms that under current dataset constraints, retrieval-based architectures such as MotionCLIP and SignCLIP provide more reliable outputs than data-hungry generative models. Despite the promising qualitative behavior of T2M-GPT, lack of motion token diversity inhibits generalization to unseen signs and limits the naturalness of synthesized gesture sequences.
Evaluation metrics are proxy-based and do not include comprehensive human or back-translation assessment, a limitation imposed by dataset scale and the intrinsic ambiguity of sign language performance metrics. Additionally, signer overlap in partitioning and a focus on isolated lexical units rather than sentence-level discourse further delimit generalization claims across out-of-vocabulary or compositional expression contexts.
Implications and Prospective Developments
The public release of SyriSign establishes a critical resource for advancing SyArSL translation research and low-resource sign language processing. The baseline analyses elucidate clear paths forward:
- Dataset expansion: Increasing the lexical inventory, signer diversity, and recording length—especially towards sentence-level, context-rich sequences—is essential for meaningful progress in generative modeling and to capture the full syntactic and semantic inventory of SyArSL.
- Linguistic completeness: Incorporating facial features and more nuanced non-manual markers is paramount for robust sign production, given their linguistic salience for distinguishing grammatical function and emotion.
- Standardized evaluation: Implementation of human assessment, back-translation fidelity, and reference-based production metrics will be necessary for rigorous cross-model comparison and deployment-readiness assessment.
These directions align with broader theoretical advances in neural sign language translation, including the move from gloss-based to gloss-free architectures and the integration of multimodal transfer learning from high-resource to low-resource sign language domains. The work implicitly argues for a larger international effort to systematically document, annotate, and process regional sign languages otherwise absent from computational infrastructure.
Conclusion
SyriSign represents a foundational contribution to the Arab sign language NLP ecosystem, providing the first parallel corpus for Arabic-to-Syrian Arabic Sign Language translation and benchmarking state-of-the-art text-to-sign production pipelines under low-resource conditions. The presented methodology highlights core challenges—data rarity, signer representation, and linguistic completeness—facing future SyArSL research. As the dataset scales and evaluation protocols mature, SyriSign will facilitate continued advances toward inclusive, automated sign language access for the Syrian DHH community.