Isharah-1000: Benchmark for CSLR

• Isharah-1000 is a subset of the full Isharah corpus featuring 15,000 sentence-level video clips with full gloss annotations and Arabic translations.
• It defines controlled evaluation protocols for signer-independent recognition and unseen-sentences recognition to test compositional generalization under realistic conditions.
• The dataset supports both raw video and pose-based keypoint pipelines, driving advancements in model architecture and data preprocessing strategies.

Searching arXiv for Isharah-1000 and related CSLR papers to ground the article in current literature. Isharah-1000 is a mid-scale, standardized subset of the Isharah continuous sign language dataset, designed for benchmarking continuous sign language recognition (CSLR) and sign language translation (SLT) under realistic, multi-scene smartphone recording conditions. It comprises 15,000 sentence-level video clips covering 1,000 target sentence types, produced by 18 fluent signers, with full gloss annotations and Arabic sentence translations (Alyami et al., 4 Jun 2025). Within the recent CSLR literature, Isharah-1000 has become a focal benchmark for two particularly demanding generalization regimes: signer-independent recognition, which tests robustness to unseen signers, and unseen-sentences recognition, which tests compositional generalization to sentence forms not observed during training (Alyami et al., 4 Jun 2025, Haque et al., 12 Aug 2025).

1. Definition, scope, and relation to the full Isharah corpus

Isharah-1000 is the 1,000-sentence, 15,000-clip subset of the broader Isharah dataset. The full Isharah corpus contains 30,000 video clips performed by 18 deaf and professional signers and was introduced as a large multi-scene dataset for CSLR collected in unconstrained environments using signers’ smartphone cameras (Alyami et al., 4 Jun 2025). Isharah is divided into three subsets—500, 1000, and 2000 sentence sets—to analyze scalability and to accommodate different computational budgets; Isharah-1000 is the middle tier intended to balance accessibility in training cost with linguistic diversity for benchmarking (Alyami et al., 4 Jun 2025).

The subset contains 1,000 target sentence types of Saudi Sign Language, full gloss annotations for every video, and Arabic translations. At the subset level, the gloss vocabulary is 685 and the Arabic text vocabulary is 1,496; the total duration is approximately 24.14 hours across the 15,000 clips (Alyami et al., 4 Jun 2025). Average clip length is roughly 5.7–6.0 seconds across splits, and average sentence length is reported at approximately 4.8–4.9 glosses and 4.3–4.4 Arabic words, depending on split (Alyami et al., 4 Jun 2025).

Its construction reflects an explicit benchmarking rationale. The subset selects approximately 1,000 frequently used Saudi Sign Language sentences spanning general and domain-specific topics, including banking, legal, healthcare, emergency services, education, and transportation (Alyami et al., 4 Jun 2025). This suggests that Isharah-1000 is intended not merely as a scale-reduced sample of the larger corpus, but as a controlled benchmark slice that retains lexical and topical breadth.

2. Data acquisition, annotation, and recording variability

Isharah-1000 was recorded in unconstrained, real-world settings, principally homes and offices, with signers using their own smartphones (Alyami et al., 4 Jun 2025). The data therefore exhibits substantial variability in lighting, backgrounds, camera distances, camera angles, resolution, and orientation. Both portrait and landscape videos are present, and the recording devices include 14 iPhones and four Android OEMs: HONOR, HUAWEI, Samsung, and TECNO (Alyami et al., 4 Jun 2025). This recording protocol differentiates Isharah-1000 from many earlier CSLR datasets collected in more controlled studio or broadcast conditions.

The collection workflow was reference-based. All sentences were first recorded by a Saudi Sign Language expert as references; participating signers then watched the expert reference videos and self-recorded the corresponding utterances on their smartphones (Alyami et al., 4 Jun 2025). An illustrated manual guided lighting, backgrounds, and clothing changes. Batch recordings of approximately 50 sentences per session were segmented into sentence clips using LosslessCut, and segmentation and verification procedures ensured correct gloss order and correct labeling by signer and sentence IDs (Alyami et al., 4 Jun 2025). Re-recording was requested for significant deviations from the references.

Annotation is gloss-centric at the sentence level. Every clip in Isharah-1000 has a gloss sequence representing the signs, and Arabic sentence translations are also provided (Alyami et al., 4 Jun 2025). Two Saudi Sign Language experts defined glossing guidelines and collaboratively annotated the data using a custom tool. Sentence boundaries are defined at the clip level, but per-gloss temporal boundaries are not provided (Alyami et al., 4 Jun 2025). The absence of frame-level gloss alignment has direct methodological consequences: models are typically trained with Connectionist Temporal Classification (CTC), which can marginalize over latent alignments without requiring explicit segmentation.

The annotation process is described as verified by experts, but inter-annotator agreement scores are not reported (Alyami et al., 4 Jun 2025). That omission is methodologically relevant for researchers concerned with annotation uncertainty, especially because the corpus is intended to support both CSLR and SLT.

3. Benchmark protocols and evaluation regimes

Isharah-1000 defines standardized evaluation protocols for two CSLR generalization problems: signer-independent recognition and unseen-sentences recognition (Alyami et al., 4 Jun 2025, Haque et al., 12 Aug 2025). These protocols are central to its role in the literature.

In the signer-independent protocol, the goal is generalization to unseen signers. The dataset paper specifies a split of 10,000 training videos, 1,000 development videos, and 4,000 test videos, with the test set drawn from four signers not seen during training (Alyami et al., 4 Jun 2025). The more recent CSLRConformer paper describes a curated challenge release of pre-extracted keypoint sequences with 10,000 labeled training samples from 13 signers, 949 development samples from one disjoint signer, and 3,800 test samples from four final unseen signers (Elden, 3 Aug 2025). The difference indicates that benchmark discussions around Isharah-1000 may refer either to the standardized video split or to a curated challenge release derived from it.

In the unseen-sentences protocol, the task is compositional generalization. Development and test sentences are entirely unseen in training, although individual glosses may appear in other contexts (Haque et al., 12 Aug 2025). The dataset paper specifies 10,000 training videos, 750 development videos, and 750 test videos, with development and test each containing 50 unique sentence types not present in training (Alyami et al., 4 Jun 2025). The split is characterized by high singleton rates and notable sentence-level out-of-vocabulary pressure, with OOV gloss rates of approximately 7.85% on development and 6.61% on test (Alyami et al., 4 Jun 2025).

Word Error Rate is the primary CSLR metric on Isharah-1000:

$\mathrm{WER} = \frac{S + D + I}{N},$

where $S$, $D$, and $I$ are substitutions, deletions, and insertions, and $N$ is the number of reference glosses (Alyami et al., 4 Jun 2025, Haque et al., 12 Aug 2025). Lower WER indicates better performance. The dataset paper also defines BLEU and ROUGE-L for SLT, but Isharah-1000’s prominence in recent literature has been especially pronounced on the CSLR side (Alyami et al., 4 Jun 2025).

Protocol	Train / Dev / Test	Evaluation focus
Signer-independent CSLR	10,000 / 1,000 / 4,000	Unseen signers
Unseen-sentences CSLR	10,000 / 750 / 750	Unseen sentence compositions

These protocols make Isharah-1000 unusual among CSLR benchmarks because they separate inter-signer robustness from compositional generalization rather than treating recognition accuracy as a single undifferentiated objective.

4. Modalities, preprocessing, and representation choices

The original Isharah-1000 benchmark is video-based. The dataset paper reports raw RGB clips segmented to sentence boundaries and states that no hand or face tracking or keypoint extraction was used in the baseline benchmark runs; the video-based models were trained end-to-end with CTC alignment (Alyami et al., 4 Jun 2025). By contrast, two later challenge-oriented papers operate on pose-based 2D keypoint sequences rather than RGB (Haque et al., 12 Aug 2025, Elden, 3 Aug 2025). This establishes Isharah-1000 as a benchmark that supports both video-centric and keypoint-centric methodological lines.

In the pose-based setting of the Signer-Invariant Conformer and Multi-Scale Fusion Transformer paper, each frame is represented by $K=86$ 2D landmarks over body, hands, and face, producing an input sequence $X=\{x_1,x_2,\dots,x_T\}$ with $x_t \in \mathbb{R}^{K \times 2}$ and a flattened tensor $X' \in \mathbb{R}^{T \times D}$ where $D=K \times 2$ (Haque et al., 12 Aug 2025). Robustness to camera distance and positional variation is enforced via normalization relative to a torso-defined bounding box, and missing landmarks caused by occlusion or blur are linearly interpolated from neighboring frames (Haque et al., 12 Aug 2025).

The CSLRConformer paper adopts a more data-centric preprocessing pipeline. It begins from 86 tracked body points per frame, uses exploratory data analysis to identify communicative points by total motion displacement, and applies a DBSCAN-based reliability filter once on a high-quality reference sample to remove four problematic keypoints, yielding a fixed 82-keypoint representation concentrated on hands, lips, and eyes (Elden, 3 Aug 2025). Frame-level normalization centers valid keypoints and rescales them by the larger of width and height of the frame-wise bounding box, while missing or invalid keypoints are set to zero rather than interpolated (Elden, 3 Aug 2025). Dynamic features are then added via velocity and acceleration, producing 492 dimensions per timestep (Elden, 3 Aug 2025).

These two pose pipelines encode different assumptions. The 86-keypoint pipeline preserves the original landmark structure and uses interpolation for missing values (Haque et al., 12 Aug 2025), whereas the 82-keypoint pipeline assumes that principled feature selection and dynamic augmentation can improve robustness under noisy real-world capture conditions (Elden, 3 Aug 2025). A plausible implication is that Isharah-1000 has become a testbed not only for model design but also for data representation strategy.

5. Baseline results and benchmark behavior

The dataset paper reports a suite of CSLR baselines on Isharah-1000: VAC, SMKD, TLP, SEN, CorrNet, Swin-MSTP, and SlowFastSign (Alyami et al., 4 Jun 2025). Their behavior differs markedly between the signer-independent and unseen-sentences regimes.

On signer-independent CSLR, Swin-MSTP is the strongest published baseline with 17.9% WER on development and 26.6% on test (Alyami et al., 4 Jun 2025). VAC and CorrNet both achieve 31.9% test WER, TLP reaches 32.0%, SlowFastSign 32.1%, SMKD 35.1%, and SEN 36.4% (Alyami et al., 4 Jun 2025). On unseen-sentences CSLR, the ordering changes: SMKD becomes strongest at 48.0% test WER, followed by VAC at 49.6%, CorrNet at 55.0%, SlowFastSign at 56.2%, SEN at 57.3%, TLP at 63.3%, and Swin-MSTP at 66.1% (Alyami et al., 4 Jun 2025).

Model	SI Test WER	US Test WER
Swin-MSTP	26.6	66.1
SMKD	35.1	48.0
VAC	31.9	49.6
CorrNet	31.9	55.0

The dataset paper explicitly notes that Swin-MSTP is strongest on signer-independent test, while simpler models such as SMKD and VAC generalize better on unseen sentences, indicating that complex models may overfit learned gloss patterns (Alyami et al., 4 Jun 2025). This observation is significant because it shows that “better” architectures on Isharah-1000 are task-dependent: an architecture optimized for signer invariance is not necessarily effective for sentence compositionality.

The same paper also reports SLT baselines, including gloss-based MMTLB and gloss-free GFSLT-VLP, with gloss-based SLT consistently outperforming gloss-free SLT on Isharah-1000 (Alyami et al., 4 Jun 2025). Although CSLR is the primary focus of subsequent work, these SLT results help position Isharah-1000 as a bridge benchmark between recognition and translation.

6. Architectural advances built on Isharah-1000

Two 2025 papers use Isharah-1000 to argue for task-specific architecture design in CSLR. The first proposes a Signer-Invariant Conformer for the signer-independent task and a Multi-Scale Fusion Transformer for the unseen-sentences task (Haque et al., 12 Aug 2025). The Signer-Invariant Conformer combines a shallow temporal encoder of 1D convolutions, sinusoidal positional encodings, a stack of conformer blocks with multi-head self-attention, depthwise 1D convolution with GLU activation, and position-wise feed-forward layers, followed by a CTC-trained classifier head without an external LLM (Haque et al., 12 Aug 2025). It achieves 7.31% WER on development and 13.07% on test for the signer-independent benchmark, improving on the previous best of 26.6% test WER from Swin-MSTP by 13.53 percentage points absolute and over 50% relative error reduction (Haque et al., 12 Aug 2025).

For unseen-sentences recognition, the same paper introduces a Multi-Scale Fusion Transformer with a joint attention module, a dual-path temporal encoder operating in parallel at original and downsampled temporal resolutions, multi-scale feature concatenation, transformer encoder layers, and an MLP classifier with GELU and dropout, again trained with CTC and decoded without an external LLM (Haque et al., 12 Aug 2025). It reports 55.08% WER on development and 47.78% on test, surpassing the previous best SMKD result of 48.0% on test (Haque et al., 12 Aug 2025). The improvement is described as incremental but meaningful because of the difficulty of unseen sentence compositions.

The second paper, CSLRConformer, targets the signer-independent challenge in a data-centric manner (Elden, 3 Aug 2025). It combines 82-keypoint feature selection, DBSCAN-based masking, per-frame normalization, position/velocity/acceleration features, a two-layer convolutional temporal subsampler, and an 8-layer Macaron-style Conformer encoder with $S$0, $S$1, 8 attention heads, dropout 0.3, and CTC training (Elden, 3 Aug 2025). It reports 5.60% WER on development and 12.01% on test, securing third place on the official competition platform (Elden, 3 Aug 2025). Error analysis attributes remaining failures to deletions of repeated glosses, insertions of plausible but contextually incorrect glosses, and substitutions among visually similar signs (Elden, 3 Aug 2025).

Together, these papers indicate that Isharah-1000 supports at least two complementary research directions. One emphasizes task-specific sequence modeling, separating signer invariance from compositional generalization (Haque et al., 12 Aug 2025). The other emphasizes data-centric refinement of the pose representation and preprocessing pipeline (Elden, 3 Aug 2025). This suggests that performance on Isharah-1000 is sensitive both to architectural inductive bias and to upstream feature quality.

7. Significance, limitations, and open directions

Isharah-1000 occupies a distinctive place among CSLR benchmarks because it combines real-world smartphone capture variability with full gloss annotations, Arabic translations, and standardized evaluation splits (Alyami et al., 4 Jun 2025). Relative to datasets such as RWTH-PHOENIX-(2014/2014T), Continuous CSL, CSL-Daily, FluentSigners-50, and TVB-HKSL-News, Isharah-1000 is characterized by unconstrained multi-scene capture, 18 signers, and full gloss plus Arabic annotation (Alyami et al., 4 Jun 2025). Its design therefore stresses robustness to variability that is often underrepresented in controlled benchmarks.

Several limitations are explicit in the literature. The unconstrained acquisition process produces large intra-class variation in camera pose, clothing, background motion, and resolution, which complicates generalization (Alyami et al., 4 Jun 2025). Singleton rates are high, including 21.76% singleton glosses in signer-independent CSLR train and 47.04% singleton Arabic words in SLT train (Alyami et al., 4 Jun 2025). The unseen-sentences split introduces notable OOV pressure at the gloss level (Alyami et al., 4 Jun 2025). Per-gloss temporal boundaries are not provided, and inter-annotator agreement is not reported (Alyami et al., 4 Jun 2025). In pose-based pipelines, dependency on upstream keypoint extraction means that pose errors propagate into recognition (Haque et al., 12 Aug 2025).

Future work noted in the model literature includes multimodal fusion with RGB cues such as hand shape and facial expressions, extension from CSLR to SLT, and unified multi-task modeling that jointly addresses signer-independent and unseen-sentences settings within a single framework (Haque et al., 12 Aug 2025). The dataset paper also notes that future work aims to add temporal boundaries aligned to glosses (Alyami et al., 4 Jun 2025). A plausible implication is that Isharah-1000 will remain important not only as a benchmark for recognition accuracy, but also as a substrate for studying modality fusion, annotation granularity, and generalization under realistic acquisition noise.

In sum, Isharah-1000 is a benchmark dataset and experimental regime rather than a single static train/test partition. It links realistic mobile capture, sentence-level gloss annotation, Arabic translation, signer-independent evaluation, and compositional generalization into a unified research platform (Alyami et al., 4 Jun 2025). Subsequent results on pose-based conformer and transformer systems have turned it into a central benchmark for measuring how well CSLR models handle unseen signers and unseen sentence structures under real-world variation (Haque et al., 12 Aug 2025, Elden, 3 Aug 2025).