---
title: 'Isharah-1000: Benchmark for CSLR'
url: https://www.emergentmind.com/topics/isharah-1000
type: topic
---

# Isharah-1000: Benchmark for CSLR

Searching arXiv for Isharah-1000 and related CSLR papers to ground the article in current literature.
Isharah-1000 is a mid-scale, standardized subset of the Isharah continuous sign language dataset, designed for benchmarking continuous sign language recognition (CSLR) and sign language translation (SLT) under realistic, multi-scene smartphone recording conditions. It comprises 15,000 sentence-level video clips covering 1,000 target sentence types, produced by 18 fluent signers, with full gloss annotations and Arabic sentence translations [2506.03615]. Within the recent CSLR literature, Isharah-1000 has become a focal benchmark for two particularly demanding generalization regimes: signer-independent recognition, which tests robustness to unseen signers, and unseen-sentences recognition, which tests compositional generalization to sentence forms not observed during training [2506.03615][2508.09372].

## 1. Definition, scope, and relation to the full Isharah corpus

Isharah-1000 is the 1,000-sentence, 15,000-clip subset of the broader Isharah dataset. The full Isharah corpus contains 30,000 video clips performed by 18 deaf and professional signers and was introduced as a large multi-scene dataset for CSLR collected in unconstrained environments using signers’ smartphone cameras [2506.03615]. Isharah is divided into three subsets—500, 1000, and 2000 sentence sets—to analyze scalability and to accommodate different computational budgets; Isharah-1000 is the middle tier intended to balance accessibility in training cost with linguistic diversity for benchmarking [2506.03615].

The subset contains 1,000 target sentence types of Saudi Sign Language, full gloss annotations for every video, and Arabic translations. At the subset level, the gloss vocabulary is 685 and the Arabic text vocabulary is 1,496; the total duration is approximately 24.14 hours across the 15,000 clips [2506.03615]. Average clip length is roughly 5.7–6.0 seconds across splits, and average sentence length is reported at approximately 4.8–4.9 glosses and 4.3–4.4 Arabic words, depending on split [2506.03615].

Its construction reflects an explicit benchmarking rationale. The subset selects approximately 1,000 frequently used Saudi Sign Language sentences spanning general and domain-specific topics, including banking, legal, healthcare, emergency services, education, and transportation [2506.03615]. This suggests that Isharah-1000 is intended not merely as a scale-reduced sample of the larger corpus, but as a controlled benchmark slice that retains lexical and topical breadth.

## 2. Data acquisition, annotation, and recording variability

Isharah-1000 was recorded in unconstrained, real-world settings, principally homes and offices, with signers using their own smartphones [2506.03615]. The data therefore exhibits substantial variability in lighting, backgrounds, camera distances, camera angles, resolution, and orientation. Both portrait and landscape videos are present, and the recording devices include 14 iPhones and four Android OEMs: HONOR, HUAWEI, Samsung, and TECNO [2506.03615]. This recording protocol differentiates Isharah-1000 from many earlier CSLR datasets collected in more controlled studio or broadcast conditions.

The collection workflow was reference-based. All sentences were first recorded by a Saudi Sign Language expert as references; participating signers then watched the expert reference videos and self-recorded the corresponding utterances on their smartphones [2506.03615]. An illustrated manual guided lighting, backgrounds, and clothing changes. Batch recordings of approximately 50 sentences per session were segmented into sentence clips using LosslessCut, and segmentation and verification procedures ensured correct gloss order and correct labeling by signer and sentence IDs [2506.03615]. Re-recording was requested for significant deviations from the references.

Annotation is gloss-centric at the sentence level. Every clip in Isharah-1000 has a gloss sequence representing the signs, and Arabic sentence translations are also provided [2506.03615]. Two Saudi Sign Language experts defined glossing guidelines and collaboratively annotated the data using a custom tool. Sentence boundaries are defined at the clip level, but per-gloss temporal boundaries are not provided [2506.03615]. The absence of frame-level gloss alignment has direct methodological consequences: models are typically trained with Connectionist Temporal Classification (CTC), which can marginalize over latent alignments without requiring explicit segmentation.

The annotation process is described as verified by experts, but inter-annotator agreement scores are not reported [2506.03615]. That omission is methodologically relevant for researchers concerned with annotation uncertainty, especially because the corpus is intended to support both CSLR and SLT.

## 3. Benchmark protocols and evaluation regimes

Isharah-1000 defines standardized evaluation protocols for two CSLR generalization problems: signer-independent recognition and unseen-sentences recognition [2506.03615][2508.09372]. These protocols are central to its role in the literature.

In the signer-independent protocol, the goal is generalization to unseen signers. The dataset paper specifies a split of 10,000 training videos, 1,000 development videos, and 4,000 test videos, with the test set drawn from four signers not seen during training [2506.03615]. The more recent CSLRConformer paper describes a curated challenge release of pre-extracted keypoint sequences with 10,000 labeled training samples from 13 signers, 949 development samples from one disjoint signer, and 3,800 test samples from four final unseen signers [2508.01791]. The difference indicates that benchmark discussions around Isharah-1000 may refer either to the standardized video split or to a curated challenge release derived from it.

In the unseen-sentences protocol, the task is compositional generalization. Development and test sentences are entirely unseen in training, although individual glosses may appear in other contexts [2508.09372]. The dataset paper specifies 10,000 training videos, 750 development videos, and 750 test videos, with development and test each containing 50 unique sentence types not present in training [2506.03615]. The split is characterized by high singleton rates and notable sentence-level out-of-vocabulary pressure, with OOV gloss rates of approximately 7.85% on development and 6.61% on test [2506.03615].

Word Error Rate is the primary CSLR metric on Isharah-1000:

$$
\mathrm{WER} = \frac{S + D + I}{N},
$$

where $S$, $D$, and $I$ are substitutions, deletions, and insertions, and $N$ is the number of reference glosses [2506.03615][2508.09372]. Lower WER indicates better performance. The dataset paper also defines BLEU and ROUGE-L for SLT, but Isharah-1000’s prominence in recent literature has been especially pronounced on the CSLR side [2506.03615].

| Protocol | Train / Dev / Test | Evaluation focus |
|---|---:|---|
| Signer-independent CSLR | 10,000 / 1,000 / 4,000 | Unseen signers |
| Unseen-sentences CSLR | 10,000 / 750 / 750 | Unseen sentence compositions |

These protocols make Isharah-1000 unusual among CSLR benchmarks because they separate inter-signer robustness from compositional generalization rather than treating recognition accuracy as a single undifferentiated objective.

## 4. Modalities, preprocessing, and representation choices

The original Isharah-1000 benchmark is video-based. The dataset paper reports raw RGB clips segmented to sentence boundaries and states that no hand or face tracking or keypoint extraction was used in the baseline benchmark runs; the video-based models were trained end-to-end with CTC alignment [2506.03615]. By contrast, two later challenge-oriented papers operate on pose-based 2D keypoint sequences rather than RGB [2508.09372][2508.01791]. This establishes Isharah-1000 as a benchmark that supports both video-centric and keypoint-centric methodological lines.

In the pose-based setting of the Signer-Invariant Conformer and Multi-Scale Fusion Transformer paper, each frame is represented by $K=86$ 2D landmarks over body, hands, and face, producing an input sequence $X=\{x_1,x_2,\dots,x_T\}$ with $x_t \in \mathbb{R}^{K \times 2}$ and a flattened tensor $X' \in \mathbb{R}^{T \times D}$ where $D=K \times 2$ [2508.09372]. Robustness to camera distance and positional variation is enforced via normalization relative to a torso-defined bounding box, and missing landmarks caused by occlusion or blur are linearly interpolated from neighboring frames [2508.09372].

The CSLRConformer paper adopts a more data-centric preprocessing pipeline. It begins from 86 tracked body points per frame, uses exploratory data analysis to identify communicative points by total motion displacement, and applies a DBSCAN-based reliability filter once on a high-quality reference sample to remove four problematic keypoints, yielding a fixed 82-keypoint representation concentrated on hands, lips, and eyes [2508.01791]. Frame-level normalization centers valid keypoints and rescales them by the larger of width and height of the frame-wise bounding box, while missing or invalid keypoints are set to zero rather than interpolated [2508.01791]. Dynamic features are then added via velocity and acceleration, producing 492 dimensions per timestep [2508.01791].

These two pose pipelines encode different assumptions. The 86-keypoint pipeline preserves the original landmark structure and uses interpolation for missing values [2508.09372], whereas the 82-keypoint pipeline assumes that principled feature selection and dynamic augmentation can improve robustness under noisy real-world capture conditions [2508.01791]. A plausible implication is that Isharah-1000 has become a testbed not only for model design but also for data representation strategy.

## 5. Baseline results and benchmark behavior

The dataset paper reports a suite of CSLR baselines on Isharah-1000: VAC, SMKD, TLP, SEN, CorrNet, Swin-MSTP, and SlowFastSign [2506.03615]. Their behavior differs markedly between the signer-independent and unseen-sentences regimes.

On signer-independent CSLR, Swin-MSTP is the strongest published baseline with 17.9% WER on development and 26.6% on test [2506.03615]. VAC and CorrNet both achieve 31.9% test WER, TLP reaches 32.0%, SlowFastSign 32.1%, SMKD 35.1%, and SEN 36.4% [2506.03615]. On unseen-sentences CSLR, the ordering changes: SMKD becomes strongest at 48.0% test WER, followed by VAC at 49.6%, CorrNet at 55.0%, SlowFastSign at 56.2%, SEN at 57.3%, TLP at 63.3%, and Swin-MSTP at 66.1% [2506.03615].

| Model | SI Test WER | US Test WER |
|---|---:|---:|
| Swin-MSTP | 26.6 | 66.1 |
| SMKD | 35.1 | 48.0 |
| VAC | 31.9 | 49.6 |
| CorrNet | 31.9 | 55.0 |

The dataset paper explicitly notes that Swin-MSTP is strongest on signer-independent test, while simpler models such as SMKD and VAC generalize better on unseen sentences, indicating that complex models may overfit learned gloss patterns [2506.03615]. This observation is significant because it shows that “better” architectures on Isharah-1000 are task-dependent: an architecture optimized for signer invariance is not necessarily effective for sentence compositionality.

The same paper also reports SLT baselines, including gloss-based MMTLB and gloss-free GFSLT-VLP, with gloss-based SLT consistently outperforming gloss-free SLT on Isharah-1000 [2506.03615]. Although CSLR is the primary focus of subsequent work, these SLT results help position Isharah-1000 as a bridge benchmark between recognition and translation.

## 6. Architectural advances built on Isharah-1000

Two 2025 papers use Isharah-1000 to argue for task-specific architecture design in CSLR. The first proposes a Signer-Invariant Conformer for the signer-independent task and a Multi-Scale Fusion Transformer for the unseen-sentences task [2508.09372]. The Signer-Invariant Conformer combines a shallow temporal encoder of 1D convolutions, sinusoidal positional encodings, a stack of conformer blocks with multi-head self-attention, depthwise 1D convolution with GLU activation, and position-wise feed-forward layers, followed by a CTC-trained classifier head without an external language model [2508.09372]. It achieves 7.31% WER on development and 13.07% on test for the signer-independent benchmark, improving on the previous best of 26.6% test WER from Swin-MSTP by 13.53 percentage points absolute and over 50% relative error reduction [2508.09372].

For unseen-sentences recognition, the same paper introduces a Multi-Scale Fusion Transformer with a joint attention module, a dual-path temporal encoder operating in parallel at original and downsampled temporal resolutions, multi-scale feature concatenation, transformer encoder layers, and an MLP classifier with GELU and dropout, again trained with CTC and decoded without an external language model [2508.09372]. It reports 55.08% WER on development and 47.78% on test, surpassing the previous best SMKD result of 48.0% on test [2508.09372]. The improvement is described as incremental but meaningful because of the difficulty of unseen sentence compositions.

The second paper, CSLRConformer, targets the signer-independent challenge in a data-centric manner [2508.01791]. It combines 82-keypoint feature selection, DBSCAN-based masking, per-frame normalization, position/velocity/acceleration features, a two-layer convolutional temporal subsampler, and an 8-layer Macaron-style Conformer encoder with $d_{\text{model}}=512$, $d_{ff}=2048$, 8 attention heads, dropout 0.3, and CTC training [2508.01791]. It reports 5.60% WER on development and 12.01% on test, securing third place on the official competition platform [2508.01791]. Error analysis attributes remaining failures to deletions of repeated glosses, insertions of plausible but contextually incorrect glosses, and substitutions among visually similar signs [2508.01791].

Together, these papers indicate that Isharah-1000 supports at least two complementary research directions. One emphasizes task-specific sequence modeling, separating signer invariance from compositional generalization [2508.09372]. The other emphasizes data-centric refinement of the pose representation and preprocessing pipeline [2508.01791]. This suggests that performance on Isharah-1000 is sensitive both to architectural inductive bias and to upstream feature quality.

## 7. Significance, limitations, and open directions

Isharah-1000 occupies a distinctive place among CSLR benchmarks because it combines real-world smartphone capture variability with full gloss annotations, Arabic translations, and standardized evaluation splits [2506.03615]. Relative to datasets such as RWTH-PHOENIX-(2014/2014T), Continuous CSL, CSL-Daily, FluentSigners-50, and TVB-HKSL-News, Isharah-1000 is characterized by unconstrained multi-scene capture, 18 signers, and full gloss plus Arabic annotation [2506.03615]. Its design therefore stresses robustness to variability that is often underrepresented in controlled benchmarks.

Several limitations are explicit in the literature. The unconstrained acquisition process produces large intra-class variation in camera pose, clothing, background motion, and resolution, which complicates generalization [2506.03615]. Singleton rates are high, including 21.76% singleton glosses in signer-independent CSLR train and 47.04% singleton Arabic words in SLT train [2506.03615]. The unseen-sentences split introduces notable OOV pressure at the gloss level [2506.03615]. Per-gloss temporal boundaries are not provided, and inter-annotator agreement is not reported [2506.03615]. In pose-based pipelines, dependency on upstream keypoint extraction means that pose errors propagate into recognition [2508.09372].

Future work noted in the model literature includes multimodal fusion with RGB cues such as hand shape and facial expressions, extension from CSLR to SLT, and unified multi-task modeling that jointly addresses signer-independent and unseen-sentences settings within a single framework [2508.09372]. The dataset paper also notes that future work aims to add temporal boundaries aligned to glosses [2506.03615]. A plausible implication is that Isharah-1000 will remain important not only as a benchmark for recognition accuracy, but also as a substrate for studying modality fusion, annotation granularity, and generalization under realistic acquisition noise.

In sum, Isharah-1000 is a benchmark dataset and experimental regime rather than a single static train/test partition. It links realistic mobile capture, sentence-level gloss annotation, Arabic translation, signer-independent evaluation, and compositional generalization into a unified research platform [2506.03615]. Subsequent results on pose-based conformer and transformer systems have turned it into a central benchmark for measuring how well CSLR models handle unseen signers and unseen sentence structures under real-world variation [2508.09372][2508.01791].

Source: https://www.emergentmind.com/topics/isharah-1000