CSLRConformer for Arabic Sign Language

• CSLRConformer is a keypoint-based CSLR system that processes 82 filtered keypoints with spatial normalization and motion derivatives.
• It employs an 8-block Conformer encoder coupled with CTC training to achieve signer-independent continuous sign language recognition.
• Empirical results show a significant reduction in WER compared to baselines, demonstrating the effectiveness of data-centric preprocessing.

CSLRConformer is a keypoint-based Continuous Sign Language Recognition (CSLR) system introduced for continuous Arabic Sign Language recognition on the Isharah dataset. It was developed for the 1st Multimodal Sign Language Recognition Workshop Challenge at ICCV 2025, specifically Task 1 on signer-independent continuous sign language recognition, and is defined by a data-centric pipeline that couples EDA-guided keypoint selection, DBSCAN-based reliability filtering, frame-level spatial normalization, and dynamic motion features with an 8-block Conformer encoder trained with CTC (Elden, 3 Aug 2025).

1. Task setting and benchmark scope

CSLRConformer addresses the signer-independent CSLR setting: given an unsegmented temporal sequence of sign-language motion, the system must transcribe the full gloss or sentence sequence without explicit temporal boundaries between signs. In the paper’s framing, the difficulty arises from fluid transitions between signs, the absence of explicit temporal segmentation, co-articulation, and signer independence. The signer-independent requirement is central, because the model must generalize to people not seen during training despite differences in body proportions, signing style, signing speed, camera framing, background, lighting, and pose-tracking quality.

The target dataset is the challenge version of Isharah-1000. The reported split is 10,000 labeled training samples from 13 signers, 949 development samples from 1 unseen signer, and 3,800 test samples from 4 unseen signers. The paper also notes that the original Isharah-1000 split had 10,000 training, 1,000 development, and 4,000 test samples, so the challenge version differs slightly in the exact video samples and counts while preserving the same 13/1/4 signer-independent protocol. Evaluation is by Word Error Rate (WER), with lower values indicating better performance (Elden, 3 Aug 2025).

The broader Isharah corpus is described as containing 30,000 video clips from 18 professional signers, recorded in unconstrained real-world environments using the signers’ own smartphone cameras, with variation in background, lighting, and camera angles. For the challenge setting discussed by CSLRConformer, however, the data provided to the model are not raw RGB videos but keypoint sequences. This distinction is important: the method operates on structured pose-like trajectories rather than directly on pixels.

2. Data-centric representation and preprocessing pipeline

The paper defines “data-centric” to mean that input representation and data quality are treated as primary design variables rather than as peripheral preprocessing. The raw input consists of 86 tracked body points over time. To quantify communicative importance, the paper measures total keypoint displacement,

$$D_i = \sum_{t=1}^{T-1} \|\text{position}_i(t+1) - \text{position}_i(t)\|_2,$$

and reports that the most active points consistently belong to the hands and face, especially the lips and eyes. Based on this EDA, combined with reliability filtering, the system reduces the feature set from 86 to 82 keypoints by removing 4 problematic keypoints.

A second stage uses DBSCAN on a high-quality reference sample to establish a static “master mask.” The stated outcome is dataset-wide dimensional consistency: the same 4 problematic keypoints are removed for every sample, and every sequence is represented with the same 82-keypoint layout. The paper does not specify the exact DBSCAN feature space, nor the parameters such as $\varepsilon$ or min_samples, so the clustering formulation is only partially documented.

After selection, each frame contains 82 keypoints with $(x,y)$ coordinates, giving 164 positional dimensions. The method then adds temporal derivatives. Velocity is defined as

$$\text{velocity}(t) = \frac{\text{position}(t+1) - \text{position}(t-1)}{2},$$

and acceleration as

$$\text{acceleration}(t) = \text{velocity}(t+1) - \text{velocity}(t-1).$$

The paper states that velocity estimation uses a 5-frame sliding window with convolution-based approximation, although the only explicit formula shown is the central-difference expression above. Concatenating position, velocity, and acceleration yields a 492-dimensional per-frame feature vector, since $164 \times 3 = 492$.

Spatial normalization is performed at the frame level. A bounding box is computed around all valid keypoints, scaling uses $$\text{scale} = \max(\text{bbox\_width}, \text{bbox\_height})$$, and the result is recentered by the mean of valid normalized keypoints. The printed normalization equation in the manuscript is syntactically broken, so the reliable description is the accompanying prose rather than the formula itself. Missing or invalid keypoints are set to zero; interpolation, Kalman smoothing, spline fitting, and sequence-level imputation are not reported. During training, the paper applies SpecAugment-style masking adapted to keypoint sequences, with time masking probability 0.08 and feature masking probability 0.2 (Elden, 3 Aug 2025).

3. Encoder design and recognition objective

At the architectural level, CSLRConformer takes the 492-dimensional frame sequence and processes it through a two-layer temporal convolutional subsampler, projects the features to 512 hidden dimensions, adds sinusoidal positional encoding, applies 8 Conformer blocks, and finishes with a linear classifier over a 1000-word vocabulary space followed by CTC training and decoding. The temporal subsampler is reported to reduce sequence length by 75%, so the output length is approximately one quarter of the input length (Elden, 3 Aug 2025).

The Conformer stack is described as a Macaron-Net-inspired sandwich structure. Each block contains four stages: a feed-forward module, multi-head self-attention, depthwise convolution, and a final feed-forward module. This matches the standard Conformer design logic introduced for ASR, in which convolution models local temporal structure and self-attention models global context (Gulati et al., 2020). In CSLRConformer, the stated rationale is that sign language contains both local temporal structure, such as fine movement transitions and handshape dynamics, and global sequence structure, such as long-range sentence context and inter-sign dependencies.

The reported hyperparameters are: 8 encoder layers, hidden dimension 512, 8 attention heads, feed-forward dimension 2048, and dropout 0.3. The paper does not specify the convolution kernel size inside the Conformer block, the exact FFN activation, the normalization type inside each submodule, dropout placement, or whether the attention uses relative positional encoding. It also does not provide the explicit formulas for sinusoidal positional encoding, MHSA, or the internal Conformer computations, even though those components are used conceptually.

Training uses Connectionist Temporal Classification. The target-sequence probability is written as

$$p(Y \mid X) = \sum_{\pi \in B^{-1}(Y)} p(\pi \mid X),$$

with path factorization

$p(\pi \mid X) = \prod_{t=1}^{T} p_t(\pi_t \mid X).$

This is the standard alignment-free formulation for sequence labeling without frame-level annotations. The paper’s output-space wording alternates among gloss, sentence, and word vocabulary, but the explicit classifier description states that the final linear layer projects hidden representations to the 1000-word vocabulary space (Elden, 3 Aug 2025).

4. Training regime, regularization, and decoding

The implementation is in PyTorch and the reported hardware is an NVIDIA A100-SXM4-40GB GPU. Optimization uses AdamW with learning rate $3 \times 10^{-4}$ and weight decay $\varepsilon$0. The learning-rate schedule is cosine annealing with a 15-epoch warmup. Batch size is 128, training runs for up to 300 epochs, and early stopping uses patience 30. Mixed precision training is explicitly mentioned in the training strategy prose.

Regularization includes dropout 0.3, layer dropout as mentioned in prose, and the SpecAugment-style time and feature masking already described. The training-progress discussion states that plots are shown over 225 epochs and that the best development WER occurs at epoch 202. Stochastic Weight Averaging was evaluated as an alternative experiment but underperformed the final model, and the paper does not state that SWA was used in the final submission.

The decoding description is incomplete. A wider-beam experiment with beam width 20 is reported, but the default beam width for the final system is not specified, nor is it stated whether the final reported results use greedy decoding or beam search. No external LLM is reported. This leaves an important reproducibility gap alongside the missing DBSCAN parameters, the malformed normalization formula, the unspecified subsampler kernel sizes and strides, and the absent default inference configuration (Elden, 3 Aug 2025).

5. Reported results and empirical interpretation

The final reported performance of the Data-Centric CSLRConformer is 5.60% WER on the development set and 12.01% WER on the test set. The abstract states that this result secured 3rd place on the official competition platform. On the test set, the paper reports 18,017 total words and 1,474 total errors, comprising 607 substitutions, 583 deletions, and 284 insertions. WER is defined in the standard way,

$\varepsilon$1

where $\varepsilon$2 is substitutions, $\varepsilon$3 is deletions, $\varepsilon$4 is insertions, and $\varepsilon$5 is the total number of reference words (Elden, 3 Aug 2025).

Against baselines reported from the original Isharah paper, the manuscript lists VAC at 18.9/31.9, SMKD at 18.5/35.1, TLP at 19.0/32.0, SEN at 19.1/36.4, CorrNet at 18.8/31.9, Swin-MSTP at 17.9/26.6, SlowFastSign at 19.0/32.1, and Data-Centric CSLRConformer at 5.60/12.01 on dev/test WER. In the explicit comparison table, the paper reports a 68.7% relative dev WER reduction and a 54.8% relative test WER reduction versus Swin-MSTP. It also notes that the abstract and introduction cite 75.1% dev and 53.6% test reductions relative to the best-performing baselines from the original Isharah dataset, so the percentage improvements vary slightly with the comparison framing.

The internal architecture comparisons are especially important for understanding the paper’s central argument. Reported dev/test WERs are 11.26/18.53 for SWA, 11.66/20.67 for Wider Beam with beam width 20, 7.86/16.33 for the Macaron-Net-inspired baseline, 7.93/14.75 for Deep Conformer, 22.31/98.88 for Squeeze-and-Excite Conformer, and 5.60/12.01 for the final Data-Centric CSLRConformer. The paper emphasizes that these alternative architecture experiments used the full 86 keypoints, whereas the final system used the 82 selected keypoints. This is the empirical basis for the paper’s claim that data-centric preprocessing and keypoint selection outperform purely architectural modifications. The same section quantifies the reduction relative to the best internal baseline, Deep Conformer, as 28.8% on dev and 22.2% on test (Elden, 3 Aug 2025).

The error analysis identifies deletions of repeated signs, insertions of plausible but incorrect glosses, substitutions among visually similar signs, difficulty with temporal redundancy, contextual disambiguation, and novel combinations and transitions of known glosses. The paper also states that the test set has zero out-of-vocabulary words relative to the train+dev vocabulary, so the residual errors are attributed not to unseen labels but to co-articulation, sequencing, and signer variability.

6. Position within Conformer-based CSLR research

CSLRConformer is best understood as a cross-domain adaptation of the Conformer encoder from ASR to pose-based CSLR. In the original Conformer formulation, the defining design is the combination of MHSA for global interactions and convolution for local temporal patterns, arranged in a Macaron-style block with feed-forward layers surrounding the contextual modules (Gulati et al., 2020). CSLRConformer preserves that local/global modeling premise but shifts the front end from acoustic frames to normalized keypoint trajectories enriched with velocity and acceleration. This suggests that the paper treats spatio-temporal motion dynamics in sign language as sufficiently analogous to the local-plus-global dependency structure that motivated Conformer in speech.

Within CSLR specifically, the paper should not be mistaken for a raw-video or linguistically hand-engineered Arabic Sign LLM. Its Arabic Sign Language specificity is described as mostly dataset- and modality-specific rather than linguistically specialized: the tailoring comes from the Isharah benchmark, the keypoint selection emphasizing hands and face, and signer-independent normalization, rather than from morphology-aware modeling, syntax-aware decoding, or explicit non-manual grammatical modules. A second common misconception is that the gains derive only from a stronger backbone. The internal comparisons argue the opposite: reducing the input from 86 to 82 keypoints, filtering out unreliable landmarks, and enriching motion features are treated as major sources of performance (Elden, 3 Aug 2025).

Related CSLR work underscores this broader context. “Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal” organizes CSLR backbones into visual, sequential, and alignment modules, and proposes auxiliary tasks such as keypoint-guided spatial attention consistency, sentence embedding consistency, and signer removal that are explicitly described as backbone-agnostic and therefore compatible in principle with Conformer-based CSLR systems (Zuo et al., 2022). A later Isharah paper, “A Signer-Invariant Conformer and Multi-Scale Fusion Transformer for Continuous Sign Language Recognition,” also adopts a pose-based Conformer encoder, using 86 two-dimensional pose, hand, and face landmarks with torso-relative normalization, a shallow 1D temporal convolutional encoder, sinusoidal positional encoding, and CTC, and reports 13.07% test WER on the signer-independent Isharah-1000 task (Haque et al., 12 Aug 2025). Together, these results indicate that Conformer-style sequence encoders have become a recurring design choice for pose-based signer-independent CSLR, while also showing that performance is highly sensitive to input curation and preprocessing.

The paper’s reproducibility profile remains mixed. It provides framework, hardware, feature dimensionality, optimizer, schedule, masking probabilities, model depth, and major hyperparameters, but it omits random seeds, DBSCAN parameters, exact subsampler and Conformer convolution specifications, exact normalization algebra because of the malformed printed formula, and the default CTC decoding configuration. A plausible implication is that the most transferable parts of the method are the EDA-driven keypoint importance analysis, the master-mask concept, frame-level normalization, motion-derivative augmentation, and the Conformer backbone itself, whereas the exact 82-keypoint mask is likely specific to the extractor and the Isharah data distribution.