SegSLR: Promptable Video Segmentation for ISLR

• The system introduces pose-derived prompts that guide SAM 2 for precise, temporally consistent segmentation, replacing coarse bounding boxes.
• It fuses multiple streams—including RGB, optical flow, and segmentation-based representations—to capture detailed manual and non-manual linguistic features.
• Empirical evaluations on ChaLearn IsoGD demonstrate significant accuracy gains over traditional methods, underscoring its robustness in isolated sign language recognition.

Searching arXiv for the specified paper and closely related work on promptable video segmentation for isolated sign language recognition. arXiv search query: "SegSLR Promptable Video Segmentation for Isolated Sign Language Recognition (Schreiber et al., 12 Sep 2025)" SegSLR is a multi-stream, pose-aware isolated sign language recognition (ISLR) system that leverages promptable zero-shot video segmentation to preserve fine-grained manual and non-manual linguistic features—especially hand shape, finger configuration, and hand orientation—while fusing them with RGB appearance and motion cues. It replaces coarse, bounding-box–based region-of-interest selection with temporally consistent, pixel-precise masks of the signer’s body and hands generated by SAM 2, a foundation model for video segmentation, guided by pose-derived point prompts. On the ChaLearn IsoGD dataset, the system is reported to achieve state-of-the-art performance (Schreiber et al., 12 Sep 2025).

1. Problem setting and design rationale

Isolated Sign Language Recognition relies on both manual parameters and non-manual parameters. The manual parameters include hand shape, orientation, position, and movement, while the non-manual parameters include body posture, facial expressions, and head movements. Prior RGB-only pipelines either process whole frames or crop bounding boxes around hands, face, or body, typically losing critical intra-hand detail due to coarse localization. Pose-only approaches capture global kinematic patterns but cannot encode fine finger configurations and texture. Hybrid methods often use pose to guide RGB crops around hands or body, yet bounding boxes are imprecise and discard hand shape and orientation cues.

SegSLR addresses this limitation by using pose keypoints as point prompts to SAM 2, obtaining high-quality, temporally consistent masklets for the signer’s body and hands. The masks focus the RGB processing on the informative regions while retaining fine pose-derived shape information in both binary masks and segmentation logits. This suggests that the system’s central claim is not merely multimodal fusion, but a specific form of fusion in which pose acts as a prompting signal for pixel-precise video segmentation rather than as a direct recognition stream (Schreiber et al., 12 Sep 2025).

A plausible implication is that SegSLR redefines the role of pose in ISLR. Instead of treating pose as an incomplete substitute for appearance, it uses pose to localize the regions from which appearance and motion should be extracted. This is the basis for the paper’s distinction from bounding-box–based hybrids and rough mask generation approaches.

2. Pose-to-prompt conversion and best-frame selection

The pipeline begins from two input representations: RGB frames $x_{1:T}$, with $T$ uniformly sampled frames per clip, and pose keypoints $K_{1:T}$, extracted via RTMW, a state-of-the-art 2D human pose estimator producing up to 116 keypoints per frame. In the algorithmic summary, the system samples $T=40$ frames and extracts central $224 \times 224$ crops.

Pose-to-prompt conversion uses RTMW keypoints per frame in two distinct ways. For body prompts, it uses all keypoints except detailed hand and face keypoints; these are positive prompts for body segmentation. For hand prompts, it uses keypoints around the first joint per finger, while negative prompts include major body and important face keypoints to discourage SAM 2 from merging hands with body or face. Undetected keypoints are ignored. Prompts are points only, both positive and negative; no boxes or mask prompts are used (Schreiber et al., 12 Sep 2025).

SegSLR then selects an optimal starting frame. For each frame $t$, it computes the average keypoint confidence $c_t$, the body spread area $a_t$ defined as the area of the bounding box covering all detected keypoints, and the face–hand overlap $o_t$ defined as the maximum IoU between a hand bounding box and the face bounding box computed from the respective keypoints. After normalization by the per-video maximum, the scalar frame score is

$$S_t = \bar{c}_t \cdot \bar{a}_t \cdot (1 - \bar{o}_t).$$

The selected frame is

$T$0

This formulation favors frames with high pose confidence, extended arm configuration, and limited face occlusion. A plausible implication is that the best-frame heuristic serves as a prompt quality control mechanism: because SAM 2 propagates bidirectionally from $T$1, errors at initialization can affect the entire masklet sequence.

3. Promptable zero-shot video segmentation

Using the prompts derived at frame $T$2, SegSLR runs SAM 2 bidirectionally, forward and backward, to segment body masks $T$3 with logits $T$4 and a hands mask $T$5 with logits $T$6. The hands are segmented jointly; the resulting mask covers both hands. SAM 2’s temporal memory attention ensures that masklets are consistent across frames, reducing flicker and handling motion and occlusion better than image-only SAM (Schreiber et al., 12 Sep 2025).

The segmented representations are used in two forms. First, binary masks are applied to RGB frames:

$T$7

Second, the raw per-pixel logits $T$8 and $T$9 are fed as inputs to I3D, treated as single-channel frames or appropriately stacked to match network input. In the paper’s formulation, the masks preserve the relevant shape information, while the segmentation logits carry global context and per-pixel confidence.

Qualitative discussion in the source emphasizes three situations: rapid hand motion, heavy background texture, and hand–face occlusion. Negative prompts on face and body help disentangle hands from face when overlapping, and even complex finger postures are reported to be captured. This suggests that the system’s segmentation component is intended not only as spatial filtering but also as a means of preserving linguistically salient shape detail that bounding boxes and skeleton dilation cannot represent.

4. Multi-stream recognition architecture and optimization

The architecture integrates four segmentation-driven streams with two conventional streams, all classified by I3D CNNs. The baseline streams are RGB appearance and optical flow, with flow computed via Zach et al. (2007). The segmentation streams are body-masked RGB, body logits, hands-masked RGB, and hands logits.

Stream	Input
RGB	$K_{1:T}$0
Optical flow	$K_{1:T}$1
Body_RGB	$K_{1:T}$2
Body_Logits	$K_{1:T}$3
Hands_RGB	$K_{1:T}$4
Hands_Logits	$K_{1:T}$5

Each $K_{1:T}$6 is an I3D CNN capturing spatiotemporal features and outputting per-class logits $K_{1:T}$7. Score-level fusion aggregates per-stream logits or probabilities. In practice, an unweighted sum or average of per-stream scores is applied:

$K_{1:T}$8

The mathematical summary also gives the equivalent alternatives

$K_{1:T}$9

or $T=40$0, with exact weighting not specified. The per-stream training loss is categorical cross-entropy,

$T=40$1

SAM 2 and RTMW are used as frozen pre-trained components; no segmentation labels are available, and no auxiliary segmentation or pose consistency losses are added. Backbones are specified as follows: SAM 2 is pre-trained on SA-1B and SA-V and used zero-shot without fine-tuning; RTMW is pre-trained on 14 datasets and used as-is; I3D CNNs are pre-trained on ImageNet and Kinetics and fine-tuned per stream. Optimization uses Adam, batch size 4, and early stopping with patience 3. Data augmentation consists of horizontal and vertical crop shifts and brightness adjustment (Schreiber et al., 12 Sep 2025).

A plausible implication is that SegSLR is architecturally modular. The segmentation and pose stages are frozen, so only the I3D streams require task-specific optimization for a new dataset.

5. Evaluation protocol and reported performance

The reported evaluation uses ChaLearn IsoGD, also described as ChaLearn249, a dataset with 249 gestures and 47,933 videos under challenging lighting and backgrounds. The train, validation, and test protocol follows the dataset’s splits. The system is trained five times, with median validation accuracy reported; mean and standard deviation across runs are also reported, and test results are reported for the model with median validation performance. The metric is top-1 accuracy (Schreiber et al., 12 Sep 2025).

SegSLR achieves the following results on ChaLearn249: validation accuracy of 71.30%, with mean 71.39% and $T=40$2 across five runs, and test accuracy of 72.76%. The paper reports improvements over several baselines. Compared with I3D-SLR, which uses RGB and flow only, SegSLR improves from 62.09% to 71.30% on validation and from 64.44% to 72.76% on test, corresponding to +9.21% and +8.32%. Compared with I3D-pseudoDepth, the improvement is from 62.50% to 71.30% on validation and from 66.20% to 72.76% on test, corresponding to +8.80% and +6.56%. Compared with Hybrid Attn-I3D-SLR, described as using moving-area focus, the improvement is from 65.02% to 71.30% on validation and from 68.89% to 72.76% on test, corresponding to +6.28% and +3.87%. Compared with TD-SLR, described as hand-focused bounding boxes, the improvement is from 67.13% to 71.30% on validation and from 70.91% to 72.76% on test, corresponding to +4.17% and +1.85%.

These comparisons situate SegSLR against RGB-only, pseudo-depth, moving-area focus, and bounding-box–based hand localization pipelines. The reported gains are consistent with the paper’s argument that pixel-precise segmentation of body and hands retains fine detail that is discarded by coarse crops.

6. Ablations, robustness, limitations, and name ambiguity

The ablation studies isolate the contribution of the segmentation-driven streams. Starting from the baseline RGB + flow configuration at 62.09% validation and 64.42% test, adding Body_RGB yields 65.51% validation and 67.33% test, a gain of +3.42% and +2.91%. Adding Body_Logits next yields 67.10% validation and 69.78% test, a further gain of +1.59% and +2.45%. Adding Hands_RGB and Hands_Logits produces 71.30% validation and 72.76% test, a further gain of +4.20% validation and +2.98% test from the previous step. The stated conclusion is that focusing first on the signer’s body improves performance, adding segmentation logits adds context, and focusing on hands yields the largest further gains, underscoring the importance of manual parameters (Schreiber et al., 12 Sep 2025).

A second ablation compares segmentation methods when only the Body_RGB stream is added. Mask R-CNN, person-only, gives 62.31% validation and 65.44% test. SAM, image-only, gives 64.14% validation and 66.46% test. SAM 2, using video segmentation, gives 65.51% validation and 67.33% test. SAM and SAM 2 outperform Mask R-CNN by up to +3.20%, attributed to better zero-shot segmentation capability and, for SAM 2, temporal consistency through memory attention. The qualitative comparison in the source states that SAM 2 reduces flicker and more consistently covers the full body than SAM.

The paper also specifies several limitations. The method relies on pose quality for prompt generation; mislocalized or missing keypoints can degrade segmentation, especially fingertips. SAM 2 segmentation depends on prompt quality, so errors in best-frame selection or prompts can propagate forward and backward. Backgrounds with similar color or texture to skin may still challenge segmentation, although prompts mitigate this. Computational cost is non-trivial because the pipeline comprises pose estimation with RTMW, SAM 2 bidirectional segmentation, and six I3D inferences; inference speed and memory footprint are not reported. There is no explicit handling of left/right hand separation because the hands are segmented jointly, and the source identifies separate masks as a potential extension. The broader-impact discussion points to continuous SLR, multimodal extensions with depth, IR, or IMU, improved prompting, semi- or weakly-supervised segmentation, and transformer-based temporal models as future directions.

A common source of confusion is the acronym itself. In the sign-language literature, SegSLR refers to “SegSLR: Promptable Video Segmentation for Isolated Sign Language Recognition” (Schreiber et al., 12 Sep 2025). In an unrelated credit-scoring context, SegSLR also refers to “Score Engineered Logistic Regression,” a sequential quadratic programming approach for constrained logistic regression scorecards (Hoadley, 2020). The two usages share an acronym but designate different methods, problem domains, and technical frameworks.