STARK: Spatio-Temporal Attention for Representation of Keypoints for Continuous Sign Language Recognition

Abstract: Continuous Sign Language Recognition (CSLR) is a crucial task for understanding the languages of deaf communities. Contemporary keypoint-based approaches typically rely on spatio-temporal encoding, where spatial interactions among keypoints are modeled using Graph Convolutional Networks or attention mechanisms, while temporal dynamics are captured using 1D convolutional networks. However, such designs often introduce a large number of parameters in both the encoder and the decoder. This paper introduces a unified spatio-temporal attention network that computes attention scores both spatially (across keypoints) and temporally (within local windows), and aggregates features to produce a local context-aware spatio-temporal representation. The proposed encoder contains approximately $70-80\%$ fewer parameters than existing state-of-the-art models while achieving comparable performance to keypoint-based methods on the Phoenix-14T dataset.

• The paper introduces STARK, a unified attention encoder that models spatial relationships among keypoints and local temporal dynamics jointly instead of using separate GCN and convolutional modules.
• On Phoenix-14T, STARK achieves 21.0% development and 21.9% test WER without pre-training, within 0.9–1.8 WER points of stronger baselines while using about 3M encoder parameters.
• The compact encoder is roughly 70–80% smaller than comparable methods, but unchanged decoder size, single-dataset evaluation, and local temporal windows leave end-to-end efficiency and generalization unresolved.

STARK: Spatio-Temporal Attention for Representation of Keypoints for Continuous Sign Language Recognition addresses a persistent inefficiency in keypoint-based Continuous Sign Language Recognition (CSLR): the reliance on large, separately parameterized spatial and temporal encoders. The paper proposes a unified attention module that computes spatial attention across keypoints within a frame and temporal attention over local windows across frames, aggregating both into a local context-aware representation. The central empirical claim is that the resulting encoder achieves performance comparable to state-of-the-art keypoint-based methods on Phoenix-14T while using roughly 70–80% fewer encoder parameters.

Motivation and positioning

Contemporary keypoint-based CSLR systems typically factorize spatio-temporal modeling into two stages: spatial interactions among body keypoints are handled by Graph Convolutional Networks or attention mechanisms, while temporal dynamics are captured by 1D convolutions or recurrent layers. Examples include CoSign, which uses ST-GCN blocks followed by 1D CNNs and Bi-LSTMs (2603.16163), MSKA, which decouples keypoints by body part and applies channel-wise attention with 1D convolution, and a conformer-based design from Haque et al. that combines multi-head self-attention over space with 1D convolution over time. The authors observe that all of these designs incur a large number of learnable parameters in both the encoder and decoder, and that the factorized spatial/temporal treatment cannot jointly learn the complex spatio-temporal dynamics of signing.

STARK departs from this factorization by computing attention scores both spatially (across intra-frame keypoints) and temporally (between the same keypoint in consecutive frames within a local window), then fusing the two attention outputs. The design is motivated by the observation that gloss recognition requires aggregating neighborhood visual information across both joints and frames, and that signer-level speed variability and recording variability demand adaptive temporal modeling rather than fixed convolutional kernels.

Method

The input is a keypoint tensor $X_{input} \in \mathbb{R}^{d \times T \times P}$, where $P$ keypoints are tracked over $T$ frames with $d$ dimensions comprising $x$, $y$ coordinates and a detector confidence score. Following MSKA, the model uses four input streams—body, left, right, and face—each encoded by its own stack of STARK blocks. The input is linearly projected and augmented with positional encoding before entering the attention modules.

The core spatio-temporal attention module proceeds as follows. Query and key representations are produced by a fully connected layer and split into $S$ subspaces. A patchify operation converts the key and feature tensors into temporally sliding patches parameterized by a kernel size and stride. Temporal attention is then computed over local temporal neighborhoods, with learnable global projection parameters $\alpha, \beta$ scaling and shifting the softmax scores per subspace and keypoint. Spatial attention is computed between keypoints within each frame using scaled dot-product attention with its own learnable projection parameters $\gamma, \delta$. The two attention outputs are aggregated by a weighted combination in which the temporal attention over patches is combined with the spatially attended features, with special handling of the central patch (index $k/2$) corresponding to the current frame. The aggregated features pass through a linear projection, residual connections, and a feed-forward layer with leaky ReLU activations. Stacked blocks progressively build higher-level representations, after which features are mean-pooled across keypoints and temporally max-pooled to produce a compact sequence of size $P$0 for decoding.

The decoder follows MSKA: the four encoder outputs are concatenated into four fusion streams (fuse, left, right, body), each decoded via a linear projection, temporal positional encoding, batch normalization, and a residual feed-forward layer. Training uses CTC loss together with a cross-distillation loss based on KL divergence over ensemble gloss probabilities, also adopted from MSKA.

Experimental results

The evaluation uses the RWTH-PHOENIX-Weather 2014T dataset (8,257 sequences, 1,066 gloss vocabulary, 9 signers of German Sign Language), with HRNet-derived keypoints. Of the 133 available keypoints, 79 are selected covering body, hands, and face, with $P$1, $P$2 coordinates and confidence scores as inputs. Augmentation includes normalization of coordinates to $P$3, random temporal resampling between $P$4 and $P$5 to model signing speed variability, and random 2D rotation within $P$6. The encoder uses four spatio-temporal attention stages with channel dimensions of 64, 96, 128, and 256 and 6 attention heads each. Training uses Adam with weight decay $P$7, cosine annealing, initial learning rate $P$8, and batch size 8; decoding uses greedy search during training and beam search with width 5 at inference. Evaluation is by Word Error Rate (WER).

Method	Pre-training	Encoder params	Dev WER	Test WER
TwoStream-SLR	Yes	—	27.1	27.2
SignBERT+	Yes	—	32.9	33.6
CoSign	No	≈10M	19.5	20.1
MSKA	No	≈15M	20.1	20.5
STARK	No	≈3M	21.0	21.9

The headline result is a favorable parameter–accuracy trade-off. STARK reaches 21.0/21.9 WER (dev/test) without any pre-training, versus 19.5/20.1 for CoSign and 20.1/20.5 for MSKA. The accuracy gap is 1.5–1.8 WER against CoSign and 0.9–1.4 WER against MSKA, while the encoder is roughly 3M parameters—about 70% smaller than CoSign's and 80% smaller than MSKA's. Notably, the parameter reduction is confined to the encoder; the decoder retains approximately 28M parameters, the same as MSKA's, so the total model is only modestly smaller. The comparison against pre-trained methods (TwoStream-SLR, SignBERT+) is not directly competitive, and the paper does not claim otherwise, since those systems benefit from large-scale pre-training that STARK forgoes.

Limitations and open questions

The paper concedes that "further study is required to fully explore the effectiveness of the proposed method." Several specific gaps follow from the reported results. First, evaluation is restricted to Phoenix-14T, a small, single-domain (weather broadcast) dataset; generalization to larger or multi-scene corpora such as Isharah is untested. Second, the 1.4–1.8 WER deficit against CoSign and MSKA is not analyzed—the paper does not report ablations isolating the contribution of the unified attention versus simply shrinking a conventional encoder, nor per-gloss error breakdowns that would indicate where the compact model loses accuracy. Third, the decoder parameter count is left unchanged, so end-to-end model compression remains an open question. Fourth, the temporal attention operates only within local windows; the interaction between window size and recognition of long, gestural glosses is not examined.

Conclusion

STARK demonstrates that a unified spatio-temporal attention mechanism over body keypoints can substitute for the conventional GCN-plus-1D-CNN factorization in CSLR, reducing encoder parameters by 70–80% relative to CoSign and MSKA while remaining within roughly 1–2 WER of those systems on Phoenix-14T without pre-training. The evidence supports the paper's efficiency claim, but the modest accuracy gap, the single-dataset evaluation, and the unchanged decoder size leave open whether the approach scales to harder benchmarks and whether the full model can be compressed comparably.