What's the Point? Spatial Grammar & Index Resolution for Sign Language Processing

Abstract: Sign LLMs are predominantly trained with gloss-sequence or text supervision, thereby under-modeling non-lexical and productive constructions. One comparatively tractable instance is spatial indexing: pointing gestures that assign discourse entities to spatial loci for subsequent co-reference, which lexicon-centric objectives largely fail to capture. We present a targeted evaluation of indexing in Sign Language Recognition, showing that despite comprising 10-15% of signing content, indexing is poorly recovered. We introduce a framework for training and evaluating indexing experts, establishing a baseline for index-aware sign language modeling. Our approach decomposes spatial reference resolution into index detection and discourse entity linking. The resulting mention representations enable automatic annotation and non-lexical structure modeling, and serve as an auxiliary indexing expert that augments a frozen SLR model at inference time.

• The paper introduces a modular pose-driven framework that isolates spatial indexing as a structured prediction task to improve sign language recognition.
• It employs an Index Proposal Network and an Entity Linking Module to dynamically detect and cluster index mentions, reducing index-specific errors significantly.
• Empirical results on multiple sign language datasets show a reduction in index-specific WER by over 36 points without degrading overall lexical accuracy.

Spatial Grammar and Index Resolution for Sign Language Processing

Motivation and Background

The paper "What's the Point? Spatial Grammar & Index Resolution for Sign Language Processing" (2606.08056) addresses a core structural limitation in current Sign Language Recognition (SLR) and Sign Language Translation (SLT) systems. Mainstream models, dominated by gloss-sequence or text-based objectives, consistently under-model the highly productive, non-lexical constructions essential in signed discourse. A central such construction is spatial indexing, in which signers associate discourse entities with specific loci in the signing space and establish referential links via pointing. These index assignments are frequent—comprising 10–15% of sign tokens—but severely under-represented and under-evaluated in prevailing SLR/SRLT benchmarks and architectures.

Spatial reference in sign languages is not merely a lexical phenomenon, but encodes core discourse and grammatical structure via three-dimensional spatial grammar. The challenge is compounded by (i) visually ambiguous forms (lexicon-like pointing tokens overlap with non-indexical signs), (ii) context dependence (spatial locus function varies with discourse), and (iii) the lack of dense, multi-level referential annotation in standard resources. The work therefore isolates the spatial indexing and resolution problem as a structured prediction task in its own right, rather than as incidental information to be implicitly recovered by generic recognition systems.

System Architecture

The proposed framework decomposes index resolution into two tightly coupled components: (1) a pose-driven Index Proposal Network (IPN) for local segment-wise detection, and (2) an online differentiable Entity Linking Module (ELM) that clusters detected index mentions and tracks referents dynamically throughout discourse. The entire system integrates modularly as an auxiliary inference-time bias, enhancing existing frozen SLR architectures.

Figure 1: Overview of the pipeline—pose segments are encoded, indexed mentions are detected, entity linking is performed, and the output augments a downstream SLR model.

The IPN leverages pose features extracted using the spatiotemporal graph backbone SL-GCN, with pose input constructed from both SMPL-X whole-body and WiLoR hand keypoints. Each gloss-aligned segment is encoded as a fixed-length spatiotemporal representation, and the IPN outputs an index likelihood. Segments surpassing a precision-calibrated threshold are selected as index mentions.

The ELM incrementally processes these mentions in sequence order, maintaining a variable-length external memory of active entities. Mention embeddings are computed via a dedicated linking projection, augmented with engineered spatial/kinematic features (pointing direction, elevation, arm reach, etc.), then scored against each entity slot. Assignment is determined by an MLP-based compatibility head, using geometric and distributional cues, with memory states updated via gated interpolation. Supervision for linking is generated at scale from parallel translations using an LLM-based (gpt-oss:120b) clustering pipeline, followed by rule-based correction to align with sign-specific referential phenomena.

At inference, index-related scores from the IPN and ELM bias the output logits of a frozen SLR system (CSLR2), boosting pointing tokens at detected loci and enforcing referential consistency across linked mentions.

Empirical Findings

The framework is trained and evaluated on BSLCP, MDGS, and BOBSL datasets, spanning multiple sign languages and annotation conventions. Quantitative results highlight extreme under-performance of state-of-the-art SLR models on index recovery under standard evaluation protocols: WER on index labels (WER$_\text{Index}$) is 91.3–98.4%, with only 6.1–15.8% segment overlap with gold reference index events under default settings. This substantiates the claim that legacy gloss-centric models do not capture productive spatial reference structure, even when overall WER remains competitive.

Augmenting a frozen CSLR2 model with the IPN+ELM pipeline produces a significant reduction in index-specific errors: WER$_\text{Index}$ is cut from 96.2% to 59.6% (a 36.6 point drop), with overall WER$_\text{All}$ reduced from 70.5% to 68.3%. Importantly, these gains are realized without any degradation to lexical accuracy (WER$_\text{Lex}$ remains stable at 77.1% vs. 77.5%). This demonstrates that targeted structured modeling and referential tracking can be deployed as modular plug-ins, substantially enhancing discourse-level behavior without retraining core recognizers.

The Sankey flow analysis further reveals robust improvements in referential accuracy for pointing tokens, with correct referential flows increasing from 6.1% (baseline) to 61–71% (with IPN+ELM) Figure 2.

Figure 3: Coreference and downstream SLR evaluation demonstrates substantial improvements in both entity clustering accuracy and index-specific WER.

Figure 2: Flow-based analysis traces ground-truth pointing tokens to the correct referential categories; the IPN+ELM system achieves markedly improved precision and recall for pronominal, locative, and deictic classes.

Figure 4: Qualitative visualization showing the system’s ability to distinguish and track multiple co-occurring referents across sentences, surpassing the baseline’s entity resolution.

Qualitative Error Analysis

Representative failure cases for the IPN Figure 5 identify two dominant error modes: (i) systematic false positives on lexical signs exhibiting index-like handshapes (e.g., what, why) in the absence of discourse context, and (ii) false negatives due to coarticulation, occlusion, or temporal imprecision, which prevent the model from isolating indexical constituents embedded within complex gestural sequences.

Figure 5: False positive and false negative samples highlight the ambiguity between index-like lexical gestures and bona fide spatial indexing.

The ELM accurately groups coreferential mentions—even in the presence of substantial label noise and segmentation errors—and maintains temporally persistent entity states even over long-range discourse gaps (Figures 7–9).

Figure 6: Sequence of correctly grouped self-referential ("me/I") index signs, affirming robustness to annotation noise.

Figure 7: Clustering remains consistent despite coverage gaps and missing ground-truth labels.

Figure 8: Simultaneous tracking of multiple referents across a discourse window—entity maintenance and separation beyond the baseline’s capabilities.

Implications and Future Directions

This work asserts that indexing is a recoverable, technology-agnostic structural layer that should be decoupled from lexicon-dominated recognition objectives. The system demonstrates that pose-based, modular referential modeling is both effective and portable—generalizing across multiple sign languages and supporting integrated operation as a plug-in rather than requiring expensive end-to-end retraining. Moreover, the approach establishes a new baseline (and evaluation protocol) for future index-aware SLR research.

The LLM-based automatic generation of entity-cluster supervision is a practical advance, enabling large-scale referential modeling even in resource-scarce settings. The results strongly suggest that future SLR/SLT architectures should integrate explicit structured discourse components for productive spatial phenomena (e.g., classifiers, depiction, role-shift), potentially in joint training scenarios.

The framework is limited by cross-domain label drift (as annotation conventions and discourse strategies differ between conversational data and interpreted broadcasts), by the reliance on precision-calibrated thresholds for downstream integration, and by the lack of ground-truth referential type labels in major benchmarks. These are natural directions for dataset and evaluation protocol refinement.

The pipeline paradigm adopted here would likely benefit (in future) from tighter coupling with attention mechanisms (e.g., for visual-linguistic context), deeper integration of non-manual features (gaze, face, torso), and more sophisticated joint learning of referential representations. Additionally, such structured modeling could readily inform or constrain sign language generation systems, enhancing the quality of output in both synthetic avatar and translation contexts.

Conclusion

The work delivers a system-level framework for spatial grammar and index resolution in SLR, demonstrating that index detection and referential clustering can be robustly modeled with pose-based methods and LLM-augmented supervision. Substantial index recovery is achieved even for frozen SLR backbones, and the pipeline outperforms prior methods in both accuracy and architectural flexibility. The modular design, principled evaluation protocols, and scalable training regimen provide a strong foundation for future research in discourse-aware sign language technologies.