Phonological Perception of Sign Language Models

Abstract: Sign languages are compositional systems where meaning arises by combining sublexical phonological parameters, such as handshape, location, and movement. While deep learning models for Sign Language Recognition (SLR) have achieved increased performance on translation benchmarks, it remains unclear whether these models distinguish abstract phonological features or merely rely on low-level statistical correlations. This work evaluates the phonological perception of SLR models trained on American Sign Language (ASL) by probing phonological sensitivity using minimal pairs and evaluating representational alignment with human behavioral data. Our results reveal that SLR models exhibit emergent phonological sensitivity, but with clear architectural trade-offs: pose-based models are sensitive to handshape contrasts, while pixel-based models better capture location changes. Furthermore, pose-based models learn latent representations that correlate with human perceptual similarity judgments (r~0.49). These findings suggest that while SLR models exhibit emergent phonology, current training paradigms are insufficient to scale them beyond their architectural inductive biases.

• The paper demonstrates that SLR models can internalize sublexical phonological features using a minimal pair probing framework.
• It compares pose-based (STGCN) and pixel-based (I3D) architectures, showing STGCN's superior handshape discrimination and I3D's advantage in spatial contexts.
• Results highlight that domain-specific ASL training boosts model generalization and alignment with human perceptual hierarchies.

Phonological Sensitivity in Sign Language Recognition Models: Probing Emergence and Alignment with Human Perception

Introduction

This paper investigates the emergence and structure of phonological knowledge in deep learning models for Sign Language Recognition (SLR), focusing on whether SLR systems internalize sublexical phonological features such as handshape, location, and movement. The work challenges the adequacy of conventional SLR metrics—primarily translation accuracy—by analyzing to what extent internal model representations encode meaningful phonological structure, as opposed to capturing spurious statistical cues. The authors propose a probing framework using minimal pairs to directly evaluate model sensitivity to distinct phonological contrasts. The study facilitates architectural comparisons and alignment with human perceptual data, highlighting emergent abstraction and domain limitations.

Methodology

The analysis employs a dataset of minimal pairs sourced from multiple American Sign Language (ASL) corpora: ASL Citizen (natural, crowd-sourced), Sem-Lex (elicited with concept prompts for lexical variability), and Handshapes in Context Stimuli (HCS; systematically controlled nonce stimuli). Minimal pairs differ by just one phonological parameter, allowing isolation and evaluation of model sensitivity to individual contrasts.

Models are evaluated on two main architectures:

1. I3D: A pixel-based Inflated 3D ConvNet operating over RGB video.
2. STGCN: A pose-based Spatio-Temporal Graph Convolutional Network ingesting skeletal pose graphs.

Models are initialized randomly, pre-trained on generic action datasets (Kinetics), or trained explicitly on ASL Citizen, allowing exploration of the effects of training domain and architecture on latent phonological differentiation.

Phonological Sensitivity Probing Framework

Phonological sensitivity is measured by extracting latent representations from the penultimate layer and computing cosine similarity distributions:

• Intra-sign similarity: The similarity between two videos of the same sign by different signers.
• Inter-sign similarity: The similarity between two minimal pair signs (differing by one phonological parameter).

A t-test assesses whether intra-sign similarities are statistically higher than inter-sign, with significance interpreted as a model's capacity to abstract the relevant phonological parameter independent of extraneous signer-specific or environmental features.

Figure 1: Phonological sensitivity auditing is operationalized by extracting penultimate-layer features and measuring the cosine similarity across minimal pairs.

Empirical Results

Architectural Trade-offs

Models trained on general action datasets show negligible phonological sensitivity, while ASL-trained models demonstrate robust differentiation of minimal pairs. However, architectural modality induces trade-offs:

• Pose-based (STGCN): Exhibits superior discrimination for handshape and movement; 86.1% win rate for handshape versus I3D's 69.3%.
• Pixel-based (I3D): Outperforms STGCN on location contrasts; 80.7% head-to-head win rate for location.

  Figure 2: Example minimal pairs across handshape and location; lower cosine similarity implies greater model sensitivity. Statistical significance (t-test) and domain transfer are visualized.

The pose/skeletal prior in STGCN aligns feature learning to articulatory geometry, hence excelling at handshape-specific nuances, while I3D retains spatial context from raw video frames, advantageous for location detection.

Alignment with Human Perception

Model latent spaces are correlated against human pairwise handshape confusion matrices, articulatory handshape distances, and the LBB2 theoretical model developed from human psycholinguistic studies:

• STGCN-ASL: Achieves the highest correlation with human signers ($r \approx 0.49$), closely mirroring the handshape distance metric ($r \approx 0.55$).
• I3D-ASL: Shows moderate correlation ($r \approx 0.31$) but diverges in the structural organization of representations.

  Figure 3: Hierarchical clustering of handshape spaces for the LBB2 perceptual model, geometric handshape distance, I3D-ASL, and STGCN-ASL, revealing topological alignment and divergence.

Hierarchical clustering on the pose-based model reveals latent spaces structured along linguistically and perceptually salient axes (compactness, finger extension), replicating both theoretical and empirical human hierarchies. Conversely, I3D organizes clusters along global visual texture, failing to recover fine-grained distinctions critical to linguistic analysis.

Domain Robustness and Generalization

Despite high intra-domain phonological sensitivity, out-of-domain performance via Sem-Lex reveals substantial drops: I3D-ASL and STGCN-ASL win rates shrink to 45.1% and 49.3%, respectively. Thus, current SLR model representations do not generalize robustly to domain shifts, in contrast to human visual perception which exhibits considerable invariance to production variability.

Figure 4: Comparative visualization of input modalities—the pixel-based I3D (left) and pose-based STGCN (right)—highlighting inductive biases and information bottlenecks in SLR pipelines.

Implications for SLR and Linguistic Representation

The findings articulate clear limitations for using SLR models as general-purpose tools in sign language and cross-linguistic research. Emergent phonological abstraction is contingent on both architecture and tight domain-specific alignment; generic action recognition data does not induce robust linguistic representations. Architectural priors induce measurable and predictable biases. While pose-based models align more closely with human-like abstraction for handshape and articulatory geometry, they underperform on spatial parameters. Pixel-based models maintain spatial differentiation but conflate fine-grained articulatory details.

The authors recommend future models integrate both modalities (dual-stream architectures) to capture the full range of sublexical phonological features. The dataset of minimal pairs curated here offers both a probing mechanism and a source for hard negative sampling in contrastive or self-supervised training, potentially enabling better phonological generalization and zero-shot transfer across sign languages.

The study also foregrounds the need for evaluation metrics beyond translation accuracy, especially in low-resource settings where phonological abstraction and robustness to spurious cues are prerequisites for practical application.

Limitations and Future Work

Three principal limitations are identified:

1. Analysis is limited to handshape, location, and movement due to data sparsity for orientation and non-manual markers.
2. Only supervised, open-weight baseline architectures were evaluated; future work should assess self-supervised and state-of-the-art proprietary systems.
3. Focus is on isolated sign recognition; continuous sign streams and co-articulation effects are unexplored.

Further, the impact of contrastive and compositional pre-training, multimodal fusion, and transfer across unrelated sign languages remain open avenues for substantive investigation.

Conclusion

Through a minimal pair probing framework and architectural analysis, this paper demonstrates that current SLR models exhibit emergent but architecture-contingent phonological perception. Pose- and pixel-based SLRs differ systematically in their representational strengths and model-induced perceptual hierarchies. There is significant, though limited, alignment with human perception in pose-based models, but both architectural class and data domain substantially constrain generalization and robustness. This work establishes a methodological foundation for more cognitively aligned, linguistically grounded, and transferable SLR systems and highlights the necessity of explicit phonological evaluation in the ongoing development of sign language technology.