ASL Minimal Translation Pairs Benchmark

• ASL Minimal Translation Pairs (ASL-MTP) is a benchmark comprising 1,275 ASL videos paired with two grammatical English translations that differ only by targeted linguistic cues.
• The benchmark is designed to diagnose model sensitivity by contrasting matched and mismatched translations based on manual, non-manual, or combined articulatory channels.
• Empirical evaluations with SHUBERT+ByT5 indicate strong reliance on manual cues while exposing model weaknesses in detecting non-manual markers, such as eyebrow raise in polar questions.

ASL Minimal Translation Pairs (ASL-MTP) is a benchmark for targeted linguistic analysis of American Sign Language (ASL) models, introduced to evaluate whether ASL-to-English translation systems use the specific ASL cues that distinguish minimally differing translations (Karabüklü et al., 29 Apr 2026). Each item pairs a single ASL video with two grammatical English sentences: a matched translation that correctly reflects the video and a mismatched translation that differs only in the cue or cues relevant to a specific linguistic phenomenon. Built from the American Sign Language Linguistic Research Project (ASLLRP), ASL-MTP contains 1,275 ASL videos organized into 9 phenomenon categories and is intended as an evaluation-only benchmark for sign-conditioned minimal pair analysis (Karabüklü et al., 29 Apr 2026).

1. Conceptual basis

ASL-MTP is defined by a contrast between translations conditioned on a fixed ASL video. This distinguishes it from traditional linguistic minimal pairs, which typically contrast acceptability in written language, with one sentence grammatical and the other ungrammatical. In ASL-MTP, both English sentences are grammatical; only one is the correct translation of the ASL input. The contrast therefore isolates whether a model uses the ASL cue or cues that distinguish the two sentences (Karabüklü et al., 29 Apr 2026).

The benchmark is motivated by the linguistic structure of sign languages, which use multiple articulatory channels. The paper distinguishes manuals, including handshape, orientation, and movement, from non-manuals, including eyebrows, head movements, and mouth actions. It further emphasizes that some phenomena depend on combinations of these channels, as in Wh-questions and conditionals. ASL-MTP is designed to diagnose whether translation models capture such phenomena and whether they are sensitive to hands, face, and upper body in a phenomenon-specific manner (Karabüklü et al., 29 Apr 2026).

The notion of minimality is operationalized narrowly. A single, phenomenon-targeted cue difference should flip which translation matches the video. The minimal contrast can be manual only, non-manual only, or combined. This makes the benchmark suitable for targeted analysis of cue use rather than general translation quality (Karabüklü et al., 29 Apr 2026).

2. Source corpus, construction, and scope

ASL-MTP is drawn from ASLLRP, described as a corpus of 2,048 linguistically annotated ASL utterances produced by 4 signers (Karabüklü et al., 29 Apr 2026). ASLLRP provides raw, high-quality ASL videos with annotations for manual signs and time-aligned non-manuals, including head position or movement, mouth movement, and eye gaze, as well as grammatical functions such as classifier, question, and conditional. Frame rate and resolution are not specified in the paper (Karabüklü et al., 29 Apr 2026).

Items were selected by querying ASLLRP for target phenomena using its linguistic annotations. The reported procedures include the use of fingerspelling glosses such as #A-N-N, classifier prefixes such as cl:, and explicit negation markers. For polar questions, items ending with ? were manually verified to ensure that they are true polar questions marked by non-manual eyebrow raise. Matched and mismatched English sentences were then constructed by minimally altering the translation, for example by replacing a number, substituting a Wh-word, switching if to when, adding or removing negation, or rewriting a question as a declarative and vice versa (Karabüklü et al., 29 Apr 2026).

The paper does not report new annotation layers, inter-annotator agreement, or coder-specific guidelines beyond use of the ASLLRP annotations and manual verification for polar questions. It also does not define official train, development, or test splits for ASL-MTP. The benchmark is instead organized into phenomena-specific subsets, with some phenomena further divided into manual plus non-manual versus non-manual-only cases (Karabüklü et al., 29 Apr 2026).

3. Phenomenon inventory

ASL-MTP spans nine phenomena designed to cover manual, non-manual, and combined cue configurations. Representative contrasts include numeral substitution, fingerspelled name substitution, classifier-based plurality contrasts, Wh-word substitution, polarity reversal through negation, conditional marking, and declarative versus polar-question alternations (Karabüklü et al., 29 Apr 2026).

Phenomenon	Count
Numbers	119
Fingerspelling	170
Classifiers	150
Wh-Questions	123
Negation vs Positive	104
Positive vs Negation	104
Conditionals	205
Declaratives vs Polar Questions	150
Polar Questions vs Declaratives	150

The phenomenon definitions are linguistically specific. Numbers are manual-primary and test whether substitution of a numeral alters the translation. Fingerspelling is also manual-primary and targets substitution of a fingerspelled proper noun or term. Classifiers are manual-primary and often context-dependent; the paper’s example uses classifier handshape to encode plurality. Wh-Questions depend on a Wh-sign and non-manuals such as brow lowering and head tilt or shake. Negation versus positive contrasts rely on the [NOT](https://www.emergentmind.com/topics/neural-organ-transplantation-not) sign and headshake, while positive versus negation tests incorrect insertion of a negation word. Conditionals use IF plus eyebrow raise, with head thrust and body forward as secondary cues, and include a non-manual-only subset of approximately 50 items. Declaratives versus polar questions and the inverse contrast are described as predominantly or exclusively non-manual, with eyebrow raise as the primary cue for interrogativity (Karabüklü et al., 29 Apr 2026).

This organization makes the dataset useful for evaluating whether a model succeeds because it detects the intended linguistic marker rather than because it exploits broader lexical or syntactic regularities. A plausible implication is that the dataset is especially informative for studying channel sensitivity in multi-stream sign LLMs, because its subsets are already aligned to distinct articulatory sources.

4. Evaluation methodology

The benchmark uses sign-conditioned minimal pair scoring via surprisal. For a phenomenon-specific dataset $D = \{(F_i, a_i, u_i)\}$, where $F_i$ is the ASL video input, $a_i$ the matched translation, and $u_i$ the minimally differing mismatched sentence, the per-token surprisal of a sentence $s_i = (x_1, \ldots, x_{|s_i|})$ conditioned on $F_i$ is defined as

$$S(s_i) = - (1/|s_i|) \sum_{t=1}^{|s_i|} \log p(x_t \mid x_{<t}, F_i)$$

The surprisal difference is

$\Delta \mathrm{Surprisal}_i = S(u_i) - S(a_i)$

and accuracy is

$$\mathrm{Accuracy} = (\# \text{ of pairs with } \Delta \mathrm{Surprisal}_i > 0) / (\# \text{ total pairs})$$

Chance level is 50% because the comparison is pairwise (Karabüklü et al., 29 Apr 2026).

The statistical protocol uses a two-tailed exact binomial test with Bonferroni correction to compare accuracies between conditions at $p < .05$. Mean $F_i$0 is reported with 95% confidence intervals, and differences versus the “All Cues” condition are tested for significance. The paper also computes BLEURT, described as a learned reference-based similarity metric, between model outputs and references. BLEURT is reported as uninformative for fine-grained phenomenon sensitivity: scores vary little across phenomena and mostly drop when hands or body are removed, failing to reveal the non-manual cue issues diagnosed by the minimal pair analysis (Karabüklü et al., 29 Apr 2026).

In practical terms, the protocol asks whether the model assigns lower surprisal to the matched translation than to the minimally mismatched one, given the same ASL video. This directly tests sign-conditioned discrimination rather than free-form generation quality.

5. Case study with SHUBERT+ByT5

The paper’s case study applies ASL-MTP to SHUBERT+ByT5, described as a state-of-the-art, publicly available ASL-to-English translation system (Karabüklü et al., 29 Apr 2026). SHUBERT is a self-supervised video encoder pretrained on 1,000 hours of ASL YouTube videos from YouTube-ASL and YouTube-SL-25 via multi-stream masked cluster prediction. Its inputs are decomposed into face crops, represented by DINOv2 image features for eyes, eyebrows, and mouth; left and right hand crops, also represented with DINOv2 features; and body pose keypoints. The decoder is ByT5-Base, jointly fine-tuned with SHUBERT on ASL-to-English translation using next-token prediction, which makes token-level probabilities available for surprisal analysis (Karabüklü et al., 29 Apr 2026).

The fine-tuning regimen proceeds in two stages. First, the model is fine-tuned on approximately 800K weakly aligned ASL-English pairs from the union of YouTube-ASL and the ASL portion of YouTube-SL-25. It is then fine-tuned on approximately 200K better-aligned pairs from How2Sign, ASL Stem Wiki, and OpenASL (Karabüklü et al., 29 Apr 2026).

Inference-time cue ablations are performed under eight conditions, with regions masked using MediaPipe keypoints: All Cues (AC), No Eyes & Brows (NE), No Mouth (NM), No Face (NF), No Hands (NH), No Hands & Mouth (NHM), No Hands & Face (NHF), and No Face & Body (NFB). The paper also reports training-time, or “controlled rearing,” ablations in which two variants are trained from scratch: NF, with no face throughout training, and NFB, with only hands retained (Karabüklü et al., 29 Apr 2026).

The reported purpose of these interventions is to determine how much the model depends on manual versus non-manual channels during both training and inference. This setup makes ASL-MTP a diagnostic framework for channel-specific failure analysis rather than only a benchmark for aggregate performance.

6. Empirical findings, limitations, and implications

The overall pattern is that the model performs above chance on 8 of 9 phenomena, performs poorly on Polar Questions vs Declaratives, and exhibits a strong declarative bias, including high performance on Declaratives vs Polar Questions (Karabüklü et al., 29 Apr 2026). The paper reports strong reliance on manual cues for manual-primary phenomena. For Numbers, accuracy is 0.82 in the All Cues condition and drops under hand ablation, including 0.61 for NH, 0.58 for NHM, and 0.55 for NHF. For Fingerspelling, AC is 0.78, hand-ablated conditions fall to 0.49 for NH, 0.46 for NHM, and 0.43 for NHF, while NFB reaches 0.68. For Classifiers, AC is 0.63, with NH at 0.53, NHM at 0.51, NHF at 0.48, and NFB at 0.53 (Karabüklü et al., 29 Apr 2026).

Results for mixed manual and non-manual phenomena are less uniform. Wh-Questions reach 0.75 under AC and show modest reductions with hands ablated, including 0.66 for NH, 0.65 for NHM, and 0.67 for NHF; removing face does not consistently reduce accuracy. Negation vs Positive reaches 0.80 under AC and declines to approximately 0.71 to 0.69 under NH, while showing less sensitivity to face removal, with NF, NM, and NE around 0.77. Positive vs Negation starts at 0.65 under AC and shows a severe drop under hand-ablated conditions, down to approximately 0.28 to 0.34. Conditionals score 0.70 under AC and 0.56 under NH, indicating that hands matter, whereas the non-manual-only conditional subset reaches 0.68 under AC and removing hands does not significantly hurt, with NH around 0.60 to 0.62 (Karabüklü et al., 29 Apr 2026).

The clearest non-manual failure concerns polar questions. For Declaratives vs Polar Questions, the model shows a strong declarative bias; in the controlled-rearing experiment, AC is 0.97, NF is 0.96, and NFB is 0.95. For Polar Questions vs Declaratives, including the non-manual-only subset, performance is near or below chance in all conditions, with AC approximately 0.09, which the paper interprets as poor sensitivity to eyebrow raise marking interrogatives (Karabüklü et al., 29 Apr 2026). BLEURT correlates poorly to moderately with surprisal-based accuracy, including $F_i$1 for Polar Questions vs Declaratives and $F_i$2 for Numbers, reinforcing the claim that minimal pairs provide more informative linguistic diagnostics (Karabüklü et al., 29 Apr 2026).

Training-time ablations largely mirror inference-time findings, and the paper concludes that train-test distribution mismatch does not explain the ablation results. Notable exceptions indicate more complex interactions. For Conditionals with manual and non-manual cues, AC is 0.70, NF is 0.89, and NFB is 0.61. For Wh-Questions, AC is 0.75, NF is 0.54, and NFB is 0.63. The paper does not present these differences as a consistent improvement pattern across phenomena (Karabüklü et al., 29 Apr 2026).

The limitations are explicit. ASL-MTP contains 1,275 items across 9 phenomena; some subsets, including the non-manual-only conditionals subset of approximately 50 items, may limit statistical power. The source corpus includes 4 signers, and broader demographic diversity is not described. Camera details, frame rate, and resolution are unspecified. The paper also notes that some cues, such as mouthing used for disambiguation, may not be uniformly annotated in ASLLRP, which can influence analyses of non-manual sensitivity (Karabüklü et al., 29 Apr 2026).

The stated implications are methodological and modeling-oriented. ASL-MTP is presented as the first phenomena-targeted minimal translation pair benchmark for sign LLMs. The results indicate that a state-of-the-art multi-channel model leverages manual cues strongly and recognizes some combined cues, yet fails to reliably use non-manual-only cues, especially eyebrow raise in polar questions. The paper recommends strengthening multi-articulator modeling, particularly for eyebrows, head, and mouth, and suggests that richer face and upper-body representations, higher-quality keypoints or learned features, and training signals that emphasize non-manual cues could reduce declarative bias and improve question detection. It also recommends future datasets with greater signer diversity, more data volume, and more explicit non-manual annotations, and proposes that semi-automatic discovery of phenomenon-specific subsets could scale ASL-MTP-like benchmarks (Karabüklü et al., 29 Apr 2026).