AulSign: LLM-Driven Sign Language Translation

• AulSign is a method for translating spoken or written language into sign language via symbolic encodings and canonical descriptions.
• It uses a three-stage pipeline—retrieval, dynamic LLM prompting, and a sign mapper—to link natural language with structured sign representations.
• The framework excels in low-resource settings, showing significant gains in metrics like BLEU, F1, and ChrF2 for both ASL and LIS.

AulSign is a method for sign language translation in low-resource settings that uses LLMs through dynamic prompting, in-context learning with sample selection, and a subsequent sign association stage rather than direct generation of sign encodings. In its explicit formulation, the name denotes Advanced Use of LLMs for Sign Language Translation, a framework designed for translation between spoken or written languages and sign languages represented symbolically—primarily Formal SignWriting (FSW)—and evaluated on English/ASL and Italian/LIS data (Bulla et al., 25 Aug 2025).

1. Definition and representational basis

AulSign addresses translation between spoken or written natural languages and sign languages by working over symbolic sign encodings rather than raw video. The method is designed to operate with structured sign representations including SignWriting / FSW, HamNoSys, and SMPL-X pose models, although the reported experiments use FSW because it is linear and symbolic, encodes symbols plus positional factors and structure, and is already used computationally in sign-language resources (Bulla et al., 25 Aug 2025).

The underlying problem formulation is explicitly low-resource. Parallel corpora aligning spoken language with sign-language data are scarce; existing resources are often domain-specific, inconsistently annotated, and linguistically incomplete. AulSign therefore does not assume that an LLM can directly generate sign encodings. The paper states that LLMs have no intrinsic knowledge of sign languages or of encodings such as FSW and HamNoSys, so the method introduces an intermediate textual layer: canonical descriptions, defined as compact, expressive, natural-language descriptions intended to map uniquely to sign entries (Bulla et al., 25 Aug 2025).

This representation is formalized through a sign vocabulary $V_s$ and a canonical-description vocabulary $V_c$. Equivalent signs are merged by an equivalence operator $\equiv$, defined so that two signs are considered equivalent if they contain the same set of symbols with the same orientation and rotation. From the available textual descriptions for a sign, the most frequent description is selected or combined into a single string, yielding a canonical description intended to refer uniquely to one sign entry. Training sentences are then converted into sequences of canonical descriptions through a mapping

$CS : D \to (V_c \cup \{\texttt{<unk>}\})^\*,$

where $D$ is the sentence corpus and $CS(d)$ is the canonical-description sequence associated with sentence $d$ (Bulla et al., 25 Aug 2025).

The canonical-description layer is central to AulSign’s design. It functions as a controlled pseudo-language between natural language and sign encoding. This distinguishes the method from gloss-only pipelines: the intermediate units are intended to be more semantically precise than gloss labels while remaining textual enough for direct use with general-purpose LLMs (Bulla et al., 25 Aug 2025).

2. Core pipeline: retrieval, prompting, and sign association

AulSign has three principal components: a Retriever, an LLM, and a Sign Mapper. In the spoken-to-sign direction, an input sentence $s$ is first embedded with a contextual embedder $e(\cdot)$, and semantically similar examples are retrieved from the training corpus $D$ and from the canonical-description vocabulary $V_c$0 using cosine similarity (Bulla et al., 25 Aug 2025).

For sentence retrieval, the similarity score is

$V_c$1

and the top-$V_c$2 examples are selected: $V_c$3 For vocabulary retrieval, the same mechanism is applied to canonical descriptions: $V_c$4 followed by

$V_c$5

These retrieved items are used to construct a dynamic prompt containing grammar rules, few-shot examples, and relevant canonical descriptions (Bulla et al., 25 Aug 2025).

The LLM then generates a pseudo-canonical output sequence

$V_c$6

where each $V_c$7 is a textual string resembling a canonical description. AulSign does not require the LLM to generate FSW directly. Instead, each generated description is mapped back to the nearest canonical description in $V_c$8 through semantic similarity: $V_c$9 The paper decomposes this as a text-to-canonical map

$\equiv$0

followed by a canonical-to-sign map

$\equiv$1

so that the overall sign-association function is

$\equiv$2

The final output is therefore a sequence of sign entries, typically serialized in FSW (Bulla et al., 25 Aug 2025).

The reverse direction, sign-to-spoken translation, mirrors this structure. An input sign sequence is first converted to canonical descriptions through the sign vocabulary and the equivalence relation, similar examples are retrieved, and a prompt is built that asks the LLM to translate the canonical-description sequence into spoken language. In this direction, no Sign Mapper is needed because the target is ordinary text (Bulla et al., 25 Aug 2025).

3. Datasets, tasks, and reported performance

AulSign is evaluated on SignBank+ for English/ASL and on the LaCAM CNR-ISTC corpus for Italian/LIS. In SignBank+, the English subset used in the experiments contains 43,705 annotated items, 19,304 unique signs, and 13,631 sentences. Three training configurations are reported: Config I with 13,275 sentences, Config II with 2,301 sentences, and Config III with 115 sentences. The test set contains 356 sentences. The baseline is the factorized Transformer model of Jiang et al. (2022), which predicts FSW symbols and positional factors (Bulla et al., 25 Aug 2025).

For spoken-to-sign evaluation, the reported metrics are F1-score at the symbol level, BLEU, ChrF2, and MAE for positional coordinates. For sign-to-spoken evaluation, the metrics are BLEU and ChrF2 (Bulla et al., 25 Aug 2025).

Before the tables, two empirical patterns are central. First, AulSign is strongest in low-resource conditions, especially on the 115-sentence ASL configuration and on the already small LIS corpus. Second, on the largest English configuration, the factorized baseline remains slightly ahead, which the paper associates with incomplete vocabulary coverage and annotation noise rather than with a collapse of the method itself (Bulla et al., 25 Aug 2025).

Setting	Baseline	AulSign
ASL text-to-sign, Config III (115 sentences)	F1 0.27; BLEU 10.94; ChrF2 39.17	F1 0.37; BLEU 18.79; ChrF2 53.91
ASL text-to-sign, Config II (2,301 sentences)	F1 0.40; BLEU 22.68; ChrF2 50.65	F1 0.41; BLEU 23.96; ChrF2 56.06
ASL text-to-sign, Config I (13,275 sentences)	F1 0.45; BLEU 29.26; ChrF2 57.82	F1 0.42; BLEU 25.40; ChrF2 56.44
LIS text-to-sign	F1 0.50; BLEU 16.40; ChrF2 45.18; MAE X 23.82; MAE Y 37.44	F1 0.63; BLEU 37.71; ChrF2 57.54; MAE X 21.22; MAE Y 33.22

Setting	Baseline	AulSign
ASL sign-to-text, Config III	BLEU 2.20; ChrF2 18.90	BLEU 26.59; ChrF2 40.76
ASL sign-to-text, Config II	BLEU 8.40; ChrF2 22.50	BLEU 24.75; ChrF2 40.26
ASL sign-to-text, Config I	BLEU 18.40; ChrF2 34.40	BLEU 23.38; ChrF2 39.21
LIS sign-to-text	BLEU 6.50; ChrF2 29.40	BLEU 17.95; ChrF2 50.42

The low-resource behavior is particularly pronounced in ASL Config III, where AulSign improves spoken-to-sign performance from F1 = 0.27, BLEU = 10.94, ChrF2 = 39.17 to F1 = 0.37, BLEU = 18.79, ChrF2 = 53.91, and sign-to-text performance from BLEU = 2.20, ChrF2 = 18.90 to BLEU = 26.59, ChrF2 = 40.76 (Bulla et al., 25 Aug 2025). On LIS, the method improves both symbolic quality and positional accuracy, with MAE X = 21.22 and MAE Y = 33.22 against the baseline’s 23.82 and 37.44 (Bulla et al., 25 Aug 2025).

4. Methodological properties and ablation findings

The ablations reported for AulSign focus on the embedder, the LLM, and the stability of the retrieval-plus-mapping mechanism. For ASL in the 115-sentence low-resource setting, varying the embedder among MPNet, MiniLM, and Mxbai produces only modest changes; the paper reports standard deviations < 2\%, indicating that the method is relatively robust to the choice of sentence embedder (Bulla et al., 25 Aug 2025).

The LLM ablation compares LLaMA-3 70B, GPT-3.5, and GPT-4.1-nano. The reported result is that GPT-3.5 gives best performance, both for text-to-sign and sign-to-text, although the differences are described as modest but consistent. This is methodologically significant because it suggests that AulSign’s gains are not reducible to a single frontier model; the retrieval, dynamic prompting, and sign-association stages contribute substantially to performance (Bulla et al., 25 Aug 2025).

AulSign’s explanation mechanism is also unusually explicit. Because the LLM outputs canonical descriptions rather than raw FSW, intermediate representations remain inspectable. The paper emphasizes that these sequences can be examined directly to identify whether an error arose from retrieval, semantic paraphrasing, or sign association. This suggests a level of diagnosability that is uncommon in end-to-end symbolic sign-language pipelines, although that implication is interpretive rather than a separately benchmarked metric (Bulla et al., 25 Aug 2025).

The method is nevertheless structurally constrained. The Sign Mapper performs nearest-neighbor matching in embedding space, and the paper identifies this as a rigidity: the one-to-one mapping between canonical description and sign is useful for precision, but it can also limit flexibility when the LLM outputs semantically adjacent phrasing or when corpora contain incomplete sign coverage (Bulla et al., 25 Aug 2025).

5. Position within sign-language AI and nomenclatural ambiguity

AulSign occupies a specific niche within sign-language AI: low-resource translation over symbolic sign encodings. This differs from video-first continuous sign language recognition systems such as ArabSign, which provides a Kinect-based continuous Arabic Sign Language dataset and encoder–decoder baselines for sentence-level recognition from RGB, depth, and skeleton data (Luqman, 2022), from AutoSign, which performs direct pose-to-text translation for Saudi Sign Language with a decoder-only transformer and no CTC/HMM alignment stage (Johnny et al., 26 Jul 2025), and from Uni-Sign, which treats multiple sign-language-understanding tasks as generative sign-to-text problems over pose+RGB inputs (Li et al., 25 Jan 2025).

The contrast is methodological rather than merely architectural. ArabSign, AutoSign, and Uni-Sign all operate on video or pose streams and treat recognition or translation as sequence modeling over spatiotemporal inputs. AulSign instead externalizes lexical knowledge into a sign vocabulary $\equiv$3, a canonical-description vocabulary $\equiv$4, retrieval indices, and an explicit Sign Mapper. It therefore shifts part of the problem from visual modeling to controlled text generation and lexical association (Bulla et al., 25 Aug 2025).

There is also a terminological complication around the name itself. In adjacent discussions, AutoSign is described as “also referred to as AulSign in your query,” and a separate Chinese bidirectional sign-language platform is described as “AulSign, as embodied by AuralLLM + SignMST-C + CNText2Sign + CNSign” (Johnny et al., 26 Jul 2025, Li et al., 1 Jan 2025). This suggests a degree of nomenclatural ambiguity in recent usage. The explicit expansion “Advanced Use of LLMs for Sign Language Translation”, however, belongs to the symbolic low-resource framework described above (Bulla et al., 25 Aug 2025).

6. Limitations, future directions, and significance

AulSign’s main limitations are openly identified. The method depends on a predefined sign vocabulary $\equiv$5 and canonical-description vocabulary $\equiv$6; signs outside the vocabulary are mapped to \texttt{<unk>}, which constrains coverage. The approach also inherits annotation problems from its source corpora: incomplete or inconsistent positional factors can reduce accuracy, and the large English configuration is specifically described as noisier, which weakens AulSign’s relative advantage there (Bulla et al., 25 Aug 2025).

The intermediate representation is also only partially expressive. Although the framework is designed to support FSW, HamNoSys, and SMPL-X, the experiments are confined to FSW, and non-manual markers are not explicitly modeled. The paper notes that non-manual features and finer phonological detail are not fully exploited. Likewise, the Sign Mapper relies on nearest-neighbor embedding similarity rather than grammar-constrained decoding, and the paper explicitly points to GRAMMAR-LLM as a possible future direction for more constrained sign-sequence generation (Bulla et al., 25 Aug 2025).

The proposed extensions are correspondingly broad: expanding vocabularies, integrating multimodal data such as video and pose, using SMPL-X for avatar generation, extending to additional sign languages and domains, and improving the treatment of non-manual features and discourse phenomena. The paper also frames AulSign in ethical terms: translation quality and reliability matter because mistranslation can exclude or misinform users; corpus bias can privilege specific dialects or domains; and vocabulary and rule design should involve Deaf signers and linguists (Bulla et al., 25 Aug 2025).

Within the sign-language literature, AulSign is therefore best understood as a retrieval-augmented, LLM-mediated translation framework for symbolic sign representations, especially valuable when aligned video–sign corpora are too small or too noisy to support purely neural end-to-end training. Its central claim is not that LLMs understand sign language natively, but that they can be made operationally useful by interposing a controlled textual layer of canonical descriptions and by grounding those descriptions in an external sign lexicon (Bulla et al., 25 Aug 2025).