Hamburg Notation System (HamNoSys)

• Hamburg Notation System (HamNoSys) is a universal, language-independent system that captures sign language phonetics through structured, block-based encoding of handshape, orientation, location, movement, and non-manual features.
• It facilitates cross-linguistic annotation and robust machine learning pipelines by converting signed utterances into standardized glyph sequences suitable for automated processing.
• Challenges include granular hand location, complex movement encoding, and label noise, prompting ongoing research into protocol improvements and computational methods.

The Hamburg Notation System (HamNoSys) is a universal, language-independent, phonetic transcription system for describing the parameter space of sign languages. Developed to provide a comprehensive, structured, and compositional representation of signed utterances, HamNoSys is used extensively in cross-linguistic linguistic research, corpus annotation, machine learning pipelines, and automated sign synthesis. It encodes the manual and non-manual articulatory parameters of signs (handshape, orientation, location, movement, and modifiers) in a standardized block-structured glyph sequence, facilitating both human annotation and computational processing across diverse sign languages (Mocialov et al., 2022, Shalev-Arkushin et al., 2022, Majchrowska et al., 2022, Ferlin et al., 2023).

1. System Structure and Symbolic Inventory

HamNoSys transcribes each sign as a sequence of up to six ordered blocks, each corresponding to a distinct phonological parameter set:

• Symmetry Operator (optional): classifies two-handed signs by the relation between hands (symmetric, mirrored, asymmetric).
• Non-Manual Features (optional): encodes facial expressions and body postures (e.g., eyebrow raise, mouth movements, head tilt).
• Handshape (mandatory): detailed by three sub-blocks—base form (12 primitives), thumb position (4 diacritics), and finger bending (6 diacritics).
• Hand Orientation (mandatory): consists of extended-finger direction (up to 26 compass points) and palm orientation (8 classes).
• Hand Location (mandatory): described as a 3D anatomical position—left/right (x), top/bottom (y, up to 47 bands), and distance from the body (z, 7 granularity levels).
• Movement/Action (mandatory): specifies path shape, directionality, repetition, and manner, with the ability to combine multiple movement primitives.

Each HamNoSys character/glyph is semi-independent and serves as a discrete token. Fully specified signs may represent both hands, each with independent parameter values. The full symbol inventory exceeds 210 glyphs, with major classes mapped to integer values for computational use (Majchrowska et al., 2022, Ferlin et al., 2023).

2. Annotation Protocols and Practical Encoding

In practical annotation, a HamNoSys string is constructed by concatenating the relevant glyphs in a left-to-right order, with each block filling its conceptual slot. For example, a typical sign is encoded as:

$$\left[\text{handshape}\right]\,\left[\text{orientation}\right]\,\left[\text{location}\right]\,\left[\text{movement}\right]\,\left[\text{modifiers}\right]$$

For two-handed signs, separate blocks are maintained per hand, and optional relief markers or additional operators (such as symmetry) are prepended as needed (Shalev-Arkushin et al., 2022, Majchrowska et al., 2022).

Automated parsers convert HamNoSys strings into fixed-length multilabel integer vectors, commonly with 25 components organized as symmetry operator, relaxed-hand flag, dominant and non-dominant hand feature blocks, and global location coordinates. This vectorization is suitable for multi-head supervised classification and as conditioning input for sequence models:

Component	Symbol Set/Range	Function
Symmetry Operator	$\{s_0,\ldots,s_8\}$	Hand coordination
Handshape Base	$\{h_0,\ldots,h_{11}\}$	12 enumerated shapes per hand
Thumb Position	$\{t_0,\ldots,t_3\}$	4 values per hand
Finger Bending	$\{d_0,\ldots,d_5\}$	6 values per hand
Extended-Finger Dir.	$\{e_0,\ldots,e_{17}\}$	18 values per hand
Palm Orientation	$\{p_0,\ldots,p_7\}$	8 values per hand
Location x/y/z	$x_0,\ldots,x_4$ etc.	Up to 5x37x6 values

Examples in (Majchrowska et al., 2022) demonstrate $>93\%$ parsing coverage and $\geq83\%$ backward uniqueness (one-to-one mapping from vector to gloss) in major sign corpora.

3. Machine Learning Integration and Feature Extraction

HamNoSys strings can be used as primary or auxiliary labels in machine learning pipelines for sign language recognition, generation, and translation. Key approaches include:

• Tokenization and Embedding: Each glyph is mapped to an integer token; embeddings (e.g., $D=128$) incorporate positional and step information for downstream transformer models (Shalev-Arkushin et al., 2022).
• Feature Extraction from Video: Automated pipelines extract hand and body keypoints using systems such as OpenPose or Mediapipe; these are normalized for scale and centering (Mocialov et al., 2022, Shalev-Arkushin et al., 2022). Handshape, orientation, and location categories are then binned or regressed directly to HamNoSys-compatible classes.
• Distance Measures: For unsupervised clustering, phoneme-like HamNoSys sequences are compared using weighted Levenshtein distances, where substitution/deletion costs are proportional to symbol distances (e.g., orientation bin difference or location band difference) (Mocialov et al., 2022).

End-to-end models have been built to animate HamNoSys sequences into pose trajectories via iterative transformer-based refinement. Weak supervision is applied by aligning predicted and observed keypoint sequences using normalized dynamic time warping (nDTW-MJE), which improves robustness to missing data and temporal variation (Shalev-Arkushin et al., 2022).

4. Cross-Linguistic and Corpus Applications

HamNoSys is intentionally language-agnostic, supporting corpora in German, Polish, Greek, French, British, and other sign languages (Ferlin et al., 2023, Shalev-Arkushin et al., 2022). Its universality enables:

• Cross-corpus Training: Models can be pre-trained on mixed-language data, with minor degradation on held-out languages (Shalev-Arkushin et al., 2022).
• Gloss Disambiguation: Integer HamNoSys multilabels maintain 83–95% decodability of sign-gloss mappings, despite omitting some movement detail (Majchrowska et al., 2022, Ferlin et al., 2023).
• Corpus Construction: Unsupervised segmentation and phoneme clustering on HamNoSys features allow mining of sign repetitions and structural motifs from "in the wild" videos, bypassing manual gloss annotation (Mocialov et al., 2022).

HamNoSys has been incorporated into several major sign language corpora (e.g., DGS, Dicta-Sign, Polish SL) and is supported by computational tools for parsing, visualization, and numerical encoding (Shalev-Arkushin et al., 2022, Majchrowska et al., 2022).

5. Limitations and Labeling Challenges

Despite its precision, HamNoSys faces several obstacles to fully reliable, large-scale annotation:

• Granularity and Subjectivity: Hand location y-bands (up to 47 categories) and z-distances (up to 7 levels) often exceed the resolution of body tracking systems and inter-annotator agreement; fine distinctions routinely collapse in practice (Ferlin et al., 2023).
• Movement Complexity: The movement block, while formally expressive, is often omitted or simplified in automated parsers due to annotation difficulty and lack of robust video-to-symbol alignment (Majchrowska et al., 2022, Ferlin et al., 2023).
• Initial Frame Selection: There is no standardized protocol for selecting the "initial" pose; annotators may disagree or use diverging heuristics, resulting in label noise for model training (Ferlin et al., 2023).
• Dominant Hand Assignment: Procedures for labeling the signer’s dominant/non-dominant hand are underspecified, complicating automated alignment (Ferlin et al., 2023).
• Label Noise: Variance within HamNoSys label classes (especially positional and movement symbols) leads to class overlap and ambiguity in supervised learning, unless extremely large datasets are available (Ferlin et al., 2023).

Automated parsing retains approximately 90% “decodability,” but much of the movement nuance needed for gloss discrimination is lost when labels are reduced to static pose vectors (Majchrowska et al., 2022, Ferlin et al., 2023).

6. Future Directions

Research recommends several technical and procedural advances to address HamNoSys limitations:

• Coarse-grained Symbol Sets: Empirical reduction of location and movement classes to match the resolution and reliability of pose extraction (Ferlin et al., 2023).
• Standardized Annotation Protocols: Fixing rules for frame selection, hand assignment, and symbol defaults to increase cross-corpus consistency (Ferlin et al., 2023).
• Computer-Aided Annotation: Integrating suggestions from pose tracking to flag inconsistent or ambiguous HamNoSys labels in real time (Ferlin et al., 2023).
• Expanded Non-Manual Feature Sets: Extending HamNoSys encoding and parser support to capture non-manual markers via automated facial keypoint extraction (Mocialov et al., 2022).
• End-to-End and Weakly Supervised Learning: Leveraging transformer architectures and sequence-to-sequence approaches that treat HamNoSys as a subunit language for sign-text translation, with tolerance for inherent label imprecision (Shalev-Arkushin et al., 2022, Ferlin et al., 2023).

A plausible implication is that refining HamNoSys protocols and symbol mappings—and integrating advanced pose estimation—would make high-fidelity, language-agnostic sign corpora feasible at scale, enhancing both linguistic analysis and machine sign recognition.

7. Summary and Significance

HamNoSys represents the most widely used universal sign language notation system for both linguistic and computational research. Its compositional, block-based inventory enables detailed yet cross-linguistically compatible modeling of sign parameters. While current manual and automated labeling workflows face challenges of granularity, subjectivity, and alignment in real-world data, research demonstrates that even pared-down HamNoSys representations support robust machine learning for sign discovery, clustering, and generation. Systematic protocol improvements and feature extraction advances are central to ongoing efforts to scale sign language technologies using HamNoSys (Mocialov et al., 2022, Shalev-Arkushin et al., 2022, Majchrowska et al., 2022, Ferlin et al., 2023).