BiLSTM-CRF Morphological Segmenter

• The paper introduces a BiLSTM-CRF model that uses bidirectional LSTM encoding and CRF decoding to effectively identify morpheme boundaries in complex languages.
• It leverages character-level embeddings and contextual hidden states to generate emission scores and enforce valid segmentation through structured transition constraints.
• Empirical evaluations across Japanese, Kurdish, and Arabic demonstrate competitive accuracies with domain adaptation, despite challenges in handling complex verb morphology.

A BiLSTM-CRF morphological segmenter is a neural sequence-labeling model that combines a bidirectional Long Short-Term Memory (BiLSTM) encoder with a Conditional Random Field (CRF) decoder to perform fine-grained word or morpheme segmentation on character sequences. The segmenter is widely applied to languages with complex or unsegmented orthographies, such as Japanese, Kurdish, and various Arabic dialects, where explicit lexical delimiters are absent and morphology is non-concatenative or highly inflectional. This model class enables data-driven, character-level discovery of word and morpheme boundaries, including the assignment of part-of-speech or other morphological features, and is effective in both high- and low-resource scenarios.

1. Model Architecture

At its core, the BiLSTM-CRF morphological segmenter processes an input character sequence using a multi-layer (often single or triple) BiLSTM encoder that captures contextualized representations from both forward and backward directions. Each character $c_t$ is embedded into a dense vector $x_t \in \mathbb{R}^d$, which is then consumed by the BiLSTM stack to produce for each position $t$ a hidden state $h_t = [\overrightarrow{h}_t; \overleftarrow{h}_t]$ where $\overrightarrow{h}$ and $\overleftarrow{h}$ are the final hidden states in each direction (e.g., $H=100$ for Arabic (Eldesouki et al., 2017), $H=256$ for Kurdish (Salehi et al., 18 Nov 2025)).

A linear projection maps each $h_t$ to emission scores $e_t$ over segmentation tags. The CRF layer parameterizes a transition matrix $T$ (or $A$), providing a global structured prediction mechanism to enforce valid boundary sequences:

$$s(\mathbf{x}, \mathbf{y}) = \sum_{t=1}^n (e_{t,y_t} + T_{y_{t-1},y_t})$$

where $\mathbf{y} = (y_1, \dots, y_n)$ is a tag sequence, and $y_0$ is a special start tag. The emission and transition scores are summed to compute the score of each possible label sequence. Inference uses Viterbi decoding to recover the most likely tag sequence.

Tag sets vary by language and task. Examples include:

• Japanese Hiragana: BIO+POS, with explicit boundary and part-of-speech tags ($T=38$) (Izutsu et al., 2022)
• Kurdish: boundary-only (end-only: $K=2$), with only the last character of each morpheme marked (Salehi et al., 18 Nov 2025)
• Arabic: BMES+WB (B, M, E, S, WB: $K=5$) for segmenting stems, affixes, and clitics (Eldesouki et al., 2017)

2. Mathematical Formulation

The segmenter's operations follow standard sequence labeling by BiLSTM-CRF:

• Character embedding:

$x_t = \mathrm{Embedding}(c_t)$

• BiLSTM encoding (final layer):

$h_t = [\overrightarrow{h}_t; \overleftarrow{h}_t]$

• Emission projection:

$e_t = W h_t + b$

• CRF sequence score:

$$s(\mathbf{x}, \mathbf{y}) = \sum_{t=1}^n (e_{t, y_t} + T_{y_{t-1}, y_t})$$

• Conditional log-likelihood loss (training objective):

$$\mathcal{L} = -\log p(\mathbf{y^*}|\mathbf{x}) = -[s(\mathbf{x}, \mathbf{y}^*) - \log \sum_{\mathbf{y'}} \exp s(\mathbf{x}, \mathbf{y'})]$$

• Inference (Viterbi decoding):

$$\hat{\mathbf{y}} = \arg\max_{\mathbf{y}} s(\mathbf{x}, \mathbf{y})$$

This framework jointly models both local tag emissions and global sequence transition constraints, enabling accurate discovery of linguistically plausible boundaries.

3. Training Procedures and Data Preparation

Data Sources and Bootstrapping

Morphological segmenters rely on curated and/or automatically preprocessed corpora. For Japanese, conversion procedures (e.g., Kanji-Kana readings via MeCab and UniDic) provide hiragana-only input with gold BIO+POS segmentation. For Kurdish, an AsoSoft Sorani corpus is filtered, Unicode-normalized, and morpheme boundaries are manually annotated in stages; initial models trained on $\sim$1,500 words are applied to the unlabeled set, followed by expert correction and iterative expansion ($\sim$4,000 labeled words in total) (Salehi et al., 18 Nov 2025). For Arabic, dialectal tweets are manually segmented using a BMES+WB scheme, with data splits of only $\sim$6,000 word tokens per dialect (Eldesouki et al., 2017).

Fine-Tuning and Adaptation

Incremental transfer learning is central to high performance in low-resource tasks. For Hiragana, models pre-trained on a larger mixed-script corpus are fine-tuned in stages on progressively more domain-specific corpora, e.g., Wikipedia $\rightarrow$ Yahoo! Answers (Izutsu et al., 2022). In Arabic, domain adaptation incorporates Modern Standard Arabic Treebank segmentations (629K tokens) to bootstrap and pre-cache canonical forms (Eldesouki et al., 2017).

Hyperparameters

Key hyperparameters vary by implementation:

Language	Layers	Hidden Dim	Dropout	Emb. Dim	Optimizer	LR	Batch Size
Japanese	1	Varied	—	Varied	SGD	0.01	32-128 sent
Kurdish	3	256	0.3	~50-100	Adam	0.001	32–64 words
Arabic	1	100	0.5	50	SGD+mom.	0.01	50 words

Early stopping on dev-set convergence is uniformly applied.

4. Evaluation Metrics and Empirical Results

Standard evaluation regimes employ word- or character-level segmentation accuracy, with supplementary F1 measures for boundary detection:

• Japanese Hiragana: macro (genre-averaged) and micro (all-character) segmentation accuracy. Best macro accuracy: 61.93%, micro: 63.01% (vs. 79.71%/80.10% for MeCab+ipadic baseline) (Izutsu et al., 2022).
• Kurdish: boundary-detection precision 0.835, recall 0.796, F1 0.815; full-word exact segmentation accuracy: nouns 90.2%, adjectives 90.1%, verbs 41.7%, others ~72%. BPE comparison showed only 28.6% coverage and 14.4% boundary agreement with BiLSTM-CRF (Salehi et al., 18 Nov 2025).
• Arabic: word-level exact segmentation accuracy of 90–95% across Egyptian, Levantine, Gulf, and Maghrebi dialects. Domain adaptation with MSA yields 1–2% absolute gain (Eldesouki et al., 2017).

A plausible implication is that performance on morphologically complex verbs remains a challenge across languages, while noun and adjective segmentations are more reliably learned.

5. Genre, Domain, and Cross-Language Adaptation

BiLSTM-CRF segmenters are sensitive to genre and domain mismatches:

• Japanese: models trained on Wikipedia excel on technical/legal genres, while fine-tuning on conversational data brings gains to dialogue-rich domains. Pre-training on mixed script aids downstream accuracy in poorly covered genres (Izutsu et al., 2022).
• Kurdish: bootstrapped annotation plus normalization enables robust segmenter training with minimal data, facilitating downstream applications such as embedding learning and subword modeling (Salehi et al., 18 Nov 2025).
• Arabic: domain adaptation from MSA and context-independence assumptions (per-token segmentation) enable rapid transfer to dialectal segmentation without sentential modeling (Eldesouki et al., 2017).

Coverage-aware evaluation is emphasized as BPE and word-level tokenization approaches do not generalize to low-resource, morphologically rich settings.

6. Strengths, Limitations, and Practical Recommendations

Strengths

• Data-driven, character-level modeling naturally discovers boundaries, avoiding explicit lexicon dependence and OOV sparsity.
• CRF enforces sequence constraints, eliminating spurious or illicit boundary sequences.
• Small-scale bootstrapping and incremental fine-tuning enable deployment in low-resource scenarios.
• Flexible tagset and layer depth accommodate language-specific segmentation conventions.

Limitations

• Reduced performance on morphologically complex forms (e.g., Kurdish verbs) owing to stem alternations and irregular morphophonology.
• Purely data-driven models may miss linguistically motivated boundary phenomena absent from small corpora.
• Statistical subword schemes (BPE) provide higher similarity scores on restricted portions of the data, but exhibit limited coverage and inconsistent boundary agreement.

Recommendations

• For low-resource or morphologically rich languages, iterate manual annotation with model-assisted correction to maximize segmentation coverage.
• Pre-train on related scripts or corpora when possible, followed by fine-tuning on gold-standard in-domain annotation.
• Apply dense CRF transition matrices and tune embedding/hidden dimensions via pilot experiments.
• For production, snapshot a cache of known word segmentations to accelerate inference when context independence is assumed.

7. Comparative Perspectives and Future Directions

The BiLSTM-CRF paradigm demonstrates competitive performance in settings with limited resources and morphologically complex languages. Its contrast with BPE and word-level tokenization reveals the need for hybrid, linguistically informed models that balance OOV coverage and boundary accuracy. The segmenter architecture generalizes across script systems, from Japanese kana to Arabic and Sorani Kurdish, and provides foundational data for downstream applications, e.g., embedding learning and neural machine translation.

Future work addresses explicit integration of morphophonemic rules, direct incorporation of context-aware features, and efficient adaptation to ultra-low-resource environments through cross-lingual transfer or self-supervised pretraining (Izutsu et al., 2022, Salehi et al., 18 Nov 2025, Eldesouki et al., 2017).