Token Classification Models in NLP

• Token classification is a method that assigns discrete labels to individual tokens using transformer-based contextual embeddings.
• These models excel in tasks such as named entity recognition, grammatical error detection, and domain-specific entity extraction with high precision.
• Advanced implementations integrate graph attention and layout features, while metrics like token-F1 and DIP ensure robust evaluation and deployment.

A token classification model is a neural architecture trained to assign discrete class labels to individual tokens within an input sequence. It is foundational in NLP tasks such as named entity recognition (NER), sequence labeling, entity extraction, and fine-grained error detection. These models, typically transformer-based, operate at token granularity, enabling both local and context-aware decisions on each input element. Token classification has supplanted earlier approaches relying on classical sequence models and feature engineering, owing to its scalability, domain transfer characteristics, and capacity for direct end-to-end optimization.

1. Architectural Principles and Model Formulation

At core, token classification extends a contextual encoder—commonly a pretrained transformer (e.g., BERT, LayoutLM, PhoBERT)—with a token-wise classification head. The encoder maps the input token sequence $x = (x_1, ..., x_L)$ to a sequence of contextualized embeddings $h = (h_1, ..., h_L)$, with each $h_i \in \mathbb{R}^d$. A projection matrix $W \in \mathbb{R}^{C \times d}$ and bias $b \in \mathbb{R}^C$ produce class logits per token:

$z_i = W h_i + b$

with probability for class $c$ given by:

$$p_i(c) = \frac{\exp(z_{i,c})}{\sum_{k=1}^C \exp(z_{i,k})}$$

The model is trained using the categorical cross-entropy loss over all valid tokens:

$$\mathcal{L}_{\text{token}} = - \frac{1}{N \cdot L} \sum_{j=1}^N \sum_{i=1}^L \sum_{c=1}^C y_{j,i,c} \log p(y_{j,i}=c | x^{(j)})$$

where $y_{j,i,c}$ is a one-hot indicator for ground-truth class and masking is applied to ignore padding or special tokens (Jafari, 2022, Islam et al., 2024).

Advanced token classification models incorporate additional relational modeling beyond sequential context. For example, TextGraphFuseGAT builds a fully-connected graph over tokens, and applies multi-head Graph Attention Networks (GAT) followed by Transformer-style decoder layers to encode global dependencies (Nguyen, 13 Oct 2025). Integration with bounding-box or layout features, as in LayoutLM, is essential for visually structured documents (Mehra et al., 28 Mar 2025).

2. Task-Specific Data Preparation and Labeling

Token classification tasks mandate precise token-level label assignment. In NER, each token is mapped to a BIO or span-based tag. In grammatical error detection (GED), each token receives tags such as O (outside error), B (begin error span), I (inside span), and M (missing-after for insertions) (Islam et al., 2024). Preprocessing includes dataset tokenization, mapping original spans to token indices post-tokenization, and propagating labels to subwords.

For tasks operating on long or structured documents, token labels may be inherited from document-level annotations (by repeating the document label for all tokens), and chunking with overlap is employed for sequences exceeding model input limits (Jafari, 2022). In error detection or similar settings, alignment procedures may be required for projecting token-level predictions back to character offsets in original text (Islam et al., 2024).

3. Training Objectives, Regularization, and Optimization

Standard training relies on minimizing per-token cross-entropy loss, with adaptations such as label smoothing to control overconfidence, especially in small or noisy datasets. For Bangla GED, label smoothing with $h = (h_1, ..., h_L)$0 or $h = (h_1, ..., h_L)$1 is deployed depending on model scale (Islam et al., 2024). Class weighting or loss regularization is used to address severe class imbalance, as in the medical abbreviation disambiguation task where rare classes are up-weighted by a factor of 100 (Cevik et al., 2022).

Auxiliary objectives can be layered, such as sentence-level error flags on [CLS] tokens via binary cross-entropy for highly noisy sentences (Islam et al., 2024). In graph-based settings, the entire encoder, graph, decoder, and classifier components are typically fine-tuned end-to-end using AdamW, with learning rates around $h = (h_1, ..., h_L)$2–$h = (h_1, ..., h_L)$3 and careful dropout regularization (Nguyen, 13 Oct 2025). Early stopping is based on validation-level metrics, most often F1 or task-specific indicators.

4. Metrics of Model Quality: Beyond F1

Conventional metrics for token classification include token-wise precision, recall, and F1-score. However, these are inadequate in settings where the business process requires all relevant entities in a document to be extracted perfectly. The Document Integrity Precision (DIP) metric was introduced to address this gap (Mehra et al., 28 Mar 2025). DIP is:

$h = (h_1, ..., h_L)$4

where $h = (h_1, ..., h_L)$5 is the count of test documents with all target tokens predicted correctly, and $h = (h_1, ..., h_L)$6 is the total number evaluated. DIP thus gives the proportion of documents requiring no manual intervention—crucial for enterprise-grade automation.

Empirically, a model may achieve token-F1 ≈ 0.97 while DIP remains 0.80 (S1_100, “known layouts” split), meaning 20% of documents still need correction. For distribution shift (S2_100, “unknown layouts”), DIP collapses to 0.23 despite token-F1 ≈ 0.85. DIP is particularly sensitive to the number of target entity fields, decaying as $h = (h_1, ..., h_L)$7 for $h = (h_1, ..., h_L)$8 fields and token accuracy $h = (h_1, ..., h_L)$9 (Mehra et al., 28 Mar 2025).

Other downstream metrics (e.g., Levenshtein Distance for error-annotated string predictions in GED) measure the average minimal edits to reach the gold-standard labeling, capturing overall correction burden (Islam et al., 2024). Weighted and macro-averaged F1 are crucial where rare entity types or senses must not be masked by class imbalance (Cevik et al., 2022, Nguyen, 13 Oct 2025).

5. Comparative Performance and Practical Implementations

Token classification consistently outperforms traditional text- or sequence-classification for fine-grained labeling:

• In medical abbreviation disambiguation, transformers with token classification yield 5–10 point higher macro-F1 than sequence classification, even after aggressive postprocessing of the latter (Cevik et al., 2022).
• In comparative experiments on English XNLI, token classification exceeds sequence-head models on both accuracy (0.827 vs. 0.814) and Quadratic Weighted Kappa (0.803 vs. 0.781), particularly when input length necessitates chunking or the task is multi-focal (Jafari, 2022).
• In Vietnamese NER, integrating token-level modeling with graph attention and transformer decoding achieves up to 0.893 Micro-F1 on complex medical domains, substantially outperforming BiLSTM+CNN+CRF and transformer-only baselines (Nguyen, 13 Oct 2025).

The table below summarizes representative architecture and result comparisons:

Study	Domain/Task	Model	Key Metric(s)	Score(s)
(Nguyen, 13 Oct 2025)	Vietnamese NER	PhoBERT + GAT + Transformer Decoder	Micro/Macro F1	0.893 / (–)
(Cevik et al., 2022)	Med. Abbr Disambig.	SciBERT (token)	Macro-F1 (MeDAL)	77.3 ± 0.7
(Mehra et al., 28 Mar 2025)	Doc Extraction	LayoutLM (token)	Token F1 / DIP	0.97 / 0.80 (“S1_100”)
(Islam et al., 2024)	Bangla GED	BanglaBERT (token)	Levenshtein Dist.	1.04 (lower is better)

6. Post-Processing, Ensembling, and Deployment Practices

Robust production systems complement token classification with structured post-processing and ensembling. In Bangla GED, ensemble strategies use span intersection across multiple checkpoints, sharply reducing false positives, with further improvement from lightweight rule-based correction modules targeting punctuation and spelling (Islam et al., 2024). Confidence thresholding over predicted token probabilities (e.g., tagging as error only if $h_i \in \mathbb{R}^d$0) is effective against overlabeling. For sequence-level aggregation in text classification, average-probability and majority-vote rules are standard (Jafari, 2022).

Production deployment guidelines recommend DIP as a gating metric for model selection and health monitoring. DIP $h_i \in \mathbb{R}^d$1 signifies readiness for near-fully automated environments, while lower DIP bands indicate the need for human-in-the-loop operation (Mehra et al., 28 Mar 2025). Early-stopping on DIP or a weighted combination with token-F1 helps prevent overfitting on trivial token-level accuracy at the expense of business utility.

7. Research Trends and Open Directions

Recent architectures expand token classification with non-local dependency modeling, as in graph-based attention networks and cross-domain transfer. Domain-specific pretraining directly influences token classification accuracy—e.g., SciBERT’s custom vocabulary and BlueBERT’s clinical corpus yield optimal results for their respective specialties (Cevik et al., 2022, Nguyen, 13 Oct 2025). Label imbalance and rare-type supervision remain active challenges, with macro-F1 the preferred metric for unbiased tracking.

Novel evaluation paradigms, such as DIP, are emerging in response to real-world requirements, exposing the inadequacy of token-level metrics for applications where failure on any single entity degrades business automation. A plausible implication is that analogous document-level or “all-or-nothing” metrics may be adopted for extractive summarization, table extraction, or span-based QA as model deployment in process automation grows (Mehra et al., 28 Mar 2025). Further, production workflows increasingly emphasize post-deployment monitoring with document-level metrics and robust retraining triggers.

• (Mehra et al., 28 Mar 2025) Improving Applicability of Deep Learning based Token Classification models during Training.
• (Islam et al., 2024) Bangla Grammatical Error Detection Leveraging Transformer-based Token Classification.
• (Cevik et al., 2022) Token Classification for Disambiguating Medical Abbreviations.
• (Jafari, 2022) Comparison Study Between Token Classification and Sequence Classification In Text Classification.
• (Nguyen, 13 Oct 2025) An Encoder-Integrated PhoBERT with Graph Attention for Vietnamese Token-Level Classification.