---
title: Token-Level Graphs for Short Text Classification
url: https://www.emergentmind.com/topics/token-level-graphs-for-short-text-classification
type: topic
---

# Token-Level Graphs for Short Text Classification

Token-level graph methods for short text classification refer to a family of techniques that construct explicit or implicit graphs at the token (word or subword) level for each short text instance, leveraging graph neural networks (GNNs) or associated architectures to learn representations or augment feature spaces for improved classification accuracy. These methods address the severe sparsity, brevity, and context limitations inherent in short texts by exploiting relationships between tokens both within texts and across the corpus, often surpassing sequential models and classical feature-based approaches on standard benchmarks.

## 1. Formal Models of Token-Level Graphs

A token-level graph for a short text $T$ is typically defined as $G=(V,E)$, where $V$ is the set of tokens (words, subwords, or syllables) identified in $T$ and $E$ is a set of edges encoding relationships among those tokens. Core construction paradigms include:

- **Sliding-window graphs**: Edges connect each token to its immediate neighbors using a window of size $k$, with adjacency $A_{ij} = 1$ if $0 < |i-j| \leq k$; self-loops $A_{ii}=1$ are standard [2412.12754][2304.11534].
- **PMI-based graphs**: Edges are weighted by corpus-level or document-level pointwise mutual information, $A_{ij} = \max\{\mathrm{PMI}(i, j), 0\}$, to capture nonlocal lexical associations [2501.09214][1909.08166][2304.11534].
- **Fully-connected graphs**: Every token is linked to every other, as in the fully-connected graphs in TextGraphFuseGAT [2510.11537].
- **Syntactic/dependency/semantic graphs**: Edges are derived from dependency trees or semantic similarity (e.g., cosine in embedding space) [2304.11534][2111.00180][2501.09214].
- **Multi-source graphs**: Graphs are constructed using various token-level sources—words, POS tags, named entities—each forming their own component subgraph [2501.09214][2111.00180].
- **Corpus-level or heterogeneous graphs**: Include nodes for words, documents, character n-grams, or even label nodes, creating inter- and intra-document edges [2210.05999][2103.14620].

The node features are derived from static embeddings (GloVe, Word2Vec), contextualized embeddings (BERT, PhoBERT, RoBERTa), or one-hot representations, often tailored to specific datasets or languages [2412.12754][2510.11537][2210.05999].

## 2. Graph Neural Architectures for Token Graphs

The learning component operates on the token graph using GNN layers. Principal GNN variants include:

- **Graph Attention Networks (GAT)**: Use learnable attention weights over neighbors; e.g., in TextGraphFuseGAT the update is
  $$
  h_i^{(k)'} = \sum_{j=1}^n \alpha_{ij}^k W^k h_j
  $$
  with multi-head concatenation to aggregate representations [2510.11537][2412.12754].
- **Graph Convolutional Networks (GCN)**: Standard updates using normalized adjacency. For two layers:
  $$
  H^{(l+1)} = \sigma(\tilde{D}^{-\frac{1}{2}} \tilde{A} \tilde{D}^{-\frac{1}{2}} H^{(l)} W^{(l)})
  $$
  where $\tilde{A} = A + I$ [2501.09214][2111.00180][2210.05999].
- **Gated or recursive propagation**: The ReGNN model uses LSTM-style gates and global graph-level nodes to alleviate over-smoothing and propagate both local and nonlocal information effectively, especially in deeper networks [1909.08166].
- **Hybrid architectures**: Integrate pretrained transformer encoders (e.g., BERT, PhoBERT) with GNNs for graph-based enhancement of contextual embeddings and further refinements via Transformer layers [2510.11537][2412.12754].
- **Heterogeneous/multigraph GNNs**: Separate architectures for word, POS, and entity graphs, pooled and fused at the document level, as in SHINE and MI-DELIGHT [2111.00180][2501.09214].

Pooling strategies for graph-level representations range from mean-pooling (node average), global attention, to hierarchical normalization and concatenation of multiple graph types [2412.12754][2111.00180][2501.09214].

## 3. Applications and Datasets

Token-level graph methods are particularly effective in sequence labeling and short-text classification. Representative applications include:

- **Named Entity Recognition (NER)**, including domain-specific NER such as PhoNER-COVID19 and VietMed-NER for Vietnamese medical ASR [2510.11537].
- **Dialogue disfluency detection** on spontaneous speech (e.g., PhoDisfluency) [2510.11537].
- **Sentiment and topic classification** in microblogs, movie reviews, search snippets, and news headlines; datasets include Twitter Sentiment, MR, Snippets, TagMyNews [2412.12754][2501.09214][2304.11534][2210.05999].
- **Zero-/few-shot node classification**: Methods like STAG quantize text-attributed graph nodes into discrete tokens compatible with LLMs, enabling both LLM-based and classical learning strategies [2507.19526].

The typical short-text regime involves sentences of length 6–40 tokens, highly imbalanced classes, and severe sparsity, where token-level graphs provide robustness and sample efficiency [2412.12754][2111.00180].

## 4. Empirical Performance and Ablation Insights

Token-level graph techniques consistently exceed or match non-graph and classical PLM baselines under low-resource and class-imbalanced conditions:

| Dataset/Task         | Best Token-Graph Model      | Accuracy / F1         | Notable Baseline       | Accuracy / F1        |
|----------------------|----------------------------|-----------------------|------------------------|----------------------|
| Twitter Sentiment    | Token-Graph [2412.12754]   | 0.837 (Acc/F1)        | Fine-tuned BERT        | 0.780 (Acc/F1)       |
| MR (Movie Reviews)   | Token-Graph [2412.12754]   | 0.702 (Acc/F1)        | BertGCN                | 0.666 (Acc)          |
| Vietnamese NER (word)| TextGraphFuseGAT [2510.11537]| 0.984 / 0.958 (F1)    | PhoBERT_large          | 0.945 / 0.931 (F1)   |
| Short Texts (various)| MI-DELIGHT [2501.09214]    | up to +4-6% ACC over SOTA| CNN/LSTM/BERT          | varies               |
| MR (TextGCN)         | WCTextGCN [2210.05999]     | 77.85 (Acc)           | BERT                   | 77.02                |

Ablations reveal that optimal token-graph GNNs are typically shallow ($L=2$), low-$k$ for neighborhood ($k=1$ or $2$), and leverage contextual or subword token embeddings. Adding dynamic or multi-source graph structures (e.g., PMI, POS, and entity) further improves performance, particularly on rare or minority classes [2510.11537][2501.09214][2210.05999]. In ablations, removal of graph structure or fusion blocks consistently leads to $2$–$16$ points drop in F1, with GAT/attention yielding largest marginal gains [2510.11537][2507.19526].

## 5. Extensions: Contrastive, Heterogeneous, and Multilingual Graphs

Recent advances explore contrastive learning over token graphs—instance-level and cluster-level contrastive objectives (e.g., MI-DELIGHT) maximize both augmentation invariance and cluster cohesion, further cementing graph methods' robustness in unlabeled and few-labeled settings [2501.09214][2112.11389]. Hierarchical heterogeneous graphs integrate token, POS, entity, and document nodes, enabling better label propagation and knowledge integration (e.g., SHINE, LiGCN) [2111.00180][2103.14620]. Techniques such as quantization of token-level graph structure into discrete tokens make LLM-based downstream inference graph-compatible (e.g., STAG) [2507.19526].

Token graphs have been extended to multilingual and cross-domain settings, for example, TextGraphFuseGAT's integration of PhoBERT supports Vietnamese benchmarks with multi-domain (COVID, speech, medical) characteristics, and future work suggests linguistically-motivated sparse graphs for multilingual adaptation [2510.11537][2412.12754].

## 6. Limitations, Challenges, and Open Directions

- **Graph construction overhead**: PMI/co-occurrence statistics over large corpora are computationally intensive; fully-connected graphs raise memory and compute requirements [2510.11537][2412.12754].
- **Oversmoothing in deep GNNs**: Deep propagation can collapse token representations; gating (LSTM/GRU) and attention alleviate, but optimal depth is typically shallow [1909.08166][2304.11534].
- **Sparsity and rare tokens**: Short texts yield small graphs; corpus-level graphs, character n-grams, or OOV-aware embeddings can mitigate, but boundary cases remain [2210.05999][2304.11534].
- **Edge semantics**: Most methods rely on syntactic or heuristic co-occurrence; dynamic or learned edge types can improve performance but add complexity [2304.11534][2111.00180].
- **Scalability**: Token-level graphs scale linearly with text length, but batch processing and very long texts demand special pooling and batching mechanisms [2412.12754].

Open research directions include optimized or adaptive sparsity for token-level graphs, dynamic edge learning, multilingual/domain adaptation, explainability via interpretable token interactions, and tight integration with transformer self-attention [2510.11537][2507.19526][2304.11534].

## 7. Representative Models and Reproducibility

Prominent models with open-source implementations and strong performance in short-text token-level graph classification include:

| Model/Method                        | Key Features                       | arXiv ID         |
|-------------------------------------|------------------------------------|------------------|
| TextGraphFuseGAT                    | PhoBERT + fully-connected GAT      | 2510.11537       |
| Token-Graph (BERT+GAT)              | PLM embedding, window-1 GAT        | 2412.12754       |
| MI-DELIGHT                          | Multi-source graph + hierarchical CL| 2501.09214      |
| SHINE                               | Heterogeneous, hierarchical GNN    | 2111.00180       |
| ReGNN                               | Gated propagation, global node     | 1909.08166       |
| STAG                                | Graph quantization to tokens       | 2507.19526       |
| WCTextGCN/WCTextGAT                 | Word and char n-gram graph         | 2210.05999       |
| ClassiNet                           | Feature-predictor graph, propagation| 1804.05260      |

These methods demonstrate the flexibility and robustness of token-level graph construction and modeling in handling the semantic sparsity, few-shot learning, and context limitations of short texts, with statistically significant improvements reported across diverse languages and domains.

Source: https://www.emergentmind.com/topics/token-level-graphs-for-short-text-classification