---
title: Token-Level Relational Structures
url: https://www.emergentmind.com/topics/token-level-relational-structures
type: topic
---

# Token-Level Relational Structures

Token-level relational structures are mathematical and computational frameworks that explicitly encode, learn, and exploit pairwise or higher-order relationships between individual tokens within an input sequence or spatial patch-based representation. Unlike coarse instance-level graphs whose nodes correspond to entire input entities, token-level relational structures operate at much finer granularity, enabling models to capture nuanced semantic, syntactic, or visual relationships—both within a single instance (inner-instance) and across multiple instances in a batch. This paradigm has significant impact in tasks such as knowledge distillation for visual models and token-level classification in natural language processing.

## 1. Definition and Motivation

Token-level relational structures formalize interactions between individual units (tokens, e.g., sub-words, image patches) by representing them as nodes in a graph, with edges modeling potential semantic or contextual dependencies. This finer granularity stands in contrast to methods that focus solely on individual token outputs (e.g., softmax logits) or to instance-level relational graphs that aggregate over entire image or document embeddings.

In knowledge distillation, token-level relational graphs allow the transfer of fine-grained information from a teacher to a student, propagating semantic similarity between local regions or co-occurring patterns ("fur" between cats and dogs) even in heavily imbalanced, long-tail datasets [2306.12442]. In sequence labeling tasks, such as named entity recognition or disfluency detection, integrating token-level relational graphs enables models to infer both local and global dependencies independent of strict sequential bias [2510.11537].

## 2. Construction of Token-Level Graphs

Token-level relational graphs can be constructed according to a range of design choices:

- **Patch/Token Extraction:** Inputs (images, sentences) are decomposed into tokens (non-overlapping patches for images, sub-word tokens for text), each represented as a vector \(T_i\) or \(h_i\).
- **Graph Topology:**
  - *k-NN Graphs (visual):* In the Token-level Relationship Graph (TRG) paradigm, nodes represent randomly sampled patch tokens across images; edges are created only between each token and its \(k\) nearest neighbors based on Gaussian-weighted Euclidean distance:
    \[
    A_{ij} =
      \begin{cases}
        \exp \left(-\frac{\|T_i - T_j\|^2}{2\sigma^2}\right), & T_i \in k\text{-NN}(T_j)\\
        0, & \text{otherwise}
      \end{cases}
    \]
    Two graphs are constructed per batch: one for fixed teacher tokens and one for student tokens [2306.12442].
  - *Fully Connected Graphs (text):* Every token is a node, and every ordered pair including self-loops forms a directed edge, yielding adjacency
    \[
    A_{ij} = 1,\quad \forall i, j \in \{1, \ldots, N\}
    \]
    This grants maximal flexibility for relational modeling via Graph Attention Networks (GATs) [2510.11537].

## 3. Learning with Token-Level Relational Structures

Token-level relational graphs support a variety of learning objectives targeting different aspects of relational knowledge.

### 3.1. Token-Wise Contextual Loss (Visual Distillation)

The contextual loss matches intra-image (inner-instance) token similarity patterns between teacher and student via contextual similarity matrices:
\[
\mathrm{CS}^{(\cdot)} = \mathrm{Softmax}\left(\frac{\mathbf F\,\mathbf F^\top}{\sqrt{D}}\right) \in \mathbb{R}^{N \times N}
\]
The student is trained to minimize mean-squared error with the teacher:
\[
\mathcal{L}_{\text{inner}} = \frac{1}{N} \|\mathrm{CS}^{\mathcal{T}} - \mathrm{CS}^{\mathcal{S}}\|_{\mathrm{F}}^2
\]
[2306.12442].

### 3.2. Local and Global Graph Losses

- **Local Structure Preserving Loss:** For each student token, the model aligns its local neighborhood distribution to that of the teacher by minimizing the KL divergence between respective normalized adjacency rows:
  \[
  \mathcal{L}_{\text{local}} = \sum_{i} \mathrm{KL}\left( \mathrm{softmax}_{j} (A_{ij}^{\mathcal{S}}) \ \| \ \mathrm{softmax}_{j} (A_{ij}^{\mathcal{T}}) \right)
  \]

- **Global Contrastive Loss:** To align global graph topology, a contrastive InfoNCE-style loss is introduced between paired student-teacher tokens (positives) and all other negatives:
  \[
  s_{ij} = \frac{\mathrm{Proj}(T_i^\mathcal{S}) \cdot T_j^\mathcal{T}}{\|\mathrm{Proj}(T_i^\mathcal{S})\| \, \|T_j^\mathcal{T}\|}
  \]
  \[
  \mathcal{L}_{\text{global}} = -\sum_{i} \log \frac{ \exp(s_{ii}/\tau_g) }{ \sum_j \exp(s_{ij}/\tau_g) }
  \]
[2306.12442].

### 3.3. Graph Attention Networks in Sequence Labeling

For token-level classification, after extracting PhoBERT embeddings \(H\), a GAT layer computes pairwise compatibility for every token pair:
\[
e_{ij} = \mathrm{LeakyReLU}\left(\mathbf{a}^\top\,[W h_i\ \|\ W h_j]\right)
\]
\[
\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k} \exp(e_{ik})}
\]
Node updates aggregate messages over all neighbors, and multiple heads are combined to produce graph-enhanced embeddings [2510.11537]. Further refinement occurs via Transformer-style self-attention before final classification.

## 4. Integration into Model Architectures

### 4.1. Knowledge Distillation with Token Graphs

The overall distillation objective in the TRG framework fuses four distinct losses:
\[
\mathcal{L}_{\text{total}} =
\mathcal{L}_{\text{CE}} + \lambda\,\mathrm{KL}(p^\mathcal{S},p^\mathcal{T}) + \alpha\,\mathcal{L}_{\text{inner}} + \beta\,\mathcal{L}_{\text{local}} + \gamma\,\mathcal{L}_{\text{global}}
\]
Optimal distillation is facilitated through additional modules for token extraction and k-NN graph construction without introducing a full graph neural network; the teacher graph stays fixed throughout [2306.12442].

### 4.2. Token-Level Classification with GATs

In TextGraphFuseGAT, PhoBERT-generated embeddings pass through a GAT layer operating on the full token graph, followed by transformer-style self-attention. The classification head applies a linear map and softmax at each token position:
\[
z_i = W^{\text{cls}} h_i^{\mathrm{dec}} + b^{\text{cls}}
\qquad
p_i = \mathrm{softmax}(z_i)
\]
with loss masked on padding/subwords [2510.11537].

## 5. Empirical Performance and Analysis

Token-level relational structures demonstrate empirical superiority across tasks and modalities.

- **Image Classification & Distillation:** On CIFAR-100, TRG achieves 76.42% Top-1 accuracy (ResNet-32×4→ResNet-8×4), surpassing other graph-based and relational distillation methods (e.g., HKD 76.21%, DKD 76.32%). For heterogeneous architectures, it remains robust (ResNet→ShuffleNet: 76.42% vs. 75.99% for HKD). On ImageNet-LT (long-tailed), TRG Top-1 accuracy (50.32%) exceeds even the teacher (50.11%), confirming enhanced minority class transfer [2306.12442].
- **Token-Level NLP Tasks:** TextGraphFuseGAT outperforms both transformer-only and hybrid neural models on Vietnamese NER and disfluency detection. For instance, on PhoNER-COVID19, model achieves Micro-F1 0.984, Macro-F1 0.958 (PhoBERTₗₐᵣgₑ baseline: Micro-F1 0.945, Macro-F1 0.931), and for VietMed-NER reaches F1 0.893 (PhoBERTₗₐᵣgₑ: 0.730, XLM-Rₗₐᵣgₑ: 0.740). Ablation studies attribute >0.16 gain in F1 to the GAT and decoder layers [2510.11537].

| Task/Model       | Baseline Metric      | Token-Level Relational Structure Metric |
|------------------|---------------------|----------------------------------------|
| CIFAR-100 (TRG)  | 76.21% (HKD)        | 76.42% (TRG)                           |
| PhoNER-COVID19   | 0.945 Micro-F1      | 0.984 Micro-F1                         |
| VietMed-NER      | 0.730 F1 (PhoBERT)  | 0.893 F1 (TextGraphFuseGAT)            |

Token-level relational structures show marked benefits in robustness to class imbalance, finer class clustering in feature space, and task generalization.

## 6. Implications for Long-Tailed and Low-Resource Scenarios

Token-level relational distillation mitigates long-tail effects by leveraging the fact that rare-class instances often contain patch tokens semantically similar to those in abundant classes (e.g., "fur," "wheel"). By pooling across all tokens, relational graphs facilitate knowledge transfer to under-represented areas without overfitting to majority classes [2306.12442]. Similarly, in low-resource NLP settings, fully connected token graphs and attention mechanisms capture long-range dependencies and syntactic phenomena not present in local sequential windows [2510.11537].

Empirical results confirm that token-level relational structures can produce student models that not only match but sometimes exceed teacher performance in challenging, imbalanced or domain-specific contexts.

## 7. Perspectives and Research Directions

Token-level relational structures unify local and global relational information in a generalizable framework, applicable to both vision and sequence learning. The introduction of direct graph-based reasoning (via k-NN or complete graphs) augments or complements transformer-style attention with explicit relational inductive biases. *A plausible implication is that further exploration of hybrid architectures—combining graph neural modules, transformers, and domain-specific pretraining—will yield enhanced interpretability and cross-domain transfer, especially for minority or rare-event phenomena.*

Key open challenges include efficient scaling of graph construction for long sequences or dense patch tokenizations, integration with dynamic or hierarchical graph structures, and systematic evaluation on multilingual or multi-modal corpora.

Source: https://www.emergentmind.com/topics/token-level-relational-structures