Hierarchical n-Gram Embeddings

• Hierarchical n-gram embeddings are structured representations that capture multi-scale linguistic dependencies by composing lower-level tokens into interpretable multi-level constructs.
• They leverage techniques such as averaging, clustering, and manifold projection to model both local and global semantic organization in language.
• These methods enhance NLP tasks like classification, translation, and similarity measurement while optimizing computational efficiency and interpretability.

Hierarchical n-gram embeddings are structured representations designed to capture multi-scale lexical, syntactic, and semantic information in natural language by organizing n-grams—sequences of n tokens—according to hierarchical principles. These models leverage compositionality, manifold projection, clustering, and geometric embedding spaces to overcome the limitations of traditional flat word or n-gram features, providing compact, interpretable, and robust embeddings for a range of linguistic tasks.

1. Foundational Principles of Hierarchical n-Gram Embeddings

Hierarchical n-gram embeddings are rooted in the idea that textual meaning is organized at multiple linguistic levels: characters, n-grams, words, phrases, and sentences. Rather than representing textual elements as independent tokens or flat bags-of-n-grams, these approaches encode dependencies and compositional relationships among sub-units, thereby aligning the representation space with linguistic hierarchy.

Techniques include averaging pre-trained word vectors to form n-gram embeddings (Lebret et al., 2014), compositional summation of sub-n-gram vectors (Kim et al., 2018), clustering via k-means to induce semantic concepts (Lebret et al., 2014), and graph-based hierarchical modeling where n-grams are nodes connected by compositional and adjoin edges (Li et al., 2022).

Central to hierarchical models is the ability to:

• Compose embeddings from lower-level units (e.g., characters to n-grams, n-grams to phrases).
• Model both local and global semantic/syntactic organization.
• Generalize to unseen or out-of-vocabulary n-grams by leveraging shared substructures.

2. Construction and Mathematical Formulation

Hierarchical n-gram embedding construction employs several mathematical strategies that reflect compositional and hierarchical relationships:

• Averaging or Summing Embeddings: For an n-gram composed of tokens $w_1, \dots, w_n$, the embedding is defined as the average: $$x_{n\text{-gram}} = \frac{1}{n} \sum_{i=1}^n x_{w_i}$$. This places every n-gram in a shared semantic space (Lebret et al., 2014).
• Compositional Summation: The segmentation-free approach computes $v_{x_{(i,j)}} = \sum_{s \in S(x_{(i,j)})} z_s$, where $S(x_{(i,j)})$ are the relevant sub-n-grams in the vocabulary (Kim et al., 2018).
• Hierarchical Graphs: Nodes represent n-grams; edges encode compositional (lower- to higher-level n-grams) and adjoin relations. The GramTransformer applies mask-attention over these graph structures, enabling encoding of dependencies (Li et al., 2022).
• Manifold Projection: Tokens or n-grams are mapped via smooth functions $\phi: \mathbb{R}^d \to \mathcal{M}$ to manifolds (e.g., Riemannian, Poincaré, Lorentzian). Hierarchical projection operators $P_h(e_i)$ use geodesic distances $d_\mathcal{M}(x_i,x_j)$ and adaptive weights $\alpha_{ij}$ to align representations along hierarchy (Martus et al., 8 Feb 2025, Patil et al., 25 May 2025).
• Clustering: K-means partitions n-gram embeddings into clusters, reducing feature dimensionality and grouping semantically similar phrases (Lebret et al., 2014).

These construction principles facilitate multi-scale representation, efficient feature reduction, and the capture of fine-grained compositional structure.

3. Hierarchical Structure and Semantic Organization

Hierarchical organization is achieved by explicitly modeling relationships between n-grams of different granularities:

• Layered Decomposition: Characters aggregate to form sub-n-grams, which are iteratively composed into longer spans (words, phrases, sentences), mirroring linguistic hierarchy (Kim et al., 2018, Wieting et al., 2016).
• Hierarchical Graphs: The GramTransformer's hierarchical n-gram graph includes adjoin and compositional edges, enabling modeling of both neighbor and containment relationships between n-grams (Li et al., 2022).
• Manifold-Based Embedding: Lexical units are mapped to structured manifolds; hierarchical bands or layers maintain semantic coherence across abstraction levels. Geodesic distances in manifolds reflect hierarchy—general concepts near the origin, specific concepts near the boundary (Dhingra et al., 2018, Martus et al., 8 Feb 2025, Patil et al., 25 May 2025).
• Clustering Concepts: K-means-induced semantic concepts serve as hierarchical units for document representation in classification (Lebret et al., 2014).

This explicit hierarchy enables the embedding of relations such as parent-child, adjacency, and compositional containment, improving semantic retention and lexical alignment across context scales.

4. Applications and Empirical Performance

Hierarchical n-gram embeddings are widely applied in:

• Document Classification: Compact document representations using clusters of semantic concepts derived from n-gram embeddings outperform LSA and LDA, and match bag-of-words baselines with far fewer features on sentiment analysis (Lebret et al., 2014).
• Zero-Shot Link Prediction: Hierarchical n-gram graphs enable robust relation embeddings for previously unseen relations in knowledge graphs, yielding state-of-the-art performance (Li et al., 2022).
• Word and Sentence Similarity: Character n-gram aggregation models (e.g., Charagram) excel at word/sentence similarity and are faster and more robust than deeper LSTM/CNN architectures (Wieting et al., 2016, Kim et al., 2018).
• Language Modeling and Machine Translation: Multi-scale character n-gram embeddings with attention mechanisms improve RNN LLMs (lower perplexity, better BLEU scores) and headline generation (Takase et al., 2019).
• Classification in Resource-Constrained Settings: Hyperdimensional computing enables efficient n-gram statistics embedding, dramatically reducing memory and computation while retaining near-baseline F1 performance (Alonso et al., 2020).
• Transformer Interpretability and Curriculum: Analysis of transformer predictions via hierarchical n-gram statistics reveals that a significant fraction (up to 79% on TinyStories) of next-token predictions can be explained via hierarchical rulesets; overfitting detection and curriculum learning effects are also observed (Nguyen, 30 Jun 2024).
• Multi-Hop Reasoning and Hierarchical Inference: Manifold-projected hierarchical embeddings facilitate accurate mixed-hop and multi-hop prediction in medical and linguistic hierarchical datasets, outperforming Euclidean baselines (Patil et al., 25 May 2025).

These applications demonstrate the versatility and empirical competitive advantage of hierarchical n-gram embedding models across domains and tasks.

5. Advances in Geometric and Manifold-Based Hierarchical Embedding

Recent approaches integrate geometric principles to model hierarchy:

• Hyperbolic Spaces: Embeddings projected into Poincaré balls or Lorentzian manifolds reflect the exponential expansion of hierarchical relationships. Entities closer to the origin denote higher-level concepts; larger norms indicate finer semantic distinctions (Dhingra et al., 2018, Patil et al., 25 May 2025).
• Learnable Curvature and Norms: Hyperbolic norms are learned, modulating the embedding space's capacity to reflect dataset complexity (Patil et al., 25 May 2025).
• Manifold Projections in Transformers: Hierarchical Lexical Manifold Projection (HLMP) ensures multi-scale semantic representation, preserving coherence across localized and global linguistic structures. Modified self-attention incorporates manifold-aware terms for dynamic adaptation (Martus et al., 8 Feb 2025).
• Robustness and Interpretability: Manifold-projected hierarchical embeddings maintain stability under adversarial perturbations, enhance interpretability via tracing token movements across the manifold, and facilitate generalization across domains (Martus et al., 8 Feb 2025).

Geometric hierarchical embedding frameworks provide scalable and expressive means to encode complex linguistic hierarchies in neural LLMs.

6. Computational Efficiency and Scalability Considerations

Hierarchical n-gram embeddings address computational bottlenecks associated with traditional models:

• Feature Reduction: Clustering and composition reduce the feature space from tens of thousands to hundreds or less (Lebret et al., 2014).
• Segmentation-Free Modeling: Processing character sequences directly avoids errors in word segmentation, improving performance in languages with ambiguous boundaries (Kim et al., 2018).
• Distributed Encodings and Hashing: Hyperdimensional computing and hashing tricks (e.g., byteSteady) support memory-efficient, scalable modeling applicable to massive input spaces, including byte-level NLP and genomics (Alonso et al., 2020, Zhang et al., 2021).
• Linear-Time Sequence Modeling: Integration with Mamba2 selective state-space models enables hierarchical hyperbolic embeddings to be trained and deployed with linear computational complexity, making long-sequence hierarchical modeling feasible (Patil et al., 25 May 2025).
• Compression Techniques: Byte-level hierarchical n-gram representations withstand moderate sequence compression (e.g., Huffman coding) without notable loss of classification accuracy, offering novel accuracy-speed trade-offs (Zhang et al., 2021).

These advances mitigate the "curse of dimensionality," optimize inference latency, and enable deployment in real-world resource-constrained scenarios.

7. Limitations, Open Directions, and Broader Impact

Despite significant empirical successes, hierarchical n-gram embeddings face challenges:

• Implicit vs. Explicit Hierarchy: While some models encode hierarchy implicitly via geometric properties (e.g., norm in hyperbolic space), extracting explicit hierarchical relations remains non-trivial (Dhingra et al., 2018).
• Task Sensitivity: Hyperbolic embeddings are advantageous for tasks involving entailment and hierarchy but may underperform in similarity-focused evaluations compared to Euclidean spaces (Dhingra et al., 2018, Patil et al., 25 May 2025).
• Complexity Control: Hierarchical rulesets' expressivity grows rapidly with context length, presenting trade-offs between approximation power and computational overhead (Nguyen, 30 Jun 2024).
• Compositional Function Design: Summing embeddings is robust but may fail to capture non-linear interactions; future work calls for more sophisticated compositional operators (Kim et al., 2018).
• Domain Adaptation and Generalization: Structured manifold projections and hierarchical graphs demonstrate promise for cross-domain adaptability and robustness but require further validation on broader NLP benchmarks (Martus et al., 8 Feb 2025).
• Model Interpretability: HLMP and geometric approaches enable tracing of semantic shifts but demand more mature interpretability frameworks for practical model diagnosis.

A plausible implication is that further exploration of adaptive curvature, explicit hierarchical supervision, and integration with self-attention mechanisms will continue to refine the utility and generalizability of hierarchical n-gram embeddings.

---

In summary, hierarchical n-gram embeddings encompass a diverse and evolving family of methods for multi-scale lexical modeling. They combine compositional principles, clustering, structured manifolds, and geometric embedding spaces to achieve robust, efficient, and interpretable representations capable of supporting state-of-the-art performance in document classification, link prediction, semantic similarity, reasoning, and LLM analysis. Continuing research in hierarchical, multi-scale, and geometry-aware neural architectures is likely to advance the field further, both technically and in terms of broad linguistic coverage.