Prefix Ngrams in Corpus Analysis

• Prefix ngrams are consecutive token sequences sharing an initial substring, essential for next-word prediction, probability estimation, and corpus pattern mining.
• Compressed trie-based indexing uses per-context remapping and succinct Elias–Fano encoding to achieve near-optimal space usage with rapid query lookup.
• FM-index approaches leverage the Burrows–Wheeler Transform and wavelet trees to support sublinear storage and arbitrary substring queries on massive corpora.

A prefix ngram is a sequence of consecutive tokens (words, characters, or bytes) that occur within a larger corpus and share a fixed prefix of length $k$, that is, all ngrams containing an initial substring $C = w_1,\dots,w_k$. Efficient indexing and querying of such prefix ngrams is fundamental in language modeling, search, and large-scale corpus analysis. State-of-the-art systems support prefix ngram queries using two broad approaches: compressed trie-based indexes with per-context remapping and high-throughput methods based on compressed full-text indexes such as the FM-index. Both paradigms enable efficient prefix enumeration, probability estimation, and corpus occurrence statistics, each optimizing for different space–time tradeoffs (Pibiri et al., 2018, Xu et al., 13 Jun 2025).

1. Formalization and Properties of Prefix Ngrams

Given a vocabulary $\Sigma$ of size $V$ and a corpus $T$, a prefix ngram query with prefix $C \in \Sigma^{k}$ seeks all ngrams of length $n \geq k+1$ observed in $T$ such that their initial $k$ tokens are precisely $C$. Formally, for context $C$, define

$$\operatorname{follow}(C) = \{ w \in \Sigma: (C, w) \text{ observed in } T \},$$

and $m_C = |\operatorname{follow}(C)|$ as the context fanout. Prefix ngram enumeration underpins tasks such as next-word prediction, probability estimation, and large-scale pattern mining. Efficient support for prefix queries crucially depends on minimizing both the space required per ngram and the time per query, particularly when $V$, $N$, and corpus size are large (Pibiri et al., 2018).

2. Compressed Trie-Based Prefix Ngram Indexing

The compressed trie approach organizes all observed ngrams up to order $N$ in a trie of depth $N$, where each level $\ell$ corresponds to the set of $\ell$-grams observed in $T$, arranged in context order. Nodes at depth $k$ enumerate all possible $k$-length prefixes $C$, and their child edges encode the set $\operatorname{follow}(C)$. Instead of absolute word IDs, a per-context integer remapping scheme maps each $w \in \operatorname{follow}(C)$ to an integer $\operatorname{rank}_C(w) \in [0, m_C-1]$. The remapped sequences form a non-decreasing sequence amenable to succinct encoding with methods such as Elias–Fano coding:

$$\sum_{C} \lceil \log_2 m_C \rceil + 2M \;\text{bits,}$$

with $M = \sum_C m_C$ the total number of trie edges. This space is near-optimal up to an additive constant and far below dense ID encodings for large $V$ (Pibiri et al., 2018).

Trie construction proceeds from a deduplicated, count-annotated list of ngrams extracted via a sliding window over the text, followed by an external sort in context order. A single scan of the sorted data suffices to remap IDs, accumulate statistical estimates (counts, probabilities, backoffs), and assign array ranges for trie traversal. Elias–Fano or Partitioned Elias–Fano (PEF) are then applied to the resulting monotone integer sequences (Pibiri et al., 2018).

3. Prefix Ngram Query Algorithms and Complexity

Prefix queries in the compressed trie consist of mapping the prefix $C$ to its ID sequence, then descending the trie level-by-level. At each level $\ell$, a binary search over the remapped child array yields the position for $w_\ell$. Upon reaching depth $k$, the block of successors of $C$ is enumerated by inverting the local remapping, yielding all $w \in \operatorname{follow}(C)$:

lookup_prefix(C): pos = 0 for ℓ in 1..k: id = vocab[C[ℓ]] [b,e) = ptrs[ℓ-1][pos..pos+1] pos = binary_search(level[ℓ].ids, b,e, id) [b,e) = ptrs[k][pos..pos+1] for i = b … e-1: out_word = inv_rank[k][i] emit C·out_word

Complexity is $O(k \cdot \log m_{\max}) + O(\#\text{output})$, where $m_{\max}$ is the largest fanout—typically $m_{\max} \ll V$ in natural language. Empirical lookup times are 1–3 μs per prefix on billion-scale datasets for $k=1,2$ (Pibiri et al., 2018). Variable-length prefixes are accommodated by storing pointers at levels up to $K$ and using a context jump table.

4. FM-Index–Based Prefix Ngram Search at Scale

The FM-index paradigm generalizes prefix ngram queries to arbitrary substring pattern queries, leveraging the Burrows–Wheeler Transform (BWT) and wavelet trees. In the FM-index, all suffixes of the corpus are sorted, and the BWT string $L$ is constructed. The LF-mapping allows for backward-search: given a pattern $Q$, the interval $[\ell, r)$ in the suffix array corresponding to all suffixes prefixed by $Q$ is computed by repeated application of rank queries on $L$:

$$\ell_i = C[q_i] + \mathrm{rank}(q_i, \ell_{i+1} - 1), \qquad r_i = C[q_i] + \mathrm{rank}(q_i, r_{i+1} - 1)$$

The interval $[\ell_0, r_0)$ then identifies all corpus positions where $Q$ occurs as a prefix. This enables immediate support for prefix ngram queries, as well as infix and suffix queries. To enumerate continuations of a prefix $P$, for each possible token $c$, check if $$\mathrm{rank}(c, r-1) - \mathrm{rank}(c, \ell-1) > 0$$ (Xu et al., 13 Jun 2025).

5. Empirical Space–Time Tradeoffs and System Comparisons

The following table summarizes space and query time across leading systems:

Method	Storage (× raw)	Query Time	Corpus Size
Suffix automaton	29×	$O(m)$	1.3 TB
Suffix array	6×	$O(m+\log n)$	12 TB
ElasticSearch	2×	$O(m)$	35 TB
FM-index	0.44×	$O(m H_0)$	46 TB

Trie-based (remapped Elias–Fano) approaches achieve $<10$ B/ngram, with prefix lookup in $1$–$3$ μs for $k=1$. FM-index methods (as in Infini-gram mini) compress 46 TB of Internet text to 0.44× its raw size, supporting pattern queries on disk with RAM usage of a few GB and counting queries in $0.4$–$8$ s for short/long input. Suffix automatons and arrays use more space but allow for faster in-RAM querying on smaller corpora. Compressed trie and FM-indexing excel for disk-based, massive-scale deployments (Pibiri et al., 2018, Xu et al., 13 Jun 2025).

6. Applications, Best Practices, and Limitations

Prefix ngram queries underpin next-token prediction, autocompletion, LLM estimation, and large-scale contamination analysis. For static ngram collections and latency-critical applications (e.g., autocomplete, speech), compressed tries with $k=1$ remapping balance space and time optimally. FM-indexing is preferred for sublinear storage and when supporting arbitrary substring queries on petabyte-scale corpora with only external memory. Sharding large corpora ($\leq$700 GB/node), memory-mapping indexes, and parallelizing across nodes are key best practices. Sampling rates in FM-index tune the tradeoff between index size and locate/reconstruction latency (Pibiri et al., 2018, Xu et al., 13 Jun 2025).

A plausible implication is that for scenarios with dynamic updates, in-RAM tries combined with periodic rebuilds offer practical viability, while entirely dynamic compressed FM-indexes remain challenging. FM-indexed systems such as Infini-gram mini have revealed large-scale benchmark contamination, demonstrating the utility of scalable, exact prefix-query systems for corpus quality control in the era of web-scale LLMs (Xu et al., 13 Jun 2025).