Fast Graph Decoder (FGD)

• The paper converts full softmax computation into a fast Euclidean nearest neighbor search via an inner-product-preserving transformation, reducing complexity from O(|V|) to O(log|V|).
• FGD is a graph-based algorithm that constructs a small-world index over transformed word embeddings to efficiently retrieve top-K tokens during decoding.
• FGD achieves significant runtime improvements in neural machine translation and language modeling with provable guarantees and minimal impact on precision.

The Fast Graph Decoder (FGD) is a graph-based algorithmic framework for accelerating one of the bottlenecks in neural sequence decoding: the softmax over large vocabularies in neural LLMs (NLMs). FGD navigates a specially constructed small-world nearest-neighbor graph over transformed word embeddings to efficiently retrieve the top-$K$ most likely output tokens for a given context. It achieves substantial speedups versus conventional full softmax computation, with negligible impact on accuracy, through a mathematically principled reduction of the top-$K$ inner product search to efficient graph-based Euclidean nearest neighbor search. FGD demonstrates strong empirical and theoretical performance, including orders-of-magnitude latency reductions in neural machine translation (NMT) and language modeling tasks, while providing provable approximation guarantees (Zhang et al., 2018).

1. Motivation and Problem Setting

Conventional beam-search decoding in NLMs requires evaluating a full softmax over vocabulary $V$ (size $|V|$) for each extension of a partial hypothesis. Given context $h \in \mathbb{R}^D$, the unnormalized logit for candidate word $i$ is $s_i = h^\top e_i + b_i$, with $e_i$ and $b_i$ the decoder output embedding and bias, respectively. The full softmax cost is $O(D |V|)$ per context, which becomes prohibitive for vocabularies in the tens or hundreds of thousands.

Crucially, only the top-$K$0 hypotheses are needed at each step of beam search, with $K$1. FGD directly capitalizes on this by seeking a sublinear-in-$K$2 algorithm to find the indices $K$3, rather than computing all $K$4 softmax scores (Zhang et al., 2018).

2. Inner-Product to Nearest-Neighbor Formulation

FGD employs an inner-product-preserving transformation (IPPT) to recast the top-$K$5 logit search as an equivalent nearest-neighbor search in Euclidean space. For each vocabulary item $K$6 with $K$7 and $K$8, define

$K$9

and

$V$0

Similarly, the context is lifted to $V$1. Then, the original logit $V$2 is shown to be

$V$3

This reduces the argmax over logits to a nearest-neighbor minimization with respect to Euclidean distance, i.e., finding the $V$4 vocabulary indices with smallest $V$5 [(Zhang et al., 2018), Lemma 3.1].

3. Graph Index Construction

The FGD index is a navigable small-world graph (typically HNSW) constructed offline from the set $V$6 over the vocabulary. Each node corresponds to a transformed embedding; local edges connect to $V$7 nearest neighbors by Euclidean distance, and additional long-range shortcuts ensure a logarithmic diameter.

Graph construction is $V$8, performed once per vocabulary. The resulting graph supports fast nearest-neighbor search via greedy and beam exploration, facilitating rapid top-$V$9 retrieval during inference (Zhang et al., 2018).

4. Fast Graph Decoder Algorithm

FGD operates in two phases:

• Offline phase (FGD-P): Transform all embeddings and biases to $|V|$0, construct the small-world graph $|V|$1.
• Online phase (FGD-I): For context $|V|$2, lift to $|V|$3, then run graph search (e.g., HNSW beam search with parameter $|V|$4) to find $|V|$5 nearest neighbors to $|V|$6 in $|V|$7. The corresponding indices yield the top-$|V|$8 logits. These can be renormalized for a top-$|V|$9 approximate softmax.

The per-context online runtime is $h \in \mathbb{R}^D$0 when $h \in \mathbb{R}^D$1 is well-formed and $h \in \mathbb{R}^D$2, a dramatic reduction from linear softmax (Zhang et al., 2018).

Pseudocode Outline

$s_i = h^\top e_i + b_i$9

5. Theoretical Guarantees

The IPPT ensures the top-$h \in \mathbb{R}^D$3 by logit are exactly the nearest neighbors in the transformed space (Theorem 3.2). For approximate search, the paper provides explicit error bounds on the relative deviation between FGD and the true full softmax probability for each target, as a function of the lowest logit recovered and the known lower bound for all scores. If precision@$h \in \mathbb{R}^D$4=1 (all actual top-$h \in \mathbb{R}^D$5 are found), the bound approaches zero as $h \in \mathbb{R}^D$6.

This theoretical foundation guarantees that FGD is provably lossless when the graph search finds the exact nearest neighbors, and quantifies the degradation for approximate retrieval (Zhang et al., 2018).

6. Empirical Evaluation and Performance

Extensive experiments demonstrate that FGD achieves substantial speedups with minimal precision loss:

• Neural Machine Translation (IWSLT’14 De$h \in \mathbb{R}^D$7En, $h \in \mathbb{R}^D$8k, $h \in \mathbb{R}^D$9): With $i$0, FGD yields a per-step softmax time of 0.43 ms (14$i$1 faster than full softmax), with BLEU within $i$2 points.
• Language Modeling (WikiText-2, $i$3 up to 80k): FGD achieves $i$4--$i$5 speedup at $i$6k, with top-10 precision exceeding 0.95 for $i$7.

The following table summarizes key results:

Task	Full Softmax Time	FGD Time ($i$8=50)	BLEU (NMT)	Top-10 Precision
NMT (IWSLT’14)	6.30 ms	0.43 ms	29.06	--
LLM	$i$9	$s_i = h^\top e_i + b_i$0	--	>0.95

A trade-off exists between $s_i = h^\top e_i + b_i$1 (search effort) and recall: larger $s_i = h^\top e_i + b_i$2 increases both recall and latency.

7. Discussion, Limitations, and Extensions

FGD delivers a scalable solution for top-$s_i = h^\top e_i + b_i$3 selection in vocabulary-intensive NLM applications, offering order-of-magnitude runtime improvements with controlled:

• Memory overhead for the graph index
• Offline construction cost (minutes for $s_i = h^\top e_i + b_i$4 on the order of $s_i = h^\top e_i + b_i$5k)
• Heuristic search (no absolute guarantee of full recall from HNSW, though empirical precision is $s_i = h^\top e_i + b_i$6 at $s_i = h^\top e_i + b_i$7 for typical settings)
• Tuning parameter $s_i = h^\top e_i + b_i$8 mediates the speed-accuracy tradeoff

Possible directions include support for online/streaming vocabulary updates, adaptation to alternative vector metrics (e.g., cosine distance), and integration into GPU-accelerated search pipelines. The FGD methodology underpins advancements in fast beam search for NMT, language modeling, and other large-vocabulary inference contexts (Zhang et al., 2018).