GPU-Accelerated Word Boosting Tree

• GPU-accelerated word boosting trees are specialized trie-based data structures engineered for real-time, context-sensitive scoring in ASR applications.
• They employ highly-parallel GPU kernels and compressed-sparse-row arrays to efficiently manage tens of thousands of phrase sequences with negligible latency.
• Empirical benchmarks demonstrate significant improvements in keyword F-score and throughput over CPU-bound methods, enabling seamless integration in production decoding.

A GPU-Accelerated Word Boosting Tree is a specialized data structure deployed on modern GPUs to provide efficient, large-scale, context-sensitive real-time scoring for select sequences of tokens—typically applied in applications such as phrase boosting for automatic speech recognition (ASR) and context-aware decoding. These systems synthesize trie-based automata with boosting score logic, implement both the data structure and inference entirely in GPU memory, and leverage highly-parallel, branch-minimal kernels—typically compiled with CUDA or Triton—to achieve negligible latency and robust throughput even for phrase repositories containing tens of thousands of sequences. The approach reliably reduces CPU bottlenecks and avoids the overheads of generalized batch search or dynamic host-device synchronization (Andrusenko et al., 9 Aug 2025).

1. Data Structure Construction and Representation

The foundational structure of a GPU-accelerated word boosting tree is a trie or finite automaton constructed over a set of context phrases, generally via the Aho–Corasick algorithm. Let $\mathcal{L}$ denote the phrase lexicon and $\Sigma$ the vocabulary. The automaton is $PT = (Q, q_0, \delta, f)$, with:

• $Q$: States associated with phrase prefixes,
• $q_0$: Root state,
• $\delta(q, t)\rightarrow q'$: Trie transitions for token $t$,
• $f(q)$: Failure links for efficient backoff during partial matches.

Each transition $(q \overset{t}{\to} q')$ is annotated with a context boost score: $$w_q(t) = \begin{cases} c_0, & \text{if depth}(q')=1\ c_0{\times}\beta + \ln(\mathrm{depth}(q')), & \text{otherwise} \end{cases}$$ where $\Sigma$0 (base context weight) and $\Sigma$1 (depth-scaling) are tunable. The structure is pruned at finals to avoid negative backoff. All transitions—including failure links—are flattened into compressed-sparse-row (CSR) style arrays: from_state, token_id, to_state, arc_score, and state_offset. This linearization enables direct mapping to device memory with maximized coalesced reads (Andrusenko et al., 9 Aug 2025).

2. GPU Kernel Design and Parallel Query Execution

Inference is performed by launching a thread for each $\Sigma$2 pair, where $\Sigma$3 indexes the beam/hypothesis and $\Sigma$4 indexes the vocabulary. At each decoding step:

• The current automaton state $\Sigma$5 and candidate token $\Sigma$6 define the search space.
• Using state_offset and token_id arrays, an in-state binary or warp search retrieves the next state and associated boost score from arc_score and to_state.
• If a transition is absent, the kernel applies a default $\Sigma$7 score and resets to the root.

The kernel emits the updated state and boost score for each $\Sigma$8 in log-additive form, suitable for shallow fusion. No device synchronization is required beyond warp-level intrinsics, as each write is disjoint. The Triton/CUDA implementation is tuned for block shapes (e.g., $\Sigma$9) that take optimal advantage of GPU SM occupancy and memory bandwidth. All computation occurs in log-space to align with decoder score conventions and minimize numeric instability (Andrusenko et al., 9 Aug 2025).

3. Mathematical Formulation and Decoding Integration

Phrase boosting is formally integrated into shallow fusion decoding as: $PT = (Q, q_0, \delta, f)$0 where $PT = (Q, q_0, \delta, f)$1 is the context-biasing/phrase-boosting model. At each step, the model computes the combined score: $PT = (Q, q_0, \delta, f)$2 with $PT = (Q, q_0, \delta, f)$3 as the log ASR score and $PT = (Q, q_0, \delta, f)$4. The reward scheme $PT = (Q, q_0, \delta, f)$5 provides monotonic depth rewards, essential for greedy search. All computation is stateless across decoding steps apart from the current automaton state index, and scaling to large phrase sets ($PT = (Q, q_0, \delta, f)$6) relies on the fixed memory layout and fully vectorized lookup operation (Andrusenko et al., 9 Aug 2025).

4. Computational Complexity and Device Utilization

Given:

• $PT = (Q, q_0, \delta, f)$7: decoding steps,
• $PT = (Q, q_0, \delta, f)$8: beam size,
• $PT = (Q, q_0, \delta, f)$9: vocabulary size ($Q$01024 typical for BPE or wordpiece),
• $Q$1: average arc count per state,
• $Q$2: physical GPU parallelism.

The complexity per utterance is:

• CPU: $Q$3 due to transition searches,
• GPU: $Q$4 wall time, with $Q$5 logical per-thread; for high $Q$6, bottleneck shifts almost entirely to memory throughput.

Device memory footprint is modest: $Q$7 arcs $Q$8 12 bytes plus $Q$9 state indices, negligible relative to typical GPU memory, even for $q_0$0 phrases. The approach scales to large context-phrase inventories with minimal runtime and negligible memory allocation overhead per step (Andrusenko et al., 9 Aug 2025).

5. Empirical Results and Comparative Performance

Comprehensive benchmarking demonstrates robust improvements:

• On CSTalks with RNN-T (beam size 8, $q_0$1): F-score for keywords increases from 44.2% (no-bias) to 82.9%; WER improves from 12.8% to 9.6%. Real-time factor remains nearly unchanged (RTFx drops by only 3.1%).
• With greedy search: F-score 70.4%, WER 10.7%, and only a 2% speed penalty.
• With 20\,000 context phrases: only a 5% runtime increase; F-score loss is minor for beam search (81.1% to 78.4%), with WER remaining well below no-bias baseline.
• Against open-source baselines, GPU-PB outperforms CPU-bound methods by large factors both in F-score and RTFx (e.g., Pyctcdecode RTFx=29, GPU-PB RTFx=1\,991 for CTC greedy) (Andrusenko et al., 9 Aug 2025).

6. Integration in Production Decoding and Broader Significance

GPU word/phrase boosting trees are integrated as drop-in modules (e.g., TurboBiasContextLM in NeMo) supporting all prevalent ASR decoding paradigms—CTC, RNN-T, AED. The phrase tree is compiled once per context set, uploaded to device memory, and applied at each decoding time step with no host intervention and negligible device overhead. Because all computation is performed with static device memory buffers, GPU contexts can be maintained across batched inference or streaming decode. This design paradigm extends naturally to any context-driven sequence decoding where latency and throughput constraints prohibit host-device round-trips or generalized LM rescoring.

A plausible implication is that this approach generalizes to other domain-specific boosting or context-sensitive decoding tasks where dynamic phrase sets, constrained recognition, or shallow fusion are required over large vocabularies and phrase inventories.

---

References:

• "TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Tree" (Andrusenko et al., 9 Aug 2025)