FlexCTC: GPU-Native CTC Decoder

• FlexCTC is an open-source, GPU-native decoding toolkit for Connectionist Temporal Classification models that delivers high-throughput, batched beam search with advanced contextualization.
• It eliminates CPU-GPU synchronization by leveraging custom CUDA kernels and CUDA Graph orchestration, thereby reducing kernel launch overhead and enhancing real-time performance.
• FlexCTC integrates n-gram language model fusion and phrase-level boosting to improve decoding accuracy and efficiency in diverse speech recognition applications.

FlexCTC is an open-source, fully GPU-native decoding toolkit purpose-built for Connectionist Temporal Classification (CTC) models, offering high-throughput, batched beam search and advanced contextualization capabilities. The system is implemented entirely in Python and PyTorch, utilizing custom CUDA kernels and orchestration via CUDA Graphs to replace traditional CPU-bound or WFST-based CTC decoders. FlexCTC supports GPU-parallelized n-gram LLM integration and phrase-level boosting, delivering efficient, accurate decoding suited for both research and production workloads (Grigoryan et al., 10 Aug 2025).

1. Architecture and Data Flow

FlexCTC’s architecture is characterized by a fully vectorized GPU-native CTC beam search decoder and a GPU-accelerated contextualization engine, encompassing two core modules: NGPU-LM for n-gram LLM fusion and GPU-PB for phrase-level boosting. All critical computation is executed on-GPU, eliminating CPU-GPU synchronization and reducing kernel launch overhead.

Input: The primary input is a log-probability tensor $D$ of shape $B \times T \times |V|$, where $B$ is batch size, $T$ the maximum number of time steps, and $|V|$ the vocabulary size. Accompanying the input is a vector $L \in \mathbb{N}^{B}$ encoding the true lengths of each utterance.

Beam Search Decoding: Decoding is performed in lock-step over $T$ frames for all $B$ utterances in the batch. At each time step $t$:

• $D_{[:,t,:]}$ is merged with accumulated beam scores (shape $B \times T \times |V|$0 for beam size $B \times T \times |V|$1).
• Insertion penalties, LLM (LM) contributions (weighted by $B \times T \times |V|$2), and boosting scores (weighted by $B \times T \times |V|$3) are summed with token-level log-probabilities.
• Top-$B \times T \times |V|$4 pruning is applied over all $B \times T \times |V|$5 candidate continuations using a flat selection followed by threshold-based truncation ($B \times T \times |V|$6). The best surviving hypotheses are maintained in a trie-based structure for efficient recombination and history tracking.
• On completion, end-of-sequence LM scores are applied, and the best hypothesis per utterance is returned.

All per-frame logic, including mask computations, score update, Top-$B \times T \times |V|$7 selection, and trie operations, are implemented as PyTorch tensor operations or via custom CUDA kernels, wrapped in a CUDA Graph for speed and determinism.

2. Decoding Algorithms and Mathematical Formulation

2.1 CTC Path Probabilities

For any path $B \times T \times |V|$8, its log-probability under the acoustic model is:

$B \times T \times |V|$9

where $B$0 is the log-probability of token $B$1 at time $B$2 for utterance $B$3.

2.2 Hypothesis Scoring with Context

Each decoded hypothesis $B$4 is scored as:

$B$5

where $B$6 and $B$7 are supplied by the n-gram LLM and phrase booster, respectively; $B$8 and $B$9 are tunable weights; $T$0 imposes an insertion penalty.

2.3 Per-Frame Beam Expansion

At each time $T$1, for beam entry $T$2 in utterance $T$3:

$T$4

The score tensor $T$5 is then flattened, Top-$T$6 candidates are selected, and scores are pruned to those within a threshold $T$7 of the maximum per utterance.

2.4 Pseudo-Code Sketch

$L \in \mathbb{N}^{B}$5

2.5 CUDA Graph Orchestration

The for-loop over $T$8 is captured as a CUDA Graph on the first batch. Subsequent batches reuse the prebuilt graph with a single cudaLaunchGraph invocation, removing kernel launch bottlenecks and CPU-GPU round-trips. Empirically, this strategy reduces kernel launch overhead by approximately 80–90%.

3. Contextualization Mechanisms

3.1 N-gram LLM Fusion (NGPU-LM)

A GPU-native n-gram LM processes $T$9 beam hypotheses in parallel, returning per-hypothesis next-token log probabilities. These LM contributions are combined with acoustic scores at each expansion, and, at $|V|$0, EOS transition scores are applied for each beam.

3.2 GPU-Accelerated Phrase Boosting (GPU-PB)

An Aho–Corasick trie is constructed for a list of user-supplied phrases (e.g., domain terms, product names). For each beam hypothesis, GPU-PB maintains and updates a trie state pointer, applying additive log-boosts to any path matching the phrase list. Boosts are proportionally applied to each prefix node, improving early detection of long phrases over standard end-only reward schemes.

Both mechanisms are fully integrated into the batched beam search and operate in GPU memory, yielding maximal throughput and minimal latency.

4. Empirical Performance and Evaluation

FlexCTC was evaluated using the Fast Conformer CTC Large model ($|V|$1115M parameters, subword BPE vocab size 1,024) across several datasets: SPGISpeech (financial, 5,000h), Earnings21 (financial, 39h, with domain term list), and MultiMed (medical, customized list, 1,000 terms). Key metrics were word error rate (WER), F-score (for boosted terms), and inverse real-time factor (RTFx). All experiments used NVIDIA A5000 GPU (float32) and Intel i9-10940X CPU.

4.1 Greedy vs Beam and Customization Effects

Mode	Customization	MultiMed WER	F-score	RTFx	Earnings21 WER	F-score	RTFx
Greedy	none	15.09%	55.5	2,804	15.02%	69.3	2,687
Greedy	LM+PB	14.83%	60.8	2,573	14.73%	72.2	2,445
Beam (8)	none	15.09%	55.5	2,306	15.02%	69.2	2,096
Beam (8)	LM+PB	13.55%	74.2	1,995	13.74%	79.6	1,885

The addition of LM and phrase boosting substantially decreases WER and increases F-score, with modest computational overhead.

4.2 Comparison to Other Decoders

Method	Beam	SPGI WER	SPGI RTFx	Earnings21 WER	Earnings21 RTFx	MultiMed WER	MultiMed RTFx
PyCTCDecode	4	4.69%	1,210	14.37%	1,129	15.45%	1,188
CUDA WFST	1,000	4.64%	1,065	14.33%	419	16.77%	778
Flashlight	4	4.48%	1,071	14.02%	1,100	14.17%	1,279
FlexCTC	4	4.48%	2,085	13.98%	2,084	14.16%	2,212
Flashlight	16	4.39%	454	13.89%	686	13.93%	631
FlexCTC	16	4.38%	1,928	13.89%	1,835	13.93%	1,956

FlexCTC consistently matches or outperforms other systems in WER while providing 2–3× higher RTFx.

4.3 Phrase Boosting Efficiency

For 100-phrase boosting on Earnings21 ($|V|$2, $|V|$3):

Method	WER	F-score (100)	RTFx
Greedy	15.02%	73.2	2,687
PyCTCDecode (100)	15.16%	76.7	28
FlexCTC (100)	14.93%	78.2	2,068

FlexCTC preserves high speed even when performing complex phrase-level context integration.

4.4 Scalability

As batch size increases (up to $|V|$4, $|V|$5), the real-time factor of beam search approaches that of greedy decoding, with only $|V|$610% overhead. Increasing beam size from $|V|$7 to $|V|$8 steadily improves WER ($|V|$9) and F-score ($L \in \mathbb{N}^{B}$0), with only a 2.5$L \in \mathbb{N}^{B}$1 drop in RTFx.

5. Extensibility and Practical Integration

FlexCTC is accessed via a PyTorch-compatible API embedded in the NeMo toolkit. Key configuration options include beam size ($L \in \mathbb{N}^{B}$2), pruning threshold ($L \in \mathbb{N}^{B}$3), insertion penalty ($L \in \mathbb{N}^{B}$4), external language/boosting models, and flags for blank-skipping or CUDA Graph utilization. Integration is demonstrated via:

$L \in \mathbb{N}^{B}$6

No compilation or specialized graph construction is required for deployment into CTC-based pipelines; all functionality is exposed via Python/CUDA with minimal extra dependencies and maximal hardware utilization.

6. Summary

FlexCTC enables high-throughput, contextually adaptive CTC decoding exclusively on GPU hardware. Its design incorporates vectorized, trie-based beam search; efficient CUDA Graph orchestration; GPU-resident n-gram LMs and phrase boosting; and user-oriented PyTorch APIs. Empirical benchmarks validate FlexCTC’s substantial improvements in real-time decoding speed (2–3× faster RTFx) with matching or superior error rates compared to CPU and WFST-based systems (Grigoryan et al., 10 Aug 2025).