---
title: 'FlexCTC: GPU-Native CTC Decoder'
url: https://www.emergentmind.com/topics/flexctc
type: topic
---

# FlexCTC: GPU-Native CTC Decoder

FlexCTC is an open-source, fully GPU-native decoding toolkit purpose-built for Connectionist Temporal Classification (CTC) models, offering high-throughput, batched beam search and advanced contextualization capabilities. The system is implemented entirely in Python and PyTorch, utilizing custom CUDA kernels and orchestration via CUDA Graphs to replace traditional CPU-bound or WFST-based CTC decoders. FlexCTC supports GPU-parallelized n-gram language model integration and phrase-level boosting, delivering efficient, accurate decoding suited for both research and production workloads [2508.07315].

## 1. Architecture and Data Flow

FlexCTC’s architecture is characterized by a fully vectorized GPU-native CTC beam search decoder and a GPU-accelerated contextualization engine, encompassing two core modules: NGPU-LM for n-gram language model fusion and GPU-PB for phrase-level boosting. All critical computation is executed on-GPU, eliminating CPU-GPU synchronization and reducing kernel launch overhead.

**Input:** The primary input is a log-probability tensor $D$ of shape $B \times T \times |V|$, where $B$ is batch size, $T$ the maximum number of time steps, and $|V|$ the vocabulary size. Accompanying the input is a vector $L \in \mathbb{N}^{B}$ encoding the true lengths of each utterance.

**Beam Search Decoding:** Decoding is performed in lock-step over $T$ frames for all $B$ utterances in the batch. At each time step $t$:
- $D_{[:,t,:]}$ is merged with accumulated beam scores (shape $B \times K$ for beam size $K$).
- Insertion penalties, language model (LM) contributions (weighted by $\alpha_{LM}$), and boosting scores (weighted by $\alpha_{BT}$) are summed with token-level log-probabilities.
- Top-$K$ pruning is applied over all $B \times K \times |V|$ candidate continuations using a flat selection followed by threshold-based truncation ($\theta$). The best surviving hypotheses are maintained in a trie-based structure for efficient recombination and history tracking.
- On completion, end-of-sequence LM scores are applied, and the best hypothesis per utterance is returned.

All per-frame logic, including mask computations, score update, Top-$K$ selection, and trie operations, are implemented as PyTorch tensor operations or via custom CUDA kernels, wrapped in a CUDA Graph for speed and determinism.

## 2. Decoding Algorithms and Mathematical Formulation

### 2.1 CTC Path Probabilities

For any path $\pi=(\pi_1,\ldots,\pi_T)$, its log-probability under the acoustic model is:
$$
\log P_\textrm{CTC}(\pi|X) = \sum_{t=1}^T \log D_{b,t,\pi_t}
$$
where $D_{b,t,v}$ is the log-probability of token $v$ at time $t$ for utterance $b$.

### 2.2 Hypothesis Scoring with Context

Each decoded hypothesis $h$ is scored as:
$$
s(h) = \log P_\textrm{CTC}(h) + \alpha_{LM} \log P_{LM}(h) + \alpha_{BT} \log P_{PB}(h) + \beta\,|h|
$$
where $\log P_{LM}$ and $\log P_{PB}$ are supplied by the n-gram language model and phrase booster, respectively; $\alpha_{LM}$ and $\alpha_{BT}$ are tunable weights; $\beta$ imposes an insertion penalty.

### 2.3 Per-Frame Beam Expansion

At each time $t$, for beam entry $k$ in utterance $b$:
$$
\log p_{b,k,v} = D_{b,t,v} + \text{acc}_{b,k}
                 + \beta\, \mathbb{1}_{v \ne B, v \ne \mathrm{last}(b,k)}
                 + \alpha_{LM} \cdot \mathrm{lm}_{b,k} \cdot \mathbb{1}_{v \notin \{B, \mathrm{last}(b,k)\}}
                 + \alpha_{BT} \cdot \mathrm{pb}_{b,k} \cdot \mathbb{1}_{v \notin \{B, \mathrm{last}(b,k)\}}
$$
The score tensor $(B, K, |V|)$ is then flattened, Top-$K$ candidates are selected, and scores are pruned to those within a threshold $\theta$ of the maximum per utterance.

### 2.4 Pseudo-Code Sketch

```python
for t in 0…T−1:
    active = (t < L)
    logp = D[:,:,t,:]  +  acc_scores[:,:,None]
    logp += β * (~repeat_or_blank)
    if use_LM:  logp[:,:,~mask] += α_LM * lm_scores[:,:,None]
    if use_PB:  logp[:,:,~mask] += α_BT * pb_scores[:,:,None]
    acc_scores, idx = TopK(logp.view(B, K*|V|), K)
    prune_mask = acc_scores < (acc_scores.max(axis=1, keepdim=True) - θ)
    acc_scores[prune_mask] = −∞
    # Update LM/PB states if necessary
    # Update beams in trie
```

### 2.5 CUDA Graph Orchestration

The for-loop over $t$ is captured as a CUDA Graph on the first batch. Subsequent batches reuse the prebuilt graph with a single `cudaLaunchGraph` invocation, removing kernel launch bottlenecks and CPU-GPU round-trips. Empirically, this strategy reduces kernel launch overhead by approximately 80–90%.

## 3. Contextualization Mechanisms

### 3.1 N-gram Language Model Fusion (NGPU-LM)

A GPU-native n-gram LM processes $B \times K$ beam hypotheses in parallel, returning per-hypothesis next-token log probabilities. These LM contributions are combined with acoustic scores at each expansion, and, at $t = T$, EOS transition scores are applied for each beam.

### 3.2 GPU-Accelerated Phrase Boosting (GPU-PB)

An Aho–Corasick trie is constructed for a list of user-supplied phrases (e.g., domain terms, product names). For each beam hypothesis, GPU-PB maintains and updates a trie state pointer, applying additive log-boosts to any path matching the phrase list. Boosts are proportionally applied to each prefix node, improving early detection of long phrases over standard end-only reward schemes.

Both mechanisms are fully integrated into the batched beam search and operate in GPU memory, yielding maximal throughput and minimal latency.

## 4. Empirical Performance and Evaluation

FlexCTC was evaluated using the Fast Conformer CTC Large model ($\approx$115M parameters, subword BPE vocab size 1,024) across several datasets: SPGISpeech (financial, 5,000h), Earnings21 (financial, 39h, with domain term list), and MultiMed (medical, customized list, 1,000 terms). Key metrics were word error rate (WER), F-score (for boosted terms), and inverse real-time factor (RTFx). All experiments used NVIDIA A5000 GPU (float32) and Intel i9-10940X CPU.

### 4.1 Greedy vs Beam and Customization Effects

| Mode      | Customization | MultiMed WER | F-score | RTFx | Earnings21 WER | F-score | RTFx |
|-----------|--------------|--------------|---------|------|----------------|---------|------|
| Greedy    | none         | 15.09%       | 55.5    | 2,804| 15.02%         | 69.3    | 2,687|
| Greedy    | LM+PB        | 14.83%       | 60.8    | 2,573| 14.73%         | 72.2    | 2,445|
| Beam (8)  | none         | 15.09%       | 55.5    | 2,306| 15.02%         | 69.2    | 2,096|
| Beam (8)  | LM+PB        | 13.55%       | 74.2    | 1,995| 13.74%         | 79.6    | 1,885|

The addition of LM and phrase boosting substantially decreases WER and increases F-score, with modest computational overhead.

### 4.2 Comparison to Other Decoders

| Method         | Beam | SPGI WER | SPGI RTFx | Earnings21 WER | Earnings21 RTFx | MultiMed WER | MultiMed RTFx |
|----------------|------|----------|-----------|----------------|-----------------|--------------|--------------|
| PyCTCDecode    | 4    | 4.69%    | 1,210     | 14.37%         | 1,129           | 15.45%       | 1,188        |
| CUDA WFST      | 1,000| 4.64%    | 1,065     | 14.33%         | 419             | 16.77%       | 778          |
| Flashlight     | 4    | 4.48%    | 1,071     | 14.02%         | 1,100           | 14.17%       | 1,279        |
| FlexCTC        | 4    | 4.48%    | 2,085     | 13.98%         | 2,084           | 14.16%       | 2,212        |
| Flashlight     | 16   | 4.39%    | 454       | 13.89%         | 686             | 13.93%       | 631          |
| FlexCTC        | 16   | 4.38%    | 1,928     | 13.89%         | 1,835           | 13.93%       | 1,956        |

FlexCTC consistently matches or outperforms other systems in WER while providing 2–3× higher RTFx.

### 4.3 Phrase Boosting Efficiency

For 100-phrase boosting on Earnings21 ($B=32$, $K=8$):

| Method           | WER    | F-score (100) | RTFx  |
|------------------|--------|--------------|-------|
| Greedy           | 15.02% | 73.2         | 2,687 |
| PyCTCDecode (100)| 15.16% | 76.7         | 28    |
| FlexCTC (100)    | 14.93% | 78.2         | 2,068 |

FlexCTC preserves high speed even when performing complex phrase-level context integration.

### 4.4 Scalability

As batch size increases (up to $B=128$, $K=4$), the real-time factor of beam search approaches that of greedy decoding, with only $\sim$10% overhead. Increasing beam size from $K=4$ to $K=128$ steadily improves WER ($13.82\% \rightarrow 13.17\%$) and F-score ($72.1\% \rightarrow 77.9\%$), with only a 2.5$\times$ drop in RTFx.

## 5. Extensibility and Practical Integration

FlexCTC is accessed via a PyTorch-compatible API embedded in the NeMo toolkit. Key configuration options include beam size ($K$), pruning threshold ($\theta$), insertion penalty ($\beta$), external language/boosting models, and flags for blank-skipping or CUDA Graph utilization. Integration is demonstrated via:

```python
from nemo.collections.asr.modules import CTCBeamDecoder

decoder = CTCBeamDecoder(
    beam_size=8,
    lm_path="6gram_subword.arpa",
    lm_weight=0.5,
    boosting_path="medical_terms.txt",
    boosting_weight=1.0,
    blank_skip_threshold=0.98,
    capture_cuda_graph=True
)

# Log-probs: Tensor[B, T, V], lengths: Tensor[B]
beam_hyps, scores = decoder(log_probs, lengths)
transcripts = [hyp.topk_transcript(0) for hyp in beam_hyps]
```

No compilation or specialized graph construction is required for deployment into CTC-based pipelines; all functionality is exposed via Python/CUDA with minimal extra dependencies and maximal hardware utilization.

## 6. Summary

FlexCTC enables high-throughput, contextually adaptive CTC decoding exclusively on GPU hardware. Its design incorporates vectorized, trie-based beam search; efficient CUDA Graph orchestration; GPU-resident n-gram LMs and phrase boosting; and user-oriented PyTorch APIs. Empirical benchmarks validate FlexCTC’s substantial improvements in real-time decoding speed (2–3× faster RTFx) with matching or superior error rates compared to CPU and WFST-based systems [2508.07315].

Source: https://www.emergentmind.com/topics/flexctc