---
title: Hierarchical-CTC Pre-training in ASR
url: https://www.emergentmind.com/topics/hierarchical-ctc-pre-training
type: topic
---

# Hierarchical-CTC Pre-training in ASR

Hierarchical-CTC pre-training refers to a family of strategies that employ auxiliary Connectionist Temporal Classification (CTC) losses at intermediate layers of a deep neural encoder, with each auxiliary loss supervising the model at a different linguistic granularity. The method is motivated by the abstraction gap in end-to-end automatic speech recognition (ASR), where directly predicting word-level sequences from speech signals poses a significant representational challenge. Hierarchical-CTC pre-training uses losses over progressively coarser subword vocabularies at deeper encoder layers, often with explicit conditioning dependencies, to guide the model toward learning effective multi-scale linguistic representations. Empirical results demonstrate improved downstream recognition performance and data efficiency, especially under limited-resource scenarios [2110.04109][1807.06234].

## 1. Architectural Principles

Hierarchical-CTC models are typically built on deep encoders (e.g., Transformers, Conformers, bidirectional LSTMs), where input speech representations pass through several stacked layers. Key architectural dimensions include:

- **Input and Front-End:** Pre-processed features frequently involve mel-filterbanks (e.g., 80-dim) and pitch (e.g., 3-dim), optionally augmented with CNN subsampling, SpecAugment, and speed perturbation [2110.04109].
- **Encoder:** For example, an 18-layer Transformer or Conformer (with $d_{\text{model}}=256$, $d_{\text{ff}}=1024$ or $2048$) can be employed [2110.04109]; bidirectional LSTMs (e.g., 5 layers, 320 units/layer/direction) are used in baseline multitask settings [1807.06234].
- **CTC Branches:** Multiple CTC “heads” branch from different encoder layers. Each branch consists of a linear projection followed by a softmax over its specific vocabulary.

A distinctive aspect of hierarchical-CTC as introduced in [2110.04109] is the **conditioning mechanism**: the prediction at each higher-level branch is explicitly conditioned on posterior distributions from preceding, lower-level branches. Concretely, with encoder output $X^{(i)}$,

$$
\tilde{X}^{(i)} = X^{(i-1)} + \text{SelfAttention}(X^{(i-1)})
$$

$$
X^{(i)} = \tilde{X}^{(i)} + \text{FeedForward}(\tilde{X}^{(i)}) + \text{Linear}(A^{(i)})
$$

where $A^{(i)} = \text{softmax}(W_o X^{(i)})$ is the lower-level branch posterior. This mechanism enforces dependency between successive subword granularities.

## 2. Loss Formulation and Optimization

The hierarchical-CTC loss framework introduces separate CTC objectives at $K$ encoder layers, each predicting a sequence at a different subword granularity. For $E$ encoder layers, losses are attached at $i_k = \lfloor kE/K \rfloor$ for $k = 1, \ldots, K$.

The per-head CTC loss:

$$
\mathcal{L}_k =
- \sum_{(X, Y^{(k)})}
\log P_{\mathrm{ctc}} \Bigl( Y^{(k)}
\,\Bigl|\,
\hat Y^{(1)},\ldots,\hat Y^{(k-1)}, X^{(i_k)} \Bigr)
$$

$P_{\mathrm{ctc}}$ marginalizes over all valid alignments of $Y^{(k)}$ given $X^{(i_k)}$ and the previous predicted sequences $\hat Y^{(j)}$.

The total hierarchical CTC loss is the average:

$$
\mathcal{L}_{\mathrm{hc-ctc}}
= \frac{1}{K} \sum_{k=1}^{K} \mathcal{L}_k
$$

In the "intermediate CTC" (SC-CTC) paradigm, identical vocabularies are used at all CTC heads, with each loss applied individually; "parallel CTC" (ParaCTC) applies all losses at the topmost layer without lower-level conditioning [2110.04109].

In hierarchical multitask learning for CTC [1807.06234], the objective is interpolated between main and auxiliary tasks:

$$
L_\text{total} = \lambda L^{\text{sub}}_\text{CTC} + (1-\lambda) L^{\text{aux}}_\text{CTC}
$$

where $\lambda \in [0,1]$.

## 3. Multi-Granular Subword Supervision

Subword unit vocabularies are constructed from training transcripts using algorithms such as SentencePiece BPE. Supervision proceeds from finer-grained tokens at shallower layers to coarser-grained tokens at deeper layers:

- **Example (LibriSpeech-100h & TEDLIUM2):**
  - $|V^{(1)}| = 256$ (character-ngram level)
  - $|V^{(2)}| = 2{,}048$ (small BPE)
  - $|V^{(3)}| = 16{,}384$ (word-level BPE)

This hierarchy balances sequence length versus target sparsity—coarse units (words) present alignment challenges and sparsity, while fine-grained units retain acoustic proximity but are compositional.

A plausible implication is that the use of multi-granular subwords allows explicit guidance in forming word-level representations, thereby reducing error rates in low-resource setups by mitigating word sparsity [2110.04109].

## 4. Training Procedure and Hyperparameterization

The primary training loop applies Adam optimizer (standard ($\beta_1=0.9$, $\beta_2=0.98$) or $\beta_2=0.999$) with layer-specific learning rate schedules, often featuring warmup and checkpoint averaging [2110.04109][1807.06234]. Data augmentation strategies, such as SpecAugment and speed perturbation, expand the effective dataset.

For pre-training in hierarchical multitask CTC [1807.06234], phone-level CTC heads are trained on a partial encoder (e.g., 3 or 4 layers). The resulting parameters initialize the lower part of a full encoder, which is then trained with the full loss (possibly in combination with auxiliary losses and interpolation). In all cases, CTC-trained decoders use greedy search for decoding; no external language model or beam search is used during evaluation in [2110.04109].

Batch sizes, learning rate schedules, and early stopping are managed using standard configurations (e.g., batch sizes 128→32, learning rate halving if WER fails to improve after three checkpoints, up to 100 epochs) [1807.06234].

## 5. Empirical Results

Hierarchical-CTC pre-training delivers consistent improvements over standard CTC and alternative multitask paradigms. Representative word error rates (WER) without language model or beam search are:

| Model         | LS-100 dev-clean/other | LS-960 dev-clean/other | TED2 dev/test  |
|---------------|-----------------------|-----------------------|----------------|
| CTC           |  11.5 / 24.8          |  4.2 / 10.0           |  11.8 / 10.7   |
| SC-CTC        |   8.9 / 21.0          |  3.2 / 8.2            |   9.4 / 8.6    |
| ParaCTC       |  10.4 / 24.0          |  4.6 / 10.3           |  10.9 / 10.2   |
| HC-CTC (ours) |   8.2 / 19.9          |  3.1 / 8.0            |   9.1 / 8.6    |

[2110.04109]

- On LibriSpeech-100h, HC-CTC reduces WER by ~2.7% relative vs. CTC and 8% relative vs. ParaCTC.
- On LibriSpeech-960h and TEDLIUM2, HC-CTC slightly outperforms SC-CTC and offers faster training/inference.

Ablations reveal that multi-granular subword vocabularies (e.g., $256 \rightarrow 2$k $\rightarrow 16$k) outperform single-granularity settings (e.g., $16$k throughout). Conditioning ablation confirms the importance of explicit dependency; removing conditioning increases WER (8.7% vs. 8.2% on LS-100 dev set).

In hierarchical multitask CTC [1807.06234], combining pretraining with auxiliary phone CTC (at $i=4$) and interpolated weighting ($\lambda=0.5$) yields the best overall results, with a 3.4% absolute WER reduction on Eval2000.

## 6. Representation Learning and Downstream Impact

Analysis of model behavior highlights that hierarchical supervision bridges the abstraction gap from acoustic to character, subword, and word-level representations. Conditioning on fine-grained token posteriors provides flexible building blocks for constructing coarse-grained (word) predictions and relaxes the conventional CTC conditional independence assumption [2110.04109].

HC-CTC training reliably sharpens CTC alignments, especially for complex or rare word sequences. Attention visualization shows that lower-level posteriors are leveraged for improved alignment.

A plausible implication is that this lightweight, generic pre-training method can bootstrap any deep encoder structure with improved intermediate representations for word-level recognition, providing better initializations for further fine-tuning (e.g., with attention-based decoders) [2110.04109]. In resource-constrained settings, this approach offers significant convergence and stability benefits [1807.06234].


## 7. Extensions and Open Directions

Potential developments include:

- **Autoregressive Decoding:** Stacking an attention-based decoder atop the HC-CTC encoder for fully end-to-end training.
- **Acoustic Units:** Replacing text-based subwords with acoustically discovered units (e.g., via self-supervised or articulatory embeddings) at lower branches.
- **Dynamic Granularity or Loss Weighting:** Adapting the number of hierarchical steps $K$ and weighting $\lambda_k$ based on utterance length or domain to improve generalization under domain shift.
- **Extremely Low-Resource Regimes:** In multitask CTC, the optimal auxiliary loss position shifts; standard MTL (auxiliary at top) is best with least data, while intermediate layers are optimal with more data [1807.06234].

Hierarchical-CTC pre-training remains a flexible and effective option for leveraging linguistic structure in end-to-end ASR systems, particularly as model depth and task complexity increase [2110.04109][1807.06234].

Source: https://www.emergentmind.com/topics/hierarchical-ctc-pre-training