---
title: 'P-Aligner: Progressive Alignment in ASR'
url: https://www.emergentmind.com/topics/p-aligner
type: topic
---

# P-Aligner: Progressive Alignment in ASR

Searching arXiv for the primary paper and closely related/disambiguating uses of “P-Aligner.”
P-Aligner, in the context of automatic speech recognition, is a convenient shorthand for the progressive alignment objective introduced as **InterAligner** in “Progressive Alignment Objectives for Aligner-Encoder based ASR” [2606.24147]. It is not the formal name used by the paper. The method extends the Aligner-Encoder family of seq2seq end-to-end ASR models by attaching an intermediate Aligner objective to an internal encoder layer so that monotonic, token-synchronous alignment emerges progressively across depth rather than abruptly at the top layer. In the reported formulation, P-Aligner is combined with an intermediate CTC objective, denoted InterCTC, to stabilize optimization. The approach targets a specific training pathology of Aligner-Encoders: alignment often forms only in a few upper layers, making training sensitive and brittle, especially on long utterances [2606.24147].

## 1. Aligner-Encoder setting and the alignment problem

Aligner-Encoders are presented as end-to-end ASR architectures that remove both decoder cross-attention and the transducer lattice. In contrast to attention-based encoder-decoder systems, which predict each output token with cross-attention over the entire encoder time lattice, and in contrast to RNN-Transducer models, which predict over a 2D acoustic-label lattice with a joint network operating on both time and label axes, Aligner-Encoders enforce a one-to-one pairing between encoder position and output position. The $u$-th token is predicted from the encoder representation at position $u$ together with an autoregressive predictor state encoding label history [2606.24147].

The formal setup begins with input features $X = (x_1,\dots,x_T)$. After subsampling, the encoder output length is $T' \ge U$, where $U$ is the number of output tokens. If the encoder hidden sequence at layer $L$ is $H^{(L)} = (h^{(L)}_1,\dots,h^{(L)}_{T'})$ and the prediction network produces label-history embeddings $g_u$ conditioned on $y_{<u}$, then a feed-forward joiner combines $(h^{(L)}_u, g_u)$ into logits $z_u$, followed by softmax:
$$
P(y_u \mid X, y_{<u}) = \operatorname{softmax}(f_{\text{joint}}(h^{(L)}_u, g_u)).
$$
The implementation uses an explicit predictor and joiner:
$$
\tilde z_u = \tanh(W_h h^{(L)}_u + W_g g_u + b), \qquad
z_u = W_o \tilde z_u + b_o,
$$
followed by $p(\cdot)=\operatorname{softmax}(z_u)$ [2606.24147].

The final-layer Aligner objective is a token-level cross-entropy,
$$
L_{\text{align}}^{(L)} = -\sum_{u=1}^{U} \log p(y_u \mid h_u^{(L)}),
$$
or, with the predictor and joiner made explicit,
$$
L_{\text{final}} = -\sum_{u=1}^{U} \log P(y_u \mid X, y_{<u}).
$$
Inference iterates over encoder positions, producing one token per encoder position with greedy or beam search and stopping at $\langle eos\rangle$; no cross-attention and no lattice are used at inference [2606.24147].

The central optimization issue is that, without progressive supervision, the encoder tends to defer the hard monotonic compression from $T'$ to $U$ until the top layers. The paper characterizes this as a **late-layer alignment bottleneck**. Early layers concentrate on local acoustic modeling, while diagonal alignment appears abruptly only in a few upper layers. This makes training brittle when the mismatch between encoder-frame length and output-token length is large, which is especially pronounced on long utterances [2606.24147].

## 2. Progressive alignment objective

P-Aligner, i.e. InterAligner, addresses that bottleneck by attaching Aligner heads at intermediate encoder depths. The stated purpose is to make the encoder learn monotonic, token-synchronous representations gradually rather than only at the final layer. Let $S_{\text{align}}$ be the set of intermediate layers receiving progressive supervision, and let $H^{(\ell)} = (h^{(\ell)}_1,\dots,h^{(\ell)}_{T'})$ be the hidden sequence at layer $\ell$. Each intermediate Aligner head has its own predictor and joiner and is trained against an intermediate target sequence
$$
y^{\text{int}} = (y^{\text{int}}_1,\dots,y^{\text{int}}_{U_{\text{int}}}),
$$
derived from the same transcript and typically constructed with a smaller BPE vocabulary so that $U_{\text{int}} > U$ but $U_{\text{int}} \le T'$ [2606.24147].

The layer-wise intermediate loss is defined as
$$
L_{\text{InterAlign}} = \sum_{\ell \in S_{\text{align}}} \alpha_\ell \left[-\sum_{u=1}^{U_{\text{int}}} \log p(y^{\text{int}}_u \mid h_u^{(\ell)})\right].
$$
With explicit predictor and joiner notation:
$$
L_{\text{int}}^{(\ell)} = -\sum_{u=1}^{U_{\text{int}}} \log P_{\text{int}}(y^{\text{int}}_u \mid X, y^{\text{int}}_{<u}; H^{(\ell)}),
$$
and
$$
L_{\text{InterAlign}} = \sum_{\ell \in S_{\text{align}}} \alpha_\ell L_{\text{int}}^{(\ell)}.
$$
The final-layer objective is combined as
$$
L_{\text{AlignTotal}} = L_{\text{final}} + L_{\text{InterAlign}},
$$
and, in the experiments, the total training loss is
$$
L(\theta) = \lambda_{\text{final}} L_{\text{final}} + \lambda_{\text{int}} L_{\text{int}} + \lambda_{\text{ctc}} L_{\text{ctc}},
$$
with standard softmax and no temperature or special normalization across heads [2606.24147].

The paper’s interpretation is explicitly curriculum-like. The intermediate head encourages an earlier mapping from $T'$ to $U_{\text{int}}$, while the top head refines from $U_{\text{int}}$ to $U$. This suggests a staged alignment regime rather than a single abrupt collapse. Attention visualizations are reported to support this account: averaged multihead maps show a sharp monotonic band corresponding to $T' \to U_{\text{int}}$ by layer 14 and then a second sharp diagonal for $U_{\text{int}} \to U$ by layer 16, indicating progressive, layerwise emergence of alignment [2606.24147].

In the reported experiments, $S_{\text{align}}=\{15\}$, so the progressive objective is instantiated with a single intermediate Aligner head at layer 15 of a 17-layer encoder. The paper further reports that layer 16 degraded performance and layer 13 was weaker than layer 15, which is interpreted as evidence that at least two top layers help convert $U_{\text{int}} \to U$ without forcing an abrupt transition [2606.24147].

## 3. InterCTC as a stabilizing objective

The progressive objective is paired with an earlier intermediate CTC loss, InterCTC, whose role is optimization stabilization. For a layer $\ell \in S_{\text{CTC}}$, the paper uses standard CTC:
$$
L_{\text{CTC}}^{(\ell)} = - \log \sum_{\pi \in B^{-1}(y)} \prod_{t=1}^{T_\ell} p(\pi_t \mid h_t^{(\ell)}),
$$
where $B$ collapses repeats and blanks to the target sequence $y$. The full objective augments the Aligner losses with weighted CTC terms:
$$
L_{\text{total}} = L_{\text{AlignTotal}} + \sum_{\ell \in S_{\text{CTC}}} \beta_\ell L_{\text{CTC}}^{(\ell)}.
$$
In the experiments, $S_{\text{CTC}}=\{12\}$ and $\lambda_{\text{ctc}}=0.1$ [2606.24147].

InterCTC is used only during training and does not alter inference, which continues to decode solely from the final Aligner head. Its stated effect is to provide early monotonic, token-predictive gradients before alignment must fully materialize. This improves gradient flow by creating shorter paths to the loss in lower layers, regularizes deep representations, and shapes encoder features before the model is required to solve the full token-synchronous alignment problem [2606.24147].

The interaction between InterCTC and P-Aligner is described as complementary. InterCTC encourages separability and monotonicity earlier in the network, and the intermediate Aligner then attaches to a cleaner sequence-to-sequence structure at its own layer. This suggests that the progressive objective is not merely another auxiliary head but part of a staged supervision scheme in which the CTC head and the intermediate Aligner operate at different depths and granularities [2606.24147].

A concise training loop is given. For each minibatch $(X,y)$, the model computes encoder states $H^{(1)},\dots,H^{(L)}$; applies the final Aligner head at layer $L$ to accumulate $L_{\text{final}}$; applies intermediate Aligner heads at $\ell \in S_{\text{align}}$ using $y^{\text{int}}$ to accumulate $L_{\text{InterAlign}}$; applies InterCTC at $\ell \in S_{\text{CTC}}$; forms
$$
L_{\text{total}} = \lambda_{\text{final}}L_{\text{final}} + L_{\text{InterAlign}} + \sum \beta_\ell L_{\text{CTC}}^{(\ell)};
$$
and then backpropagates and updates parameters [2606.24147].

## 4. Architecture, tokenization, and training configuration

The reported system uses a Conformer-L encoder with 17 layers and approximately 118M parameters. The prediction network is a one-layer LSTM over token embeddings, and the joiner is a feed-forward MLP combining $W_h h_u$ and $W_g g_u$ with a $\tanh$ nonlinearity and an output projection $W_o$, followed by softmax. Attention visualizations average over 8 heads, indicating multihead self-attention in the encoder. Head placement is fixed as follows: the final Aligner head at layer 17, the InterAligner head at layer 15, and InterCTC at layer 12 [2606.24147].

Training is conducted on LibriSpeech 960h and Common Voice 16.1 English, with punctuation removed in train and test for Common Voice. Decoding uses beam width 6 for all reported results and always uses the final head only. Optimization follows Transformer warmup/decay with 20k warmup steps. The peak learning rate is 0.0020 when no target vocabulary $\le 256$ is used and 0.0025 otherwise. The effective batch size is approximately 2 hours of audio. LibriSpeech is trained for 100 epochs with model averaging over the 10 best checkpoints, and Common Voice for 50 epochs with the same 10-best averaging [2606.24147].

The tokenization scheme is BPE. Default vocabulary sizes are 1024 for the final Aligner head and 256 for the InterAligner head; InterCTC matches the intermediate vocabulary when InterAligner is used, and is 1024 otherwise. The loss weight $\lambda_{\text{ctc}}=0.1$ is fixed. For InterAligner systems, $(\lambda_{\text{final}},\lambda_{\text{int}})$ are tuned, and the best-performing settings emphasize the intermediate head. The paper explicitly gives $\lambda_{\text{final}}=0.5$ and $\lambda_{\text{int}}=1.0$ as an example of such a setting [2606.24147].

Ablation results support two linked design choices. First, matching the InterAligner and InterCTC vocabularies helps. Second, emphasizing the intermediate loss with $\lambda_{\text{int}} > \lambda_{\text{final}}$ improves both final-head and intermediate-head performance. This suggests that the progressive objective is most effective when it is not treated as a weak auxiliary term but as a major supervisory signal for the middle-to-upper encoder stack [2606.24147].

## 5. Empirical results and ablations

The main LibriSpeech results are reported as word error rate on test-clean and test-other. A final-only Aligner achieves **5.0 / 7.8**. Adding InterCTC improves performance to **3.4 / 6.0**. Adding InterAligner on top of InterCTC further improves the result to **3.1 / 5.6** [2606.24147]. For reference, the paper notes that Stooke et al. report **4.8 / 6.5** for a final-only Aligner under a larger training budget [2606.24147].

The length-bin breakdown is central to the paper’s argument. On test-clean, the final-only Aligner obtains **3.2**, **5.7**, and **23.4** WER on utterances of length **<17s**, **17–21s**, and **>21s**, respectively. With InterCTC the corresponding values are **2.3**, **2.3**, and **17.0**. With InterAligner they are **2.4**, **2.9**, and **11.6**. On test-other, the same systems achieve **7.0 / 8.2 / 24.0**, then **5.4 / 5.3 / 18.0**, then **5.2 / 5.5 / 13.5** across the same bins. The largest gains therefore occur on utterances longer than 21 seconds, which the paper presents as direct support for the progressive-alignment motivation [2606.24147].

On Common Voice 16.1 English test WER, the reported values are **12.4** for the final-only Aligner, **11.2** with InterCTC, and **10.9** with InterAligner [2606.24147]. The gains are smaller in absolute magnitude than on LibriSpeech but remain consistent with the same training narrative.

Several ablations further characterize the method:

| Configuration | Final WER | Intermediate WER |
|---|---:|---:|
| final=1024, inter=256, CTC=256, $\lambda_{\text{final}}/\lambda_{\text{int}}=0.5/1.0$ | 3.1 / 5.6 | 3.0 / 5.5 |
| final=1024, inter=256, CTC=256, $\lambda_{\text{final}}/\lambda_{\text{int}}=1.0/0.5$ | 3.1 / 5.8 | 5.7 / 7.0 |
| final=1024, inter=256, CTC=1024, $\lambda_{\text{final}}/\lambda_{\text{int}}=1.0/0.5$ | 3.2 / 5.8 | 4.0 / 6.1 |

These ablations are reported to show that matching intermediate tokenizations helps and that emphasizing the intermediate loss improves both heads [2606.24147].

A second set of tokenization ablations compares final=1024/inter=1024/CTC=1024, final=1024/inter=256/CTC=256, final=1024/inter=64/CTC=64, and final=256 without InterAligner. Smaller intermediate vocabularies improve final performance, while training only a final head with final=256 performs much worse than the hierarchical setup. The paper therefore argues that the improvement is not due to tokenization alone but to the staged objective itself [2606.24147].

Training cost increases modestly because of the extra forward and backward passes through the intermediate heads and the CTC criterion, and memory overhead comes from storing intermediate activations for the selected layers. Inference cost and memory remain unchanged because decoding uses only the final head [2606.24147].

## 6. Interpretation, limitations, and terminological ambiguity

The paper interprets P-Aligner as a form of deep supervision specifically tailored to alignment formation inside Aligner-Encoders. It differs from generic intermediate losses in that it supervises a token-synchronous objective at intermediate depth using a finer-grained target sequence. InterCTC complements this with a classical monotonic objective earlier in the network. The combined system can therefore be read as decomposing the alignment problem into staged subproblems across depth: first a coarser monotonic compression and then a finer token-level refinement [2606.24147].

The reported limitations are practical rather than conceptual. The method adds training-time heads and associated hyperparameters, including layer placement, loss weights, and vocabulary sizes. Poor placement can degrade performance; the layer-16 ablation is given as an explicit example. The paper identifies future directions including adaptive layer weighting, layer-wise curricula, more flexible multi-layer $S_{\text{align}}$ sets, integrating monotonic constraints, and combining with external language models [2606.24147].

A frequent source of confusion is the name itself. In this ASR setting, **P-Aligner is not the term introduced by the paper**; the formal name is **InterAligner**, and “P-Aligner” is only a convenient shorthand for “Progressive Aligner” [2606.24147]. That ambiguity matters because the arXiv record also contains unrelated methods using similar terminology. “P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis” denotes an instruction-rewriting module for LLMs, not an ASR model [2508.04626]. “pyro-align” is a multiple-sequence alignment system for pyrosequencing reads rather than a speech recognizer [0901.2753]. “Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models” refers to a parameter-efficient alignment method for LLMs, again unrelated to encoder-side ASR alignment [2312.05503]. In the context of seq2seq speech recognition, however, P-Aligner most precisely denotes the intermediate progressive alignment objective named InterAligner [2606.24147].

Source: https://www.emergentmind.com/topics/p-aligner