P-Aligner: Progressive Alignment in ASR

• P-Aligner is a progressive alignment approach that introduces intermediate supervision to gradually enforce token-synchronous representations across encoder layers.
• It combines intermediate CTC losses with aligner heads to stabilize optimization and mitigate the late-layer alignment bottleneck in end-to-end ASR models.
• Empirical results show significant WER reductions on long utterances, validating the staged supervision strategy for improved speech recognition performance.

Searching arXiv for the primary paper and closely related/disambiguating uses of “P-Aligner.” P-Aligner, in the context of automatic speech recognition, is a convenient shorthand for the progressive alignment objective introduced as InterAligner in “Progressive Alignment Objectives for Aligner-Encoder based ASR” (Lee et al., 23 Jun 2026). It is not the formal name used by the paper. The method extends the Aligner-Encoder family of seq2seq end-to-end ASR models by attaching an intermediate Aligner objective to an internal encoder layer so that monotonic, token-synchronous alignment emerges progressively across depth rather than abruptly at the top layer. In the reported formulation, P-Aligner is combined with an intermediate CTC objective, denoted InterCTC, to stabilize optimization. The approach targets a specific training pathology of Aligner-Encoders: alignment often forms only in a few upper layers, making training sensitive and brittle, especially on long utterances (Lee et al., 23 Jun 2026).

1. Aligner-Encoder setting and the alignment problem

Aligner-Encoders are presented as end-to-end ASR architectures that remove both decoder cross-attention and the transducer lattice. In contrast to attention-based encoder-decoder systems, which predict each output token with cross-attention over the entire encoder time lattice, and in contrast to RNN-Transducer models, which predict over a 2D acoustic-label lattice with a joint network operating on both time and label axes, Aligner-Encoders enforce a one-to-one pairing between encoder position and output position. The $u$-th token is predicted from the encoder representation at position $u$ together with an autoregressive predictor state encoding label history (Lee et al., 23 Jun 2026).

The formal setup begins with input features $X = (x_1,\dots,x_T)$. After subsampling, the encoder output length is $T' \ge U$, where $U$ is the number of output tokens. If the encoder hidden sequence at layer $L$ is $H^{(L)} = (h^{(L)}_1,\dots,h^{(L)}_{T'})$ and the prediction network produces label-history embeddings $g_u$ conditioned on $y_{<u}$, then a feed-forward joiner combines $(h^{(L)}_u, g_u)$ into logits $u$0, followed by softmax:

$u$1

The implementation uses an explicit predictor and joiner:

$u$2

followed by $u$3 (Lee et al., 23 Jun 2026).

The final-layer Aligner objective is a token-level cross-entropy,

$u$4

or, with the predictor and joiner made explicit,

$u$5

Inference iterates over encoder positions, producing one token per encoder position with greedy or beam search and stopping at $u$6; no cross-attention and no lattice are used at inference (Lee et al., 23 Jun 2026).

The central optimization issue is that, without progressive supervision, the encoder tends to defer the hard monotonic compression from $u$7 to $u$8 until the top layers. The paper characterizes this as a late-layer alignment bottleneck. Early layers concentrate on local acoustic modeling, while diagonal alignment appears abruptly only in a few upper layers. This makes training brittle when the mismatch between encoder-frame length and output-token length is large, which is especially pronounced on long utterances (Lee et al., 23 Jun 2026).

2. Progressive alignment objective

P-Aligner, i.e. InterAligner, addresses that bottleneck by attaching Aligner heads at intermediate encoder depths. The stated purpose is to make the encoder learn monotonic, token-synchronous representations gradually rather than only at the final layer. Let $u$9 be the set of intermediate layers receiving progressive supervision, and let $X = (x_1,\dots,x_T)$0 be the hidden sequence at layer $X = (x_1,\dots,x_T)$1. Each intermediate Aligner head has its own predictor and joiner and is trained against an intermediate target sequence

$X = (x_1,\dots,x_T)$2

derived from the same transcript and typically constructed with a smaller BPE vocabulary so that $X = (x_1,\dots,x_T)$3 but $X = (x_1,\dots,x_T)$4 (Lee et al., 23 Jun 2026).

The layer-wise intermediate loss is defined as

$X = (x_1,\dots,x_T)$5

With explicit predictor and joiner notation:

$X = (x_1,\dots,x_T)$6

and

$X = (x_1,\dots,x_T)$7

The final-layer objective is combined as

$X = (x_1,\dots,x_T)$8

and, in the experiments, the total training loss is

$X = (x_1,\dots,x_T)$9

with standard softmax and no temperature or special normalization across heads (Lee et al., 23 Jun 2026).

The paper’s interpretation is explicitly curriculum-like. The intermediate head encourages an earlier mapping from $T' \ge U$0 to $T' \ge U$1, while the top head refines from $T' \ge U$2 to $T' \ge U$3. This suggests a staged alignment regime rather than a single abrupt collapse. Attention visualizations are reported to support this account: averaged multihead maps show a sharp monotonic band corresponding to $T' \ge U$4 by layer 14 and then a second sharp diagonal for $T' \ge U$5 by layer 16, indicating progressive, layerwise emergence of alignment (Lee et al., 23 Jun 2026).

In the reported experiments, $T' \ge U$6, so the progressive objective is instantiated with a single intermediate Aligner head at layer 15 of a 17-layer encoder. The paper further reports that layer 16 degraded performance and layer 13 was weaker than layer 15, which is interpreted as evidence that at least two top layers help convert $T' \ge U$7 without forcing an abrupt transition (Lee et al., 23 Jun 2026).

3. InterCTC as a stabilizing objective

The progressive objective is paired with an earlier intermediate CTC loss, InterCTC, whose role is optimization stabilization. For a layer $T' \ge U$8, the paper uses standard CTC:

$T' \ge U$9

where $U$0 collapses repeats and blanks to the target sequence $U$1. The full objective augments the Aligner losses with weighted CTC terms:

$U$2

In the experiments, $U$3 and $U$4 (Lee et al., 23 Jun 2026).

InterCTC is used only during training and does not alter inference, which continues to decode solely from the final Aligner head. Its stated effect is to provide early monotonic, token-predictive gradients before alignment must fully materialize. This improves gradient flow by creating shorter paths to the loss in lower layers, regularizes deep representations, and shapes encoder features before the model is required to solve the full token-synchronous alignment problem (Lee et al., 23 Jun 2026).

The interaction between InterCTC and P-Aligner is described as complementary. InterCTC encourages separability and monotonicity earlier in the network, and the intermediate Aligner then attaches to a cleaner sequence-to-sequence structure at its own layer. This suggests that the progressive objective is not merely another auxiliary head but part of a staged supervision scheme in which the CTC head and the intermediate Aligner operate at different depths and granularities (Lee et al., 23 Jun 2026).

A concise training loop is given. For each minibatch $U$5, the model computes encoder states $U$6; applies the final Aligner head at layer $U$7 to accumulate $U$8; applies intermediate Aligner heads at $U$9 using $L$0 to accumulate $L$1; applies InterCTC at $L$2; forms

$L$3

and then backpropagates and updates parameters (Lee et al., 23 Jun 2026).

4. Architecture, tokenization, and training configuration

The reported system uses a Conformer-L encoder with 17 layers and approximately 118M parameters. The prediction network is a one-layer LSTM over token embeddings, and the joiner is a feed-forward MLP combining $L$4 and $L$5 with a $L$6 nonlinearity and an output projection $L$7, followed by softmax. Attention visualizations average over 8 heads, indicating multihead self-attention in the encoder. Head placement is fixed as follows: the final Aligner head at layer 17, the InterAligner head at layer 15, and InterCTC at layer 12 (Lee et al., 23 Jun 2026).

Training is conducted on LibriSpeech 960h and Common Voice 16.1 English, with punctuation removed in train and test for Common Voice. Decoding uses beam width 6 for all reported results and always uses the final head only. Optimization follows Transformer warmup/decay with 20k warmup steps. The peak learning rate is 0.0020 when no target vocabulary $L$8 is used and 0.0025 otherwise. The effective batch size is approximately 2 hours of audio. LibriSpeech is trained for 100 epochs with model averaging over the 10 best checkpoints, and Common Voice for 50 epochs with the same 10-best averaging (Lee et al., 23 Jun 2026).

The tokenization scheme is BPE. Default vocabulary sizes are 1024 for the final Aligner head and 256 for the InterAligner head; InterCTC matches the intermediate vocabulary when InterAligner is used, and is 1024 otherwise. The loss weight $L$9 is fixed. For InterAligner systems, $H^{(L)} = (h^{(L)}_1,\dots,h^{(L)}_{T'})$0 are tuned, and the best-performing settings emphasize the intermediate head. The paper explicitly gives $H^{(L)} = (h^{(L)}_1,\dots,h^{(L)}_{T'})$1 and $H^{(L)} = (h^{(L)}_1,\dots,h^{(L)}_{T'})$2 as an example of such a setting (Lee et al., 23 Jun 2026).

Ablation results support two linked design choices. First, matching the InterAligner and InterCTC vocabularies helps. Second, emphasizing the intermediate loss with $H^{(L)} = (h^{(L)}_1,\dots,h^{(L)}_{T'})$3 improves both final-head and intermediate-head performance. This suggests that the progressive objective is most effective when it is not treated as a weak auxiliary term but as a major supervisory signal for the middle-to-upper encoder stack (Lee et al., 23 Jun 2026).

5. Empirical results and ablations

The main LibriSpeech results are reported as word error rate on test-clean and test-other. A final-only Aligner achieves 5.0 / 7.8. Adding InterCTC improves performance to 3.4 / 6.0. Adding InterAligner on top of InterCTC further improves the result to 3.1 / 5.6 (Lee et al., 23 Jun 2026). For reference, the paper notes that Stooke et al. report 4.8 / 6.5 for a final-only Aligner under a larger training budget (Lee et al., 23 Jun 2026).

The length-bin breakdown is central to the paper’s argument. On test-clean, the final-only Aligner obtains 3.2, 5.7, and 23.4 WER on utterances of length <17s, 17–21s, and >21s, respectively. With InterCTC the corresponding values are 2.3, 2.3, and 17.0. With InterAligner they are 2.4, 2.9, and 11.6. On test-other, the same systems achieve 7.0 / 8.2 / 24.0, then 5.4 / 5.3 / 18.0, then 5.2 / 5.5 / 13.5 across the same bins. The largest gains therefore occur on utterances longer than 21 seconds, which the paper presents as direct support for the progressive-alignment motivation (Lee et al., 23 Jun 2026).

On Common Voice 16.1 English test WER, the reported values are 12.4 for the final-only Aligner, 11.2 with InterCTC, and 10.9 with InterAligner (Lee et al., 23 Jun 2026). The gains are smaller in absolute magnitude than on LibriSpeech but remain consistent with the same training narrative.

Several ablations further characterize the method:

Configuration	Final WER	Intermediate WER
final=1024, inter=256, CTC=256, $H^{(L)} = (h^{(L)}_1,\dots,h^{(L)}_{T'})$4	3.1 / 5.6	3.0 / 5.5
final=1024, inter=256, CTC=256, $H^{(L)} = (h^{(L)}_1,\dots,h^{(L)}_{T'})$5	3.1 / 5.8	5.7 / 7.0
final=1024, inter=256, CTC=1024, $H^{(L)} = (h^{(L)}_1,\dots,h^{(L)}_{T'})$6	3.2 / 5.8	4.0 / 6.1

These ablations are reported to show that matching intermediate tokenizations helps and that emphasizing the intermediate loss improves both heads (Lee et al., 23 Jun 2026).

A second set of tokenization ablations compares final=1024/inter=1024/CTC=1024, final=1024/inter=256/CTC=256, final=1024/inter=64/CTC=64, and final=256 without InterAligner. Smaller intermediate vocabularies improve final performance, while training only a final head with final=256 performs much worse than the hierarchical setup. The paper therefore argues that the improvement is not due to tokenization alone but to the staged objective itself (Lee et al., 23 Jun 2026).

Training cost increases modestly because of the extra forward and backward passes through the intermediate heads and the CTC criterion, and memory overhead comes from storing intermediate activations for the selected layers. Inference cost and memory remain unchanged because decoding uses only the final head (Lee et al., 23 Jun 2026).

6. Interpretation, limitations, and terminological ambiguity

The paper interprets P-Aligner as a form of deep supervision specifically tailored to alignment formation inside Aligner-Encoders. It differs from generic intermediate losses in that it supervises a token-synchronous objective at intermediate depth using a finer-grained target sequence. InterCTC complements this with a classical monotonic objective earlier in the network. The combined system can therefore be read as decomposing the alignment problem into staged subproblems across depth: first a coarser monotonic compression and then a finer token-level refinement (Lee et al., 23 Jun 2026).

The reported limitations are practical rather than conceptual. The method adds training-time heads and associated hyperparameters, including layer placement, loss weights, and vocabulary sizes. Poor placement can degrade performance; the layer-16 ablation is given as an explicit example. The paper identifies future directions including adaptive layer weighting, layer-wise curricula, more flexible multi-layer $H^{(L)} = (h^{(L)}_1,\dots,h^{(L)}_{T'})$7 sets, integrating monotonic constraints, and combining with external LLMs (Lee et al., 23 Jun 2026).

A frequent source of confusion is the name itself. In this ASR setting, P-Aligner is not the term introduced by the paper; the formal name is InterAligner, and “P-Aligner” is only a convenient shorthand for “Progressive Aligner” (Lee et al., 23 Jun 2026). That ambiguity matters because the arXiv record also contains unrelated methods using similar terminology. “P-Aligner: Enabling Pre-Alignment of LLMs via Principled Instruction Synthesis” denotes an instruction-rewriting module for LLMs, not an ASR model (Song et al., 6 Aug 2025). “pyro-align” is a multiple-sequence alignment system for pyrosequencing reads rather than a speech recognizer (0901.2753). “Aligner: One Global Token is Worth Millions of Parameters When Aligning LLMs” refers to a parameter-efficient alignment method for LLMs, again unrelated to encoder-side ASR alignment (Ziheng et al., 2023). In the context of seq2seq speech recognition, however, P-Aligner most precisely denotes the intermediate progressive alignment objective named InterAligner (Lee et al., 23 Jun 2026).