Progressive Alignment Representation Training (PART)

Updated 12 July 2026

PART is a design pattern that stages alignment objectives across spatial patches, modalities, encoder depths, and training phases.
It replaces monolithic alignment targets with progressive, relational constraints to improve performance across vision, multimodal, and ASR systems.
Implementations include off-grid patch translation, frozen universal projections with alignment layers, intermediate ASR supervision, and multilingual speech-to-text curricula.

Progressive Alignment Representation Training (PART) denotes a cluster of recent alignment-oriented training formulations rather than a single canonical algorithm. In the papers considered here, the label is used for a self-supervised method that learns relative spatial composition from off-grid patches (Ayoughi et al., 4 Jun 2025), the progressive modality-alignment pipeline underlying OneEncoder (Faye et al., 2024), and a multi-stage curriculum for multilingual speech-to-text systems built on top of LLMs (Zhang et al., 24 Sep 2025); a closely related reuse of the acronym in end-to-end ASR denotes “Progressive Alignment Objectives,” implemented as InterAligner + InterCTC (Lee et al., 23 Jun 2026). Taken together, these uses suggest a recurring emphasis on replacing monolithic or absolute alignment targets with staged, relational, or progressively refined alignment.

1. Scope of the term and its current research usage

In the papers considered here, PART is attached to four distinct technical settings.

Variant	Domain	Core formulation
PART in "How PARTs assemble into wholes"	Self-supervised vision and structured signals	Predict pairwise continuous relative translations between randomly sampled off-grid patches
PART in OneEncoder	Multimodal alignment	Train a shared Universal Projection on image–text, freeze it, then add new modalities via small Alignment Layers
PART in Aligner-Encoder ASR	End-to-end ASR	Add InterAligner and InterCTC so monotonic alignment forms progressively across encoder depth
PART in multilingual speech-to-text with LLMs	Multilingual ASR and S2TT	Separate within-language alignment from cross-language alignment through a three-stage schedule

The term therefore does not identify a single shared architecture, loss, or benchmark suite. Rather, it is used for methods that make alignment an explicit training object and distribute that alignment across space, modality, depth, or training stage. This suggests that PART is best understood as a design pattern centered on progressive constraint formation, with the exact meaning determined by the application domain.

2. Off-grid relative composition in self-supervised representation learning

In "How PARTs assemble into wholes: Learning the relative composition of images," PART is a self-supervised pretraining method designed to learn relative spatial composition rather than absolute spatial coordinates. The method replaces the usual objective of predicting where a patch belongs in a fixed grid with the objective of predicting how two randomly sampled patches relate in continuous image space. This responds to a limitation of grid-based methods such as Jigsaw, MAE-style masking, MP3, and DropPos: real objects are not arranged on a rigid lattice, and their parts can appear at variable locations, overlap, vary in scale and aspect ratio, and be partially occluded or deformed (Ayoughi et al., 4 Jun 2025).

Given an image $I \in \mathbb{R}^{H \times W \times C}$ , PART extracts $N$ random patches from arbitrary locations and resizes each patch to a common size $P \times P$ . This induces two effects emphasized in the paper: implicit masking, because unsampled regions are omitted, and loss of absolute coordinate information, because resizing removes direct cues about original size and location. For a reference patch and target patch with centers $(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ and $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ , the target is a continuous displacement vector,

$\theta_{\mathrm{ref},\mathrm{tgt}}= \begin{bmatrix} \Delta x\ \Delta y \end{bmatrix} = \begin{bmatrix} (x_{\mathrm{tgt}}-x_{\mathrm{ref}})/w_{\mathrm{ref}}\ (y_{\mathrm{tgt}}-y_{\mathrm{ref}})/h_{\mathrm{ref}} \end{bmatrix},$

with $w_{\mathrm{ref}}$ and $h_{\mathrm{ref}}$ effectively $P$ after resizing. The model therefore regresses a continuous relative translation rather than a discrete grid label.

Architecturally, the method uses a ViT backbone without positional embeddings during pretraining and a cross-attention relative encoder. If the resized patches yield tokens $X \in \mathbb{R}^{N \times d}$ and embeddings $N$ 0, and if $N$ 1 indexes selected patch pairs, then pairwise embeddings are concatenated as $N$ 2. The paper writes the attention-style formulation as

$N$ 3

and trains with mean squared error,

$N$ 4

The cross-attention head serves two roles stated explicitly in the paper: it enables information sharing across all patches, not only the queried pair, and it imposes an additional masking effect because translation is predicted for only a subset of pairs.

Empirically, the method is strongest on tasks that depend on local spatial precision. On COCO detection with ViT-B pretrained on ImageNet-1K, reported AP $N$ 5 values are 40.1 for MAE, 41.8 for MP3, 42.1 for DropPos, and 42.4 for PART; PART also reports 62.5 AP $N$ 6 and 46.8 AP $N$ 7. On PhysioNet 2018 sleep staging, Cohen’s kappa rises from 0.531 for the supervised baseline to 0.616 for PART, compared with 0.553 for MP3, 0.582 for DropPos, and 0.595 for MAE. On ImageNet-1K classification, PART reaches 82.7, which is competitive with MP3 at 82.6–82.9 and above the supervised baseline at 81.8, but below MAE at 83.6 and DropPos at 83.0; on CIFAR-100, PART reaches 83.0, above the supervised baseline at 81.8 and MP3 at 82.6, but below MAE at 84.5. The paper’s ablation comparing random off-grid sampling with a grid-based variant (“PART-grid”) reports 42.4 vs 41.4 on COCO, 83.0 vs 82.1 on CIFAR-100, 82.7 vs 82.43 on ImageNet-1K, and 0.616 vs 0.500 on time series, supporting the claim that off-grid sampling better captures local spatial detail. Additional capabilities discussed include off-grid reconstruction, extension to $N$ 8, patch uncertainty, and an emergent negative symmetry under patch-order reversal.

3. Progressive multimodal alignment in OneEncoder

In OneEncoder, PART refers to Progressive Alignment Representation Training as a lightweight, frozen, progressive alignment pipeline for four modalities: image, text, audio, and video. The framework begins with pretrained modality-specific encoders—ViT for images, BERT for text, Wav2Vec2 for audio, and VideoMAE for video—which remain frozen and act only as feature extractors. Stage 1 trains a lightweight Universal Projection (UP) module on image–text pairs, chosen because image–text data are the most abundant and serve as a semantic anchor. After this stage, UP is frozen. New modalities are then added by training only small modality-specific Alignment Layers (ALs) that map the new modality into a space compatible with the already aligned image/text space; the new representation is then passed through the frozen UP (Faye et al., 2024).

This yields the transitive alignment logic stated explicitly in the paper: if audio is aligned with text, and text is already aligned with image, then audio becomes aligned with image as well. The same process is repeated for video. The resulting pipeline is summarized in the paper as: image–text train UP, freeze UP, train AL for audio, freeze UP, train AL for video, and then repeat for future modalities. UP is implemented as a small Transformer encoder and uses learnable modality tokens $N$ 9, with the modality-conditioned projection written as

$P \times P$ 0

where $P \times P$ 1 may denote concatenation, addition, or cross-attention. In the reported setup, UP has 4M parameters, and the modality-specific AL has about 65,792 parameters.

Optimization uses a symmetric InfoNCE-style contrastive loss adapted from ConVIRT-style text–image representation learning. The paper uses AdamW with learning rate $P \times P$ 2, $P \times P$ 3, $P \times P$ 4, weight decay $P \times P$ 5, and a learned temperature $P \times P$ 6. The training rule is stage-specific: first update only UP and modality tokens for image/text, then freeze UP and train only AL and the new modality token for the newly introduced modality. Stage-1 image–text alignment uses COCO Captions, Flickr30K, and TextCaps; audio alignment uses LibriSpeech Speech Recognition Alignment (SRA); video alignment uses MSR-VTT.

The reported results emphasize efficiency and competitive performance. Against CLIP, OneEncoder-1 reports zero-shot classification scores of 78.15 on CIFAR-10 vs 62.12, 69.23 on Oxford-IIIT Pets vs 58.27, 58.18 on CIFAR-100 vs 53.06, 56.20 on Caltech-101 vs 52.17, and 52.27 on Tiny ImageNet vs 47.15. In linear classification, the paper states that OneEncoder-1 outperforms CLIP across all reported image and text datasets, including CIFAR-10 at 90.16 vs 81.21, SST-2 at 82.12 vs 80.15, and TREC at 79.10 vs 78.24; in fine-tuning, it reports CIFAR-10 at 96.01 vs 86.76, Oxford-IIIT Pets at 92.32 vs 81.90, and SST-2 at 86.11 vs 85.15. The key efficiency claim is that OneEncoder trains only the 4M-parameter UP, whereas CLIP retrains roughly 196M parameters in the comparison setup. Against AudioCLIP, OneEncoder-1 improves text-audio retrieval and image-audio zero-shot classification, with examples including AudioSet P@1 of 5.37 vs 4.27, UrbanSound8K mAP of 79.65 vs 78.27, ESC-50 R@1 of 80.12 vs 78.21, and CIFAR-10 image-audio zero-shot classification at 77.01 vs 61.28. Against CLIP4Clip and X-CLIP, the video-aligned model reports MSVD text-to-video R@5 of 80.76 and video-to-text R@5 of 91.62, and on LSMDC a video-to-text R@1 of 27.01, exceeding X-CLIP’s 26.9. The paper also characterizes the framework as an “open-world alignment framework” because new modalities can be attached without retraining the entire system.

4. Progressive alignment objectives in Aligner-Encoder ASR

A separate ASR paper reuses the acronym PART to denote Progressive Alignment Objectives for Aligner-Encoder based ASR, implemented as InterAligner + InterCTC. The starting point is the Aligner-Encoder formulation in which the encoder must learn a monotonic alignment internally, pairing the $P \times P$ 7-th output token directly with the $P \times P$ 8-th encoder position. The final-only objective is

$P \times P$ 9

applied only on the first $(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ 0 encoder outputs. The paper argues that this creates a late-layer alignment bottleneck: if $(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ 1, the encoder must compress and reorganize acoustic information so that the first $(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ 2 positions become token-aligned while the rest are unused, and clear diagonal alignment often appears abruptly only in the upper few layers of a deep encoder (Lee et al., 23 Jun 2026).

InterAligner addresses this by attaching an intermediate Aligner head to an upper-intermediate layer and supervising it with a longer, finer-grained target sequence $(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ 3, yielding

$(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ 4

The staged mapping is described as

$(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ 5

rather than the final-only mapping $(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ 6. InterCTC is added at an earlier layer as an auxiliary stabilizer, and the total objective is

$(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ 7

with $(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ 8 in all experiments.

The reported architecture is a 17-layer Conformer-L of about 118M parameters, with a 1-layer LSTM predictor, an MLP joiner with tanh nonlinearity, beam width 6, 100 epochs on LibriSpeech 960h, 50 epochs on Common Voice English 16.1, model averaging over the 10 best checkpoints, batch size roughly 2 hours of audio, and Transformer warmup/decay with 20k warmup steps. In the main configuration, the final Aligner is attached at layer 17, InterAligner at layer 15, and InterCTC at layer 12; the paper reports that layer 15 is the best placement for InterAligner.

On LibriSpeech, the key progression is from a final-only Aligner at 5.0/7.8 WER on test-clean/test-other, to 3.4/6.0 with InterCTC, and to 3.1/5.6 with InterAligner. The longest-utterance results provide the strongest support for the method’s motivation. On test-clean for utterances $(x_{\mathrm{ref}}, y_{\mathrm{ref}})$ 9s, WER drops from 23.4 for the final-only model to 17.0 with InterCTC and 11.6 with InterAligner; on test-other for utterances $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ 0s, it drops from 24.0 to 18.0 to 13.5. The paper’s matched-vocabulary ablation finds that using the same intermediate vocabulary for InterAligner and InterCTC performs best, with a strong configuration of final $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ 1 BPE, intermediate $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ 2 BPE, and CTC $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ 3 BPE. Best reported loss weights are $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ 4. On Common Voice, the same trend holds: 12.4 for the final-only model, 11.2 with InterCTC, and 10.9 with InterAligner, with improvements reported as statistically significant at the 1% level.

5. Multilingual speech-to-text PART with LLMs

In "PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs," PART is a training recipe for multilingual speech-to-text systems that separates within-language alignment from cross-language alignment. The architecture consists of a pre-trained multilingual speech encoder, a lightweight adapter, and a multilingual LLM. Given speech signal $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ 5, log-mel spectrogram features $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ 6 are processed by a multilingual speech encoder with parameters $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ 7, then by an adapter with parameters $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ 8, producing aligned embeddings $(x_{\mathrm{tgt}}, y_{\mathrm{tgt}})$ 9, which are concatenated with an instruction token embedding $\theta_{\mathrm{ref},\mathrm{tgt}}= \begin{bmatrix} \Delta x\ \Delta y \end{bmatrix} = \begin{bmatrix} (x_{\mathrm{tgt}}-x_{\mathrm{ref}})/w_{\mathrm{ref}}\ (y_{\mathrm{tgt}}-y_{\mathrm{ref}})/h_{\mathrm{ref}} \end{bmatrix},$ 0 and consumed by the multilingual LLM with parameters $\theta_{\mathrm{ref},\mathrm{tgt}}= \begin{bmatrix} \Delta x\ \Delta y \end{bmatrix} = \begin{bmatrix} (x_{\mathrm{tgt}}-x_{\mathrm{ref}})/w_{\mathrm{ref}}\ (y_{\mathrm{tgt}}-y_{\mathrm{ref}})/h_{\mathrm{ref}} \end{bmatrix},$ 1. The adapter is described as two transformer layers plus one CNN layer, the speech encoder is SenseVoice-large, and the LLM backbone is Qwen2.5 (Zhang et al., 24 Sep 2025).

All stages optimize the same autoregressive conditional likelihood,

with

The novelty lies not in the loss itself but in the training schedule. Stage 1 freezes both the speech encoder and the LLM and trains only the adapter on monolingual ASR data $\theta_{\mathrm{ref},\mathrm{tgt}}= \begin{bmatrix} \Delta x\ \Delta y \end{bmatrix} = \begin{bmatrix} (x_{\mathrm{tgt}}-x_{\mathrm{ref}})/w_{\mathrm{ref}}\ (y_{\mathrm{tgt}}-y_{\mathrm{ref}})/h_{\mathrm{ref}} \end{bmatrix},$ 4, yielding a coarse speech-text bridge for each language individually. Stage 2 remains monolingual but progressively unfreezes the speech encoder: first the last 8 layers are unfrozen and trained jointly with the adapter, then the entire speech encoder is unfrozen. Stage 3 introduces cross-lingual tasks, activates the LLM, and jointly optimizes the speech encoder, adapter, and LLM on $\theta_{\mathrm{ref},\mathrm{tgt}}= \begin{bmatrix} \Delta x\ \Delta y \end{bmatrix} = \begin{bmatrix} (x_{\mathrm{tgt}}-x_{\mathrm{ref}})/w_{\mathrm{ref}}\ (y_{\mathrm{tgt}}-y_{\mathrm{ref}})/h_{\mathrm{ref}} \end{bmatrix},$ 5. The paper also states that later text-based tasks are introduced to enhance multilingual understanding and generation.

The underlying argument is that multilingual speech-text systems fail when they try to align all languages too aggressively and too early. Within-language alignment is therefore learned first, using only monolingual ASR, so that language-specific phonetic and lexical structure is preserved. Cross-language alignment is deferred until the speech-text interface is stable. Dynamic activation of LLM parameters is central to this logic: the LLM is frozen in stages 1 and 2, then unfrozen in stage 3 so that it can contribute multilingual semantic knowledge when the speech representations are already better organized. The paper presents this as a way to avoid both over-converging speech representations across languages and overburdening the speech encoder.

The training data comprise 810k hours of monolingual ASR data across 10 languages—Chinese, English, Japanese, Korean, Cantonese, German, French, Russian, Spanish, and Italian—and 434k hours of cross-lingual speech-text data covering directions such as zh-en, ja-en, de-en, fr-en, es-en, it-en, ru-en, en-zh, en-ja, en-de, en-sv, en-id, and en-ar. The S2TT data come from CoVoST, TED-LIUM, MuST-C, translations of ASR transcripts into target languages, and additional commercial data. Evaluation uses LibriSpeech, Wenetspeech, Fleurs, CommonVoice 15, and CoVoST2, with WER for English and other word-tokenized languages, CER for Chinese, Japanese, Korean, and Cantonese, BLEU for translation, Whisper-style normalization for ASR post-processing, character tokenization for Chinese and Japanese translation evaluation, and the 13a tokenizer for other languages. The reported speech encoder has about 700M parameters, two model sizes are used—PART-2B with Qwen2.5 1.5B and PART-8B with Qwen2.5 7B—and training uses 256 NVIDIA A800 GPUs for 3 epochs with greedy decoding at inference.

The main ASR highlights are an average WER reduction on Fleurs from 6.35 to 3.73, described as a 41% reduction, and on CommonVoice 15 from 9.18 to 6.29, described as a 32% reduction. For PART-8B on Fleurs, the paper reports language-specific WERs including zh 3.9, en 4.0, ja 3.1, ko 2.5, yue 3.7, de 4.4, fr 4.7, ru 5.2, es 2.8, and it 3.0. On CoVoST2, the model is reported as especially strong on low-resource directions, with en→sv, en→id, and en→ar gains of 3–8 BLEU over the two-stage baseline. Ablations compare a two-stage approach with the proposed three-stage schedule and compare mixed ASR+S2TT training from the outset with the PART strategy in which the first two stages use only ASR and the final stage uses both ASR and S2TT; the three-stage, alignment-first schedule performs better on both WER and BLEU. A progressive unfreezing ablation reports WER 6.4 and BLEU 31.7 for last8→full, versus WER 6.3 and BLEU 32.0 for full unfreezing.

6. Cross-cutting principles, distinctions, and limitations

Across these papers, the aligned objects differ substantially: off-grid patches in visual SSL, modality-specific embeddings in multimodal representation learning, encoder positions and token sequences in Aligner-Encoder ASR, and multilingual speech/text representations in speech-LLM systems. This suggests that PART is less a single algorithmic family than a recurrent research strategy for decomposing difficult alignment problems into simpler stages or relative constraints.

Several shared motifs nevertheless recur. One is the replacement of absolute or all-at-once objectives with relational or staged supervision. In the visual formulation, absolute grid index prediction is replaced by continuous relative translation (Ayoughi et al., 4 Jun 2025). In OneEncoder, end-to-end retraining of all modality-specific encoders is replaced by a frozen shared coordinate system plus small modality-specific alignment layers (Faye et al., 2024). In Aligner-Encoder ASR, a final-only monotonic alignment objective is replaced by intermediate alignment and CTC supervision across depth (Lee et al., 23 Jun 2026). In multilingual speech-to-text, simultaneous multilingual convergence is replaced by a curriculum that first stabilizes within-language alignment, then activates cross-language alignment and LLM adaptation (Zhang et al., 24 Sep 2025).

The limitations are correspondingly domain-specific. The visual PART paper states that classification gains are competitive but not always superior to the best grid-based SSL methods, and that the method adds complexity in sampling and pairwise prediction (Ayoughi et al., 4 Jun 2025). OneEncoder’s practical advantage depends on strong frozen pretrained encoders and on the assumption that new modalities can be attached transitively through the existing space (Faye et al., 2024). The ASR formulation is tightly tied to Aligner-Encoder architectures and derives its clearest gains on long utterances, where the late-layer bottleneck is most severe (Lee et al., 23 Jun 2026). The multilingual speech-LLM formulation still relies on a strong pre-trained multilingual speech encoder, a multilingual LLM, large-scale data, and substantial compute, and the paper notes that some implementation details of the later text-based tasks are not fully expanded in the provided description (Zhang et al., 24 Sep 2025).

A plausible implication is that future use of the acronym will continue to depend on domain-specific alignment failures: rigid grids in vision, heavy paired-data requirements in multimodal learning, abrupt late-layer alignment in ASR, or premature cross-language collapse in multilingual speech-text systems. What unifies these otherwise heterogeneous formulations is the view that alignment is not merely an emergent by-product of representation learning, but a training object that can be staged, factorized, and constrained deliberately.