Progressive Modality Alignment

• Progressive modality alignment is a multimodal learning strategy that establishes cross-modal correspondence step-by-step to maintain modality-specific structure.
• It uses design patterns like anchor-first, reliable-first, and coarse-to-fine refinement to sequentially build alignment between diverse representations.
• Empirical results in sentiment analysis, medical imaging, and speech recognition show that progressive alignment mitigates noise and boosts task performance.

Searching arXiv for papers on progressive modality alignment and closely related formulations. {"queries":[{"q":"\"progressive modality alignment\" multimodal","limit":10},{"q":"progressive alignment multimodal arXiv","limit":10},{"q":"\"Progressive\" modality alignment multimodal arXiv 2024 2025","limit":10}]} I found multiple relevant arXiv entries spanning multimodal sentiment analysis, multimodal understanding/generation, medical image-text pretraining, test-time adaptation, and lightweight unified alignment frameworks. I will ground the encyclopedia entry in these papers and the provided source block, citing the relevant arXiv IDs directly. Progressive modality alignment denotes a family of multimodal learning strategies in which cross-modal correspondence is established in stages rather than by one-shot fusion. Across recent work, the progression may proceed from a stable anchor space to newly introduced modalities, from low-cost or reliable modalities to high-cost or unreliable ones, from coarse global correspondence to fine local grounding, or from disentangled and denoised representations to fused multimodal decision spaces. The common premise is that direct fusion can be unstable when modalities are missing, shifted, noisy, weakly paired, safety-critical, or semantically heterogeneous, whereas staged alignment can preserve modality-specific structure while gradually improving shared semantics (Faye et al., 2024, Bao et al., 10 Mar 2026, Yan et al., 25 Feb 2025).

1. Conceptual scope and definitions

In current usage, “alignment” is not a single invariant object. In some papers it is a shared latent space learned by contrastive training, as in image–text anchoring followed by progressive attachment of audio and video in OneEncoder (Faye et al., 2024). In others it is a task-specific representational geometry: phoneme space in open-modality speech recognition (Cheng et al., 2023), safety-concept space in PSA-VLM (Liu et al., 2024), word–pixel correspondence in PLAN (Yan et al., 25 Feb 2025), or layerwise vision–language similarity measured by mutual-kNN in UniFork (Li et al., 20 Jun 2025). Progressive modality alignment therefore refers less to a fixed loss than to an ordering principle governing when, where, and how cross-modal agreement is enforced.

A recurring distinction is between alignment and fusion. Several systems first stabilize or purify modality-specific representations and only later combine them. PMF freezes alignment-irrelevant entity features before weighted multi-modal fusion (Huang et al., 2024). PIA first constructs clothing-irrelevant identity representations with DBDL and only then performs prototype-based visible–infrared alignment with BPL (Xu et al., 3 Apr 2026). BriMPR first calibrates unimodal global distributions and only afterward refines cross-modal interaction (Li et al., 28 Nov 2025). This suggests that progressive modality alignment is often used as an intermediate control layer between raw modality encoding and final task inference.

The term “progressive” also has multiple operational meanings. It can denote stage-wise module activation, as in Math-PUMA’s three-stage training and PART’s adapter-first, encoder-next, LLM-later schedule (Zhuang et al., 2024, Zhang et al., 24 Sep 2025). It can denote iteration, as in ProgInteract, PLAN, and one-stage coarse-to-fine dense registration (Bao et al., 10 Mar 2026, Yan et al., 25 Feb 2025, Wang et al., 2023). It can denote curriculum over data quality or ambiguity, as in PFA on the ASE dataset and PICA for domain-generalized OVOD (Hsu et al., 2024, Xu et al., 29 Mar 2026). It can also denote staged clinical acquisition, where earlier low-cost features are aligned to later high-cost stages for Alzheimer’s subtype diagnosis (Liu et al., 2024).

2. Recurrent design principles

Recent formulations repeatedly instantiate a small set of design patterns.

Pattern	Mechanism	Representative papers
Anchor-first alignment	Learn a shared space on a strong paired subset, then freeze it and attach new modalities	(Faye et al., 2024, Cheng et al., 2023)
Reliable-first alignment	Select a dominant or cleaner modality and align others toward it	(Bao et al., 10 Mar 2026, Zhuang et al., 2024)
Purify-before-align	Remove nuisance factors before cross-modal matching	(Xu et al., 3 Apr 2026, Huang et al., 2024)
Coarse-to-fine refinement	Start with global or shallow correction, then refine local or instance-level agreement	(Li et al., 28 Nov 2025, Yan et al., 25 Feb 2025, Wang et al., 2023)
Difficulty-aware curriculum	Introduce hard, ambiguous, or low-signal samples gradually	(Hsu et al., 2024, Xu et al., 29 Mar 2026)

The anchor-first pattern appears most explicitly in OneEncoder. It first trains a lightweight Universal Projection module on image–text pairs, freezes that module, and then aligns audio and video with only a small Alignment Layer and modality token per new modality (Faye et al., 2024). OpenSR uses an analogous preservation principle in a different domain: AV-Hubert pretraining establishes audio–visual phoneme-space alignment, after which AV-Fusion and the encoder are frozen so that a decoder trained on target-domain audio can transfer zero-shot to visual-only and audio-visual inference (Cheng et al., 2023).

The reliable-first pattern appears when modalities are incomplete or asymmetric. PRLF uses an Adaptive Modality Reliability Estimator combining classification confidence and Fisher information to choose a dominant modality, then uses ProgInteract to align auxiliary modalities toward it under missing-modality conditions (Bao et al., 10 Mar 2026). Math-PUMA operationalizes a similar asymmetry at the task level: text-rich math formulations serve as the stronger modality condition whose next-token distributions supervise weaker, more vision-rich versions of the same problem (Zhuang et al., 2024).

Purify-before-align is prominent when raw alignment would reinforce spurious factors. In CMCC-ReID, direct visible–infrared alignment is risky because clothing variation can dominate identity cues; PIA therefore disentangles identity and clothing first, then aligns identity features with intra- and inter-modality prototypes (Xu et al., 3 Apr 2026). In multi-modal entity alignment, PMF progressively freezes modality features whose relevance score drops to zero, preventing gradient updates from alignment-irrelevant content before joint fusion and cross-modal association learning (Huang et al., 2024).

3. Architectural realizations

One major architectural family uses frozen encoders plus lightweight trainable bridges. OneEncoder freezes ViT, BERT, Wav2Vec2, and VideoMAE, trains a transformer-based Universal Projection of about 4M parameters for image–text alignment, and later adds only a two-layer MLP Alignment Layer of about 65,792 parameters for each new modality (Faye et al., 2024). PART keeps the same high-level speech-LLM stack but varies which modules are trainable by stage: adapter only, then adapter plus speech encoder, then adapter plus speech encoder plus LLM (Zhang et al., 24 Sep 2025). These systems treat progression as a parameter-allocation policy as much as a representation-learning policy.

A second family is based on iterative interaction and controlled overwrite. In PRLF, each modality undergoes self-refinement, then weighted cross-modal interaction, and finally gated decomposition into projected shared content and residual complementary content. The update rule keeps the dominant modality stable while progressively adjusting auxiliary modalities through projection, residual denoising, and a phase constraint that encourages orthogonality between shared and residual parts (Bao et al., 10 Mar 2026). BriMPR uses a similar two-level logic under test-time shift: Prompt-driven Modality-specific Global Feature Alignment first matches target means and standard deviations to source statistics, and Inter-modal Interaction Enhancement for Alignment Refinement then uses masked recombination and instance-wise contrastive learning to tighten semantic agreement (Li et al., 28 Nov 2025).

A third family localizes progression inside the aligned structure itself. PLAN begins with a word–pixel similarity matrix, estimates word and pixel importance, constructs a co-importance matrix, reweights similarities, extracts keyword–region pairs, and repeats the process over multiple rounds with a smoothed similarity update. Alignment is thus not merely global image–text matching but iterative soft region recognition (Yan et al., 25 Feb 2025). PromViL extends this logic to compositional reasoning: lower-level grounded expressions are used as explicit context for higher-level expressions, so grounding a complex phrase is conditioned on previously grounded simpler phrases (Le et al., 2024).

Depth-structured architectures express progression along the network rather than the training schedule. UniFork measures alignment between mean-pooled visual and textual features with mutual-kNN across Transformer depth and argues that understanding prefers monotonically increasing alignment, while generation prefers an increase-then-decrease profile. Its Y-shaped design shares shallow layers and forks into deeper task-specific branches so that the two tasks need not share the same alignment trajectory (Li et al., 20 Jun 2025). This paper is notable because it treats progressive alignment as an observable representational dynamic rather than only a training prescription.

Concept-bottleneck architectures represent another variation. PSA-VLM inserts a Safety Projector, Safety Tokens, and a Safety Head between the vision encoder and the LLM. Stage I trains safety type and level classification while the VLM backbone is frozen; Stage II unfreezes the LLM so that generation becomes conditioned on the learned safety concepts (Liu et al., 2024). Here progression moves from concept extraction to response shaping, and “alignment” means making the visual pathway consistent with explicit safety concepts rather than only with generic language semantics.

4. Objectives, schedules, and optimization motifs

Despite architectural diversity, several objective motifs recur. Contrastive learning remains central. OneEncoder uses a symmetric InfoNCE-style objective to align image and text and later the anchor modality with each new modality (Faye et al., 2024). PFA on the ASE dataset aligns image embeddings with both LLM-derived and VLM-derived text embeddings through symmetric image-to-text and text-to-image contrastive losses, summing $L_{itc}^{LLM}$ and $L_{itc}^{vlm}$ into $L_{itc}^{total}$ (Hsu et al., 2024). PLAN combines global CLIP-style contrastive loss with local word–region contrastive alignment (Yan et al., 25 Feb 2025). PICA employs a bidirectional InfoNCE curriculum loss between region features and adaptive pseudo-word prototypes (Xu et al., 29 Mar 2026).

Another motif is asymmetry in supervision. PRLF computes modality importance as

$$\mu^{(t,i)} = (1 - w^{(t,i)}) \, \hat{\alpha}^{(i)} + w^{(t,i)} \, \hat{\beta}^{(i)},$$

where $\hat{\alpha}$ is normalized classifier confidence and $\hat{\beta}$ is normalized Fisher-information trace, then uses the highest-$\mu$ modality as the guide for alignment (Bao et al., 10 Mar 2026). Math-PUMA makes the stronger problem presentation supervise the weaker one through forward and reverse KL divergence between next-token prediction distributions, combined with a hard supervised loss

$$\mathcal{L}=\lambda_{\text{KL}}(\alpha_{\text{KL}}\mathcal{L}_{\text{FKL}}+(1-\alpha_{\text{KL}})\mathcal{L}_{\text{RKL}})\tau^2+(1-\lambda_{\text{KL}})\mathcal{L}_{\text{hard}},$$

with $\alpha_{\text{KL}}=0.2$, $\tau=1.0$, and $L_{itc}^{vlm}$0 (Zhuang et al., 2024).

Distribution matching and geometric constraints form a third motif. BriMPR aligns layerwise target and source feature means and standard deviations with $L_{itc}^{vlm}$1, then adds masked recombination and inter-modal instance-wise contrastive learning, yielding

$L_{itc}^{vlm}$2

(Li et al., 28 Nov 2025). PRLF adds a phase constraint $L_{itc}^{vlm}$3 so that the projected shared component and residual component are encouraged to be orthogonal, explicitly limiting over-alignment (Bao et al., 10 Mar 2026). PMF supplements cross-KG supervision with a cross-modal association loss that pulls together modality embeddings of the same entity and pushes away negatives across modalities (Huang et al., 2024).

Schedules are equally important. ASE PFA divides $L_{itc}^{vlm}$4 into sub-blocks and reports better performance for the finer schedule $L_{itc}^{vlm}$5 than for direct whole-set alignment or $L_{itc}^{vlm}$6 (Hsu et al., 2024). PICA progressively shifts sampling from easier to harder ambiguity tiers with $L_{itc}^{vlm}$7, while penalizing the bottom $L_{itc}^{vlm}$8 percentile of signal strength with $L_{itc}^{vlm}$9 (Xu et al., 29 Mar 2026). PMF increases the freezing threshold $L_{itc}^{total}$0 from $L_{itc}^{total}$1 to $L_{itc}^{total}$2 over epochs (Huang et al., 2024). These schedules indicate that progression is often encoded not only in architecture but also in the temporal exposure of the optimizer to increasingly difficult or less reliable cross-modal relations.

5. Application domains and empirical evidence

Progressive modality alignment now appears across a wide range of tasks. In multimodal sentiment analysis with incomplete modalities, PRLF is evaluated under inter-modality missingness on CMU-MOSI, CMU-MOSEI, and SIMS, and under intra-modality missingness with frame drop rates $L_{itc}^{total}$3. Its ablations show that removing Progressive Interaction causes the largest drop, and increasing interaction steps from 1 to 4 improves performance while step 5 slightly degrades it; 4 iterations is reported as the optimal balance in that setting (Bao et al., 10 Mar 2026). In multimodal test-time adaptation, BriMPR raises Kinetics50-C severity-5 average accuracy from 60.5 to 65.9, raises VGGSound-C unimodal audio corruption performance from 25.0 to 36.5, achieves 40.9 under multimodal shift on Kinetics50-C, 20.7 on VGGSound-C, and reports 58.2 ACC / 57.6 F1 on MOSI→SIMS (Li et al., 28 Nov 2025).

In reasoning-heavy vision–language settings, Math-PUMA reports that Math-PUMA-Qwen2-7B reaches 33.6 on MathVerse ALL and 47.9 overall on MathVista, while Stage 2 is identified as the key modality-alignment stage for reducing the gap between text-rich and vision-rich formulations (Zhuang et al., 2024). PromViL improves compositional grounding on CompoVL-hard from 55.37 with Kosmos-2 to 64.07, and its ablation shows that removing intermediate reasoning levels degrades performance from 64.07 to 61.22, indicating that the intermediate progressive alignments are not redundant (Le et al., 2024). UniFork shows that alignment trajectories themselves can be diagnostic: understanding prefers deeper cross-modal convergence, whereas generation requires later-layer decoupling to recover spatial detail (Li et al., 20 Jun 2025).

Medical and safety applications further broaden the concept. PLAN reports that phrase-grounding performance improves as iterations increase from 1 to 3 and then slightly declines for 4 and 5, selecting $L_{itc}^{total}$4 as optimal; it also reports Prec@1 of 55.88 on MIMIC-5x200 retrieval and 0.8820 zero-shot classification accuracy on COVID (Yan et al., 25 Feb 2025). PSA-VLM reports 8.46 on RTVLM for PSA-VLM-13B + LoRA, while maintaining general capability measured on MMBench, SEEDBench, and MME; its safety head is identified as the dominant contributor in ablation (Liu et al., 2024). In industrial defect classification, direct alignment yields 84.76 / 70.45, whereas PFA with $L_{itc}^{total}$5 reaches 92.52 / 80.77, providing a clear example where gradual alignment outperforms both no alignment and one-shot alignment (Hsu et al., 2024). In clinical diagnosis, a progressive framework for Alzheimer’s subtype diagnosis explicitly proposes aligning earlier low-cost-stage features with later high-cost-stage features so that accurate diagnosis can be obtained without acquiring later-stage modalities (Liu et al., 2024).

Speech and open-vocabulary perception offer further evidence. OpenSR reports 36.0% WER for zero-shot lip reading, 25.0% WER for full-shot lip reading, and 2.7% WER for audio-visual speech recognition on LRS2, attributing transfer to maintained audio–visual phoneme-space alignment under backbone freezing (Cheng et al., 2023). PICA reaches $L_{itc}^{total}$6 on OV-COCO-C and 19.7 on OV-COCO-O, and its analyses report lower AI-gap and higher gradient direction cosine similarity than baselines, supporting the view that stable latent semantic anchoring is central under domain shift (Xu et al., 29 Mar 2026). OneEncoder shows that a lightweight progressive strategy can be competitive even outside specialized multimodal pretraining regimes, improving zero-shot CIFAR-10 from CLIP’s 62.12 to 78.15 and reaching strong MSVD retrieval metrics such as text-to-video R@5 = 80.76 and video-to-text R@5 = 91.62 (Faye et al., 2024).

6. Misconceptions, limits, and unresolved issues

A common misconception is that progressive modality alignment simply means “more alignment is better.” The recent literature does not support that view. UniFork explicitly argues that understanding and generation require different depth-dependent alignment patterns, with generation benefiting from late-layer decoupling rather than monotonic convergence (Li et al., 20 Jun 2025). PRLF adds orthogonality-constrained residual preservation to avoid overwriting complementary modality content (Bao et al., 10 Mar 2026). PLAN reports degradation beyond three refinement rounds, interpreting it as over-refinement that may remove important words (Yan et al., 25 Feb 2025). These results indicate that useful progression is usually selective and task-conditional, not maximal.

Another misconception is that progression is equivalent to late fusion. In many systems, the critical operation occurs before fusion or even instead of conventional fusion. PMF freezes harmful modality channels before joint representation formation (Huang et al., 2024). OpenSR preserves pretraining geometry by freezing the multimodal backbone rather than adapting all branches jointly (Cheng et al., 2023). PSA-VLM inserts an interpretable safety bottleneck that constrains what the visual modality can transmit to the LLM (Liu et al., 2024). This suggests that progressive alignment often acts as a regularization and routing mechanism, not merely as a fusion schedule.

The main limitations are likewise recurrent. Several methods depend on trustworthy anchors or statistics: OneEncoder depends on a strong initial image–text anchor (Faye et al., 2024), BriMPR depends on pre-estimated source statistics (Li et al., 28 Nov 2025), and PICA weakens on extreme stylization such as Sketch and Tattoo where signal-strength proxies become unreliable (Xu et al., 29 Mar 2026). Schedule sensitivity is also substantial. PIA reports the best results when Stage II begins at epoch 55, with earlier alignment harming performance (Xu et al., 3 Apr 2026). PMF’s threshold analysis shows that too large a $L_{itc}^{total}$7 can freeze useful information (Huang et al., 2024). These findings imply that progression is itself a hyperparameterized inductive bias whose benefits depend on when purification, calibration, alignment, and fusion are allowed to interact.

A plausible implication is that future work will continue separating modality-wise calibration from cross-modal refinement, and will treat modality reliability, alignment granularity, and schedule design as first-class modeling choices rather than secondary implementation details. The literature surveyed here already points in that direction: progression can occur across modules, depths, samples, domains, concepts, or acquisition stages, but in each case it is used to control when shared semantics should dominate and when modality-specific structure should be preserved.