Auxiliary Modality Learning in Multimodal Systems

• Auxiliary modality learning is a multimodal strategy that uses training-only auxiliary signals to guide models operating with fewer modalities at test time.
• It employs techniques like teacher–student transfer, auxiliary reconstruction, and adversarial alignment to bridge the gap caused by missing inputs.
• This approach enhances robustness in applications such as video recognition, speech translation, and medical imaging under incomplete modality conditions.

Searching arXiv for recent and foundational papers on auxiliary modality learning and closely related training-only modality frameworks. arXiv search query: "auxiliary modality learning training only modalities cross-modal distillation hallucination missing modality" Auxiliary modality learning denotes a family of multimodal learning formulations in which one or more modalities are available during training but absent, unreliable, or intentionally discarded during inference. The central objective is to exploit those auxiliary signals to improve the representation, prediction, or optimization of a target model that must ultimately operate with a reduced modality set. Across recent work, this objective is realized through several recurring mechanisms: cross-modal knowledge distillation, auxiliary reconstruction or back-reconstruction, adversarial feature enrichment, progressive curriculum design, prototype or center alignment, stochastic modality dropping, and teacher–student transfer from modality-complete to modality-reduced settings. The resulting systems span video recognition and retrieval, speech translation, medical image reconstruction and segmentation, visible–infrared re-identification, multimodal captioning, low-light video enhancement, collaborative multi-agent perception, and geometric foundation models (Chen et al., 2023).

1. Problem formulation and scope

In the formulation summarized for AMID, auxiliary-modality learning addresses the setting where “extra” modalities are present only during training, while inference is constrained to a subset of target modalities. A full-modality teacher model $M(x_1,\dots,x_n)$ is trained on all modalities, a student $S(x_1,\dots,x_k)$ is restricted to the target modalities used at test time, and an auxiliary model $A(x_{k+1},\dots,x_n)$ encodes the training-only modalities. The stated goal is to transfer as much multimodal information as possible from $M$ to $S$ so that the student approaches teacher-level performance without generating or hallucinating the missing modalities at inference (Chen et al., 2023).

This general pattern recurs with different operational emphases. In action recognition, RGB and optical flow are treated as source modalities while skeletons are auxiliary and used only during training; the model is trained so that source features compensate for the loss of skeletons at test time (Song et al., 2020). In multimodal brain tumor segmentation, only a small pool of subjects has full modalities, while most subjects are partial-modality cases; meta-learning is used to enhance partial-modality representations toward full-modality representations, and a discriminator encourages fused features to mimic a full-modality setting (Konwer et al., 2023). In multimodal video paragraph captioning, ASR transcripts and event boundaries are supportive auxiliary inputs that may be missing unpredictably at test time; the proposed framework therefore trains for both modality-complete and modality-deficient operation (Chen et al., 2024).

A related but distinct regime uses an auxiliary modality not merely as privileged training information but as a structural bridge. In visible–infrared person re-identification, MUN constructs a learned auxiliary modality by fusing visible and infrared features, and PMT uses gray-scale images as an auxiliary modality to progressively bridge RGB and infrared (Yu et al., 2023); (Lu et al., 2022). In PAN-sharpening, panchromatic imagery acts as an auxiliary high-resolution source whose structural cues are injected into multi-spectral reconstruction through joint reconstruction and cross-modality attention (Do et al., 29 May 2025). In these cases, the auxiliary modality is not simply omitted because it is unavailable; it also serves to regularize alignment, preserve shared structure, or isolate modality-invariant cues.

The literature therefore does not restrict auxiliary modality learning to a single algorithmic template. Rather, it encompasses training-only modality exploitation, intermediate bridging modalities, auxiliary tasks defined over privileged signals, and frameworks designed for arbitrary modality subsets during testing (Peng et al., 13 Nov 2025).

2. Core learning paradigms

A dominant paradigm is teacher–student transfer. AMID formalizes this directly with a full-modality teacher, a target-modality student, and an auxiliary-modality encoder, combining mutual-information maximization with conditional-entropy minimization so that the teacher remains informative and the student closes the teacher–student gap (Chen et al., 2023). CAML extends this paradigm from a single-agent setting to multi-agent systems: a teacher aggregates all agents and all training modalities, while a student uses only the test-time modalities but still aggregates across agents, with knowledge transfer implemented through a Kullback–Leibler distillation term (Liu et al., 25 Feb 2025). DistillAM in MR-VPC similarly trains a full-modality teacher on clean modality-complete data and uses its generated captions as pseudo-labels for a student trained under random modality omission (Chen et al., 2024). In speech translation, the auxiliary text-translation branch serves as an online teacher for speech translation through online knowledge distillation from MT to ST (Tang et al., 2021).

A second paradigm is auxiliary reconstruction or back-reconstruction. PAN-Crafter runs the same U-Net–style model in two training modes: an MS mode for HRMS reconstruction and a PAN mode for reconstructing a multi-band PAN replication. The auxiliary PAN branch is weighted within a joint $\ell_1$ loss and forces intermediate features to encode high-frequency PAN structure; at inference the PAN mode is disabled and the network runs only in MS mode (Do et al., 29 May 2025). This is auxiliary modality learning in the sense that the privileged modality contributes a training objective that shapes the features used by the primary task. A related design appears in AMNet for low-light video enhancement, where missing event or infrared features are replaced by implicit auxiliary representations generated from RGB by a Spatial–Spectral Dual-Gated Translator; feature-level distillation then aligns these generated features with real auxiliary features (Liang et al., 9 Jun 2026).

A third paradigm is adversarial or discriminator-based feature enrichment. In the brain tumor segmentation framework, the discriminator predicts which modalities are present from the fused bottleneck representation, while the generator is trained against a dummy all-ones label so that partial-modality fused features resemble full-modality features in latent space (Konwer et al., 2023). AMID likewise uses adversarial learning to minimize the conditional entropy of teacher representations given student representations, employing two discriminators to align student outputs with the teacher distribution (Chen et al., 2023).

A fourth paradigm uses progressive or curriculum-based modulation. PMT first trains on gray-scale and infrared, then switches to visible and infrared, so that early optimization emphasizes structural cues without color bias (Lu et al., 2022). The BUPTCampus AuxNet uses a curriculum factor $\alpha(E)=\frac{\cos(\pi E)+\phi}{2(1+\phi)}$ to give strong early weight to an easier auxiliary branch defined on single-camera samples and then gradually shift emphasis to the harder primary task (Du et al., 2023).

These paradigms are compatible rather than mutually exclusive. Several systems combine auxiliary branches, stochastic dropping, distillation, and explicit alignment objectives to preserve performance under inference-time modality loss (Chen et al., 2024).

3. Information-theoretic, adversarial, and alignment formulations

AMID supplies one of the clearest mathematical formulations. It identifies a shortcut in naive mutual-information maximization between teacher and student representations, namely the degenerate “weak-teacher” solution in which the teacher collapses its outputs while still achieving deceptively high $I(Z_T;Z_S)$. To prevent this, AMID adds an auxiliary-modality mutual-information term $I(Z_T;Z_A)$, thereby forcing the teacher to remain informative about the auxiliary modalities. It further minimizes $H(Z_T \mid Z_S)$ to reduce the information gap between teacher and student. The overall objective is

$S(x_1,\dots,x_k)$0

with contrastive lower bounds used for the mutual-information terms and adversarial learning used for conditional-entropy minimization (Chen et al., 2023).

Alignment-based formulations appear in several other settings. MUN introduces an identity-alignment loss $S(x_1,\dots,x_k)$1 over per-identity centers from visible, infrared, and auxiliary modalities, together with a modality-alignment loss $S(x_1,\dots,x_k)$2 based on modality prototypes updated by exponential moving average and aligned using squared Maximum Mean Discrepancy. The auxiliary modality is explicitly described as a “bridge” between visible and infrared distributions (Yu et al., 2023). PMT’s Modality-Shared Enhancement Loss minimizes squared discrepancies between intra-modality and cross-modality distances, while its Discriminative Center Loss compresses intra-class distances relative to hard negatives (Lu et al., 2022). In MAC-VR, automatically extracted modality-specific tags are embedded into auxiliary latent concepts and aligned back to the video and text concepts through an auxiliary alignment loss $S(x_1,\dots,x_k)$3 (Fragomeni et al., 2 Apr 2025).

Adversarial formulations are used when the desired full-modality state is defined implicitly rather than by explicit reconstruction. In the brain tumor segmentation framework, a missing-modality detector attempts to infer modality presence from fused bottleneck features, while the generator is trained with

$S(x_1,\dots,x_k)$4

where $S(x_1,\dots,x_k)$5 denotes the all-modalities-present target (Konwer et al., 2023). This creates a latent-space hallucination objective without synthesizing missing images.

A notable implication is that auxiliary modality learning does not require explicit generation of the missing raw signal. Some methods, such as AMID, avoid modality hallucination at inference entirely (Chen et al., 2023). Others, such as AMNet, generate implicit auxiliary features rather than pixel-space reconstructions (Liang et al., 9 Jun 2026). This suggests a conceptual distinction between exploiting privileged multimodal information and reconstructing absent observations.

4. Architectural strategies

Architecturally, auxiliary modality learning often combines modality-specific shallow processing with deeper shared or fused representations. MARIO uses a two-stream, dual-domain Dense-UNet in both k-space and image domains, with one stream for the auxiliary modality and one for the target modality; T1-guided attention modules inject auxiliary structure hierarchically at multiple scales (Feng et al., 2021). MUN duplicates only the first ResNet stage for visible and infrared inputs and shares later stages across visible, infrared, and the learned auxiliary feature branch (Yu et al., 2023). The pose-auxiliary VI-ReID model uses separate shallow layers for RGB and IR, followed by a shared backbone, then splits into a pose estimation branch and a ReID branch, with pose-derived attention masking the ReID features (Miao et al., 2022).

Other architectures make auxiliary interaction explicit through token- or graph-level mechanisms. CORECT constructs three nodes per utterance—lexical, acoustic, and visual—and combines modality-specific temporal edges with intra-utterance multimodal edges in a relational temporal graph. A separate Pairwise Cross-Modal transformer computes six pairwise attention streams, which function as auxiliary cross-modality interactions for emotion recognition (Nguyen et al., 2023). MAC-VR disentangles video, text, visual-tag, and textual-tag embeddings into latent concepts and uses those auxiliary concepts to weight and align modality-specific concept factors (Fragomeni et al., 2 Apr 2025).

Several recent systems are designed explicitly for arbitrary auxiliary modality combinations. MR-VPC concatenates video embeddings with the embeddings of any available auxiliary inputs, and its DropAM procedure randomly replaces ASR transcripts or event boundaries with null sequences during training (Chen et al., 2024). OmniVGGT generalizes this idea within a 3D foundation model: a GeoAdapter injects depth maps and camera intrinsics/extrinsics into a VGGT backbone, while stochastic multimodal fusion randomly samples modality subsets per training instance so that testing can use any subset from zero to all available auxiliary geometric modalities (Peng et al., 13 Nov 2025). AMNet for low-light video enhancement similarly includes real event and infrared encoders when those modalities are present and uses the S2DG translator to synthesize implicit features when they are absent (Liang et al., 9 Jun 2026).

The following table summarizes several recurring architectural roles for auxiliary modalities.

Role of auxiliary modality	Representative mechanism	Example papers
Privileged teacher signal	Full-modality teacher guiding reduced-modality student	(Chen et al., 2023, Liu et al., 25 Feb 2025, Tang et al., 2021)
Auxiliary reconstruction target	Joint primary reconstruction and auxiliary back-reconstruction	(Do et al., 29 May 2025)
Latent-space full-modality surrogate	Discriminator encourages partial inputs to resemble full-modality features	(Konwer et al., 2023)
Bridge modality	Learned or hand-crafted intermediate modality reduces domain gap	(Yu et al., 2023, Lu et al., 2022)
Auxiliary task branch	Parallel pose, translation, or tag-alignment objective regularizes the main task	(Miao et al., 2022, Fragomeni et al., 2 Apr 2025)
Arbitrary-subset robustness	Random modality dropping or stochastic subset sampling	(Chen et al., 2024, Peng et al., 13 Nov 2025, Liang et al., 9 Jun 2026)

5. Representative application domains

In video understanding, AMID reports improvements across video recognition, video retrieval, and emotion classification. On UCF51 and ActivityNet for video recognition, the visual-only baseline is 66.7% / 48.8% Top-1 accuracy, while AMID achieves 73.8% / 53.6%. On video retrieval, AMID reaches 73.6/77.1% and 46.4/67.2% for $S(x_1,\dots,x_k)$6 on UCF51 and ActivityNet, respectively (Chen et al., 2023). MCN addresses action recognition with RGB, optical flow, and skeletons; on NTU cross-subject, S-Res-LSTM improves RGB from 79.6 to 82.0 and flow from 85.8 to 87.6, while two-stream fusion reaches 89.5 / 95.2 on NTU cross-subject / cross-view using only RGB+Flow at test time (Song et al., 2020).

In language generation and translation, auxiliary text translation improves speech translation. The full proposal in the speech translation study reaches 26.8 BLEU on MuST-C EN→DE, 31.0 on EN→ES, and 37.4 on EN→FR, exceeding the vanilla joint-training baseline by 2.7, 2.0, and 2.3 BLEU, respectively (Tang et al., 2021). MR-VPC transfers the same broad principle to video paragraph captioning: MVPC with all modalities reaches 74.13 CIDEr on YouCook2 and 43.14 on ActivityNet, while MR-VPC remains robust under missing modalities, including 38.37 CIDEr for video-only inference on YouCook2 compared with approximately 3.4 for Vid2Seq-only (Chen et al., 2024).

Medical imaging provides multiple distinct instantiations. MARIO uses a fully sampled auxiliary MR modality to guide reconstruction of an undersampled target modality. On IXI, under 1D-Random 3× undersampling, MoDL obtains 32.6 dB / 0.926, DuDoRNet 33.9 dB / 0.945, and MARIO 36.9 dB / 0.969 (Feng et al., 2021). The modality-agnostic brain tumor segmentation method reports, on BRATS2018 with 50% of subjects used as full-modality data, average DSC values of 87.12% for WT, 79.12% for TC, and 62.53% for ET across fifteen missing-modality cases, exceeding RFNet and ACN (Konwer et al., 2023).

In cross-spectral person re-identification, auxiliary modality learning serves both as supervision and as domain bridging. MUN improves SYSU-MM01 single-shot all-search from a no-auxiliary baseline of Rank-1 57.49% / mAP 55.83% to 76.24 / 73.81 with all components (Yu et al., 2023). PMT reports 67.53% / 64.98% rank-1 / mAP on SYSU-MM01 all-search and 84.83% / 76.55% on RegDB Visible→Thermal, with gray-scale used as an auxiliary bridge modality (Lu et al., 2022). The pose-auxiliary VI-ReID model reaches 65.8% Rank-1 / 64.5% mAP on SYSU-MM01 all-search and 93.4% / 89.0% on RegDB Visible→Thermal (Miao et al., 2022).

Other domains show the breadth of the concept rather than a single canonical formulation. PAN-Crafter uses PAN as auxiliary self-supervision for PAN-sharpening and reports improvements on WorldView-3 full resolution from PSNR 37.44 dB to 37.96 dB, SSIM 0.973 to 0.976, and HQNR 0.951 to 0.958 relative to CANConv (Do et al., 29 May 2025). CAML extends auxiliary modality learning to collaborative decision-making and semantic segmentation in multi-agent systems, reporting up to a 58.13% improvement in accident detection and up to a 10.61% improvement in mIoU on aerial-ground robot semantic segmentation (Liu et al., 25 Feb 2025). OmniVGGT shows that auxiliary geometric modalities can be integrated into a 3D foundation model while preserving RGB-only capability and inference speed comparable to VGGT (Peng et al., 13 Nov 2025).

6. Robustness, test-time adaptation, and broader implications

A central research motive is robustness to incomplete or changing modality availability. MR-VPC addresses the assumption that a specific auxiliary modality will always be present, arguing instead for resilience to random absence of ASR or event boundaries through DropAM and DistillAM (Chen et al., 2024). OmniVGGT formalizes this more generally with stochastic multimodal fusion, randomly masking camera and depth inputs per instance while also forcing RGB-only batches with probability $S(x_1,\dots,x_k)$7 (Peng et al., 13 Nov 2025). AMNet supports all four combinations of presence and absence for event and infrared modalities by replacing any missing branch with an implicit feature produced from RGB, and reports that full-modality training yields 30.35 dB even when inferring RGB only, rising to 31.29 dB when both events and IR are present at test time (Liang et al., 9 Jun 2026).

Some work pushes adaptation beyond fixed training. Highlight-TTA proposes test-time adaptation for video highlight detection through an auxiliary task of cross-modality hallucinations and a meta-auxiliary training scheme, with adaptation performed on the test video using the auxiliary task (Islam et al., 6 Aug 2025). The abstract establishes the role of an auxiliary modality-related task in dynamic test-time specialization, but the prompt does not include the technical sections needed to characterize the learning objective, architecture, or adaptation algorithm in detail. This suggests a further extension of auxiliary modality learning from training-time privileged information toward test-time self-adaptation.

A common misconception is that auxiliary modality learning is equivalent to multimodal fusion. The cited literature indicates otherwise. Many methods explicitly discard the auxiliary branch at inference, retain only the student or target branch, or learn a model that can flexibly operate with arbitrary modality subsets (Chen et al., 2023); (Song et al., 2020); (Chen et al., 2024). Another misconception is that improvements necessarily derive from semantic alignment with the auxiliary modality. SimpAux provides a cautionary counterexample: in few-shot learning with language-conditioned CBN, the reported gain on CUB can be matched by replacing the language-conditioned bridge input with a random vector, leading the authors to conclude that the observed benefit was due to the additional compute and parameters rather than the language signal itself (Armengol-Estapé et al., 2024). This constitutes an explicit limitation study within the broader area.

The broader implication is that auxiliary modality learning is best understood as a design principle rather than a single method: use privileged or structurally informative side modalities to shape a model that remains effective when those signals are missing, expensive, or intentionally excluded at deployment. The literature shows that this principle can be instantiated through distillation, alignment, adversarial latent supervision, stochastic subset training, progressive curricula, or auxiliary branch objectives, with the exact choice determined by modality structure, task geometry, and the nature of the train–test modality mismatch (Chen et al., 2023).

7. Open issues and methodological cautions

Several papers point to unresolved methodological issues. First, gains attributed to auxiliary modalities may actually stem from extra capacity or optimization effects. SimpAux is explicit on this point, finding that improvements do not reproduce across benchmarks and, when they do, are explained by the bridge network’s additional parameters and compute (Armengol-Estapé et al., 2024). This suggests that compute-matched and random-conditioning controls are necessary when evaluating auxiliary-modality benefits.

Second, modality alignment can fail in degenerate ways. AMID’s weak-teacher analysis shows that maximizing only $S(x_1,\dots,x_k)$8 permits teacher collapse; the auxiliary-modality mutual-information term is introduced precisely to prevent this (Chen et al., 2023). A plausible implication is that auxiliary modality learning objectives must preserve auxiliary information content, not merely cross-modal agreement.

Third, robustness to modality omission is not guaranteed by multimodal training alone. MR-VPC introduces DropAM because models trained only with complete modalities remain fragile under missing-modality test conditions (Chen et al., 2024). OmniVGGT similarly adopts stochastic subset sampling rather than assuming that auxiliary geometry will always be available (Peng et al., 13 Nov 2025). These results suggest that train–test mismatch in modality availability must be modeled explicitly.

Fourth, auxiliary inputs can be noisy, spatially misaligned, or only weakly correlated with the target modality. PAN-Crafter is motivated by cross-modality misalignment between PAN and MS caused by sensor placement, acquisition timing, and resolution disparity, and addresses this with modality-consistent alignment rather than per-pixel reconstruction assumptions (Do et al., 29 May 2025). In multi-agent settings, CAML notes that misaligned or noisy auxiliary modalities can degrade performance, and that robust alignment or attention-based gating may be needed (Liu et al., 25 Feb 2025).

Finally, the area includes distinct notions of “auxiliary.” In some work the auxiliary signal is a missing test-time modality; in other work it is a constructed bridge modality, an auxiliary task over modality-shared cues such as pose, or a training-time tag representation produced by a foundation model (Yu et al., 2023); (Miao et al., 2022); (Fragomeni et al., 2 Apr 2025). This terminological breadth is useful but can obscure comparisons unless the training-time and inference-time role of the auxiliary signal is specified precisely.

Taken together, the current literature portrays auxiliary modality learning as a technically diverse and rapidly expanding area centered on one persistent problem: how to capture the informational advantage of multimodal supervision without assuming that all modalities remain available, reliable, or economical at deployment.