Modality-Agnostic Input Channel

• Modality-Agnostic Input Channel is an interface that accepts diverse, arbitrary, or evolving modalities and converts them into a unified representation for downstream processing.
• It employs architectural patterns like isolated fusion, symmetric aggregation, shared decoders, and dynamic channel mapping to integrate data from different sensors and support continual learning.
• Robust training methods such as modality dropout and cross-modal distillation ensure resilience against missing, corrupted, or novel modalities.

A modality-agnostic input channel is a model component or interface that accepts different modalities, arbitrary modality subsets, or evolving modality sets and maps them into a unified representation that a downstream network can process without redesigning the core predictor. In recent work, this idea appears as a fused feature tensor produced from separate MRI streams, a binary edge-map pathway that conditions a frozen foundation model, a dynamic channel-inflation mechanism for continual learning, a shared modality-agnostic feature map for semantic segmentation, and a shared decoder that operates on camera-only, LiDAR-only, or joint features (Shu et al., 12 Apr 2026, Hollet et al., 6 Aug 2025, Sadegheih et al., 20 Jan 2026, Zheng et al., 2024, Cha et al., 2024).

1. Conceptual scope and meanings

The literature uses “modality-agnostic” in several non-equivalent but related senses. In multi-modal medical segmentation, it often denotes an input or fusion block that can accept an arbitrary subset of modalities and produce a single tensor for a standard backbone, as in Modality-Isolated Gated Fusion (MIGF), or a decoder-compatible feature map formed by averaging modality-specific encoders, as in MAG-MS (Shu et al., 12 Apr 2026, He et al., 2023). In continual learning, it can mean that the architecture does not need prior knowledge of the maximum modality set and can expand on-the-fly when a novel modality appears, which is the role of input-layer inflation in CLMU-Net (Sadegheih et al., 20 Jan 2026). In modality-incomplete semantic segmentation, it denotes a feature path that can combine any subset of available sensors into a modality-agnostic representation $f_{ma}$, allowing the same segmentation head to operate under all $2^4-1=15$ non-empty modality combinations (Zheng et al., 2024).

A second meaning emphasizes representation rather than fusion. The concept-centric framework defines a modality-agnostic knowledge space $\mathcal{K}\subset\mathbb{R}^d$ populated by concept boxes $\Omega_y$, while modality-specific projection models map vision or language inputs to boxes in the same space (Geng et al., 2024). MA-ViT similarly seeks modality-agnostic liveness features in a single-branch transformer by combining modality tokens, spectrum embeddings, Modal-Disentangle Attention, and Cross-Modal Attention (Liu et al., 2023). In SeamlessM4T v2, modality-agnostic behavior would mean that speech and text are mapped to essentially the same internal representation for a shared decoder, although neuron-level analysis finds only partial achievement of that goal (Nakai et al., 24 Jan 2026).

A third meaning concerns robustness under missing, corrupted, or unseen modalities. Edge2Prompt replaces raw intensities with modality-agnostic binary edge maps and uses them to generate logit prompts for SAM-2, enabling cross-modality transfer from CT to MR and vice versa (Hollet et al., 6 Aug 2025). The brain-lesion U-Net with an extra modality-agnostic channel or path uses synthetic “artificial modalities” so that previously unseen MRI sequences can be injected into a dedicated agnostic slot at inference (Addison et al., 11 Sep 2025). MEFormer trains a shared transformer decoder on camera-only, LiDAR-only, and joint branches so that detection remains strong when one sensor is missing (Cha et al., 2024).

These usages suggest that “modality-agnostic input channel” is best treated as an umbrella term for interfaces that decouple downstream reasoning from any fixed raw-input configuration. A plausible implication is that modality agnosticism is not a single architectural recipe but a family of invariance and flexibility mechanisms defined by the failure modes a system is expected to tolerate.

2. Architectural patterns

A useful synthesis is that current designs fall into a small number of recurring architectural patterns.

Pattern	Representative implementation	Key mechanism
Isolated multi-stream fusion	MIGF	Separate encoders, bias-free convolutions, learned scalar weights, gate, fused tensor
Symmetric set-style fusion	MAG-MS, MAGIC, MAGIC++	Mean aggregation or set-like processing over available modalities
Shared backbone or decoder	Any2Seg, MA-ViT, MEFormer	Same encoder or decoder weights reused across modality combinations
Dynamic channel mapping	CLMU-Net	Input-layer inflation with zero-filled absent channels
Dedicated agnostic slot	AGN-MOD-SEG	One extra input channel or shallow path for unseen modalities
Representation-space hub	Concept-centric framework	Modality-specific projections into a shared concept space

Strict modality isolation is one of the clearest patterns. MIGF explicitly replaces early concatenation with three dedicated encoders $f_m$, one per modality, and enforces zero-in zero-out behavior by using bias-free convolutions so that $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$. No concatenation, attention, or cross-modal convolutions occur prior to the MIGF block; interaction happens only through learned scalar weights $\alpha_m$ and a feature-level gate $G$, after which the fused feature $F_{\text{MIGF}}$ is passed to an otherwise standard UNet, nnUNet, or Mamba backbone (Shu et al., 12 Apr 2026). This is an explicitly architecture-agnostic front end.

Symmetric aggregation is another recurrent design. MAG-MS assigns each modality its own encoder but uses a shared decoder and mean-pooled fused feature $$\tilde{\mathbf{F}}_S = \frac{1}{|S|}\sum_{i\in S}\mathrm{enc}_i(x_i)$$ for any subset $2^4-1=15$0, so missing modalities are omitted rather than imputed (He et al., 2023). MAGIC processes modalities in a batch-like fashion, not by concatenating them along channels, and builds a semantic feature $2^4-1=15$1 through a Multi-modal Aggregation Module followed by similarity-based ranking in an Arbitrary-modal Selection Module (Zheng et al., 2024). MAGIC++ replaces MAGIC’s learned benchmark with a parameter-free mean benchmark $2^4-1=15$2 at each scale, then performs hierarchical modality selection before a Multi-Modal Interaction Module refines robust and fragile features (Zheng et al., 2024).

Shared-weight architectures provide a different route to agnosticism. Any2Seg uses a shared SegFormer encoder across modalities and a Modality-agnostic Feature Fusion module that reweights modality-specific features based on inter-modal correlation and then performs fine-grained winner-take-all feature selection to construct $2^4-1=15$3 (Zheng et al., 2024). MA-ViT uses a single ViT backbone with modality-specific tokens and embeddings but a shared transformer path, rather than multi-branch halfway or late fusion (Liu et al., 2023). MEFormer adopts a shared transformer decoder $2^4-1=15$4 for three branches—multi-modal, LiDAR-only, and camera-only—so the same object-query decoding logic is trained under different modality subsets (Cha et al., 2024).

Dynamic and open-ended channel schemes address settings where the modality universe itself can change. CLMU-Net expands the input convolution from $2^4-1=15$5 to $2^4-1=15$6 channels as new modalities appear, copying old weights and zero-initializing new slices, while zero-filled channels represent absent modalities for any current case (Sadegheih et al., 20 Jan 2026). The brain-lesion segmentation model with a modality-agnostic extra path keeps dedicated modality-specific channels for seen modalities but adds one extra channel $2^4-1=15$7 to receive synthetic or unseen sequences, optionally processed by a shallow three-convolution path before concatenation into the main U-Net (Addison et al., 11 Sep 2025).

A related but more abstract pattern replaces fusion by projection into a common semantic hub. In the concept-centric framework, every concept $2^4-1=15$8 is represented by a box $2^4-1=15$9 in $\mathcal{K}\subset\mathbb{R}^d$0, and each modality-specific projection model maps its input to a box $\mathcal{K}\subset\mathbb{R}^d$1 in the same space, so downstream reasoning reduces to entailment probabilities between boxes (Geng et al., 2024). This suggests a different interpretation of modality-agnostic input channels: not as sensor fusion layers, but as modality-specific adapters into an explicitly shared knowledge space.

3. Training and alignment mechanisms

Robust modality-agnostic behavior is rarely obtained from architecture alone. Most systems pair the input channel design with targeted training procedures that force compensation, alignment, or semantic consistency.

Modality dropout is among the most common strategies. MIGF uses ModDrop during training only: with probability $\mathcal{K}\subset\mathbb{R}^d$2 per sample, one of the three modalities is selected uniformly at random and replaced by all zeros, with no explicit missing-modality indicator (Shu et al., 12 Apr 2026). CLMU-Net applies Random Modality Drop to available channels in both current and replay samples, encouraging redundancy-aware features under changing modality combinations (Sadegheih et al., 20 Jan 2026). The unseen-modality MRI model also uses Modality Dropout in conjunction with its agnostic channel, so the network cannot rely on fixed channel–modality correspondences (Addison et al., 11 Sep 2025). These results collectively support the design principle that missing-modality robustness improves when the model repeatedly sees incomplete inputs during optimization.

Distillation and teacher-student alignment are another major class. MAG-MS uses the fused all-modality branch as a teacher and each single-modality branch as a student, combining Dice plus cross-entropy supervision with output-level KL divergence $\mathcal{K}\subset\mathbb{R}^d$3 and feature-level $\mathcal{K}\subset\mathbb{R}^d$4 to make unimodal pathways approximate the multi-modal fused representation (He et al., 2023). Any2Seg introduces Language-guided Semantic Correlation Distillation, where a frozen multi-modal vision-LLM supplies both inter-modal correlation targets and intra-modal class-relation targets, yielding $\mathcal{K}\subset\mathbb{R}^d$5 (Zheng et al., 2024). The concept-centric framework pretrains a modality-agnostic concept space by matching predicted and empirical entailment probabilities between concepts via symmetric KL-based losses, then trains projection models to place inputs into that same probabilistic geometry (Geng et al., 2024).

Several systems rely on modality-neutral intermediate representations rather than raw intensities. Edge2Prompt first performs histogram equalization, bilinear filtering, Canny edge detection, and thresholding to obtain binary edge maps $\mathcal{K}\subset\mathbb{R}^d$6, then learns a U-Net prompt generator whose logits condition frozen SAM-2 (Hollet et al., 6 Aug 2025). AGN-MOD-SEG uses synthetic “artificial modalities” created by region-aware intensity augmentations—Lesion Switch, Inversion, MixUp, Scale, and Shift—so that the agnostic channel learns lesion-versus-brain cues across varied contrasts rather than one fixed MRI sequence (Addison et al., 11 Sep 2025). SeqAug operates one layer higher, perturbing feature sequences $\mathcal{K}\subset\mathbb{R}^d$7 by selecting a subset of feature dimensions and permuting them along the temporal axis, providing a modality-agnostic augmentation for text, audio, and visual sequence features (Georgiou et al., 2023).

Cross-modal pretraining also serves as a powerful supervisory prior. The deepfake detector pretrains an audio-visual speech recognition backbone with CTC loss so that lip–speech correlations, phoneme–viseme timing, and unimodal speech structure are encoded before dual-label authenticity prediction (Yu et al., 2023). MEFormer trains a shared decoder with separate losses for fused, LiDAR-only, and camera-only branches, effectively treating modality availability as a multi-task condition rather than a special case (Cha et al., 2024). A plausible implication is that modality-agnostic input channels are most effective when the training objective explicitly exposes the downstream model to the same combinatorial variability that deployment will produce.

4. Empirical behavior and evaluation protocols

A defining feature of this line of work is that evaluation usually extends beyond the ideal all-modality regime. Performance is commonly reported under arbitrary modality subsets, missing-modality conditions, synthetic artifacts, or modality-transfer scenarios, and these protocols often change the qualitative ranking of methods.

In prostate MRI segmentation, MIGF was evaluated under seven validation scenarios—ideal input, missing T2W, missing HBV, missing ADC, and three artifact cases—and improved the seven-scenario average Ranking Score for nnUNet from $\mathcal{K}\subset\mathbb{R}^d$8 to $\mathcal{K}\subset\mathbb{R}^d$9, for UNet from $\Omega_y$0 to $\Omega_y$1, and for Mamba from $\Omega_y$2 to $\Omega_y$3. Under the ideal scenario, the best model, MIGFNet-nnUNet, achieved $\Omega_y$4 (Shu et al., 12 Apr 2026). Mechanistic analysis showed that the learned modality weights converged to an approximately stable prior—about $\Omega_y$5 T2W, $\Omega_y$6 HBV, and $\Omega_y$7 ADC—rather than to highly sample-dependent routing. This directly challenges the common assumption that robustness must arise from dynamic quality estimation.

Across arbitrary-modality semantic segmentation, gains are often much larger than in full-modality testing. Any2Seg improved the average over all non-empty subsets on DELIVER from $\Omega_y$8 mIoU for CMNeXt to $\Omega_y$9 mIoU, a $f_m$0 mIoU improvement, while also increasing full-modality performance (Zheng et al., 2024). MAGIC reported a mean mIoU of $f_m$1 on DELIVER in the modality-agnostic setting versus $f_m$2 for CMNeXt, a $f_m$3 mIoU gain, while reducing parameters by $f_m$4 under the common multi-modal setting (Zheng et al., 2024). MAGIC++ further raised DELIVER mean MaSS mIoU from $f_m$5 to $f_m$6, with especially large gains for sparse combinations such as RGB-only, event-only, and LiDAR-only (Zheng et al., 2024). These findings indicate that subset-averaged evaluation is not a peripheral stress test but a primary metric for the quality of the input channel design.

Out-of-distribution and unseen-modality experiments expose a different dimension of agnosticism. Edge2Prompt achieved a mean Dice Score of $f_m$7 on OOD tasks, outperforming U-Net baselines by $f_m$8 and other self-prompting methods by $f_m$9; in its cross-modality regime, CT-trained $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$0 MR test produced Dice $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$1 and MR-trained $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$2 CT test produced Dice $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$3 (Hollet et al., 6 Aug 2025). The unseen-modality brain-lesion model showed that feeding an unseen FLAIR sequence through the agnostic path in WMH raised Dice from $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$4 without FLAIR to $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$5 with FLAIR, while in ISLES22 an unseen DWI improved the agnostic-path model from $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$6 to $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$7 (Addison et al., 11 Sep 2025). CLMU-Net reported an average Dice score improvement of $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$8 in heterogeneous-modality continual learning and, in the ILI ablation, improved $$x_m=\mathbf{0}\Rightarrow F_m=f_m(\mathbf{0})=\mathbf{0}$$9 AVG from $\alpha_m$0 to $\alpha_m$1 and $\alpha_m$2 AVG from $\alpha_m$3 to $\alpha_m$4 without prior knowledge of the maximum modality set (Sadegheih et al., 20 Jan 2026).

Robustness under sensor loss or corruption is equally central outside medical imaging. In 3D detection, MEFormer achieved $\alpha_m$5 NDS and $\alpha_m$6 mAP on nuScenes validation, and under missing modalities its single multi-modal model outperformed both camera-only and LiDAR-only comparators: for camera-only inference it reached $\alpha_m$7 NDS and $\alpha_m$8 mAP, and for LiDAR-only $\alpha_m$9 NDS and $G$0 mAP (Cha et al., 2024). In face anti-spoofing, MA-ViT allowed a single model trained jointly on RGB, depth, and IR to operate in fixed-modal and flexible-modal scenarios, and on MmFA its TPR@FPR$G$1 rose from $G$2 for ViT to $G$3, while ACER dropped from $G$4 to $G$5 (Liu et al., 2023). These results suggest that the practical value of modality-agnostic input channels lies as much in graceful degradation as in peak all-sensor accuracy.

5. Domain-specific realizations

The most extensive body of work appears in medical imaging. Prostate MRI, liver CT/MR, infant brain MRI, brain lesion segmentation, and tumor or white-matter lesion segmentation all motivate architectures that tolerate missing acquisitions, artifacts, abbreviated protocols, or cross-cohort heterogeneity (Shu et al., 12 Apr 2026, Hollet et al., 6 Aug 2025, He et al., 2023, Sadegheih et al., 20 Jan 2026, Addison et al., 11 Sep 2025). The design choices vary—from separate bias-free streams and early post-encoding fusion, to mean-pooled shared decoders, to synthetic contrast channels—but the operational goal is consistent: preserve as much modality-specific utility as possible while making the downstream predictor insensitive to which sequences actually arrive.

Autonomous-driving and robotics perception uses the same idea under a different sensor mix. Any2Seg, MAGIC, and MAGIC++ treat RGB, depth, LiDAR, event, or polarization inputs as interchangeable contributors to semantic segmentation rather than as RGB-plus-auxiliary streams (Zheng et al., 2024, Zheng et al., 2024, Zheng et al., 2024). MEFormer does the same at the object level, where shared object queries are decoded under camera-only, LiDAR-only, or fused conditions and then ensembled with proximity-aware cross-attention (Cha et al., 2024). In these systems, modality-agnostic input channels function as anti-fragility mechanisms against night, rain, beam reduction, blur, occlusion, and other sensor-specific failures.

Speech, language, and audio-visual systems expose a complementary set of issues. The audio-visual deepfake framework is fake-modality-agnostic because it predicts real/fake labels independently for audio and video and remains usable when one modality is missing (Yu et al., 2023). SeamlessM4T v2 targets a speech-text shared decoder, but neuron-level analysis finds incomplete modality invariance, with modality-selective structure concentrated in decoder cross-attention key and value projections (Nakai et al., 24 Jan 2026). The concept-centric framework generalizes the input-channel idea further: the modality-agnostic interface is not a fusion tensor but a box-structured concept space $G$6 into which both images and text are projected for image-text matching and visual question answering (Geng et al., 2024).

Long-context encoding and communication broaden the notion beyond fusion. MAELRE uses a unified encoder with progressive token merging and approximate-to-exact attention switching for text, time series, audio, and images, reducing long-context cost on a single device while retaining a mostly modality-neutral internal architecture (Parag et al., 25 Jul 2025). Implicit-JSCC treats modality agnosticism at the communication interface: the same overfitted optimization pipeline directly writes channel symbols and a tiny decoder for each source instance, without modality-specific training datasets or pre-trained models, and the method is validated on both images and audio (Wu et al., 24 Dec 2025). This suggests that modality-agnostic input channels can be understood either as representational front ends for perception or as modality-neutral encoders for transmission and long-range processing.

6. Limitations, tensions, and open problems

One recurring tension is between invariance and discriminability. The unseen-modality brain-lesion model explicitly argues that aggressive modality invariance can weaken modality-specific discriminative power, and therefore keeps dual pathways: modality-specific channels for familiar sequences and an agnostic path for unseen ones (Addison et al., 11 Sep 2025). MAG-MS, by contrast, aligns unimodal features to a fused teacher but still relies on separate encoders for each modality, not a fully shared front end (He et al., 2023). This suggests that successful modality-agnostic input channels often preserve some form of modality-specific specialization rather than abolishing it.

A second limitation is that many systems are agnostic only within a predefined modality universe. Any2Seg, MAGIC, and MA-ViT support arbitrary subsets of modalities seen during training, but they do not claim zero-shot addition of wholly novel sensor types (Zheng et al., 2024, Zheng et al., 2024, Liu et al., 2023). CLMU-Net partially addresses this with input-layer inflation when new modalities appear, and AGN-MOD-SEG addresses unseen MRI sequences with a dedicated agnostic slot, but both require training procedures specifically designed for that flexibility (Sadegheih et al., 20 Jan 2026, Addison et al., 11 Sep 2025). A plausible implication is that “arbitrary subset” and “unseen modality” should be treated as distinct capabilities rather than interchangeable definitions of agnosticism.

A third issue is that claimed agnosticism may be only partial at the mechanistic level. The SeamlessM4T analysis finds that encoder representations become increasingly language-agnostic but that the shared decoder still relies on sharply localized modality-selective neurons in cross-attention, especially for speech-conditioned decoding and non-dominant scripts (Nakai et al., 24 Jan 2026). MIGF’s robustness gains do not arise from adaptive per-sample quality routing, since the learned gate behaves like a stable modality prior rather than a dynamic quality estimator (Shu et al., 12 Apr 2026). These findings caution against equating architectural symmetry with true modality invariance.

Finally, evaluation itself remains an open methodological issue. Many papers now test all non-empty modality combinations, synthetic artifacts, or OOD transfers, but the exact protocols differ substantially (Zheng et al., 2024, Zheng et al., 2024, Hollet et al., 6 Aug 2025). Edge-based pipelines can fail in very low-contrast or noisy regions, replay-based continual learners raise privacy and storage concerns, and dynamic-channel models incur computational overhead as modality sets accumulate (Hollet et al., 6 Aug 2025, Sadegheih et al., 20 Jan 2026). The broader trend nonetheless points to a common design principle: robust multi-modal systems increasingly benefit from structuring the input channel so that missing or corrupted modalities are contained early, semantically aligned representations are learned explicitly, and downstream reasoning is reused across modality configurations rather than retrained for each one.