SEMI: Sample-Efficient Modality Integration

• The paper introduces a hypernetwork-based adaptation that integrates new modalities into a frozen LLM with very little paired data.
• It employs a two-phase approach where a shared MLP projector is initially trained on high-resource datasets and later refined using LoRA-based low-rank corrections.
• Empirical evaluations show SEMI requires up to 64× less paired data than traditional projector training, demonstrating significant sample efficiency.

Searching arXiv for the primary SEMI paper and closely related modality-integration work. Primary search: (İnce et al., 4 Sep 2025) and related multimodal modality-integration papers. Sample-Efficient Modality Integration (SEMI) denotes a method for integrating a new modality into a pre-existing LLM with very little paired data, rather than retraining a multimodal foundation model from scratch. In the formulation introduced in "Sample-efficient Integration of New Modalities into LLMs," the central problem is that the space of possible modalities is large and evolving over time, while low-resource modalities often lack the paired data needed for conventional projector training. SEMI addresses this by learning a hypernetwork that adapts a shared projector between frozen modality-specific encoders and a frozen LLM, conditioning the adaptation on only a few examples from the new modality at inference time (İnce et al., 4 Sep 2025).

1. Problem setting and formal objective

SEMI assumes a set of $M$ training modalities, each with a fixed, frozen encoder

$\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$

a frozen LLM decoder whose token-embedding space is $\mathbb{R}^{h_d}$, and a learned projector

$$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$

Given a new low-resource modality encoder $\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$ and only $k \ll |X_{new}|$ paired samples $\{(x_i,y_i)\}_{i=1}^k$, the goal is to produce an adapted projector $\mathrm{proj}_{\psi'}$ that enables the frozen LLM to generate the correct text $y$ from $x$ with minimal additional data (İnce et al., 4 Sep 2025).

The baseline alternative is projector fine-tuning from scratch:

$\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$0

SEMI replaces this with a hypernetwork $\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$1 that outputs an adaptation $\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$2 at inference time and forms

$\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$3

where $\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$4 has LoRA form of rank $\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$5. The training loss for the hypernetwork is cross-entropy over high-resource modalities, with the hypernetwork conditioned on an instruction embedding and a small support set of modality-text pairs. In the formal exposition, the high-resource training distribution includes modality encodings transformed by a random orthogonal matrix $\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$6, and the hypernetwork input is

$\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$7

This formulation makes the key distinction between SEMI and conventional multimodal adaptation explicit. The method does not retrain the LLM, and it does not require dense paired data for every new modality. Instead, it learns how to infer a modality-conditioned projector update from a small in-context support set.

2. Hypernetwork and shared-projector architecture

SEMI is organized in two phases. In Phase I, the projector $\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$8 is trained as a 2-layer MLP with first layer $\mathrm{enc}_m: X_m \to \mathbb{R}^{h_e},$9 and bias $\mathbb{R}^{h_d}$0, followed by GELU, then second layer $\mathbb{R}^{h_d}$1 and bias $\mathbb{R}^{h_d}$2. The projector parameters $\mathbb{R}^{h_d}$3 are trained on large paired datasets, specifically COCO for images, AudioCaps for audio, and OpenVid for video, using cross-entropy (İnce et al., 4 Sep 2025).

In Phase II, the hypernetwork $\mathbb{R}^{h_d}$4 receives an interleaved context composed of one frozen text-instruction embedding $\mathbb{R}^{h_d}$5 and $\mathbb{R}^{h_d}$6 pairs of modality encoding plus ground-truth text,

$\mathbb{R}^{h_d}$7

For generating LoRA adapters for only $\mathbb{R}^{h_d}$8, the hypernetwork prepends $\mathbb{R}^{h_d}$9 special adapter-tokens to the sequence, with $$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$0 if both $$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$1 and $$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$2 are generated. It then adds sinusoidal positional embeddings, applies one self-attention layer with a single head, and sends each special token through its own linear head. These heads produce

• $$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$3,
• $$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$4, so that $$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$5 is rank-$$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$6,
• and $$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$7 if biases are adapted.

The architectural premise is that a single shared projector can be reused across modalities if the correct low-rank correction is generated from a small demonstration set. A plausible implication is that SEMI converts modality integration into a conditional adapter-generation problem rather than a full re-estimation problem.

3. Isometric augmentation, inference-time adaptation, and optimization

A central component of SEMI is modality multiplication through random isometries. For each batch, the method samples

$$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$8

and feeds both $$\mathrm{proj}_\psi: \mathbb{R}^{h_e} \to \mathbb{R}^{h_d}.$$9 to the hypernetwork and $\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$0 to the projector. Because orthogonal $\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$1 preserves inner products and norms, the transformation leaves local structure intact while altering global orientation (İnce et al., 4 Sep 2025).

At inference time, SEMI freezes $\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$2, $\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$3, and all encoders. Given $\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$4 pairs $\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$5 and one instruction $\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$6, it partitions the pairs into batches of size at most $\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$7, forms an interleaved context for each batch, computes a batchwise adapter $\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$8, averages adapters across batches,

$\mathrm{enc}_{new}: X_{new} \to \mathbb{R}^{h_e}$9

and merges the result into the projector,

$k \ll |X_{new}|$0

The adapted projector may then be optionally fine-tuned on the same $k \ll |X_{new}|$1 pairs with standard cross-entropy and AdamW for a few epochs.

The reported optimization protocol is specific. Phase I uses COCO (590K), AudioCaps (45K), and OpenVid (59K), with AdamW, $k \ll |X_{new}|$2, $k \ll |X_{new}|$3, weight decay $k \ll |X_{new}|$4, learning rate $k \ll |X_{new}|$5, $k \ll |X_{new}|$6k warmup steps, and cosine decay over $k \ll |X_{new}|$7 epochs, approximately $k \ll |X_{new}|$8K steps. Phase II uses ShareGPT4V (~35K), Clotho-Detail (3.9K), and ShareGPT4Video (~39K), with the same optimizer settings, learning rate $k \ll |X_{new}|$9, $\{(x_i,y_i)\}_{i=1}^k$0k warmup, cosine decay over $\{(x_i,y_i)\}_{i=1}^k$1 epochs, approximately $\{(x_i,y_i)\}_{i=1}^k$2K steps, and batch size $\{(x_i,y_i)\}_{i=1}^k$3 real modalities $\{(x_i,y_i)\}_{i=1}^k$4 subset-samples with gradient accumulation to achieve effective batch $\{(x_i,y_i)\}_{i=1}^k$5. The generated adapter rank is $\{(x_i,y_i)\}_{i=1}^k$6 and $\{(x_i,y_i)\}_{i=1}^k$7.

The stated regularizers are likewise architectural rather than heuristic: LoRA constrains the adaptation to low rank, while isometric augmentation and text grounding are intended to prevent overfitting to spurious encoder idiosyncrasies.

4. Empirical sample-efficiency profile

SEMI is evaluated on five held-out modalities: satellite images on SydneyCaptions with three encoders (ViT-B/32, ViT-L/14, RN-50), astronomical images on CAPDELS with ConvNeXt Nano, Tiny, and Base, IMU sensor data on SensorCaps, molecules on ChEBI-20, and an unseen audio encoder on SoundBible (İnce et al., 4 Sep 2025). The abstract further states that these experiments involve satellite images, astronomical images, inertial measurements, and molecules, and that SEMI operates with encoders of arbitrary embedding dimensionality.

The key empirical claim is that SEMI produces a substantial reduction in required paired data for new-modality integration. To reach the same accuracy as 32-shot SEMI, training the projector from scratch often requires at least 2048 samples, described as $\{(x_i,y_i)\}_{i=1}^k$8 more data. Relative to the best baseline, full fine-tuning of the shared projector, SEMI is reported to save about $\{(x_i,y_i)\}_{i=1}^k$9 samples in low-data regimes. The reported trend is that, when plotted as CIDEr or BLEU versus $\mathrm{proj}_{\psi'}$0, SEMI shows steeper gains in the $\mathrm{proj}_{\psi'}$1 regime, whereas baselines plateau or even degrade at very low $\mathrm{proj}_{\psi'}$2.

Held-out modality / encoder	Samples needed for projector-scratch to match SEMI(32)
Satellite (ViT-B/32)	$\mathrm{proj}_{\psi'}$3
Astronomical (ConvNeXt-Tiny)	$\mathrm{proj}_{\psi'}$4
IMU (LIMU-BERT)	$\mathrm{proj}_{\psi'}$5
Molecule (MolCA)	$\mathrm{proj}_{\psi'}$6
Audio (BLAT)	$\mathrm{proj}_{\psi'}$7

These results support the specific sense in which the method is “sample-efficient”: the gain is not merely improved accuracy at fixed data volume, but a reduction in the number of paired examples required to obtain the same accuracy as conventional projector training.

5. Relationship to adjacent multimodal research

SEMI, in the sense of (İnce et al., 4 Sep 2025), is distinct from several neighboring multimodal research programs that also target scarcity. In semi-supervised multimodal brain tumor segmentation, Chung et al. introduce a Modality-specific Enhancing Module (MEM) and a Complementary Information Fusion (CIF) module to strengthen modality-specific semantic cues and adaptively exchange complementary knowledge between MRI modalities under 1%, 5%, and 10% labeled data; the problem there is not the post hoc addition of a new modality to a frozen LLM, but segmentation under extreme label scarcity and cross-modality discrepancy (Chung et al., 10 Dec 2025).

A different neighboring line is robust semi-supervised multimodal medical image segmentation via Cross Modality Collaboration. That framework uses modality-specific 3D encoders, a Modality-Independent Awareness module, a fusion layer, a channel-wise semantic consistency loss, and a contrastive anatomical-similar consistency loss to handle scarce labeled data and misaligned modalities across cardiac, abdominal multi-organ, and thyroid-associated orbitopathy segmentations. The emphasis is robustness to misalignment and exploitation of unlabeled data with 10% and 20% labels, rather than inference-time generation of an adapter for a previously unseen modality (Zhou et al., 2024).

Another related problem is missing-modality robustness in semi-supervised multi-modal semantic segmentation. There, Linear Fusion blends transformer tokens across RGB and depth branches by

$\mathrm{proj}_{\psi'}$8

and M3L uses a multi-modal teacher with masked-modality learning to improve both label efficiency and robustness to missing modalities. The reported setting concerns RGB-D segmentation with unlabeled data and test-time modality absence, not few-shot adaptation of a shared projector for a novel modality (Maheshwari et al., 2023).

Cohort-based Active Modality Acquisition addresses yet another bottleneck: deciding which test samples should receive additional modalities under a budget. It formalizes acquisition decisions $\mathrm{proj}_{\psi'}$9 under $y$0, and proposes imputation-based acquisition functions such as expected KL-divergence. This is a cohort-level selection problem for costly modalities rather than a mechanism for integrating a new modality into an LLM (Rheude et al., 22 May 2025).

Taken together, these works suggest a broader landscape of sample-efficient multimodal learning. One branch, exemplified by (İnce et al., 4 Sep 2025), studies few-shot integration of genuinely new modalities into a frozen generative backbone. Other branches study label scarcity, misalignment, missing modalities, or acquisition budgets when the modality set is already known.

6. Limitations, misconceptions, and open directions

SEMI carries several explicit constraints. It assumes that an encoder already exists for the new modality, meaning that modality-only data was plentiful enough to train $y$1 in advance (İnce et al., 4 Sep 2025). A common misconception is therefore that SEMI removes the need for prior modality modeling altogether; the formulation does not do so. It reduces the paired-data burden for integrating a modality into the LLM, but it does not eliminate the need for a modality-specific encoder.

The current method also adapts only one modality at a time and only generates text. Extending the framework to simultaneous multi-modal inputs such as text+image+audio is identified as an open problem. Another stated limitation is that adapting deeper projector layers proved brittle; in the reported experiments, only the first MLP layer is adapted via LoRA. On molecules, the authors note occasional chemistry-specific hallucinations.

The future directions named in the source are concrete. They include extending adapters to cross-attention or Q-Former layers in the LLM, learning to integrate multiple new modalities jointly in order to exploit synergy, and reducing inference cost of the hypernetwork through approaches such as distillation (İnce et al., 4 Sep 2025). This suggests that the core idea of SEMI is not tied to a single projector architecture, but to the more general principle of conditioning lightweight adaptation on a small support set of modality-text examples.