IMC: Inherent-Enhanced Multi-modal Calibration

• IMC is a multi-modal calibration framework that detects and corrects noise and bias in vision and language models without additional retraining.
• It leverages model-internal features with methods like KMeans clustering and PCA-based corrections to perform targeted denoising.
• IMC incorporates iterative text refinement and decoder fine-tuning for scientific surrogate models, significantly improving prediction accuracy and bias suppression.

Inherent-enhanced Multi-modal Calibration (IMC) is a framework for robust multi-modal learning that applies targeted calibration to mitigate input noise and cross-modal bias using the latent capabilities of deep learning models. IMC operates across both vision and language modalities in medical multi-modal LLMs (MLLMs), as well as in bias suppression tasks for scientific predictive modeling. In both usage contexts, IMC exploits model-internal representations to perceive perturbations, then performs calibration using either training-free latent steering or decoder fine-tuning, yielding improved robustness without the need for costly retraining or modality-specific external calibrators (Xu et al., 26 Dec 2025, Kustowski et al., 2021).

1. Core Principles of IMC

IMC is founded on the “perceive-and-calibrate” paradigm: first, the system detects the existence and type of perturbation in each modality; second, it applies a corrective transformation, either directly in the latent feature space (vision) or through multi-agent iterative text editing (language). This principle enables targeted denoising for each modality and is realized without additional model training by leveraging inherent structure and self-assessment capabilities discovered during large-scale pretraining (Xu et al., 26 Dec 2025).

In scientific surrogate modeling, IMC is instantiated as a two-stage process: pretraining a simulation-based multi-modal autoencoder to capture cross-modal correlations, followed by transfer learning—where only the decoder’s innermost layers are retrained to absorb experimental bias—thereby correcting both scalar and image output predictions based on limited real data (Kustowski et al., 2021).

2. Perturbation-aware Denoising Calibration (PDC) for Vision

The visual pipeline of IMC centers on prototype-guided denoising using model-internal feature clusters:

• Noise Perception: For image $v$, layer-wise features $f_l(v) \in \mathbb{R}^d$ are collected across modalities ($$m \in \{\mathrm{CT}, \mathrm{MRI}, \mathrm{X-ray}\}$$) and noise states ($\delta \in \Delta$), clustering embeddings into $K$ prototypes per (modality, noise, layer) triplet: $c^{l,k}_{(\delta, m)} = KMeans_k(\{f_l(v)\})$.
• Noise Identification: At inference, the model computes $f_l(\hat{v})$ and selects the nearest prototype $(\hat{\delta}^l, \hat{m}^l, \hat{k}^l)$ by minimizing $\|\hat{f}_l - c^{l,k}_{(\delta, m)}\|_2$. Majority voting across $L$ layers yields predicted noise and modality.
• Calibration Step: When $(\hat{\delta}, \hat{m}) \neq (\delta_0, m)$ (corrupted input), a calibration vector $p^{l,k}_{(\hat{\delta},\hat{m})}$ is computed as the principal direction of clean–corrupt difference vectors within the cluster using PCA:

$$\phi_{(\hat{\delta}, \hat{m}), j}^{l, k} = f^{l, k}_{(\delta_0, \hat{m}), j} - f^{l, k}_{(\hat{\delta}, \hat{m}), j}$$

$$p_{(\hat{\delta}, \hat{m})}^{l, k} = \mathrm{PCA}(\{\phi_{(\hat{\delta}, \hat{m}), j}^{l, k}\})$$

The correction: $$\tilde{f}_l = \hat{f}_l + \alpha p^{l,k}_{(\hat{\delta}, \hat{m})}$$, with $\alpha \approx 0.05$.

• Training-Free Deployment: No loss terms or end-to-end fine-tuning are introduced; a small set of labeled exemplars suffices to construct prototype and calibration pools (Xu et al., 26 Dec 2025).

3. Self-instantiated Multi-agent System (SMS) for Text Denoising

IMC’s text branch implements a cooperative hierarchical editing structure modeled on iterative human proofreading:

• Micro Loop: An agent classifies and denoises character- and sentence-level noise in $q_{\mathrm{in}}$, while a residual noise checker verifies and iterates until all noise is removed. $k$ such micro loops yield candidate denoised texts $\{q^{(1)}, ..., q^{(k)}\}$.
• Macro Loop: An agent selects the optimal candidate, using both linguistic and visual context; a validator confirms grammatical and logical correctness, feeding validated outputs into successive editing rounds (typically $n = 2$).
• Joint Vision-Language Calibration: The final output is assembled from denoised text and calibrated image features, enabling robust downstream reasoning in MLLMs.
• Pseudocode Implementation:

Function SMS_Denoise(text q, image_feature f_img, k, n): q_curr = q for round = 1…n: candidates = [] for i in 1…k: x = q_curr repeat: x = Agent_A_ClassifyAndDenoise(x) until Agent_B_CheckResidualNoise(x) = false candidates.append(x) q_sel = Agent_C_SelectOptimal(candidates, f_img) if Agent_D_Validate(q_sel): q_curr = q_sel reduce sampling temperature return q_curr

(Xu et al., 26 Dec 2025)

4. IMC for Multi-modal Simulation Calibration and Bias Suppression

IMC extends beyond medical MLLMs to scientific surrogate modeling, where it suppresses simulation bias via cross-modal calibration:

• Surrogate Model Construction: Pretraining a multi-modal autoencoder $(E, D)$ on simulation outputs $$(y_{\mathrm{sca}}, y_{\mathrm{img}}) \to z \to (\hat{y}_{\mathrm{sca}}, \hat{y}_{\mathrm{img}})$$, coupled with a forward-inverse mapping $(F: x \to z, I: z\to x)$.
• Transfer Learning: On sparse experimental data, only the decoder’s innermost parameters $\theta$ are retrained using a weighted loss over scalar and image modalities, subject to regularization:

$$L_{TL}(\theta) = \sum_{j} ||y_j^{\mathrm{img}} - D_\theta(F(x_j))_{\mathrm{img}}||_2^2 + \gamma_{\mathrm{sca}} \sum_{j} \left\| \frac{y_{j, \mathrm{sca}} - D_\theta(F(x_j))_{\mathrm{sca}}}{\sigma_{j, \mathrm{sca}}} \right\|_2^2 + \lambda ||\theta||_2^2$$

• Bias Correction: Systematic bias $$\delta(x) = y^{\mathrm{exp}}(x) - \hat{y}^{\mathrm{sim}}(x)$$ is absorbed via decoder fine-tuning; the latent $z$ remains fixed, preserving cross-modal correlations.
• Network Architecture: Scalar and image encoders, shared latent vectors, forward/inverse networks, and modality-specific decoders, typically trained on $>90,000$ simulations then adapted with $<10$ experiments (Kustowski et al., 2021).

5. Benchmarks, Results, and Experimental Validation

IMC is evaluated on robustness to multi-modal noise and bias via dedicated benchmarks:

Task/Metric	Performance Impact (Baseline)	IMC Recovery / Gain
MRI aliasing (ROUGE)	-27.6 pts	+13.3 pts recovered
X-ray movement (Accuracy)	-8.3 pts	+6.9 pts recovered
Multi-modal bias (χ²/N, sim)	>1 (uncalibrated)	<1 for most scalars

• Datasets include SLAKE and OmniMed (medical MLLM robustness), as well as inertial confinement fusion (ICF) experimental splits for bias-suppression validation.
• Competing baselines include Prompt-only, self-denoising prompting, single-direction latent steering; IMC outperforms each by 4–10 points in ROUGE or accuracy across ablations on prototype number ($K$ best at 8), SMS loop rounds ($n=2$ sufficient).
• Cross-validation on real experiments demonstrates IMC’s reduction of systematic bias in scalar and image predictions; synthetic validation confirms correction of global shifts, even under altered physics (Xu et al., 26 Dec 2025, Kustowski et al., 2021).

6. Discussion, Assumptions, and Limitations

IMC’s success relies on leveraging pre-existing model structure:

• Perceptual Latent Spaces: By organizing representations into modality- and noise-specific clusters, IMC enables targeted correction, avoiding the inefficiency and safety risks of end-to-end retraining.
• Multi-agent Iteration: SMS’s layered agent hierarchy prevents context collapse and mimics human editing workflows.
• Transfer Learning for Surrogates: Decoder-only calibration maintains latent encodings, preserving critical cross-modal correlations while correcting bias in final outputs.

Key limitations include dependency on availability of labeled noise exemplars to compute prototypes/PCA vectors for PDC, high inference cost for layer-wise computations and repeated denoising loops, and restricted handling of corruptions outside the precomputed pool. In scientific surrogate calibration, IMC may be insufficient if biases are highly input-dependent or latent space alignment between simulation and experiment is poor; full fine-tuning or advanced regularization may be required given larger experimental datasets.

A plausible implication is that IMC’s training-free calibration paradigm can generalize to diverse domains involving multi-modal, biased, and noisy data, provided model-internal representations encode stable cross-modal relationships and the correction can be effected at the output mapping level.