---
title: 'StaRFM: Robust Foundation Models'
url: https://www.emergentmind.com/topics/starfm
type: topic
---

# StaRFM: Robust Foundation Models

Searching arXiv for StaRFM and the cited paper to ground the article in the latest preprint record.
StaRFM denotes **Stable and Robust Foundation Models**, a **unified training-time framework** for improving the robustness of foundation models to **distribution shift** and their calibration under uncertainty across **vision-language classification** and **medical image segmentation**. It is introduced in “Calibrated and Robust Foundation Models for Vision-Language and Medical Image Tasks Under Distribution Shift” [2507.09222]. The term is potentially ambiguous in arXiv search contexts: the paper “The Star Formation in Radio Survey: 3 - 33 GHz Imaging of Nearby Galaxy Nuclei and Extranuclear Star-forming Regions” [2004.10230] belongs to the **Star Formation in Radio Survey (SFRS)** and explicitly states that it does **not** use the abbreviation “StaRFM.” In current arXiv usage reflected by the supplied sources, “StaRFM” properly refers to the foundation-model framework rather than the radio-survey paper.

## 1. Conceptual scope and naming

StaRFM is presented as a **single plug-and-play framework** intended for two settings that are usually treated separately: **CLIP-like vision-language classification** and **SAM/MedSAM-like medical segmentation**. Its stated motivation is that the same two failure modes recur in both domains. The first is **covariate shift**, described as a mismatch between training and test distributions. The second is **confidence misalignment**, meaning that the model can be highly confident even when wrong. The framework addresses both by adding two regularizers directly to the task loss: a **Fisher Information Penalty (FIP)** and a **Confidence Misalignment Penalty (CMP)** [2507.09222].

The paper positions StaRFM as a generalization of earlier CalShift-style ideas from 2D classification to a broader setting that includes **3D medical segmentation**. This extension is central to the framework’s identity. In the vision-language case, the emphasis is on stabilizing CLIP adaptation under few-shot and domain-generalization conditions. In medical imaging, the emphasis is on multi-center variability, scanner and protocol differences, and dense voxel-level uncertainty.

A plausible implication is that the term “unified” is not merely rhetorical. In the paper’s formulation, the same pair of regularization principles is reused across different modalities and output structures, with the task-specific differences handled through how Fisher sensitivity and confidence misalignment are instantiated.

## 2. Targeted failure modes: distribution shift and confidence misalignment

The distribution-shift problem is formalized in the paper as
$$
P_{\text{tr}}(y,x) \neq P_{\text{ts}}(y,x),
$$
and, for the covariate-shift case,
$$
P_{\text{src}}(x) \neq P_{\text{tgt}}(x), \qquad P_{\text{src}}(y\mid x)=P_{\text{tgt}}(y\mid x).
$$
Within this framing, StaRFM is designed primarily for settings where the conditional labeling mechanism is preserved while the input distribution changes [2507.09222].

For **CLIP and related vision-language models**, the paper identifies shift in the learned image-text embedding space when web-scale pretraining data do not match the downstream domain. Few-shot adaptation is singled out as especially vulnerable because prompt tuning or lightweight adaptation can overfit limited support examples and distort pretrained features. For **SAM and medical segmentation models**, the corresponding issue is distribution mismatch between natural-image pretraining and medical-imaging deployment, where the paper highlights intensity statistics, texture, anatomical structure, scanner protocol, resolution, and site-specific acquisition effects.

The second failure mode is **confidence misalignment** or **miscalibration**. In classification, this means softmax confidence can substantially exceed empirical correctness under shift. In segmentation, the paper treats the problem as spatially localized: unreliable confidence is particularly important at lesion boundaries and in ambiguous regions, where overlap metrics alone may hide clinically relevant uncertainty.

The framework’s practical rationale follows from this dual diagnosis. The paper argues that many existing methods either seek robustness without calibration or perform post hoc calibration without addressing underlying representation instability, and that such methods are often domain-specific. StaRFM is therefore intended as a training-time mechanism that jointly regularizes representation sensitivity and predictive confidence.

## 3. Methodological structure: FIP and CMP

StaRFM retains the base task loss and augments it with **FIP** and **CMP**. For vision-language tasks, the paper writes the objective over image \(x\), text prompt \(t\), and label \(y\) as
$$
\min_\theta \mathbb{E}_{(x,t,y)\sim\mathcal{D}_{\text{tgt}}}\left[\ell(y, f_\theta(x,t))\right].
$$
For medical segmentation, with volumetric input \(x \in \mathbb{R}^{H\times W\times D}\) and binary mask \(\mathbf{M}\),
$$
\min_\theta \mathbb{E}_{(x,\mathbf{M}^*)\sim\mathcal{D}_{\text{tgt}}}\left[\mathcal{L}_{\text{seg}}(f_\theta(x), \mathbf{M}^*)\right].
$$
The framework then adds a Fisher-based robustness term and a confidence-misalignment term to these baselines [2507.09222].

The **Fisher Information Penalty** is introduced through
$$
I(\theta) = -\mathbb{E}_{x\sim\mathcal{D}_{\text{tgt}}}\left[\nabla_\theta^2 \log p(x;\theta)\right],
$$
while elsewhere the paper also uses the score-covariance form
$$
I(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{\text{src}}} \left[ \nabla_\theta \log p(x;\theta)\nabla_\theta \log p(x;\theta)^\top \right].
$$
The intended interpretation is that Fisher information measures parameter sensitivity to changes in the data distribution, so penalizing it should discourage unstable, shift-sensitive parameter regions.

For CLIP, FIP is defined over both image and text pathways:
$$
I_{\text{vision}}(\theta)=I(\theta_{\text{img}})+I(\theta_{\text{text}}).
$$
The underlying CLIP loss is given as
$$
\mathcal{L}_{\text{CLIP}}=\frac{1}{2}\left(\mathcal{L}_{\text{img}}+\mathcal{L}_{\text{txt}}\right),
$$
with the standard contrastive terms
$$
\mathcal{L}_{\text{txt}}= -\frac{1}{N}\sum_{i=1}^N \log \frac{ \exp(\mathrm{sim}(\mathbf{t}_i,\mathbf{i}_i)/\tau) }{ \sum_{j=1}^N \exp(\mathrm{sim}(\mathbf{t}_i,\mathbf{i}_j)/\tau) },
$$
and
$$
\mathcal{L}_{\text{img}}= -\frac{1}{N}\sum_{i=1}^N \log \frac{ \exp(\mathrm{sim}(\mathbf{i}_i,\mathbf{t}_i)/\tau) }{ \sum_{j=1}^N \exp(\mathrm{sim}(\mathbf{i}_i,\mathbf{t}_j)/\tau) }.
$$
StaRFM’s vision objective is then
$$
\mathcal{L}_{\text{vision}} = \mathcal{L}_{\text{CLIP}} +\lambda_1 I_{\text{vision}}(\theta) +\lambda_2 \mathrm{CMP}_{\text{vision}}.
$$

For 3D medical data, the distinctive extension is **patch-wise FIP**:
$$
I_{\text{3D}}(\theta) = \frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}} I(\theta_p), \qquad \mathcal{P}=\{\text{16}\times\text{16}\times\text{16 patches}\}.
$$
The stated idea is that large, spatially structured volumes require local regularization over volumetric patches. The manuscript also reports **\(128\times128\times128\) patches** in the implementation section, creating an explicit inconsistency. The most conservative reading is that the method is patch-wise in conception, while the equation and implementation summary do not perfectly align.

The **Confidence Misalignment Penalty** is defined generally as
$$
\mathrm{CMP} = \sum_{y' \neq y} \frac{P(y'|x)} {\sum_{y_j \neq y'} P(y_j|x)} \cdot \mathbf{1}\!\left[P(y'|x) > P(y|x)\right].
$$
It is intended to penalize cases where an incorrect class is assigned higher probability than the true class. For image-level classification,
$$
\mathrm{CMP}_{\text{vision}} = \mathrm{CMP}\big(P(y\mid x_{\text{img}}, t)\big).
$$
For voxel-level segmentation, the paper reformulates CMP as
$$
\mathrm{CMP}_{\text{3D}} = \frac{1}{|\mathcal{V}|} \sum_{v\in\mathcal{V}} \frac{P(y_v' \mid x_v)}{1 - P(y_v' \mid x_v)}, \qquad \mathcal{V}=\text{voxels}.
$$
This is the mechanism used to calibrate uncertain volumetric predictions.

In the medical setting, the base loss is
$$
\mathcal{L}_{\text{Dice}} = 1-\frac{2|\mathbf{M}\cap \mathbf{M}^*|}{|\mathbf{M}|+|\mathbf{M}^*|},
$$
$$
\mathcal{L}_{\text{BCE}} = -\frac{1}{|\mathcal{V}|} \sum_{v\in\mathcal{V}} \left[ y_v^* \log p_v + (1-y_v^*)\log(1-p_v) \right],
$$
$$
\mathcal{L}_{\text{SAM}} = \mathcal{L}_{\text{Dice}}+\mathcal{L}_{\text{BCE}},
$$
and the StaRFM segmentation objective becomes
$$
\mathcal{L}_{\text{medSAM}} = \mathcal{L}_{\text{SAM}} +\lambda_1 I_{\text{3D}}(\theta) +\lambda_2 \mathrm{CMP}_{\text{3D}}.
$$

The paper describes these additions as requiring **minimal architectural changes**. In vision-language experiments, the regularizers are integrated into CLIP adaptation; in medical imaging, they are integrated into SAM/MedSAM-style fine-tuning, described in the text and figures as applied to the **text encoder** for CLIP and the **SAM prompt encoder** for medical segmentation. The manuscript does not provide a more granular layer-by-layer integration specification.

## 4. Theoretical framing and stated guarantees

The main formal result included in the manuscript is **Proposition 1**, titled *Fisher Regularization for Domain Adaptation* [2507.09222]. Under covariate shift,
$$
P_{\text{src}}(x)\neq P_{\text{tgt}}(x), \qquad P_{\text{src}}(y|x)=P_{\text{tgt}}(y|x),
$$
and for a model \(f_\theta\) with Fisher information matrix
$$
I(\theta) = \mathbb{E}_{x\sim\mathcal{D}_{\text{src}}} \left[ \nabla_\theta \log p(x;\theta)\nabla_\theta \log p(x;\theta)^\top \right],
$$
the target-risk bound is stated as
$$
\mathcal{R}_{\text{tgt}}(\theta) \le \mathcal{R}_{\text{src}}(\theta) + \frac{1}{2}\sqrt{\mathrm{tr}\!\left(I(\theta)\Sigma_{\text{shift}}\right)} + \mathcal{O}(n^{-1/2}),
$$
where
$$
\Sigma_{\text{shift}} = \mathbb{E}_{x\sim\mathcal{D}_{\text{tgt}}} \left[ (\nabla_\theta\log p(x;\theta)-\mu_\theta)(\cdot)^\top \right],
$$
and
$$
\mu_\theta=\mathbb{E}[\nabla_\theta \log p(x;\theta)].
$$
The notation for \(\Sigma_{\text{shift}}\) is explicitly incomplete in the manuscript, so the covariance definition is not fully typeset.

The proof sketch proceeds by expressing target risk through importance weighting,
$$
\mathcal{R}_{\text{tgt}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}_{\text{src}}} \left[ \frac{P_{\text{tgt}}(x)}{P_{\text{src}}(x)} \ell(y,f_\theta(x)) \right],
$$
then approximating it as source risk plus a covariance term,
$$
\mathcal{R}_{\text{tgt}}(\theta) \approx \mathcal{R}_{\text{src}}(\theta) + \mathrm{Cov}\!\left( \frac{P_{\text{tgt}}(x)}{P_{\text{src}}(x)}, \ell(y,f_\theta(x)) \right),
$$
which is bounded via Cauchy–Schwarz and linked to KL divergence and Fisher-structured loss sensitivity:
$$
\mathrm{Cov}(\cdot) \le \sqrt{ \mathbb{V}\!\left[\frac{P_{\text{tgt}}(x)}{P_{\text{src}}(x)}\right] \mathbb{V}[\ell(y,f_\theta(x))] } \le \frac{1}{2}\sqrt{ D_{\mathrm{KL}}(\mathcal{D}_{\text{tgt}}\|\mathcal{D}_{\text{src}}) \cdot (I(\theta)\Sigma_\ell) }.
$$
The regularized objective is then motivated as
$$
\min_\theta \mathcal{R}_{\text{src}}(\theta)+\lambda \|I(\theta)\|_*,
$$
with \(\|\cdot\|_*\) the nuclear norm.

The paper repeatedly states that it derives **PAC-Bayes bounds** and that CMP minimizes calibration error through **Brier score optimization**. However, the supplied manuscript material includes only the Fisher-based proposition above; it does not include a formal PAC-Bayes theorem statement, an explicit Brier score equation, or a formal theorem proving CMP/Brier equivalence or an upper bound on ECE. The most precise summary is therefore that the paper **claims** PAC-Bayes-style and Brier-score-based theoretical grounding, while the included formal derivation in the supplied text is specifically the Fisher-based domain-adaptation bound.

The limitations of the theory are also stated clearly. It does **not** cover arbitrary distribution shift beyond covariate shift, assumes finite divergence or bounded shift, does **not** guarantee improved performance under extreme domain gaps, and does **not** address label shift \(P_{\text{src}}(y)\neq P_{\text{tgt}}(y)\). This suggests that the framework’s theoretical support is strongest for moderate covariate-shift settings rather than unrestricted domain transfer.

## 5. Architectures, benchmarks, and empirical performance

StaRFM is evaluated in two experimental regimes [2507.09222].

For **vision-language classification**, the backbone is **CLIP ViT-B/16**. The paper evaluates on **19 vision datasets**. The **11 standard vision benchmarks** are ImageNet, Caltech101, OxfordPets, Flowers102, Food101, SUN397, DTD, EuroSAT, UCF101, FGVCAircraft, and StanfordCars. The **8 domain adaptation or distribution-shift benchmarks** are PACS, Office-Home, VLCS, DomainNet, ImageNet-V2, ImageNet-R, ImageNet-A, and ImageNet-S. Baselines include zero-shot CLIP, CoOp, CalShift, and comparisons involving CoCoOp, MaPLe, KgCoOp, and ProDA. The reported vision training details are Adam, learning rate \(1\times 10^{-4}\), batch size 32, and weights \(\lambda_1=0.4\), \(\lambda_2=0.5\), although the hyperparameter-tuning summary later states a final configuration of \(\lambda_1=0.4\), \(\lambda_2=0.4\), which is an acknowledged inconsistency.

For **medical segmentation**, the backbone is **SAM ViT-H**. The paper uses three brain MRI datasets: **BraTS 2023**, described as 1,250 multi-parametric MRI scans from 60 centers for tumor segmentation; **ATLAS v2.0**, described as 1,128 T1-weighted scans from 22 sites for chronic stroke lesion segmentation; and **IBSR18**, described as legacy low-resolution 1.5T T1 scans used for robustness to low-resolution and distribution differences. The medical benchmarks emphasize multi-center scanner and protocol shift, cross-site variation, and low-resolution cross-dataset robustness.

The reported metrics for vision are **accuracy** and **ECE**, with
$$
\mathrm{ECE} = \sum_{b=1}^{B}\frac{|S_b|}{N} \left| \mathrm{acc}(S_b)-\mathrm{conf}(S_b) \right|, \qquad B=10.
$$
For medical segmentation, the reported metrics are **Dice Similarity Coefficient (DSC)**, **95% Hausdorff Distance (HD95)**, **ECE**, **Domain Generalization Gap (DGG)**, and cross-site variance of DSC.

On ImageNet few-shot experiments, FIP improves accuracy most strongly for **CoOp**: baseline average **78.8**, **CoOp + FIM average 81.2**, for a gain of **+3.1 points average**; the paper also reports \(72.4 \rightarrow 76.1\) at 0-shot, \(78.2 \rightarrow 80.9\) at 2-shot, and \(83.6 \rightarrow 85.2\) at 16-shot. For **CoCoOp**, the average goes from \(79.6 \rightarrow 81.4\) (**+2.3**); for **MaPLe**, \(80.2 \rightarrow 80.7\) (**+0.6**), with slight degradations at 4/8/16-shot; for **KgCoOp**, \(79.4 \rightarrow 79.7\) (**+0.4**); and for **ProDA**, \(78.0 \rightarrow 78.7\) (**+0.9**). CMP improves calibration on ImageNet few-shot; for example, **CoOp** ECE changes from \(4.10 \rightarrow 3.77\), with an average reduction of **8.05%** relative.

Across the **11 standard datasets**, the paper reports an average **accuracy improvement of 3.2%** from FIP. Examples include UCF101 \(78.6 \rightarrow 84.3\), Flowers102 \(79.2 \rightarrow 85.5\), and ImageNet \(79.5 \rightarrow 84.9\), while underperformance is also reported on StanfordCars \(59.2 \rightarrow 54.3\) and EuroSAT \(53.8 \rightarrow 49.2\). For calibration, CMP reduces average ECE by **5.70%** relative, although the paper also notes that calibration worsens on Flowers102 \(2.96 \rightarrow 3.16\).

On the **domain-shift vision benchmarks**, CoOp + FIM improves accuracy over CoOp on all reported datasets: PACS \(96.5 \rightarrow 98.0\), Office-Home \(82.1 \rightarrow 85.6\), VLCS \(82.5 \rightarrow 86.0\), DomainNet \(58.8 \rightarrow 60.0\), ImageNet-V2 \(64.2 \rightarrow 65.5\), ImageNet-S \(47.9 \rightarrow 48.8\), ImageNet-A \(49.7 \rightarrow 50.5\), and ImageNet-R \(75.2 \rightarrow 76.8\). The paper also reports ECE improvements on shifted benchmarks such as PACS \(2.02 \rightarrow 1.91\) and Office-Home \(2.92 \rightarrow 2.75\). It further claims a **40% lower cross-domain performance gap** than prior benchmarking methods, though the exact comparative table is not shown in the supplied text.

The strongest empirical case appears in **medical segmentation**. On **BraTS**, the reported results are: Vanilla SAM, DSC WT \(68.2\), DSC TC \(52.7\), HD95 \(8.7\) mm, ECE \(12.4\%\); SAM-FT, DSC WT \(82.4\), DSC TC \(76.8\), HD95 \(5.2\) mm, ECE \(8.6\%\); and **StaRFM**, DSC WT **84.7**, DSC TC **78.3**, HD95 **4.5 mm**, ECE **6.2\%**. On **ATLAS**, the results are: Vanilla SAM, DSC \(52.4\), HD95 \(9.1\) mm, ECE \(13.7\%\); SAM-FT, DSC \(74.6\), HD95 \(6.3\) mm, ECE \(9.4\%\); and **StaRFM**, DSC **78.9**, HD95 **4.8 mm**, ECE **5.9\%**. These results underwrite the abstract-level summary of **84.7\% DSC and 4.8mm HD95 in medical segmentation**, and **28\% lower ECE** relative to the stated baseline regime.

The framework’s computational overhead is described as primarily a training-time effect. Patch-wise 3D FIP increases memory usage by **\(\sim 18\%\)** versus vanilla SAM on BraTS with \(128^3\) patches. No explicit inference-time overhead is discussed.

## 6. Ablations, caveats, and interpretive significance

The ablation studies emphasize that StaRFM’s two regularizers are not interchangeable and need moderation in their weighting [2507.09222]. In vision, using **CoOp** on 11 standard datasets, **CMP alone** changes average accuracy from **71.2** to **71.9** (**+1.0**), but average ECE worsens from **3.16** to **3.24**, a **+2.53%** relative change. By contrast, **FIP alone** changes average accuracy from **71.2** to **72.6** (**+2.0**) and improves ECE from **3.16** to **3.10**, about **1.90% down**. The paper interprets this as evidence that FIP stabilizes the model and makes CMP more effective.

Hyperparameter sensitivity is reported explicitly. For \(\lambda_1\) with \(\lambda_2=0\), the best reported value is **0.4**, and values greater than **0.4** degrade performance due to over-regularization. For \(\lambda_2\) with \(\lambda_1=0\), the best reported value is **0.4**, and larger values worsen ECE again. In medical ablations, the paper notes that over-regularizing FIP with \(\lambda_1>0.5\) hurts DSC by about **1.2\%**, while over-regularizing CMP with \(\lambda_2>0.6\) reduces DSC by over-suppressing uncertain boundary voxels.

The medical ablations also clarify how the two penalties behave under different sources of difficulty. On **BraTS**, the paper reports SAM-FT at DSC \(82.4\), HD95 \(5.2\), ECE \(8.6\), DGG \(12.3\); SAM + FIP at DSC \(83.1\), HD95 \(4.9\), ECE \(7.9\), DGG \(9.8\); SAM + CMP at DSC \(83.5\), HD95 \(5.1\), ECE \(6.8\), DGG \(10.2\); and StaRFM at DSC \(84.7\), HD95 \(4.5\), ECE \(6.2\), DGG \(7.4\). On **ATLAS**, the corresponding values are SAM-FT \(74.6, 6.3, 9.4, 14.2\); SAM + FIP \(76.2, 5.8, 8.7, 10.5\); SAM + CMP \(75.8, 6.1, 7.6, 11.3\); and StaRFM \(78.9, 4.8, 5.9, 8.1\). The paper’s interpretation is that **FIP alone** is stronger on ATLAS, where scanner-induced shift dominates, while **CMP alone** is slightly better on BraTS than FIP for DSC, likely because ambiguous tumor boundaries benefit more from voxel-level calibration.

The paper explicitly lists four main limitations. First, the theory assumes **bounded distribution shift** with finite \(D_{\mathrm{KL}}(\mathcal{D}_{\text{tgt}}\|\mathcal{D}_{\text{src}})\), so extreme shifts may violate the assumptions. Second, the method is **hyperparameter-sensitive**. Third, **patch-wise 3D FIP increases memory by about 18%**. Fourth, StaRFM does **not handle label shift**, since it assumes
$$
P_{\text{src}}(y|x)=P_{\text{tgt}}(y|x),
$$
rather than
$$
P_{\text{src}}(y)\neq P_{\text{tgt}}(y).
$$

Several manuscript-level inconsistencies are also relevant to interpretation. The patch size is written as \(16\times16\times16\) in Eq. (7) but as \(128\times128\times128\) in the implementation section. The reported vision hyperparameters list \(\lambda_2=0.5\) in one place and \(\lambda_2=0.4\) in another. The formal theory for CMP and Brier score is claimed more strongly than it is shown in the supplied text. These issues do not negate the reported empirical pattern, but they do delimit how precisely the method can be reconstructed from the paper alone.

Within those boundaries, StaRFM’s significance lies in its attempt to treat **robustness under covariate shift** and **calibration under uncertainty** as a single optimization problem spanning both image-text classification and voxel-wise medical segmentation. The framework’s empirical profile suggests that its strongest use case is one where raw transfer performance is not sufficient: few-shot CLIP adaptation and multi-site medical segmentation, especially when deployment conditions differ from adaptation data and confidence reliability matters alongside accuracy.

Source: https://www.emergentmind.com/topics/starfm