---
title: Cross-Model Stability Transfer
url: https://www.emergentmind.com/topics/cross-model-stability-transfer
type: topic
---

# Cross-Model Stability Transfer

Cross-model stability transfer refers to the principled study, measurement, and facilitation of stability and semantic alignment of learned internal representations across different neural architectures or model instantiations. This field addresses the key question of how, why, and under what structural conditions representations, feature spaces, or task-specific adapters can not only generalize within a model but also transfer—often under simple mappings—across models with different priors, parameterizations, or inductive biases [2508.03649].

## 1. Conceptual Foundations: Representational Stability and Alignment

**Representational stability** denotes the sensitivity of a model's learned features to changes in architectural or procedural priors. If models $\mathcal{A}_1$ and $\mathcal{A}_2$ are trained on the same task, with respective mappings $f_1(x), f_2(x): \mathcal X \to \mathbb R^d$, then stability asks that there exists a transformation $\mathcal{T}$ such that $\|\mathcal{T}(f_1(x)) - f_2(x)\|$ is small for typical $x$.

**Alignment** quantifies the possibility of finding a (frequently linear) transform $\mathcal{T}$ with $\mathcal{T} \circ f_1 \approx f_2$. Standard alignment metrics include Centered Kernel Alignment (CKA),
principal-angle subspace overlaps, Procrustes error, and cross-model probe accuracy [2508.03649].

This general framework is instantiated in diverse research lines:
- Cross-architecture feature geometry [2508.03649]
- Portability of concept steering vectors in LLMs [2501.02009]
- Adapter transfer and subspace-based parameterization [2501.16559]
- Transferring unstable features for robust classifier training [2106.07847]

## 2. Structural Decomposition: Shaping Operators and Corrective Paths

One robust class of models for cross-model stability transfer is the dual-pathway decomposition:
\[
  f_{i}(x) = S_{i} W_{i} x + C_{i}(x)
\]
where $S_{i}$ (shaping operator) encodes inductive bias—block-sparse, low-rank, or spectral structure—while $C_{i}(x)$ is a residual corrective path [2508.03649]. This decomposition separates the model into an explicitly structured pathway reflecting architectural priors and a learned component which adapts to the residual semantic content required by the task.

Empirical and theoretical analyses show that closeness of $S_1, S_2$ and controlled drift in $C_1, C_2$ directly imply stable, transferable representations across models [2508.03649]. Analogous structure appears in LoRA-X, which explicitly constrains low-rank adapters to operate within the dominant singular subspace of the base model's weights, enabling projection-based, data-free cross-model transfer [2501.16559].

## 3. Theoretical Results: Alignment Bounds and Mapping Conditions

Provable guarantees characterize when and how cross-model stability transfer succeeds. Let $\|S_2 - S_1\| \le \epsilon_S$, $\|C_2(x) - C_1(x)\| \le \epsilon_C$, and $\sigma_{\min}(S_1) \ge \sigma_0 > 0$. Then, the linear mapping $\mathcal{T} = S_2 S_1^{-1}$ yields:
\[
  \|\mathcal{T}(f_1(x)) - f_2(x)\| \le \frac{\epsilon_S}{\sigma_0}\|W_1 x\| + \epsilon_C
\]
for all $x$, and, integrated over the data distribution $\mu$,
\[
  \mathbb{E}_x \|\mathcal{T}(f_1(x)) - f_2(x)\| \le \frac{\epsilon_S}{\sigma_0} \mathbb{E}_\mu \|W_1 x\| + \epsilon_C
\]
[2508.03649]. This formalizes the intuition that stability transfer is possible whenever model structures (e.g., shaping operators) remain close in operator norm and residual paths are well-controlled.

Analogous stability is observed in LoRA-X, where the projection of the source adapter into the target model's subspace ensures small Frobenius-norm error provided subspace alignment metrics $sim_l, sim_r \approx 1$: 
\[
  \|\Delta W_{t \leftarrow s} - \Delta W_s\|_F \le (1-sim_l)\|\Delta W_s\|_F + (1-sim_r)\|\Delta W_s\|_F
\]
[2501.16559].

## 4. Empirical Measurement and Protocols

Empirical assessment of cross-model stability transfer relies on suites of geometric and functional metrics computed over learned representations on fixed datasets.

**Key geometric metrics:**
- Linear CKA between embedding matrices $X, Y$
- Principal-angle (subspace) overlap for top-$k$ singular vectors
- Procrustes alignment error: $\min_{R \in O(d)} \|RX-Y\|_F$
- Numerical similarity of learned linear transformations across concept steering tasks [2508.03649, 2501.02009]

**Key protocols:**
1. Train several models with different architectures or initializations on identical tasks/datasets.
2. Compute penultimate-layer or relevant embeddings from a validation set.
3. Measure alignment metrics and cross-model probe accuracy (e.g., logistic regression trained on one model's representations evaluated on another's).
4. In testbed cases (e.g., LoRA-X, LLM SV transfer), evaluate transferability of adapters or steering vectors via end-task metrics (e.g., human preference score, generated image quality, LLM output alignment) [2501.16559, 2501.02009].

Table: Summary of Representative Cross-Model Alignment Results [2508.03649]

| Model Pair         | CKA (mean ± std) | Procrustes Error | Transfer Accuracy (%) |
|--------------------|:----------------:|:----------------:|:--------------------:|
| PGNN – MLP         |   0.78 ± 0.02    |   0.12 ± 0.01    |   88.1 ± 1.1         |
| PGNN – CNN         |   0.62 ± 0.03    |   0.26 ± 0.02    |      —               |
| MLP – CNN          |   0.55 ± 0.04    |      —           |   75.6 ± 2.0         |
| PGNN_NoStruct–MLP  |   0.69 ± 0.03    |      —           |   82.0 ± 1.4         |

These results indicate that explicit structural shaping ($S$) in the architecture both increases metric alignment and functional interoperability [2508.03649].

## 5. Methodological Extensions: Adapter, Steering Vector, and Feature-space Transfer

- **LoRA-X** parameterizes and transfers low-rank adapters as changes in the dominant SVD subspace of pretrained layer weights, yielding stable and data-free adapter transfer whenever the subspace similarity between source and target layers exceeds a threshold [2501.16559].
- **Concept Steering Vectors (SVs) in LLMs** can be ported across different architectures using a learned ordinary-least-squares linear mapping, which is found empirically to generalize across concepts and model families (SSIM ≈ 0.87–0.95 among alignment transformations). This permits robust behavioral control transfer, e.g., modulating "harmfulness" or "sycophancy" by porting direction vectors across models without gradient access [2501.02009].
- **TOFU (Transfer of Unstable Features)** extracts "nuisance" (unstable) features from source tasks and uses them to partition target datasets, enforcing robustness by group-DRO minimax training over those clusters [2106.07847]. Unlike feature reuse, this pipeline transfers the *notion* of instability/spuriousness rather than the absolute feature weights.

## 6. Applications and Implications: Distillation, Modular Pipelines, and Robustness

Cross-model stability transfer supports a variety of system-level capabilities:
- **Model distillation:** Teacher-student alignment is facilitated if both share the same shaping operator, concentrating distillation loss on residuals [2508.03649].
- **Plug-and-play modularity:** When modules are structured around a common shaping prior, they can be swapped between models—encoder-decoder or pipeline settings—with predictable bounds on transfer error [2508.03649].
- **Parameter-efficient adaptation:** LoRA-X demonstrates that adapters can be transferred *without retraining* if SVD subspaces align, enabling scalable fine-tuning across model updates and derivative architectures [2501.16559].
- **Stable semantic interface:** Steering vectors align across LLMs via a single mapping. Safety and control functions (e.g., refusing unsafe queries) can thus be ported among closed or proprietary models [2501.02009].
- **Robustness to spurious correlations:** By transferring only the unstable-feature partition, models can enforce invariance to known nuisance factors without propagating spurious dependencies [2106.07847].

**Design recommendations:** Explicitly separate shaping (inductive bias) from residual adaptation in layers, regularize shaping operators during model updates, ensure their spectra remain well-conditioned, and align on stable, low-frequency semantic subspaces where possible [2508.03649].

## 7. Limitations and Future Directions

Current methods succeed predominantly when the architectures share strong structural similarities or task-induced priors (e.g., singular subspaces, orthogonal latent semantics) [2501.16559, 2501.02009, 2508.03649]. Deterioration is observed for highly divergent architectures or when weak prior alignment translates to low subspace or representation similarity.

Open directions include:
- Extending linear mapping to non-linear or kernel alignment for more distant models
- Automatic modulation parameter selection in concept steering
- Standardized cross-model benchmarks encompassing broad concept and task taxonomies
- Theoretical delineation of the regimes in which linear/projection-based transfer breaks down.

The field of cross-model stability transfer thus provides both conceptual and algorithmic tools for building increasingly interoperable, robust, and modular machine learning systems, with direct implications for foundation model scaling, open safety protocols, and long-term reliability [2508.03649, 2501.02009, 2501.16559, 2106.07847].

Source: https://www.emergentmind.com/topics/cross-model-stability-transfer