Inter-layer Fisher Integration (IFI)

• IFI is a method that computes layer-specific merging coefficients using the Fisher Information Matrix to control interpolation error in long-to-short reasoning.
• It employs a data-free diagonal Fisher computation via random token inputs, eliminating the need for calibration data.
• Empirical benchmarks show that IFI improves accuracy and reduces required token lengths compared to calibration-based merging methods.

Inter-layer Fisher Integration (IFI) is a theoretically grounded, data-free model merging algorithm designed to combine the capabilities of specialized LLMs for Long-to-Short (L2S) reasoning. IFI—referred to as FIM-Merging in its originating work—assigns per-layer merging strengths using the Fisher Information Matrix (FIM) as a proxy for the local curvature of the model objective, substantially improving L2S merging accuracy and efficiency relative to prior methods. IFI allows merging without requiring domain-specific calibration data, relying only on randomness injected by sampled token sequences, and demonstrates superior performance on benchmark tasks in L2S settings compared to uniform or calibration-based approaches (Xia, 23 Mar 2026).

1. Theoretical Foundations

IFI is motivated by the systematic failure of linear "Task Arithmetic" merging in the L2S regime, where merging error cannot be controlled by uniform coefficients due to pronounced inter-layer nonlinearity. The core theoretical result is Proposition 1, which establishes a tight, layer-wise bound on interpolation error for parameter interpolation along a task vector $\delta = \theta_1 - \theta_0$ between base ($\theta_0$) and long-chain reasoning ($\theta_1$) models. Given a twice-differentiable model output function $f:\mathbb{R}^d \rightarrow \mathbb{R}^m$, the Task-Arithmetic interpolation error at coefficient $\alpha$ is

$$E(\alpha) = \| f(\theta_0 + \alpha \delta) - [ f(\theta_0) + \alpha (f(\theta_1) - f(\theta_0)) ] \|_2,$$

and Proposition 1 states

$$E(\alpha) \leq \frac{\alpha(1-\alpha)}{2} \|\delta\|_2^2 \sup_{t \in [0,1]} \| H_f(\theta_0 + t\delta) \|_2,$$

where $H_f$ is the parameter Hessian. This bound evidences the critical role of the per-layer Hessian in determining safe merging magnitudes, justifying the need for non-uniform, layer-adaptive merging coefficients.

A key insight is the Fisher-Hessian equivalence for maximum-likelihood models at local optima. Classical results (Amari 1998) show

$$F(\theta^*) = \mathbb{E}_{x} [ \nabla \log p(x|\theta^*) \nabla \log p(x|\theta^*)^\top ] = - \mathbb{E}_{x} [ H_{\log p}( \theta^* ) ],$$

so the FIM provides a computable proxy (under a diagonal, per-layer approximation) for the local Hessian norm $\| H_f^l \|_2$. This supports using the diagonal per-layer FIM to set adaptive merging strengths.

2. Diagonal Fisher Computation: Data-Free Approach

IFI employs a data-free protocol for estimating the diagonal Fisher for each layer. For each parameter subset $\theta_0$0 (layer $\theta_0$1), the diagonal FIM is computed as

$\theta_0$2

where $\theta_0$3 are input sequences of random tokens uniformly sampled from the model's vocabulary, padded to a fixed length. For each $\theta_0$4, negative log-likelihood loss $\theta_0$5 is computed, gradients $\theta_0$6 are accumulated, and entries of $\theta_0$7 are summed to form $\theta_0$8. Empirically, $\theta_0$9–$\theta_1$0 random samples suffice for stable and robust per-layer ranking. No domain-specific or calibration data are used in this process.

3. Layer-Adaptive Merging Algorithm

Layer-wise adaptation in IFI operates by scaling the task vector in each layer according to the estimated local interpolation risk. For each layer $\theta_1$1:

1. Define the task vector and its squared norm: $\theta_1$2, $\theta_1$3.
2. Compute log importance: $\theta_1$4.
3. Center scores: $\theta_1$5.
4. Normalize: $\theta_1$6.
5. Gate via sigmoid: $\theta_1$7, where $\theta_1$8 is the sigmoid.
6. Set merging strength: $\theta_1$9.

Layers with high $f:\mathbb{R}^d \rightarrow \mathbb{R}^m$0 receive lower $f:\mathbb{R}^d \rightarrow \mathbb{R}^m$1 (more conservative merging), and vice versa. Additional TIES-based elementwise pruning with FIM-weighting is applied, in which the lowest-importance fraction of $f:\mathbb{R}^d \rightarrow \mathbb{R}^m$2 entries is zeroed. Gate layers may use additional scaling factors, and output-norm calibration is enforced per layer.

A full pseudocode specification of FIM-TIES is provided in the originating work (Xia, 23 Mar 2026).

4. Empirical Performance on Long-to-Short Reasoning

IFI's empirical evaluation centers on two primary benchmarks, 1.5B and 7B parameter L2S scenarios. On the 1.5B L2S suite:

• DeepSeek-R1-1.5B (long-CoT) yields average response length $f:\mathbb{R}^d \rightarrow \mathbb{R}^m$3 tokens at 43.3% accuracy.
• ACM-TIES achieves 43.3% at 1489 tokens.
• FIM-TIES achieves 47.3 ± 0.3% accuracy (increase of 3.9 points over ACM-TIES) at only 411 tokens (a 92.6% reduction in length).
• On MATH500: 74.9% (FIM-TIES) vs 71.4% (ACM-TIES).
• On AIME24: 20.0% (FIM-TIES) vs 10.0% (ACM-TIES).

On the 7B benchmark:

• FIM-TIES matches or exceeds ACM-TIES on five of six tasks.
• MATH500: 90.2% (FIM-TIES) vs 84.0% (ACM-TIES).
• OlympiadBench: 47.9% (FIM-TIES) vs 46.4% (ACM-TIES).
• GSM8K: both 92.2%.
• On AIME24 (greedy), FIM-TIES 26.7% vs ACM-TIES 33.3%; with self-consistency, FIM-TIES surpasses ACM-TIES (36.7% vs 33.3%).

Task Arithmetic (uniform coefficient merging) consistently underperforms due to its failure to accommodate layerwise nonlinearity. ACM-TIES relies on mutual information scores with calibration data, whereas IFI infers layer adaptation from fixed random tokens, supporting a fully data-free protocol.

5. Comparison with Prior Merging Methods

IFI advances upon previous merging strategies by providing an explicit theoretical justification for the superiority of layerwise, non-uniform merging. Activation-based or Task Arithmetic methods assume implicitly linear output variation with respect to the merging coefficient, which is systematically violated in L2S regimes, leading to substantial interpolation error. IFI employs the FIM-Hessian connection to set per-layer adaptation strengths grounded in risk control for nonlinearity. Unlike ACM-TIES, which requires held-out calibration data to estimate per-layer mutual information, IFI achieves consistent and strong results using only random token inputs.

A summary of key methodological differences is detailed below:

Method	Per-layer adaptation	Calibration data required	Theoretical bound
Task Arithmetic	No	No	None; assumes linearity
ACM-TIES	Yes	Yes	Empirical, no explicit bound
IFI (FIM-Merging)	Yes	No	Hessian-bound via FIM proxy

6. Practical Implications and Scope

IFI demonstrates a robust approach to model merging that eliminates the dependence on domain data while providing layer-resolved error control. The data-free estimation of the diagonal Fisher via random token input enables broad applicability in settings where calibration data is costly or unavailable. The theoretical framework also unifies and explains why mutual information and other layerwise signals empirically benefit merging, as they act as surrogates for local curvature.

A plausible implication is that IFI's approach may generalize to other forms of model merging or ensembling where complex, nonlinear parameter trajectories are present, and explicit risk control over interpolation error is required.

7. Context in Model Merging Literature

IFI (FIM-Merging) is the first method to directly implement a theoretically derived layerwise merging bound in LLM merging via an efficiently computable proxy, providing empirical and explanatory advances for L2S model composition. Its successful elimination of calibration data dependency and improvement in trade-off metrics position it as a significant development in the model merging landscape, specifically in the context of L2S reasoning for LLMs (Xia, 23 Mar 2026).