Test-Time Adaptive Merging (T^3)

• The paper introduces T^3 as a dynamic, test-time model merging approach that computes adaptive interpolation weights based on metrics like JS divergence and entropy minimization.
• The methodology leverages diverse adaptation techniques including scalar, vector, binary mask, and latent-space adaptations to refine model merging under distribution shifts.
• Empirical results indicate that per-sample or batch-adaptive merging improves performance in zero-shot classification, continual learning, and robustness against corruptions.

Searching arXiv for papers on test-time adaptive model merging and related methods. Test-Time Task Adaptive Merging, commonly abbreviated as $T^3$, denotes a family of inference-time model-composition methods in which a merged model is not fixed once offline, but is adapted at deployment using signals available from the current test setting. In the literature summarized here, the central adaptive object may be a scalar interpolation coefficient, a task-wise or layer-wise vector of merge coefficients, a binary mask over singular directions, a router over task vectors, or an input-conditioned gate over low-rank experts. The shared premise is that static merging is often insufficient because task vectors, checkpoints, or experts interact differently across inputs, tasks, layers, and distribution shifts. Recent work therefore treats model merging not as a one-time arithmetic operation, but as a conditional decision problem driven by unlabeled test data, predictive disagreement, latent fingerprints, evidential uncertainty, or current-task seed batches (Yang et al., 2023, Lee et al., 28 Mar 2025, Yang et al., 22 May 2025, Qiu et al., 17 May 2025, Imam et al., 31 Oct 2025, Xie et al., 4 Mar 2026).

1. Definition and scope

The most direct named instance of $T^3$ is "T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis" (Imam et al., 31 Oct 2025). That work studies inference-time merging of two medical vision-LLMs: a pretrained/generalist model $f_{\text{pt}}$ with parameters $\theta_{\text{pt}}$, and a fine-tuned/expert model $f_{\text{ft}}$ with parameters $\theta_{\text{ft}}$. The merged parameters for input $x$ are defined as

$$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$

where $\lambda(x)$ is computed at test time from the Jensen–Shannon divergence between the two models’ predictive distributions on that same sample (Imam et al., 31 Oct 2025). In this narrow sense, $T^3$ is a backpropagation-free, per-sample or per-batch adaptive parameter interpolation rule.

The broader research landscape uses the same underlying principle without always using the name $T^3$0. AdaMerging adapts task-wise or layer-wise coefficients over task vectors using entropy minimization on unlabeled test samples (Yang et al., 2023). AdaRank adapts binary masks over singular components of task vectors during test time, again using entropy minimization, thereby selecting which spectral directions should be retained in the merged model (Lee et al., 28 Mar 2025). CodeMerge computes merge coefficients for adaptation checkpoints through latent-space fingerprints and ridge leverage scores rather than direct multi-model inference (Yang et al., 22 May 2025). MINGLE adapts gates over low-rank experts using a small unlabeled seed batch from the current task in a continual setting (Qiu et al., 17 May 2025). BD-Merging trains a debiased router that allocates task-wise or layer-wise weights on a per-sample basis under distribution shift (Xie et al., 4 Mar 2026).

This suggests a useful working distinction. In a narrow usage, $T^3$1 refers to per-sample or per-batch test-time interpolation such as the medical VLM method in (Imam et al., 31 Oct 2025). In a broader usage, it denotes test-time adaptive merging more generally: methods that condition model composition on unlabeled target data or current input statistics rather than relying on a fixed offline merge. A plausible implication is that the field is converging on a common design pattern even when nomenclature differs.

2. Mathematical formulations of adaptive merging

A recurrent starting point is the shared-base model-merging setup. AdaMerging assumes a pretrained model $T^3$2 and $T^3$3 fine-tuned task models $T^3$4, defining task vectors

$T^3$5

Static task arithmetic forms

$T^3$6

whereas AdaMerging replaces the single global coefficient with task-wise coefficients $T^3$7,

$T^3$8

or with layer-wise coefficients,

$T^3$9

These coefficients are optimized at test time by minimizing Shannon entropy on unlabeled samples from the multi-task setup (Yang et al., 2023).

AdaRank begins from the same task-vector formalism but replaces scalar coefficient adaptation with adaptive selection in singular space. For task $f_{\text{pt}}$0 and layer $f_{\text{pt}}$1,

$f_{\text{pt}}$2

Instead of top-$f_{\text{pt}}$3 truncation, AdaRank introduces a binary mask $f_{\text{pt}}$4, yielding

$f_{\text{pt}}$5

The mask is learned at test time by entropy minimization, with a straight-through estimator applied to an underlying continuous parameterization (Lee et al., 28 Mar 2025). This reframes test-time adaptation as structural selection over spectral components rather than only coefficient tuning.

BD-Merging takes a different route. With shared backbone $f_{\text{pt}}$6 and task vectors $f_{\text{pt}}$7, it predicts per-sample task weights

$f_{\text{pt}}$8

from token embeddings $f_{\text{pt}}$9, and forms

$\theta_{\text{pt}}$0

The paper explicitly states that the router allocates task-specific or layer-specific weights on a per-sample basis (Xie et al., 4 Mar 2026). This is among the clearest examples of sample-adaptive parameter-space merging in the broader $\theta_{\text{pt}}$1 sense.

The named $\theta_{\text{pt}}$2 method in medical VLMs uses only two models, but the logic is analogous. It computes

$\theta_{\text{pt}}$3

then maps this to

$\theta_{\text{pt}}$4

and merges

$\theta_{\text{pt}}$5

A batch-wise extension averages coefficients across a batch,

$\theta_{\text{pt}}$6

and uses one parameter interpolation per batch (Imam et al., 31 Oct 2025).

3. Adaptation signals at test time

The principal distinction among $\theta_{\text{pt}}$7-style methods is the signal used to infer how merging should change at inference.

AdaMerging uses entropy minimization on unlabeled test samples. For sample $\theta_{\text{pt}}$8 with predicted distribution $\theta_{\text{pt}}$9, the entropy is

$f_{\text{ft}}$0

and the merge coefficients are optimized to minimize the sum of these entropies over current test batches (Yang et al., 2023). The method reports an average Spearman correlation of $f_{\text{ft}}$1 between entropy and prediction loss across the multi-task data, providing the paper’s empirical justification for entropy as a surrogate objective (Yang et al., 2023).

AdaRank uses the same general entropy-minimization principle but applies it to binary spectral masks and optionally to layerwise scaling coefficients $f_{\text{ft}}$2. Its objective is

$f_{\text{ft}}$3

The significant shift is therefore not the adaptation signal but the adaptation space (Lee et al., 28 Mar 2025).

The medical $f_{\text{ft}}$4 method avoids optimization entirely. Its signal is disagreement between the pretrained and expert models, quantified by Jensen–Shannon divergence. The paper argues that entropy ratio or confidence ratio fail to capture high-confidence disagreement, whereas JS divergence compares the full predictive distributions jointly (Imam et al., 31 Oct 2025). This makes the method training-free and backpropagation-free at inference.

CodeMerge adopts a latent proxy rather than output entropy or current-sample behavioral similarity. Each checkpoint $f_{\text{ft}}$5 is associated with a fingerprint

$f_{\text{ft}}$6

derived from source-model penultimate activations. At time $f_{\text{ft}}$7, ridge leverage scores

$f_{\text{ft}}$8

are used to select and weight checkpoints (Yang et al., 22 May 2025). The paper reports Pearson $f_{\text{ft}}$9 and Kendall $\theta_{\text{ft}}$0 between pairwise fingerprint differences and model weight differences across SparseDrive and SECOND, which is the empirical premise allowing latent-space retrieval to approximate weight-space selection (Yang et al., 22 May 2025).

BD-Merging uses evidential uncertainty and neighborhood discrepancy. A joint evidential head produces Dirichlet parameters $\theta_{\text{ft}}$1, predictive probabilities $\theta_{\text{ft}}$2, and uncertainty mass $\theta_{\text{ft}}$3 or $\theta_{\text{ft}}$4 depending on task-specific or unified-label notation (Xie et al., 4 Mar 2026). It then computes an Adjacency Discrepancy Score

$\theta_{\text{ft}}$5

and uses ADS to drive a discrepancy-aware contrastive objective for training a debiased router (Xie et al., 4 Mar 2026). This is a more elaborate adaptation signal than plain entropy minimization and is explicitly targeted at robustness under distribution shift.

MINGLE uses a current-task seed batch and a teacher KL objective. For current task $\theta_{\text{ft}}$6, it minimizes

$\theta_{\text{ft}}$7

where the current task’s standalone fine-tuned model $\theta_{\text{ft}}$8 serves as teacher (Qiu et al., 17 May 2025). This is a distinct test-time guidance signal: the merge is adapted not toward generic confidence but toward imitation of the current expert on unlabeled target samples.

4. Architectural variants of $\theta_{\text{ft}}$9-style merging

The literature supports several distinct architectural interpretations of test-time adaptive merging.

One class performs direct parameter interpolation between complete models. The named $x$0 method for medical VLMs belongs here: a current coefficient $x$1 directly determines interpolation between $x$2 and $x$3 (Imam et al., 31 Oct 2025). This is the simplest realization and is especially suitable when the merge involves only two compatible checkpoints.

A second class adapts coefficients over task vectors derived from multiple fine-tuned models. AdaMerging is the canonical example, with task-wise or layer-wise scalar coefficients over $x$4 (Yang et al., 2023). BD-Merging extends this to per-sample router-predicted coefficients over task vectors or layer-specific task vectors (Xie et al., 4 Mar 2026). These methods keep the representation of expertise in the same coordinate system as standard task arithmetic.

A third class performs structural selection within task vectors. AdaRank adapts binary inclusion masks over singular components of each task vector, per task and per layer (Lee et al., 28 Mar 2025). This can be interpreted as task-adaptive control over parameter subspaces rather than over whole-task deltas.

A fourth class adapts over expert modules rather than dense vectors. MINGLE represents each task by a low-rank expert $x$5, with merged layer output

$x$6

and adapts only the current task gate $x$7 at test time (Qiu et al., 17 May 2025). TTMM, from "Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging," merges LoRA adapters rather than full dense checkpoints: $x$8 where the prompt embedding $x$9 routes over cluster centroids $$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$0 (Bertolissi et al., 20 May 2025). Although framed as Test-Time Model Merging rather than $$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$1, it is a prompt-level adaptive merge and therefore a fine-grained instance of the same general paradigm.

A fifth class uses intra-model adaptive transfer rather than inter-model merging. Hi-Vec equips a single model with a hierarchy of linear heads, selects a head by gradient norm,

$$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$2

then merges the adapted selected head into similar heads via

$$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$3

The paper explicitly calls each hierarchical layer a task vector, though this usage differs from the standard residual-delta definition (Ambekar et al., 11 Aug 2025). This suggests that $$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$4 can be interpreted not only as inter-checkpoint merging but also as test-time adaptive sharing among internal parameter components.

5. Empirical performance and benchmark evidence

The named $$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$5 medical VLM paper evaluates across four medical imaging modalities and three deployment regimes: in-domain, base-to-novel, and corruption shifts (Imam et al., 31 Oct 2025). Averaged over four medical datasets, the reported Top-1 accuracies are: pretrained CLIP $$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$6, expert CLIP $$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$7, DaWin $$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$8, $$\theta_{\mathrm{merged}}(x) = (1-\lambda(x))\,\theta_{\mathrm{pt}} + \lambda(x)\,\theta_{\mathrm{ft}},$$9 $\lambda(x)$0, and $\lambda(x)$1 $\lambda(x)$2. The corresponding normalized error values are $\lambda(x)$3, $\lambda(x)$4, $\lambda(x)$5, $\lambda(x)$6, and $\lambda(x)$7, respectively (Imam et al., 31 Oct 2025). On Cell Microscopy, the mean accuracy rises from DaWin’s $\lambda(x)$8 to $\lambda(x)$9 with $T^3$0 and $T^3$1 with $T^3$2 (Imam et al., 31 Oct 2025). These results support the specific claim that sample- or batch-adaptive weight interpolation can outperform both static merging and prior dynamic entropy-based weighting in medical zero-shot classification.

AdaMerging reports strong gains over conventional task arithmetic on eight-task CLIP image classification. For ViT-B/32, Task Arithmetic achieves $T^3$3, TIES-Merging $T^3$4, Layer-wise AdaMerging $T^3$5, and Layer-wise AdaMerging++ $T^3$6. For ViT-L/14, Task Arithmetic gives $T^3$7, TIES-Merging $T^3$8, AdaMerging $T^3$9, and AdaMerging++ $T^3$00 (Yang et al., 2023). The paper also reports robustness improvements under corruption shifts and improvements on unseen downstream tasks (Yang et al., 2023). This is evidence that batch-level test-time coefficient adaptation already yields substantial gains, even without per-sample routing.

AdaRank extends this with adaptive spectral selection. On vision benchmarks merging CLIP ViT-B/32 and ViT-L/14 fine-tuned on 8, 14, and 20 tasks, CART+AdaRank reports $T^3$01 on ViT-B/32 and $T^3$02 on ViT-L/14, compared with static CART’s $T^3$03 and $T^3$04 (Lee et al., 28 Mar 2025). On language, RoBERTa average rises from $T^3$05 with CART to $T^3$06 with CART+AdaRank, and GPT-2 from $T^3$07 to $T^3$08 (Lee et al., 28 Mar 2025). Particularly notable is the ablation on ViT-B/32 8-task merging: starting from TA/CART, no adaptation gives $T^3$09, learning $T^3$10 only gives $T^3$11, learning $T^3$12 only gives $T^3$13, and learning both yields $T^3$14 (Lee et al., 28 Mar 2025). This indicates that structural adaptation in singular space can matter at least as much as scalar coefficient adaptation.

CodeMerge evaluates in online 3D perception under corruptions and cross-dataset transfer. On nuScenes-C end-to-end autonomous driving, averaged over corruptions, it improves 3D detection from MOS’s $T^3$15 to $T^3$16 mAP and from $T^3$17 to $T^3$18 NDS; AMOTA rises from $T^3$19 to $T^3$20, AMOTP decreases from $T^3$21 to $T^3$22, and recall increases from $T^3$23 to $T^3$24 (Yang et al., 22 May 2025). On Waymo $T^3$25 KITTI and nuScenes $T^3$26 KITTI transfer, it exceeds MOS by clear margins in AP$T^3$27 and AP$T^3$28 (Yang et al., 22 May 2025). Because CodeMerge computes weights from a compact codebook rather than repeated forward passes through all checkpoints, it shows how $T^3$29-style adaptivity can be implemented in high-cost settings.

BD-Merging reports strong robustness under corruptions and unseen-task generalization. In the layer-wise setting, it achieves clean average accuracy $T^3$30, corrupted $T^3$31 $T^3$32, corrupted $T^3$33 $T^3$34, and corrupted $T^3$35 $T^3$36 (Xie et al., 4 Mar 2026). On a 4-seen / 4-unseen split, it reports seen Avg Acc $T^3$37 and unseen Avg Acc $T^3$38, compared with AdaMerging’s $T^3$39/$T^3$40 and Twin-Merging’s $T^3$41/$T^3$42 (Xie et al., 4 Mar 2026). Its ablation shows the router is indispensable: full BD-Merging gives clean $T^3$43, corrupted $T^3$44, while removing the router drops these to $T^3$45 and $T^3$46 (Xie et al., 4 Mar 2026).

MINGLE provides the continual-learning variant of the story. On ViT-B/32 with 8 tasks, OPCM yields $T^3$47 ACC and $T^3$48 BWT, whereas MINGLE reaches $T^3$49 ACC and $T^3$50 BWT (Qiu et al., 17 May 2025). On ViT-B/16 with 20 tasks, OPCM gives $T^3$51 ACC and $T^3$52 BWT, while MINGLE reaches $T^3$53 ACC and $T^3$54 BWT (Qiu et al., 17 May 2025). The ablation shows that test-time adaptation alone improves ACC but can severely worsen forgetting, and that null-space constrained gating is central to balancing plasticity and stability (Qiu et al., 17 May 2025).

TTMM provides a prompt-level language-model analogue. With Llama-3.2-1B on Wikipedia, base perplexity is $T^3$55, single-model fine-tuning gives $T^3$56, TTMM with 10 experts gives $T^3$57, and TTT gives $T^3$58. On GitHub Python, base is $T^3$59, fine-tuned $T^3$60, TTMM with 10 experts $T^3$61, and TTT $T^3$62 (Bertolissi et al., 20 May 2025). The paper reports that TTMM is more than 100x faster than TTT at test time for the 1B-parameter model by amortizing optimization into offline expert construction (Bertolissi et al., 20 May 2025). This supports a broader interpretation of $T^3$63: adaptive merging can act as an amortized surrogate for test-time training.

6. Limitations, controversies, and adjacent directions

A major dividing line concerns granularity. AdaMerging is clearly test-time adaptive, but mainly at the level of a single merged model tuned to a current test mixture rather than per-sample dynamic routing (Yang et al., 2023). AdaRank similarly adapts a shared merged model for the current evaluation setting, not a different mask per incoming sample (Lee et al., 28 Mar 2025). By contrast, BD-Merging and the medical $T^3$64 method are explicitly per-sample or per-batch adaptive (Xie et al., 4 Mar 2026, Imam et al., 31 Oct 2025). This suggests that the phrase "task adaptive" is used with varying strictness across the literature.

Another major issue is whether test-time adaptation is necessary at all. WUDI-Merging offers a deliberately static, data-free counterpoint. It defines task vectors $T^3$65, interference vectors $T^3$66, and an interference objective based on how merged and task-specific task vectors act on task-relevant hidden inputs (Cheng et al., 11 Mar 2025). The method is entirely offline and uses no test data, yet reports state-of-the-art performance in data-free model merging scenarios, with an abstract claim of average $T^3$67 improvement versus baseline methods and $T^3$68 better performance than mainstream test-time adaptation approaches (Cheng et al., 11 Mar 2025). A plausible implication is that $T^3$69-style methods may benefit from strong offline interference reduction as an initialization rather than replacing static merging entirely.

The question of what counts as "task" is also unsettled. In TTMM, each prompt is effectively its own task (Bertolissi et al., 20 May 2025). In CodeMerge, the latent environment or domain state plays the role of task (Yang et al., 22 May 2025). In Hi-Vec, hierarchical classifier heads are treated as task vectors, though they are not standard task deltas (Ambekar et al., 11 Aug 2025). In MINGLE, the current sequential task and its class vocabulary are known (Qiu et al., 17 May 2025). In the medical VLM $T^3$70 paper, the task is the current sample’s location along the generalist–expert specialization axis (Imam et al., 31 Oct 2025). This suggests that $T^3$71 is better understood as a conditional merging principle than as a single fixed task formalism.

There are also practical constraints. Entropy-minimization methods require unlabeled target batches and can suffer from overconfident collapse, a limitation AdaMerging does not explicitly regularize away (Yang et al., 2023). AdaRank requires SVD of task vectors and test-time optimization over binary masks (Lee et al., 28 Mar 2025). BD-Merging requires neighborhood construction and a joint evidential head, making purely single-sample low-latency deployment more difficult (Xie et al., 4 Mar 2026). CodeMerge depends on the availability of stored checkpoints and a meaningful source-feature fingerprint space (Yang et al., 22 May 2025). MINGLE assumes shared initialization, same backbone architecture, current task class prompts, and retention of protected subspace statistics (Qiu et al., 17 May 2025). The medical $T^3$72 method assumes two compatible checkpoints and classification-style predictive distributions, and its entropy-threshold extrapolation rule is heuristic (Imam et al., 31 Oct 2025).

Finally, some adjacent work is relevant without being a direct $T^3$73 method. S4T studies test-time training in a single multi-task model and shows that task relations should remain synchronized during target-time adaptation; its synchronization metrics and cross-task consistency losses are relevant to $T^3$74 because they frame multi-task adaptation as a relational coordination problem rather than independent confidence maximization (Jeong et al., 10 Jul 2025). MergeRepair studies static and continual merging of LoRA adapters in code LLMs for automated program repair, but explicitly reports no empirical results yet and does not implement test-time task-conditioned merging (Dehghan et al., 2024). It is therefore more a precursor in deployment-time adapter composition than a demonstrated $T^3$75 system.

Overall, Test-Time Task Adaptive Merging denotes a transition from static model arithmetic to conditional model composition. The literature now spans backpropagation-free coefficient computation from predictive disagreement (Imam et al., 31 Oct 2025), entropy-based adaptation of task-wise and layer-wise weights (Yang et al., 2023), singular-space mask optimization (Lee et al., 28 Mar 2025), latent retrieval over checkpoint memories (Yang et al., 22 May 2025), prompt-conditioned low-rank expert merging (Bertolissi et al., 20 May 2025), continual low-rank gating with interference control (Qiu et al., 17 May 2025), and bias-aware per-sample routing under distribution shift (Xie et al., 4 Mar 2026). This suggests that $T^3$76 is less a single algorithm than a research program: the systematic study of how model-merging decisions should depend on the current input, batch, stream, or deployment regime.