Transferability of DARTS to larger and alternative language-model architectures

Determine whether Decoder-Aware Representation Tuning via Surgery (DARTS) transfers effectively from Llama-2-7B to larger language models, such as 13B- and 70B-parameter models, and to different decoder architectures, such as Mistral.

Background

The paper evaluates DARTS exclusively on merged Llama-2-7B models constructed from instruction-following, mathematical-reasoning, and code-generation fine-tunes. DARTS is designed to correct decoder-model representation bias using entropy-weighted L1 loss and a position-dependent additive correction, but the experiments do not establish whether these mechanisms remain effective as model scale or architecture changes.

The unresolved issue concerns the generality of DARTS beyond the specific Llama-2-7B setting, including transfer to larger models with substantially different parameter counts and to architectures such as Mistral. Establishing this would clarify whether the reported improvements reflect broadly applicable decoder-model behavior or properties specific to Llama-2-7B.

References

We evaluate exclusively on Llama-2-7B. Whether DARTS transfers to larger models (13B, 70B) or different architectures (Mistral~\citep{jiang2023mistral}) remains untested.

— DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging  (2608.28547 - Sharma et al., 28 Aug 2026) in Section Limitations, item 2 (Scale)

We show these two mechanisms on the Llama-3-8B substrate; whether Mistral-7B fine-tuning installs analogous or different mechanisms requires a matched base-model control on that architecture, which we leave to future work.

— Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation  (2609.01604 - Vasava et al., 1 Sep 2026) in Section 4, paragraph “An End-to-End Two-Stage Evaluation Pipeline”