Generalization of surrogate expressiveness

Determine when increasing the expressiveness of the polynomial surrogate beyond the quadratic approximation improves generalization for linear task-vector merging of large language models over the coefficient simplex.

Background

The paper evaluates quadratic and cubic simplex-lattice interpolants for selecting coefficients when merging eight language-model experts. Although the cubic surrogate uses substantially more evaluations, it produces only small in-distribution gains and lower out-of-distribution performance in the reported experiments. The authors therefore identify the conditions under which a more expressive surrogate generalizes better as unresolved.