Scaling RGBM to many specialists and heterogeneous bases

Investigate scaling Residual-Governed Branch-and-Merge and its associated Fisher-geometric and residual-spectrum analyses to substantially more than ten specialists and to specialists with heterogeneous base models.

Background

The paper evaluates the Fisher-geometric equivalence between multi-teacher on-policy distillation and Fisher merging, together with the Residual-Governed Branch-and-Merge procedure, on configurations involving two or three specialists and two model families. Its theoretical diagnostic relies on local first-order reward superposition, while its empirical validation does not establish behavior for substantially larger specialist counts or for specialists derived from heterogeneous bases. The authors therefore identify extending the approach to these settings as unresolved future work.

References

While evaluating K = 2 ∼ 3 specialists across two model families (Gao et al., 2026; Yang et al., 2026b; Sun et al., 2026), scaling to K ≫ 10 or heterogeneous bases remains future work.

— A Fisher-Geometric View of Multi-Teacher On-Policy Distillation  (2610.26525 - Kaiyuan et al., 19 Oct 2026) in Appendix D, Limitations