Held-out effectiveness of weight training for Gemma 4 12B

Determine whether training the Gemma 4 12B weights on trajectories generated by a harness that has barely moved yields gains that transfer from development tasks to held-out DeepPlanning tasks.

Background

Gemma 4 12B has nearly saturated delivery at 98% and exhibits F4 planning-capacity failures as its largest failure class, so the paper’s diagnostic rule directs the next intervention toward weight training rather than further harness evolution. A same-night evaluation found only a small, within-noise harness gain.

The current training recipe produced gains on development tasks, but those gains did not transfer to held-out tasks. The effectiveness of the weights branch for this model therefore remains unresolved.

References

This model enters with delivery at 98\% and F4 the largest class (40\% of trajectories), the composition the rule assigns to the weights, so a small harness gain is the branch the rule predicts; the current training recipe, on trajectories from a harness that had barely moved, bought development-task gains that did not carry to held-out, so on this model the weights branch remains open.

— Harness Evolution Hits a Ceiling: When Weight Training Should Begin  (2610.11655 - Tian et al., 8 Oct 2026) in Appendix, Section “Further Lines in Detail,” subsection “Gemma~4~12B”