Mechanistic explanation of the Gemma-specific guard-pass decline

Identify the mechanism responsible for the Gemma-3-4B and Gemma-3-12B models’ substantial decline in leakage-guard pass rates after procedural assembly fine-tuning.

Background

The paper evaluates procedural assembly fine-tuning as a cross-domain test of whether natural-language serialization skills transfer beyond arithmetic. Although assembly fine-tuning improves generation and extraction quality for the open-weight models overall, it reduces the leakage-guard pass rate for Gemma-3-4B and Gemma-3-12B by 22.9 and 17.5 percentage points, respectively. The authors report that these two models account for nearly all of the aggregate decline, while other models remain relatively stable or improve.

The paper does not provide a mechanistic explanation for this model-specific degradation. Determining why procedural assembly fine-tuning causes this adverse effect in the Gemma models would clarify how structural fine-tuning changes the interaction between serialization behavior and the leakage guard, and could inform model-specific training or guard-design strategies.