Distinguishing genuine mask-parameterization differences from seed variation

Determine whether the small performance differences between binary straight-through-estimator masks and continuous masks in LearnedMask-LoRA reflect genuine differences between the parameterizations or merely random seed variation.

Background

The paper compares the binary straight-through-estimator mask used in LearnedMask-LoRA with a continuous mask in the first training stage. On the larger models, the two variants have very similar AUROC values, but the continuous-mask results are based on a single seed.

Because the observed gaps are small and the experimental design does not provide multi-seed estimates for the continuous variant, the authors explicitly state that they cannot determine whether the differences are substantive or stochastic.

References

On all five $\geq8$B models the two parameterizations are within $0.007$ AUROC of each other ($+0.004$ on Ministral, $-0.007$ on LLaMA3, $-0.003$ on Qwen3, $+0.005$ on GPT-oss, $-0.001$ on Gemma); the continuous-mask row is a single seed, so we cannot resolve whether these gaps reflect a real difference or seed variation, and we treat the two parameterizations as indistinguishable at this granularity.

— Local Sparsity Enables Unsupervised LLM Safety Detection  (2609.20129 - Chen et al., 17 Sep 2026) in Appendix, Section ‘Ablations for LearnedMask-LoRA’, subsection ‘Mask Parameterization: Continuous vs. Binary STE’