Establish the causal mechanism of premature EOS after encoder pruning

Determine whether pruning layers from the Qwen3-ASR-0.6B audio encoder shifts the conditioning audio embeddings sufficiently to weaken the pretrained decoder’s acoustic evidence and cause premature end-of-sequence predictions, using direct measurements of embedding statistics and causal EOS analysis.

Background

Pruning the Qwen3-ASR-0.6B audio encoder can produce very short outputs and large deletion errors because the pruned encoder and bridge provide altered audio-placeholder embeddings to the causal language-model decoder. The paper hypothesizes that these shifted conditioning embeddings weaken the decoder’s acoustic evidence, but the reported safeguards—training the tied output embedding, enforcing a minimum generation length, and filtering degenerate rollouts—do not establish the underlying mechanism.

Resolving this problem would require directly measuring the relevant embedding mean and variance changes and testing whether those changes causally affect EOS probabilities and deletion errors. Such analysis would distinguish a mechanistic explanation from an empirically effective mitigation strategy.

References

We hypothesize that pruning shifts these conditioning embeddings, weakening acoustic evidence for the pretrained decoder. We do not directly measure embedding mean/variance or EOS causality, so this account motivates the safeguards rather than constituting a mechanistic proof.

— X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation  (2609.11412 - Zhang et al., 10 Sep 2026) in Appendix, Section 2, “Premature-EOS Safeguards” (label sec:eos)