Selective filtering for improved rationale distillation

Determine whether a more selective contamination filter for hindsight-guided rationale distillation can recover additional diagnostic-accuracy gains beyond those achieved by removing all explicit ground-truth phrase slots.

Background

The paper trains a 1.5B Qwen student on traces generated by an 8B DeepSeek-R1-Distill-LLaMA teacher that receives the ground-truth disease label during trace generation. These traces can contain explicit “ground truth is X” phrases, which the unfiltered student may reproduce at inference, sometimes copying an incorrect disease label into its diagnosis tag.

StudentF uses a static regular-expression filter that removes all such ground-truth slots before supervised fine-tuning. This substantially reduces contamination and yields a statistically significant overall accuracy improvement, but the all-or-nothing filter also removes correct label hints that might otherwise improve some predictions. The unresolved problem is whether a more selective filtering strategy could retain beneficial information while preventing harmful reproduction of incorrect labels.

References

Aggressive filtering is also inherently all-or-nothing. Training-time labels are always correct, so a static filter cannot know in advance which phrases a student would have reproduced correctly at inference. What the data establish is that removing all such phrases yields a positive net effect; whether a more selective filter could recover additional gains remains open.

— Hindsight-Guided Rationale Distillation for Rare Disease Diagnosis  (2610.03176 - Singh et al., 2 Oct 2026) in Section 4, Discussion, paragraph “Why the filtering gain is bounded”