Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)

Matched-input comparison of ordinary knowledge distillation

Determine whether applying ordinary knowledge distillation to both original and evidence-removed inputs yields results that differ from the Evidence-Conditioned Dependency Distillation approach, thereby establishing the effect of the matched-input training comparison.

Background

Evidence-Conditioned Dependency Distillation (ECDD) trains compact medical vision-LLMs using the original image together with an evidence-removed control, while ordinary knowledge distillation in the reported baselines is applied only to the original image. The experiments therefore compare ECDD against original-image distillation baselines, but do not evaluate ordinary knowledge distillation under the same paired-input setup.

The unresolved comparison is concrete: ordinary knowledge distillation could be applied to both the original and evidence-removed inputs, and its resulting evidence-dependency behavior could then be compared with ECDD. The paper explicitly identifies this matched-input experiment as not yet performed, so the relative contribution of ECDD's dependency objective versus simply exposing ordinary distillation to both endpoints remains undetermined.

References

Finally, the present comparisons do not include ordinary KD applied to both original and evidence-removed inputs. The current ablations therefore support the design choices within ECDD, while leaving this matched-input comparison open.

— When Evidence Vanishes: Distilling Visual-Evidence Dependency in Medical Vision-Language Models  (2610.23469 - Lan et al., 16 Oct 2026) in Section 4.6, Limitations