Characterize the geometry of distilled text-embedding spaces

Characterize more finely how distillation changes the geometry of T5Gemma-2 text embeddings, particularly the distances and connectivity among plausible candidate embeddings under imperfect contextual inputs, to explain the resulting improvement in diffusibility.

Background

The paper argues that raw T5Gemma-2 embeddings separate plausible alternative words, making continuous diffusion trajectories likely to terminate at invalid or ambiguous embeddings. Distillation with the teacher decoder’s soft probabilities pulls plausible candidates closer together and improves the validity and quality of generated embeddings.

The appendix provides an initial geometric analysis by replacing context tokens and measuring candidate distances, finding that the distilled student keeps candidates closer when the context is imperfect. However, the authors explicitly defer a more detailed characterization of this embedding geometry.

References

We leave a more fine-grained analysis of the geometry for future work.

— Scaling and Distilling Text Embeddings for Better Diffusibility  (2610.01016 - Zhang et al., 1 Oct 2026) in Appendix, Section “How distillation changes the embeddings,” subsection “Downstream performance of the distilled embeddings”