Effect of Text Length on Embedding Inversion and Information Retention

Determine whether longer MS-Marco passages encoded by the Contriever text encoder indeed contain more information that makes exact reconstruction harder and causes the resulting embeddings to discard more details from the input, thereby reducing inversion performance with increasing text length.

Background

The authors evaluate how the length of original texts affects inversion performance using the Contriever encoder on MS-Marco passages. They bucket texts by token count and observe that both F1 scores and cosine similarity generally decrease as length increases up to 64 tokens.

They propose a conjectured explanation for the observed trend, suggesting that longer texts carry more information, which hampers exact reconstruction and leads embeddings to discard more input details. This explanation remains conjectural and invites empirical verification or theoretical justification.

References

Our conjectured explanation is that longer texts contain more information, making exact reconstruction harder and leading to embeddings that discard more details from the input.

— Universal Zero-shot Embedding Inversion  (2504.00147 - Zhang et al., 31 Mar 2025) in Subsection "Effect of Text Length" (Section 5); Table \ref{tab:length_effect}

The main open challenge is exact (mean) sufficiency, which is unlikely to hold for pre-trained or learned embedding maps, and even small amounts of residual information can affect the T1E at larger sample sizes, as the simulation suggests. A diagnostic quantifying how much relevant information about $Y$ or $X{\omega_X}$ is left in the source text, rather than certifying that none is, would be of considerable interest.

— Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speeches  (2609.00946 - Simnacher et al., 1 Sep 2026) in Section 5, Conclusion