Evaluate structure-preserving target transformations as a retrieval mitigation

Evaluate whether preserving a molecule’s structure while rescaling its target so that the published benchmark number is no longer the answer can interrupt verbatim retrieval without making the molecular-property prediction task impossible.

Background

The paper tests character substitution of molecular structure strings, but this intervention destroys chemical information along with molecular identity. The authors therefore cannot determine whether retrieval can be suppressed while retaining a chemically meaningful task. They report that an alternative factorial experiment using non-monotone label transformations was inconclusive because both retrieving and non-retrieving models had similarly low rank correlation.

References

We could not test the obvious alternative, keeping the structure and rescaling the target so that the published number is no longer the answer, at this shot count: in a $3 \times 4 \times 2$ structure/label factorial at 60 in-context examples, the two non-monotone relabellings left a retrieving model and a matched non-retrieving one at the same low rank correlation, so those cells measure the inability of any model to perform the task rather than anything about retrieval.

— Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models  (2609.05381 - Busch et al., 4 Sep 2026) in Section 4, “What remains after blinding”