Generalization of curriculum-trained retrieval beyond the training token band

Determine whether the retrieval curriculum for bidirectional recurrent diffusion language models can install a general retrieval faculty that transfers to held-out token bands, rather than only a binding circuit over embeddings encountered during retrieval training.

Background

The converted recurrent diffusion models achieve retrieval on token bands used during retrieval training but score 0.000 on disjoint held-out bands. This failure persists even when keys and values are resampled from a randomized pool, indicating that the learned behavior is tied to token coverage rather than memorized fixed key–value pairs.

The paper proposes pool-size annealing—first learning retrieval on a narrow token pool and then widening the pool—as a possible remedy, but the reported annealing run never reached its first onset within the available budget. Consequently, the authors do not establish whether the curriculum can produce retrieval that generalizes across token identities.