Investigate decoupled generation and embedding training

Investigate the effects of decoupling the training batches used for causal language modelling and contrastive embedding learning in the Generative Embedding Model.

Background

GEM jointly optimizes a causal language-modelling loss and a contrastive embedding loss. Although the objective permits separate batches for these two losses, the experiments use a shared batch of prompt-response pairs, partly because shared hidden states reduce training cost.

The authors leave open the effect of using a decoupled training strategy, potentially with different batch sizes for generation and embedding. This problem is relevant to understanding whether the observed retrieval and generation trade-offs depend on the shared-batch design.

References

Investigating the effects of a decoupled training strategy is left for future work.

— GEM: A Generative Embedding Model Bridging Reasoning and Retrieval  (2608.13200 - Shen et al., 13 Aug 2026) in Appendix, Section 1, subsection “Training”

We conjecture that these non-reasoning samples regularise GEM, mitigating overfitting to its reasoning.

— GEM: A Generative Embedding Model Bridging Reasoning and Retrieval  (2608.13200 - Shen et al., 13 Aug 2026) in Section 4, subsection “Results and Analysis,” paragraph on component effects in the training data

They leave open how to stabilize retrieval targets when one shared model handles generation and encoding.

— DEPT: Document Embedding Preservation Tuning for Unified Query Expansion and Retrieval  (2608.17632 - Wang et al., 18 Aug 2026) in Section "Related Work," paragraph "LLM-Based Query Expansion"