Fully disentangle constituent streams from a mixed hidden state

Develop a mechanism that fully disentangles the mixed hidden state produced by linearly averaging two input embedding streams into their constituent streams, thereby overcoming the geometric-mean obstruction to separately decoding both continuations from a single forward pass.

Background

The paper shows that fine-tuning can make the output distribution for averaged embeddings approximate the arithmetic mean of the independent stream distributions. However, standard decoding does not separately recover both streams: when logits are approximately averaged, token probabilities behave proportionally to the geometric mean of the two independent distributions. Consequently, a token that is likely in one stream but unlikely in the other is suppressed in the mixed distribution.

The authors introduce Joint Contrastive decoding as a proof of concept, using a small auxiliary model to provide per-stream guidance, but report a substantial residual gap relative to single-stream decoding. They therefore identify complete recovery of both constituent continuations as an unresolved problem.

References

Overcoming this obstruction fully remains an open problem for future research; however, below we propose a proof-of-concept decoding method as a suggestion for the community (with additional decoding variants, including a parameter-free two-head approach and inference-time logit arithmetic, deferred to Appendix~\ref{app:decoding_extra}).

— Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs  (2609.29845 - Tikhonov et al., 24 Sep 2026) in Section 5, paragraph “The geometric-mean obstruction” (Section 5.1)