Self-conditioned Flow Map Language Models via Fixed-point Flows

Abstract: Self-conditioning is a core technique that enhances continuous flow-based LLMs, where the model learns to denoise generated text by conditioning on its own denoising estimate. While empirically successful, its performance improvements are poorly understood. Moreover, there is growing interest in the use of few-step generators based on flow maps, for which how to leverage self-conditioning is unclear. Here, we show that flow LLMs with self-conditioning solve a fixed-point iteration that bootstraps the performance of the learned denoiser. We use this viewpoint to formulate fixed-point flows, a two-dimensional class of self-conditioned flows, where the first dimension represents the flow process and the second represents the fixed-point iteration. We show that fixed-point flows define valid flow maps, and show that they can be distilled from self-conditioned flow models by compressing both fixed-point iterations and the flow process, the former with fixed-point distillation and the latter with flow map distillation. Our resulting flow map LLM, FMLM$^\star$, outperforms state-of-the-art self-conditioned models and few-step models in one- and few-step generation on OpenWebText. Code is available at https://github.com/Ugness/self-conditioned-fmlm.

• The paper demonstrates that self-conditioned flows implement fixed-point iterations, converging exponentially towards optimal denoising solutions.
• It introduces a fixed-point flow abstraction that decouples flow evolution from self-conditioning, facilitating efficient one- or few-step sampling.
• Empirical results show that fixed-point distillation produces models like FMLM★, which achieve superior performance under entropy constraints on OpenWebText.

Self-conditioned Flow Map LLMs via Fixed-point Flows: An Expert Synopsis

Introduction and Motivation

Self-conditioning has emerged as a critical technique in advancing flow-based LLMs, particularly in the context of non-autoregressive text generation. These models utilize continuous flows to perform denoising in an embedding space, enabling efficient parallel token generation via deterministic velocity fields. Notably, the incorporation of self-conditioning—conditioning a model’s denoising predictions on its own previous outputs—has empirically yielded significant performance gains in recent state-of-the-art flow LLMs, yet the underlying mechanism for these improvements remained theoretically opaque. Furthermore, as the research community intensifies efforts on few-step generation through flow maps, there remains a gap in the understanding and exploitation of self-conditioning within flow map distillation paradigms.

The paper "Self-conditioned Flow Map LLMs via Fixed-point Flows" (2607.00714) systematically addresses these gaps: it establishes that self-conditioned flow LMs perform a fixed-point iteration per timestep to refine denoising, formalizes this process via fixed-point flows, and elaborates a framework for their distillation into flow maps, leading to improved one-step and few-step language generation.

Theoretical Contributions: Fixed-point Flows and Self-conditioning

The central theoretical innovation is the fixed-point perspective on self-conditioning. The self-conditioned denoiser $\hat{D}_t({\bf x}, {\bf z})$ is shown to induce a fixed-point iteration:

${\bf z}^{j+1} = \hat{D}_t({\bf x}, {\bf z}^j),$

starting from an initialization (e.g., ${\bf z}^0 = 0$). Under a contractivity assumption, this iteration is shown to converge exponentially to a unique fixed point, which constitutes a self-corrected estimate of the ideal denoiser $D_t({\bf x})$. This is formalized and proved in their Proposition 2.1 (learned fixed-point convergence).

Figure 1: Convergence towards the fixed point across fixed-point iterations.

Crucially, this perspective allows for the replacement of the self-conditioning state with its limiting fixed point, yielding a fixed-point velocity and thereby a first-order ODE generative process that is autonomous in ${\bf x}$, rather than conditioned on past denoising states. The resulting framework, termed fixed-point flows, forms the basis for both efficient sampling and model distillation.

A significant practical implication is that cold-start (initialization at zero) and warm-start (bootstrapping from previous timestep's fixed-point) sampling both converge to the same denoiser given sufficient iterations, with warm-start only improving convergence speed. This unification provides a theoretical explanation for the empirical effectiveness of warm-started self-conditioning and demonstrates that self-conditioning does not fundamentally change the underlying generative process, but rather accelerates convergence toward the Bayes-optimal denoiser.

Figure 2: Warm-start and cold-start sampling with 1 and 100 fixed-point iterations.

Distillation Framework: From Flows to Flow Maps

Building on the fixed-point flows, the authors introduce the distillation of self-conditioned models into flow map LLMs. The two-dimensional structure—flow progression over time and convergence along fixed-point iteration—is exploited to distill both axes: flow map distillation (across time) and fixed-point distillation (along the self-conditioning axis).

The two-time denoiser $\delta_{s,t}$ parameterization allows the learning of flow maps, satisfying critical semigroup and consistency properties. Notably, the authors provide both online and offline distillation routes:

• Offline: Distilling a separately trained fixed-point denoiser.
• Online: Joint distillation using few (as low as nine) fixed-point iterations during training, directly from the self-conditioned teacher.

This yields FMLM$^\star$, a flow map LM that is capable of one- or few-step text generation while incorporating the accuracy advantages of self-conditioning. Essential to this process is that the distilled flow maps preserve entropy and data distribution fidelity, maintaining proximity to the original model's quality.

Empirical Evaluation

Experiments on OpenWebText rigorously evaluate the hypotheses:

• Convergence and refinement: The self-conditioned models (ELF, LangFlow) exhibit clear fixed-point behavior, with generation quality (measured by generative perplexity, gPPL) significantly improving as the number of fixed-point iterations is increased—demonstrating iterative refinement toward the Bayes-optimal denoiser.
• Initialization impact: When sufficient iterations are performed, cold and warm starts yield indistinguishable results, confirming that the benefit of warm-starts is solely acceleration of convergence.
• Self-conditioning is removable: Distilled fixed-point denoisers (e.g., ELF$^\star$) match the teacher model in gPPL-entropy frontiers, confirming that self-conditioning can be fully compressed out without loss using consistent fixed-point distillation objectives.
• Few-step generation: FMLM$^\star$ achieves superior gPPL across one, two, and four-step settings when compared to discrete and continuous diffusion/flow baselines, while rigorously preserving entropy at the data level. The online distillation achieves near-parity with offline in a fraction of the time.

The following table summarizes key quantitative results relevant to the state-of-the-art (from the paper):

Model	Steps	gPPL ↓	Entropy
FMLM$^\star$	1	112.5	5.37
FMLM${\bf z}^{j+1} = \hat{D}_t({\bf x}, {\bf z}^j),$0	2	94.7	5.45
FMLM${\bf z}^{j+1} = \hat{D}_t({\bf x}, {\bf z}^j),$1	4	75.2	5.41

Strong claims supported by the results:

• FMLM${\bf z}^{j+1} = \hat{D}_t({\bf x}, {\bf z}^j),$2 outperforms prior best self-conditioned and few-step models in the one- and few-step settings under entropy control.
• Self-conditioning can be removed or compressed without degradation, both in autonomous flows and flow map LMs.

Implications and Future Directions

Practical implications: The fixed-point flows perspective provides a precise protocol for compressing the benefits of self-conditioning into models that support efficient (few-step) sampling—critical for scaling generative LMs to large sequences or modalities requiring parallel generation. Importantly, it removes the ambiguity in how to distill self-conditioned models without cycling dependencies across steps, enabling more modular and efficient model deployment.

Theoretical implications: The formal linking of self-conditioning with contractive fixed-point iteration clarifies why and when self-conditioning succeeds, offering a lens through which similar architectures may be analyzed in diffusion-based generative modeling across data types. It grounds the improvements in a well-understood fixed-point contraction framework, making connections to deep equilibrium models and related implicit architectures.

Speculation for future work: The approach could be generalized:

• To self-distillation: simultaneously training the self-conditioned denoiser, its fixed-point version, and the flow map in end-to-end or synergistic fashion.
• Beyond language: modal extensions to vision, graph, and other structured or discrete spaces, exploiting the fixed-point formalism to inform model design and distillation.
• To adaptive and stable training: using contractivity as a diagnostic or explicit constraint for designing robust, scalable iterative architectures.

Conclusion

This work gives a rigorous theoretical and empirical treatment of self-conditioning in flow LMs, delivering a principled, systematically applicable framework for leveraging fixed-point flows in few-step generative text modeling. The strong numerical results and detailed analysis substantiate the role of fixed-point iteration as the mechanism enabling self-conditioning's empirical gains—ultimately enabling practical, high-fidelity, and efficient non-autoregressive generation.