Fixed-Point Distillation

• Fixed-point distillation is a process where a model or resource is refined until it remains invariant under a specific update operator.
• It is applied in various settings such as discrete diffusion image generation, self-conditioned language flows, recursive meta-distillation, and quantum Fourier state purification.
• The method uniquely employs teacher corrections and contraction mechanisms to guarantee convergence and improve model consistency compared to standard knowledge distillation.

Searching arXiv for papers on fixed-point distillation and closely related formulations. I’ll retrieve a few relevant arXiv records to ground the article in current literature. Fixed-point distillation denotes a family of distillation procedures in which a student model, latent state, or resource is trained or iteratively refined so that a specified operator leaves it invariant, contracts it toward a unique attractor, or monotonically amplifies the desired component. In the recent literature, the label covers at least five technically distinct settings: one-step distillation of discrete diffusion image generators, identity-preserving score distillation sampling, self-conditioned flow LLMs, operator-theoretic recursive meta-distillation, and iterative purification of Fourier ancilla states for phase kickback. The term is therefore not uniform across fields: in some works it is an explicit fixed-point equation, in others a fixed-point iterator, and in still others a fixed-point-like purification or consistency process (Wang et al., 20 May 2026, Kim et al., 27 Feb 2025, Yoo et al., 1 Jul 2026, Flouro et al., 19 Jan 2026, Jones, 2013).

1. Definitional core and scope

The central fixed-point idea is that the distilled object should already be stable under a refinement map. In discrete diffusion, Fixed-Point Distillation (FPD) states this directly as

$$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}\,p_\theta,\qquad \forall\,r\in(0,1),$$

so that if $\hat z\sim p_\theta$, partial re-masking followed by one frozen-teacher refinement step should preserve the student distribution. In self-conditioned flow LLMs, the corresponding condition is

${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$

with the fixed-point denoiser defined by $D_t^\star({\bf x})\coloneqq {\bf z}^\star$. In recursive meta-distillation, the fixed point is a distributional equilibrium of the operator $S_{g+1}=T(S_g;T_0,\ldots,T_k)$, and the simple anchored case yields the unique fixed point $p^{(S^*)}=p^{(T_0)}$ (Wang et al., 20 May 2026, Yoo et al., 1 Jul 2026, Flouro et al., 19 Jan 2026).

A second recurring element is that fixed-point distillation is not merely model compression. FPD uses the teacher as a local correction operator rather than as a provider of full trajectories; IDS uses a fixed-point iterative regularization (FPR) to repair the score used in score distillation sampling; recursive meta-distillation characterizes when iterative KD is mathematically well-posed rather than error-accumulating; and Fourier-state distillation repeatedly suppresses sidebands so that the dominant Fourier component becomes increasingly dominant. This suggests that “fixed-point distillation” is best understood as a structural property of the update rule, not a single architecture or loss (Wang et al., 20 May 2026, Kim et al., 27 Feb 2025, Flouro et al., 19 Jan 2026, Jones, 2013).

2. Canonical formulations

The most concise way to compare the literature is to identify the object being driven to a fixed point and the mechanism used to do so.

Setting	Fixed-point object	Mechanism
FPD	$$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$	corrupt student draft, one teacher refinement, lifted drift loss, hard-forward STE
IDS	source-consistent posterior mean under FPR	minimize $$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$
Fixed-point flows	${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$	iterate self-conditioning, then distill into $D^\star$
Recursive meta-distillation	$\hat z\sim p_\theta$0	anchored meta-teacher with KL contraction
Fourier-state distillation	dominant Fourier component after purification rounds	modular addition, postselection, coefficient squaring

These formulations differ in what is stabilized. FPD stabilizes a student distribution under a corrupt-then-refine map. IDS stabilizes the relation between a source image and the posterior mean implied by the source-conditioned score. Fixed-point flows stabilize the self-conditioning variable at each flow time. Recursive meta-distillation stabilizes a sequence of student distributions via teacher anchoring. Fourier-state distillation stabilizes a resource state under repeated purification, with the desired spectral component amplified at each round (Wang et al., 20 May 2026, Kim et al., 27 Feb 2025, Yoo et al., 1 Jul 2026, Flouro et al., 19 Jan 2026, Jones, 2013).

A common misconception is to treat all teacher-student consistency objectives as fixed-point distillation. The literature is narrower. An explicit fixed-point statement requires either an invariance equation, an iterative map whose fixed point is the target, or a contraction/purification mechanism with monotone convergence. Later sections discuss related approaches that are consistency-like or teacher-guided but not explicitly fixed-point in this sense.

3. Fixed-point distillation in image and 3D generation

In discrete diffusion image generation, FPD is an end-to-end framework for one-step distillation that constructs local correction targets by partially corrupting the student’s one-step draft and refining it with a single teacher step. It lifts discrete tokens into continuous features through the VQ codebook embedding $\hat z\sim p_\theta$1, decoder $\hat z\sim p_\theta$2, and frozen backbone $\hat z\sim p_\theta$3, and then applies a multi-bandwidth drift loss. The empirical drift vector is

$\hat z\sim p_\theta$4

and the training loss is

$\hat z\sim p_\theta$5

with $\hat z\sim p_\theta$6 in experiments. To backpropagate through discrete sampling, FPD uses the hard-forward, soft-backward straight-through estimator

$\hat z\sim p_\theta$7

so that the teacher and decoder always see valid discrete codebook tokens. The frozen teacher is MaskGIT for class-conditional generation and MaskGen-L for text-to-image generation; the frozen feature backbone is DINOv3 ViT-B/16 using blocks $\hat z\sim p_\theta$8 and patch-grid features. On ImageNet-256, FPD reports FID $\hat z\sim p_\theta$9, IS ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$0, Precision ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$1, and Recall ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$2; on GenEval, MaskGen-FPD reports Overall ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$3 with ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$4 step and ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$5B parameters. An ablation shows that if the teacher refines a random sequence rather than the student’s own draft, the overall score drops from ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$6 to ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$7, which directly supports the state-dependent fixed-point interpretation (Wang et al., 20 May 2026).

In score distillation sampling for image editing and editable NeRF, IDS introduces fixed-point iterative regularization to repair the source-conditioned score before applying the editing update. The forward diffusion and classifier-free guidance are

${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$8

${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star),$9

and the posterior mean used by FPR is

$D_t^\star({\bf x})\coloneqq {\bf z}^\star$0

FPR minimizes

$D_t^\star({\bf x})\coloneqq {\bf z}^\star$1

with Euclidean loss as default, updating the noisy latent by gradient descent and then extracting a guided noise

$D_t^\star({\bf x})\coloneqq {\bf z}^\star$2

The fixed-point principle is that if the score is correctly aligned with the source image, then the posterior mean inferred from that score should resemble the source image. On cat-to-others tasks, IDS reports cat2pig IoU $D_t^\star({\bf x})\coloneqq {\bf z}^\star$3 and LPIPS $D_t^\star({\bf x})\coloneqq {\bf z}^\star$4, and cat2squirrel IoU $D_t^\star({\bf x})\coloneqq {\bf z}^\star$5 and LPIPS $D_t^\star({\bf x})\coloneqq {\bf z}^\star$6; on InstructPix2Pix it reports PSNR $D_t^\star({\bf x})\coloneqq {\bf z}^\star$7 and LPIPS $D_t^\star({\bf x})\coloneqq {\bf z}^\star$8. In editable NeRF, IDS reports CLIP score $D_t^\star({\bf x})\coloneqq {\bf z}^\star$9 versus $S_{g+1}=T(S_g;T_0,\ldots,T_k)$0 for DDS and $S_{g+1}=T(S_g;T_0,\ldots,T_k)$1 for CDS. The paper repeatedly describes FPR as a self-correcting process that reprojects the intermediate state onto a source-consistent manifold (Kim et al., 27 Feb 2025).

Taken together, these two lines of work show two distinct uses of fixed-point distillation in generative modeling. FPD treats the student sample as a point that should already be locally stable under teacher refinement, whereas IDS treats the score itself as an object that must be iteratively corrected until posterior-mean consistency with the source holds. Both reject simple trajectory imitation as the sole mechanism of distillation.

4. Self-conditioned flows and recursive knowledge refinement

For self-conditioned flow-based LLMs, the fixed-point interpretation is unusually explicit. A self-conditioned denoiser is written as $S_{g+1}=T(S_g;T_0,\ldots,T_k)$2, and self-conditioning defines the iteration

$S_{g+1}=T(S_g;T_0,\ldots,T_k)$3

The corresponding fixed-point denoiser is

$S_{g+1}=T(S_g;T_0,\ldots,T_k)$4

Replacing the denoiser in the flow velocity gives

$S_{g+1}=T(S_g;T_0,\ldots,T_k)$5

so that after solving the inner fixed-point problem, the outer dynamics again form an ordinary time-dependent ODE with valid flow map $S_{g+1}=T(S_g;T_0,\ldots,T_k)$6. The fixed-point distillation objective for learning a self-conditioning-free denoiser is

$S_{g+1}=T(S_g;T_0,\ldots,T_k)$7

where $S_{g+1}=T(S_g;T_0,\ldots,T_k)$8 is computed by fixed-point iteration from a cold start. This can then be combined with flow map distillation through a two-time denoiser $S_{g+1}=T(S_g;T_0,\ldots,T_k)$9. On OpenWebText, the paper reports that ELF $p^{(S^*)}=p^{(T_0)}$0 steps$p^{(S^*)}=p^{(T_0)}$1 improves gPPL from $p^{(S^*)}=p^{(T_0)}$2 at $p^{(S^*)}=p^{(T_0)}$3 FPI to $p^{(S^*)}=p^{(T_0)}$4 at $p^{(S^*)}=p^{(T_0)}$5 FPIs, and LangFlow $p^{(S^*)}=p^{(T_0)}$6 steps$p^{(S^*)}=p^{(T_0)}$7 improves from $p^{(S^*)}=p^{(T_0)}$8 at $p^{(S^*)}=p^{(T_0)}$9 FPI to $$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$0 at $$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$1 FPIs. The distilled FMLM$$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$2 reports gPPL $$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$3 and entropy $$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$4 in $$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$5 step, gPPL $$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$6 and entropy $$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$7 in $$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$8 steps, and gPPL $$p_\theta=\big(\mathcal{T}_\phi^{r}\circ\mathcal{M}_r\big)_{\#}p_\theta$$9 and entropy $$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$0 in $$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$1 steps, outperforming the compared few-step baselines that preserve entropy (Yoo et al., 1 Jul 2026).

Recursive meta-distillation provides a more abstract, operator-theoretic account of fixed points in KD. The framework defines a meta-teacher construction operator

$$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$2

subject to Axiom 1 (Convexity Preservation), Axiom 2 (Positivity Inheritance), Axiom 3 (Teacher Anchoring), Axiom 4 (Continuity), and Axiom 5 (Monotonicity in Anchor Weight). The canonical anchored mixture is

$$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$3

With

$$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$4

the contraction theorem gives

$$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$5

and for the canonical operator,

$$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$6

Hence

$$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$7

so the process converges geometrically to the unique fixed point $$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$8 under realizability and convexity assumptions. The same framework states that without anchoring, recursion becomes pure self-training and can drift, with the failure mode summarized as

$$\mathcal{L}_{\text{FPR}}=d(\mathbf{z}^{\text{src}}, \mathbf{z}_{0|t}^{\text{src}})$$9

This work is foundational rather than algorithmic, but it supplies a precise criterion for when recursive distillation is contractive rather than error-accumulating (Flouro et al., 19 Jan 2026).

These two literatures occupy different levels of description. Fixed-point flows show how an inner self-conditioning iterator can be compressed into a self-conditioning-free denoiser and then into a few-step flow map. Recursive meta-distillation shows how repeated KD across generations can be well-posed if and only if the operator remains anchored to a base teacher. The former is constructive; the latter is axiomatic.

5. Fixed-point-like purification in quantum computing

In fault-tolerant quantum computing, the paper on Fourier states does not use the standard magic-state terminology of fixed-point distillation, but it presents a repeat-until-success distillation protocol whose convergence mechanism is fixed-point-like. The ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$0-qubit Fourier state is

${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$1

and satisfies

${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$2

For the fundamental state ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$3, the paper constructs a Clifford-only approximation

${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$4

with overlap bounded below by about ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$5 for all ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$6:

${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$7

The distillation step is two-to-one. Two approximate Fourier states are added modulo ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$8, the first register is measured in the Fourier basis, and one postselects on outcome ${\bf z}^\star = \hat{D}_t({\bf x},{\bf z}^\star)$9, which can be tested Clifford-only because $D^\star$0 is just $D^\star$1. If the input Fourier-basis coefficients are $D^\star$2 and $D^\star$3, the success probability is

$D^\star$4

and conditioned on success the output coefficients satisfy

$D^\star$5

In the symmetric case this becomes

$D^\star$6

so each round squares the Fourier-basis weights. That is the central purification mechanism: the dominant component becomes increasingly dominant, and the unwanted sidebands are suppressed.

The paper derives an approximate error law after $D^\star$7 rounds,

$D^\star$8

with ratio exactly $D^\star$9, so the error shrinks double-exponentially in the number of rounds. The required number of rounds scales as $\hat z\sim p_\theta$00, the Toffoli cost for the fundamental Fourier state is $\hat z\sim p_\theta$01, arbitrary Fourier states require up to $\hat z\sim p_\theta$02 gates, and the width is at most $\hat z\sim p_\theta$03 qubits. The significance is not merely asymptotic. Because Fourier ancillae enable phase kickback and are preserved after use, the paper argues that phase kickback may be the current lowest-overhead method for generating arbitrary phase rotations when Toffoli gates are as cheap as or cheaper than $\hat z\sim p_\theta$04 gates (Jones, 2013).

This quantum case clarifies a broader point: fixed-point distillation need not always appear as an explicit self-consistency equation. It can also arise as an iterative purification rule in which repeated successful rounds monotonically amplify the target component and suppress the rest.

6. Related approaches, boundary cases, and misconceptions

Some recent teacher-student methods are closely related to fixed-point distillation but do not meet the stronger criterion of an explicit fixed-point update or convergence statement. CasPoinTr is a two-stage point cloud completion framework built on AdaPoinTr. It formulates completion as

$\hat z\sim p_\theta$05

where $\hat z\sim p_\theta$06 is Shape Reconstruction, $\hat z\sim p_\theta$07 is an auxiliary feature encoder, and $\hat z\sim p_\theta$08 is Fused Completion. Its distillation targets are encoder features:

$\hat z\sim p_\theta$09

with total loss

$\hat z\sim p_\theta$10

The teacher has the same architecture as CasPoinTr but is trained with privileged inputs from denser point clouds, and for Teacher B the privileged input is sampled from GT at resolution $\hat z\sim p_\theta$11 rather than full GT resolution $\hat z\sim p_\theta$12 in order to avoid the shortcut problem and too large teacher-student gap. The paper explicitly states that it “does not define an explicit fixed-point update equation or iterative consistency constraint in the style of classical fixed-point distillation,” and characterizes the method instead as having “consistency/distillation flavor.” Its quantitative comparison with AdaPoinTr on ShapeNet-55 reports $\hat z\sim p_\theta$13, $\hat z\sim p_\theta$14, and F1 Score $\hat z\sim p_\theta$15, versus AdaPoinTr’s $\hat z\sim p_\theta$16, $\hat z\sim p_\theta$17, and $\hat z\sim p_\theta$18 (Yang et al., 27 Sep 2025).

ReDiF, by contrast, is explicitly positioned against fixed reconstruction or consistency losses. It treats few-step diffusion distillation as a policy optimization problem, with the student as a policy over denoising actions in an MDP and the teacher as a frozen reference. The RL objective is

$\hat z\sim p_\theta$19

and the framework uses PPO or GRPO, optionally with a divergence penalty

$\hat z\sim p_\theta$20

The paper states unambiguously that ReDiF is “not formulated as a fixed-point distillation objective” and is “not primarily” a consistency method, even though it discusses teacher-student agreement and larger denoising steps. In experiments with a Stable Diffusion v1.5 teacher using $\hat z\sim p_\theta$21 denoising steps and a $\hat z\sim p_\theta$22-step student, the best reported COCO result is ReDiF(PPO with Rényi) with FID $\hat z\sim p_\theta$23 and CLIPScore $\hat z\sim p_\theta$24, while the best reported LAION result is ReDiF(GRPO) with FID $\hat z\sim p_\theta$25 and CLIPScore $\hat z\sim p_\theta$26 (Tighkhorshid et al., 28 Dec 2025).

These boundary cases matter because they delimit the concept. Fixed-point distillation is not equivalent to knowledge distillation in general, not equivalent to consistency regularization in general, and not equivalent to teacher-guided training in general. The stronger notion requires one of three structures that are repeatedly visible across the literature: explicit invariance under an operator, iterative refinement to a self-consistent solution, or a contraction/purification mechanism with a unique attractor or monotone component amplification. Where those structures are absent, the more accurate description is “consistency-like,” “teacher-guided,” or “RL-based distillation,” not fixed-point distillation.