Fixed-Point Masked Generative Modeling

Abstract: Masked Generative Models (MGMs) enable parallel decoding and achieve strong performance across modalities, but require full-sequence bidirectional transformers at every step, making training costly and degrading quality under low sampling budgets. Existing work improves efficiency via better samplers or cheaper fixed-depth denoisers, but they still allocate a fixed amount of denoiser computation to each refinement step. We introduce Fixed-Point Masked Generative Models (FP-MGMs), which replace part of the denoiser with a fixed-point solver over shared attention layers to enable adaptive depth with fewer parameters. To make it more effective for masked generation, we first introduce a cross-step consistency loss, which aligns hidden representations at neighboring denoising steps and, second, three-state reuse (3SR) which warm-starts the solver using the previous solution by treating differently unchanged, still-masked, and newly revealed tokens respectively. Together, these components define our complete training-to-inference framework for fixed-point masked generation, \emph{CoFRe}. We also show that pre-trained MGMs can be converted into FP-MGMs with short fine-tuning, avoiding full retraining. Across modalities, CoFRe improves the quality and cost trade-off. On OpenWebText, CoFRe reduces parameters by 38.8\%, training time by 11.5\%, and VRAM by 16.9\%, while improving generative perplexity from 830.8 to 101.8 at a budget of $96$ transformer-block forward passes, compared to MDLM. In ImageNette, CoFRe reduces training time by 48.6\% and VRAM by 50.7\%, while improving FID in all sample budgets tested. Overall, CoFRe offers a practical framework for cheaper training and stronger low-budget masked generation.

• The paper presents a novel fixed-point solver integrated with a three-state token reuse mechanism to improve efficiency and generative quality.
• It introduces cross-step consistency regularization to align hidden representations across denoising steps, reducing compute demands.
• Empirical results demonstrate significant improvements in generative perplexity, FID scores, and parameter efficiency on benchmarks like OpenWebText and ImageNette.

Fixed-Point Masked Generative Modeling: Architecture, Algorithms, and Empirical Performance

Introduction and Motivation

Masked Generative Models (MGMs) have established themselves as versatile frameworks for parallel generation across text and vision domains, supporting strong sample quality, adaptive sequence lengths, and robust multimodal capabilities. However, their computational demands are substantial: each refinement step requires a full bidirectional transformer pass over the entire sequence, leading to high training cost in terms of parameters, VRAM, and wall-clock time. Moreover, sampling quality degrades sharply under constrained compute budgets (i.e., limited transformer-block forward passes), rendering MGMs less competitive than autoregressive baselines particularly in low-latency deployment scenarios.

Previous attempts to improve MGMs have focused on alternative samplers, fixed-depth masked architectures, and continuous-depth or DEQ-style implicit models, each offering partial remedies but failing to deliver a holistic quality-cost solution. This paper proposes Fixed-Point Masked Generative Models (FP-MGMs), which directly address architectural inefficiencies by replacing part of the denoiser with a fixed-point solver over shared transformer blocks, enabling controllable effective depth, efficient parameter reuse, and adaptation to step difficulty. The FP-MGM framework introduces (i) a cross-step consistency regularization for aligning hidden representations at neighboring denoising steps, and (ii) three-state reuse for token-aware solver initialization, yielding the CoFRe recipe for masked generation.

FP-MGM Architecture: Implicit Depth and Token-Aware Initialization

FP-MGMs operate by decomposing the denoiser into an explicit preprocessing stack, an input-conditioning projection, an implicit fixed-point block, and a postprocessing stack. The fixed-point block is repeatedly applied, approximating equilibrium representations at each refinement step, with effective denoiser depth controlled by the number of solver iterations rather than architectural depth. This reduces both parameter count and computational waste, and enables adaptive compute allocation per-step.

The solver initialization is critical in discrete masked contexts, where token states change abruptly across steps. Three-state reuse stratifies initialization by token transition type: unchanged visible tokens fully reuse previous solutions, still-masked tokens partially reuse them (with coefficient interpolation as masking diminishes), and newly revealed tokens rely more on the current input. This token-dependent interpolation is empirically validated to reduce initialization error and improve generative metrics.

Figure 1: FP-MGM architecture and training. During training, the middle transformer stack is replaced with an iterated fixed-point block; cross-step consistency aligns hidden representations at neighboring denoising steps.

Cross-Step Consistency: Stabilizing Low-Budget Generation

Architectural improvements alone are insufficient for strong low-budget generation: abrupt input changes induce logit and representation drift across denoising steps. The cross-step consistency loss, $\mathcal{L}_{\mathrm{CONS}}$, aligns representations between student and teacher states at correlated noise levels (where the teacher context is strictly cleaner). This regularization operates as cross-time self-distillation in hidden space, sharpening masked-token predictions and mitigating drift.

Empirical results indicate that this mechanism drives substantial improvements in generative perplexity, particularly in low-budget regimes (e.g., sample quality improves from 193.1 to 101.8 at budget 96 and 47.0 to 37.8 at budget 768 for text generation on OWT).

Figure 2: FP-MDLM and CoFRe improve the quality-cost trade-off on OpenWebText: CoFRe achieves superior generative perplexity at all budgets and uses fewer parameters, less training, and less VRAM.

Model Adaptation and Short-Distillation

An important property of FP-MGMs is their ability to adapt pretrained MGM checkpoints into the fixed-point architecture via short teacher-student adaptation. Selected transformer layers (identified via CKA analysis for block structure redundancy) are mapped to the explicit and implicit components, and adaptation proceeds over only a fraction ($<$5%) of the original training steps. Empirical results demonstrate that this process yields better generative perplexity at all budgets, and makes fixed-point denoisers practical without full retraining.

Figure 3: Pretrained MDLM checkpoints are converted into FP-MDLM architectures by mapping layers to explicit preprocessing, fixed-point, and postprocessing blocks.

Empirical Results: Quality-Cost Trade-Off Across Modalities

Across OpenWebText and ImageNette benchmarks, CoFRe consistently outperforms previous masked generative baselines on both quality and efficiency metrics:

• On OpenWebText, parameters are reduced by 38.8%, training time by 11.5%, and VRAM by 16.9%. Generative perplexity is improved from 830.8 to 101.8 at budget 96, and entropy remains non-degenerate, avoiding sample collapse.
• On ImageNette, training time decreases by 48.6% and VRAM by 50.7% relative to MaskGIT-Large, while FID is improved at all budgets.

Notably, CoFRe remains competitive in downstream evaluation tasks and achieves strong quality-diversity tradeoff scores (MAUVE), matching or exceeding state-of-the-art distilled or accelerated diffusion LLMs.

Figure 4: Generative perplexity as function of sampling budget on OpenWebText. CoFRe outperforms other accelerated diffusion LLMs and distillation baselines across all budgets.

Figure 5: Generated samples using CoFRe at moderate transformer-block budget (460), demonstrating high image realism and diversity.

Ablations and Component Studies

Ablation studies isolate the contributions of implicit depth, three-state reuse, and cross-step consistency. Three-state reuse proves superior to uniform or full reuse, especially in low-/medium-budget regimes; lagged logit analyses, fixed-point residual analysis, and quality-diversity heatmaps further support the design. Component sweeps confirm that regularization and token-aware initialization are complementary, and that non-uniform solver depth allocation (decreasing schedule) achieves optimal trade-off between compute and generative quality.

Figure 6: Heatmaps of generative perplexity and entropy for varying allocation between denoising steps and fixed-point iterations. Optimal tradeoffs are achieved with sufficient denoising steps and moderate implicit depth.

Theoretical and Practical Implications

FP-MGMs extend masked generative modeling in two principal dimensions:

• Parameter-efficient adaptive computation: Fixed-point solvers enable architectural depth to be decoupled from parameter count, supporting model scaling under compute constraints, and reminiscent of techniques developed for looped and DEQ-style transformers [baiDeepEquilibriumModels2019; Prairie2026ParcaeSL].
• Stabilized low-budget generation: Cross-step regularization generalizes consistency distillation and mitigates representation collapse, substantially improving accelerated generation, which is critical in deployment scenarios and for non-autoregressive discrete modalities.

Future work should explore larger scales (beyond OWT/ImageNette), adaptive stopping in fixed-point solvers, more general reuse mechanisms for non-monotone trajectories (e.g., remasking models [wang2025remasking]), and further synergy with explicit route selection or learned unmasking policies.

Conclusion

FP-MGMs and the CoFRe recipe introduce a practical and theoretically motivated solution to the masked generative modeling quality-cost bottleneck. The fixed-point block reduces denoiser parameters and compute waste; cross-step consistency regularization and three-state reuse yield superior low-budget generation. Adaptation from pretrained MGM checkpoints is tractable and effective, lowering barriers for flexible deployment. CoFRe demonstrates substantial improvements in generative perplexity, image FID, training efficiency, and sample diversity. The framework advances the frontier in parallel generative modeling and sets the stage for further research into scalable and efficient architectures for text, vision, and multimodal generation.

• (2605.31215) Fixed-Point Masked Generative Modeling
• [sahoo_simple_2024] Simple and Effective Masked Diffusion LLMs
• [changMaskGITMaskedGenerative2022] MaskGIT: Masked Generative Image Transformer
• [baiDeepEquilibriumModels2019] Deep Equilibrium Models
• [bai_fixed_2024] Fixed Point Diffusion Models
• [deschenauxPartitionGenerativeModeling2026] Partition Generative Modeling: Masked Modeling Without Masks
• [deschenaux2024beyond] Beyond autoregression: Fast LLMs via Self-Distillation Through Time

---

Figure 7: Fixed-point residual analysis. Mean relative residual decreases with FP iterations, demonstrating convergence to equilibrium; three-state reuse initializes closer to fixed point than uniform reuse.