One Pass Is Not Enough: Recursive Latent Refinement for Generative Models

Abstract: Despite remarkable progress, image generation is far from solved. The dominant metric, FID, conflates sample fidelity with mode coverage and is close to being saturated. Yet a model can still exhibit mode collapse while achieving a low FID, since a handful of sharp, near-duplicate images can outscore a model that faithfully covers the full data distribution. We argue that precision and recall are essential complements to FID, and that because FID is already saturated, the more meaningful goal is to improve diversity and coverage. Achieving high recall requires a model that explicitly prioritizes mode coverage, unlike most generative models, which optimize sample fidelity. We introduce RTM, which replaces the single-pass latent mapping in style-based generators with an iterative refinement process, and show that this consistently improves both quality and diversity. Integrated with Implicit Maximum Likelihood Estimation (IMLE), which optimizes mode coverage by design, RTM achieves the highest precision and recall among current state-of-the-art approaches while maintaining competitive FID, with improvements across CIFAR-10, CelebA-HQ at 256x256, and nine few-shot benchmarks. RTM also improves StyleGAN2 and StyleGAN2-ADA on CIFAR-10 and AFHQ-v1 at 512x512, demonstrating that the benefit is not specific to IMLE. Unlike flow-matching baselines that achieve competitive FID at the expense of coverage, recursive refinement improves both quality and diversity simultaneously.

• The paper introduces a Recursive Token Mapper (RTM) that recursively refines latent representations to improve both image fidelity and mode coverage.
• The method builds on IMLE and style-based GANs, achieving lower FID scores and higher precision/recall on benchmarks like CIFAR-10 and CelebA-HQ.
• RTM offers efficient memory usage and parameter sharing while preventing mode collapse, paving the way for scalable and diverse image synthesis.

Recursive Latent Refinement in Generative Models: The Recursive Token Mapper

Motivation and Background

Despite substantial advances in deep generative modeling, state-of-the-art evaluation metrics such as FID are nearly saturated and fail to fully capture important aspects of generative performance. FID conflates sample fidelity and mode coverage, allowing models to demonstrate low FID while suffering from mode collapse—a lack of diversity or coverage of the data distribution. The paper “One Pass Is Not Enough: Recursive Latent Refinement for Generative Models” (2605.15309) identifies this critical shortcoming and instead emphasizes the necessity of measuring generation quality through complementary metrics: Precision (fidelity) and Recall (coverage), as well as FID.

In this context, the authors target mode coverage explicitly by building upon Implicit Maximum Likelihood Estimation (IMLE), which guarantees explicit coverage of the data distribution. They introduce the Recursive Token Mapper (RTM), a novel latent mapping architecture that overcomes bottlenecks of conventional one-pass MLP mappers in style-based generators such as StyleGAN. The RTM recursively refines the latent representation, allowing for sequential, hierarchical correction of the initial noise vector, which results in improvements to both sample quality and distributional coverage.

Recursive Token Mapper: Architecture and Motivation

Traditionally, style-based GANs and all prior IMLE-based models utilize a shallow MLP mapping network that processes a noise vector $z$ to a style vector $w$ in a single pass. This mapping is required to capture all necessary factors of variation, from coarse structure to fine details, in a single forward path. This monolithic processing restricts the mapping’s expressivity and makes the model highly sensitive to errors in latent placement.

The RTM addresses these limitations by adopting a recursive structure inspired by recent advances in recursive and iterative architectures (e.g., Tiny Recursive Model [jolicoeurmartineau2025trm], HRM [wang2025hrm]). RTM introduces two levels of recursion: an inner cycle ($L$ steps) for fast adaptation of a local latent state, and an outer cycle ($H$ steps) for higher-level refinement. At each inner step, a shared MLP-Mixer block (with optional token-attention) updates a latent representation, anchored by continual injection of the original noise tokens, and modification is propagated to the outer state. The recursion deepens the computation while maintaining a compact parameter count due to weight sharing.

This recursive mapping network fundamentally transforms the latent refinement process:

• Sequential refinement: Coarse factors (e.g., layout, pose) can be established early and progressively refined (e.g., texture, fine detail).
• Increased effective depth: More computational layers per forward pass at fixed parameter count, permitting greater functional expressivity without overfitting.
• Efficient memory usage: Short-gradient optimization is used where only the final step is differentiated, reducing memory footprint and enabling deep recursions.

  Figure 1: Unconditional AFHQ-v1 ($512{\times}512$) samples from StyleGAN2-ADA without RTM (left) vs. with RTM (right). RTM demonstrates improved FID and recall, indicating superior fidelity and mode coverage.

Integration with Direct-Latent Generators and Comparisons

RTM is integrated seamlessly both with IMLE-based pipelines (specifically, RS-IMLE) and with adversarial pipelines (StyleGAN2, StyleGAN2-ADA). In the IMLE setting, the training leverages a pool of random latents and pairs each with a training image through nearest-neighbor search in perceptual feature space, ensuring coverage. The generator is then optimized so the generated image from the matched latent approaches the real image. This structure precludes mode collapse by design but historically lagged in fidelity, limited partly by the quality of the mapping network.

Replacing the vanilla MLP mapper with RTM elevates both quality and coverage across a broad empirical suite:

• On CIFAR-10 ($32\times32$), RS-IMLE + RTM achieves state-of-the-art Recall (0.773) and Precision (0.896), outperforming all compared GAN, diffusion, score-based, and flow-matching models, with FID reduced by 30% relative to the RS-IMLE baseline.
• On CelebA-HQ ($256\times256$), RTM continues to produce the highest Precision (0.952) and Recall (0.592) among both VAE/score-based and GAN-model baselines, with substantial reductions in FID.
• In few-shot regimes, RTM consistently halves FID compared to matched RS-IMLE baselines across nine diverse small data benchmarks, without per-dataset tuning.

  Figure 3: Random samples from RS-IMLE + RTM on Shells, showing diverse and high-fidelity outputs in the few-shot regime.

Qualitative Evidence: Coverage, Diversity, and Faithfulness

Visual analysis underscores the RTM’s claims regarding diversity and mode coverage. The produced samples demonstrate high intra-class variability and maintain semantically meaningful diversity, including in challenging few-shot settings.

Figure 5: Random samples from RS-IMLE + RTM on Dog, evidencing improved generative diversity.

Additionally, nearest-neighbor analyses show that RTM-generated neighbors cluster tightly around the query’s semantic attributes and class, further confirming that recursive latent refinement prevents detrimental mode concentration.

Figure 7: RS-IMLE+RTM neighbors more faithfully match the query across gender, skin tone, age, and hair attributes; the baseline often fails to preserve such features.

Ablations and Theoretical Perspective

A critical ablation examines whether the improvements of RTM are explained by depth alone or by recursive parameter sharing. A naïvely deep (32-layer) non-recursive MLP actually worsens recall and modal coverage compared to the otherwise compact RTM, while using 13× more parameters. This supports the claim that recursion, rather than raw depth, provides the crucial inductive bias.

From a theoretical standpoint, the RTM maintains the IMLE family’s core guarantee: as a composition of continuous, differentiable mappings, it does not alter the coverage guarantee. The number of recursive cycles ($H$, $L$) acts solely as a compute-time hyperparameter, not affecting model parameterization, which means that inference precision and compute can be easily traded off post-training.

Implications and Future Directions

The Recursive Token Mapper presents clear improvements for both IMLE-family and adversarially trained style-based generators, indicating that recursive latent refinement is of broad utility within image generation. Practically, by supporting one-step inference with improved mode coverage, RTM is attractive for efficient, high-diversity image synthesis tasks and data augmentation pipelines. Theoretically, this work positions recursive computation as an essential inductive bias for continuous-generation tasks, as has already been shown in discrete and reasoning contexts.

The authors also identify several open avenues for further exploration, notably:

• Dynamic computation allocation: Extending the architecture with a learned halting policy, as in HRM and TRM, enabling the model to allocate computation based on the complexity or rarity of target modes.
• Scaling: Overcoming IMLE’s prohibitive cost for massive datasets like ImageNet, enabling full-scale deployment on truly large and diverse data distributions.
• Broader applicational domains: Adapting recursive latent refinement into non-StyleGAN architectures, or integrating with multimodal or text-conditioned generation.

Conclusion

The Recursive Token Mapper represents a significant architectural advance in the mapping-network design for generative models, establishing the importance of recursive latent refinement for both sample quality and data-distribution coverage. The empirical evidence demonstrates strong performance across standard and challenging settings, with benefits extending to both IMLE and adversarial pipelines. The recursive, parameter-shared refinement paradigm introduced here is likely to further inform the design of efficient, faithful, and flexible generative models.