Beyond Generative Priors: Minority Sampling with JEPA-Guided Diffusion

Abstract: Minority sampling aims to generate low-density instances on a data manifold and is of central importance in applications such as medical diagnosis, anomaly detection, and creative AI. Existing approaches, however, define minority samples relative to generative priors learned from training data, confining rarity to model-specific notions that may poorly reflect real-world semantics. In this work, we propose a world-centric perspective on minority sampling, which defines rarity with respect to real-world priors rather than generator-induced densities. To this end, we introduce JEPA guidance, a diffusion sampling framework guided by a Joint-Embedding Predictive Architecture (JEPA) -- a class of world models that encode broad, semantically rich representations. JEPA guidance steers diffusion trajectories toward low-density regions under the implicit density induced by the JEPA, thereby aligning generated minorities with real-world semantic rarity. To make JEPA guidance computationally practical, we develop principled approximation strategies accompanied by theoretical error bounds, significantly reducing the overhead of guidance computation. Extensive experiments across unconditional, class-conditional, and text-to-image generation demonstrate that JEPA guidance consistently improves the fidelity and semantic validity of minority samples, outperforming generator-centric baselines in capturing real-world notions of rarity. Code is available at https://github.com/soobin-um/jepa-guidance.

• The paper introduces a novel world-centric minority sampling method that leverages JEPA guidance to generate semantically rare samples beyond traditional statistical outlier definitions.
• The methodology integrates JEPA encoders into the diffusion process via an energy-based gradient, with randomized SVD efficiently approximating the dominant singular values.
• The paper demonstrates that JEPA-guided diffusion improves downstream classification and fairness while highlighting dual-use risks related to prior-induced biases.

JEPA-Guided Diffusion: World-Centric Minority Sampling

Motivation and Problem Formulation

Traditional minority sampling in generative modeling is defined by rarity relative to the modeled data distribution, usually via generative priors. This restricts the notion of "minority" to statistical outliers or low-density regions within a specific training dataset, which often fails to capture semantically meaningful atypicality rooted in real-world contexts. The paper "Beyond Generative Priors: Minority Sampling with JEPA-Guided Diffusion" (2605.24631) addresses this limitation by introducing a world-centric approach where minority samples are defined relative to priors encoded by Joint-Embedding Predictive Architectures (JEPAs), i.e., world models trained on vast, diverse data.

This formulation shifts the sampling objective to producing semantically rare instances with respect to global representations, aligning minority sample generation with real-world semantics instead of generator-induced statistical novelty.

Figure 1: JEPA guidance steers diffusion sampling toward world-centric minorities, contrasting existing methods that target low-density regions solely within the generative prior.

Generator-Centric vs World-Centric Minority Definitions

The paper formally distinguishes generator-centric minority sampling, which identifies low-density instances based on the generative model’s density $p_\theta(x)$, from world-centric minority sampling, which targets points with low density under the JEPA-induced implicit prior $p_\psi(x)$. World models like JEPA (typically DINOv2 encoders) represent broader, real-world visual structures. Empirical comparisons reveal generator-centric approaches often produce diverse, contextually unusual samples, whereas world-centric methods concentrate on semantically meaningful, globally rare objects (e.g., stealth aircraft).

Figure 2: Generator-centric minorities highlight contextual outliers, while world-centric minorities capture genuine semantic rarity across four ImageNet classes.

JEPA Guidance: Algorithmic Framework

JEPA guidance integrates the JEPA encoder into the diffusion process via an energy-based gradient. At each sampling step, the JEPA SCORE (sum of log singular values of the encoder’s Jacobian), a proxy for the representation density, is computed on the denoised estimate. Sampling is guided toward regions of low JEPA SCORE, thus producing world-centric minorities.

Exact computation of the JEPA SCORE is computationally expensive due to the large Jacobian matrix. The paper leverages randomized SVD to efficiently approximate the dominant singular values. Theoretical bounds are provided for the approximation error, demonstrating negligible impact on guidance quality when $k$ (rank for approximation) is chosen appropriately.

Deferred guidance is proposed to mitigate domain gap issues, applying JEPA guidance only during later stages of the diffusion trajectory, after conditioning information has been reliably incorporated. This enables integration with conditional models without requiring the JEPA encoder to access conditioning (e.g., class or text).

Empirical Evaluation

Comprehensive experiments are conducted across unconditional, class-conditional, and text-to-image generation settings using CelebA, ImageNet, Stable Diffusion v1.5, and SDXL-Lightning. The approach is benchmarked against canonical and recent minority sampling baselines (SGMS, BnS, MinorityPrompt, etc.), standard samplers (ADM, DDIM), and diversity mechanisms.

JEPA guidance achieves consistently lower JEPA SCORE values (higher semantic atypicality) while maintaining high image fidelity and alignment metrics. Notably, generator-centric approaches and standard sampling methods do not reliably produce world-centric minorities as measured by the JEPA SCORE. JEPA guidance shows superior precision/recall, density, and coverage statistics when evaluated against reference sets of real-world minorities.

Figure 3: World-centric minorities generated by JEPA-guided diffusion exhibit globally atypical semantics versus baselines in SDXL-Lightning text-to-image tasks.

Figure 4: JEPA guidance produces high-fidelity, semantically rare samples on CelebA, outperforming generator-centric baselines.

Analysis and Practical Implications

Ablation studies demonstrate robustness across JEPA encoder architectures, guidance strength, deferral ratios, and RSVD ranks. The leading singular values contribute most to image-dependent variation, with truncation errors dominated by image-agnostic offsets. Computational overhead is manageable, with deferred and intermittent guidance strategies further enhancing practicality.

JEPA-guided minority samples prove more effective for data-augmented downstream classification, boosting classifier generalization and attribute prediction performance with fewer synthetic instances.

The paper discusses potential dual-use risks: reversing the guidance direction could reinforce prior-induced biases or homogeneity. Responsible application and societal consideration are emphasized for fairness and inclusivity in generative tasks.

Theoretical Implications and Future Directions

This work operationalizes the concept of real-world priors in sample generation, demonstrating that world-centric representations (as encoded by JEPA) can directly inform atypicality. It bridges the gap between generative modeling and large-scale representation learning, offering a scalable mechanism for semantic rarity. Future research may explore more efficient Jacobian computations, amortized guidance, and fusion of multiple world models to enrich atypicality notions.

Conclusion

JEPA-guided diffusion sampling establishes a principled, efficient mechanism for generating minority samples defined with respect to real-world priors. It successfully decouples minority sample generation from idiosyncratic training distributions, yielding high-fidelity, semantically rare samples across diverse generative settings. The methodology provides a robust framework for creative content generation, robustness evaluation, and fairness-enabling applications, indicating promising directions in leveraging world models for guided generative sampling.

Figure 5: Generated minorities across four ImageNet classes with JEPA guidance, visually confirming semantic atypicality and diversity.