2000 character limit reached
Boosting Latent Diffusion Models via Disentangled Representation Alignment
Published 9 Jan 2026 in cs.CV | (2601.05823v1)
Abstract: Latent Diffusion Models (LDMs) generate high-quality images by operating in a compressed latent space, typically obtained through image tokenizers such as Variational Autoencoders (VAEs). In pursuit of a generation-friendly VAE, recent studies have explored leveraging Vision Foundation Models (VFMs) as representation alignment targets for VAEs, mirroring the approach commonly adopted for LDMs. Although this yields certain performance gains, using the same alignment target for both VAEs and LDMs overlooks their fundamentally different representational requirements. We advocate that while LDMs benefit from latents retaining high-level semantic concepts, VAEs should excel in semantic disentanglement, enabling encoding of attribute-level information in a structured way. To address this, we propose the Semantic disentangled VAE (Send-VAE), explicitly optimized for disentangled representation learning through aligning its latent space with the semantic hierarchy of pre-trained VFMs. Our approach employs a non-linear mapper network to transform VAE latents, aligning them with VFMs to bridge the gap between attribute-level disentanglement and high-level semantics, facilitating effective guidance for VAE learning. We evaluate semantic disentanglement via linear probing on attribute prediction tasks, showing strong correlation with improved generation performance. Finally, using Send-VAE, we train flow-based transformers SiTs; experiments show Send-VAE significantly speeds up training and achieves a state-of-the-art FID of 1.21 and 1.75 with and without classifier-free guidance on ImageNet 256x256.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What this paper is about (in simple terms)
This paper is about making AI image generators faster and better. It focuses on a key part of many image generators called a VAE (Variational Autoencoder), which turns big images into small codes (like zipping a file) so the generator can work more easily. The authors argue that to help image generators do their job well, the VAE should learn to store clear, separate “pieces” of meaning (like color, shape, pose, texture) in its tiny code. They call their improved VAE the Semantic-disentangled VAE, or Send-VAE.
What questions the paper asks
- What makes a “good” VAE for image generation?
- Is it enough to make the VAE copy the same kind of features used by the big image model that guides the generator?
- Or is something else more important—like keeping different image attributes cleanly separated in the VAE’s tiny code (this is called “semantic disentanglement”)?
- If we train a VAE to better separate these attributes, will image generators train faster and make higher-quality pictures?
How they approached the problem
First, some everyday explanations:
- VAE: Think of it as a zipper for images: it compresses an image into a small code and can unzip it back. If that small code is neatly organized, later steps (like image generation) become easier.
- Diffusion models (like LDMs): These models start with random noise and gradually “denoise” it into a realistic image, using the VAE’s compact code instead of full-size images to save time.
- Vision Foundation Models (VFMs): Very strong, pre-trained vision models (like CLIP or DINO) that “understand” images well. They’re like expert teachers for visual concepts.
- Semantic disentanglement: Storing different attributes (e.g., “red,” “striped,” “smiling,” “wearing hat”) in a way that’s clean and separate, so a simple rule can pick them out.
What others did before:
- Earlier work tried to make the VAE’s code look similar to features from VFMs (the “teacher”), hoping that would help the generator. It helped a bit, but the authors say VAEs and generators need different things.
What this paper does differently:
- Hypothesis: A VAE that clearly separates fine-grained attributes in its code is best for generation.
- Test: They use a simple check called “linear probing.” Imagine drawing a straight line to separate examples with a certain attribute from those without it. If a straight line works well, the VAE’s code is neatly organized for that attribute.
- New method (Send-VAE): Instead of directly forcing the VAE’s code to match the teacher’s high-level features, they insert a “mapper” network between them. This mapper is like a translator that turns the VAE’s code into something the teacher understands, without destroying the VAE’s ability to keep attributes neatly separated.
- Mapper details (kept simple): It’s a small vision-transformer-based translator that: 1) Takes the VAE’s latent code (sometimes with a bit of noise added, like data augmentation). 2) Translates it patch-by-patch (small image tiles) into features comparable to the teacher’s. 3) Aligns them by making the translated features point in the same direction as the teacher’s features (using cosine similarity).
In short, the mapper bridges the “representation gap” so the VAE learns from the teacher while still organizing attribute-level details cleanly.
What they found and why it matters
Main findings:
- Strong correlation: VAEs whose codes make attribute prediction easy (via linear probing) also lead to better image generation quality. In other words, better semantic disentanglement → better generative results.
- Faster training: Using Send-VAE, diffusion transformers (they use a model called SiT) learn faster. After only 80 training epochs, they already get much better scores than baselines.
- State-of-the-art quality: On the ImageNet 256×256 benchmark, Send-VAE reaches new best FID scores:
- FID 1.75 without classifier-free guidance (CFG)
- FID 1.21 with CFG
- Lower FID is better; it means the generated images look closer to real photos.
- Consistent gains across foundation models: Aligning through the mapper helps regardless of whether the teacher is CLIP, I-JEPA, DINOv2, or DINOv3 (DINO models worked best here).
- Slight trade-off: Send-VAE’s reconstructions (unzipping back to the original image) are a tiny bit worse in low-level detail than some baselines. That’s expected because it focuses on clean, meaningful attributes rather than every pixel-level texture. But for generation, this trade-off pays off.
Why this matters:
- It reframes what a “good” VAE should do for generators: not just copy high-level semantics, but structure attribute-level information cleanly.
- It offers a simple, practical test (linear probing on attributes) to judge if a VAE will be good for generation—useful for researchers and engineers.
What this could change going forward
- Better, faster training of image generators: With Send-VAE, teams can train high-quality models in less time, saving compute and energy.
- Clearer VAE design goals: Future VAEs for generation may focus on semantic disentanglement rather than only matching big vision models’ features.
- A new evaluation habit: Linear probing on attribute datasets (like faces, clothing, or animals) can become a quick, predictive tool to screen VAEs before full generator training.
- Broader impact: The idea of using a translator (mapper) to align different model “languages” could help in other areas where two systems need different kinds of internal representations.
In one sentence: By teaching the VAE to organize image attributes clearly—and using a smart translator to learn from big vision models—this paper shows we can train image generators faster and produce even more realistic images.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, concrete list of what remains missing, uncertain, or unexplored in the paper, framed to be actionable for future research.
- Establish causal evidence (beyond correlation) that increased semantic disentanglement in VAE latents directly improves downstream generative performance; e.g., controlled interventions that vary disentanglement while holding reconstruction quality, compression rate, and latent dimensionality constant.
- Validate disentanglement using standard metrics and benchmarks from the disentanglement literature (e.g., MIG, DCI, SAP on dSprites/3DShapes) rather than relying solely on linear probing; quantify agreement or divergence between these metrics and generation improvements.
- Clarify the theoretical mechanism by which attribute-level disentanglement aids diffusion training (e.g., formal analysis of denoising dynamics in disentangled latent spaces, or information-theoretic arguments linking attribute separability and sample efficiency).
- Isolate Send-VAE’s contribution by reporting downstream results without REPA and REPA-E (and with alternative representation-regularizers), to disentangle gains due to the tokenizer vs. the denoising network alignment.
- Generalize beyond ImageNet 256×256: evaluate on diverse datasets (e.g., CIFAR-10, LSUN, FFHQ), higher resolutions (512, 1024), and text-conditioned diffusion (e.g., SD/DiT with prompts) to test robustness and applicability across domains and tasks.
- Quantify convergence acceleration with compute-aware metrics (wall-clock time, GPU hours, FLOPs, memory footprint) and training curves (loss, FID/IS vs. iterations); currently “faster training” is claimed without detailed cost accounting.
- Characterize the reconstruction–generation trade-off more rigorously: measure how alignment weighting affects rFID vs. gFID, and whether reduced low-level fidelity harms tasks like super-resolution, inpainting, or inversion.
- Explore the mapper architecture design space beyond depth: ablate patch size, positional encodings, attention variants (cross-attention to VFM tokens), CNN/MLP-Mixer backbones, regularization (dropout, weight decay), and parameter count to balance semantic transfer and overfitting.
- Compare alignment objectives: patch-wise cosine vs. InfoNCE/contrastive losses, MSE/distillation to specific VFM layers, structural/adversarial alignment (e.g., SARA), multi-scale and hierarchical (coarse-to-fine) alignment; analyze which objectives best encourage attribute-level disentanglement.
- Study alignment target selection within VFMs: which layers (early/mid/late) and feature granularities yield the best attribute separability; assess benefits of multi-layer or hierarchical alignment that explicitly targets “semantic hierarchy.”
- Systematically ablate the alignment weight (λ_align), training schedules, and curricula (e.g., gradually increasing λ_align, progressive mapper capacity), given only a single weight setting (1.0) is reported.
- Analyze the noise injection strategy in alignment: sensitivity to noise magnitude, timestep schedule, and distribution; determine whether matching the downstream diffusion noise schedule further improves transfer.
- Examine latent design factors: downsampling rate (4× vs. 16×), channel dimensionality, and multi-scale latents; quantify how compression settings affect disentanglement, reconstruction, and generative quality.
- Report statistical rigor: multiple seeds, confidence intervals, and significance tests for gFID/sFID/IS and linear probing; correlation coefficients in Fig. 2 are based on few models and may be unstable.
- Clarify inference-time requirements and artifacts: confirm that VFMs and the mapper are not needed at inference, and analyze any residual effects on sampling speed or memory; provide a clean deployment path and quantify overhead during training.
- Evaluate sampling efficiency: does Send-VAE enable fewer sampling steps (lower NFE) for the same quality; provide speed–quality curves and compare to baselines at matched NFE.
- Test OOD robustness and domain shift: how well does Send-VAE transfer to domains different from VFM pretraining (e.g., medical, satellite, artistic styles); measure performance under distribution shifts and dataset biases.
- Assess diversity and controllability: beyond FID/IS, evaluate precision–recall trade-offs, mode coverage, and attribute controllability in generation (e.g., targeted edits via latent manipulations) to confirm practical benefits of disentangled latents.
- Extend to other generative backbones: U-Net-based LDMs, DiT, and autoregressive models (VQGAN/VQVAE, LFQ); test whether Send-VAE-like alignment benefits discrete tokenizers and AR training regimes.
- Provide failure case analysis: when does Send-VAE hurt performance (e.g., overly aggressive alignment causing loss of fine details or reduced recall); outline mitigation strategies (e.g., adaptive λ_align, selective layer alignment).
- Investigate biases and safety: aligning to VFMs (e.g., DINO) may inherit biases; evaluate fairness across attributes (e.g., CelebA demographic subgroups), content safety, and robustness to adversarial perturbations.
- Make “semantic hierarchy” concrete: propose and test explicit hierarchical alignment schemes (layer-wise, multi-scale, class/attribute heads) to verify the paper’s claim of aligning to a hierarchy rather than a single-level representation.
- Document full training hyperparameters and implementation details (e.g., mapper initialization, optimizer settings, normalization choices) to improve reproducibility; include code for attribute probing and alignment pipelines.
- Explore semi-/weakly-supervised alternatives: can small amounts of attribute supervision (or pseudo-labels) improve disentanglement more directly than VFM-only alignment; compare to β-VAE/FactorVAE-style inductive biases.
- Analyze long-horizon training (800 epochs) more thoroughly: why does the gap narrow with more training; does Send-VAE mainly help early-stage optimization, and can curricula/transfers preserve late-stage gains?
Practical Applications
Immediate Applications
The paper’s methods and findings enable several deployable use cases across sectors. Below are actionable applications, each with suggested tools/workflows and feasibility notes.
- Drop-in tokenizer upgrade for diffusion pipelines to reduce training time and improve quality
- Sectors: Software/AI platforms, Media/Advertising, Gaming/Film, E-commerce
- Tools/Products/Workflow:
- Replace the VAE in latent diffusion training with Send-VAE from the open-source repo (https://github.com/Kwai-Kolors/Send-VAE)
- Train/fine-tune VAE with the non-linear mapper aligned to a VFM (prefer DINOv2/v3 per ablations), inject latent noise, then train the diffusion transformer (e.g., SiT/DiT) with REPA-style alignment
- Use the paper’s sampler defaults (e.g., SDE Euler–Maruyama, ~250 NFE) as a baseline
- Assumptions/Dependencies: Availability and licensing of VFMs (DINOv2/v3); demonstrated SOTA on ImageNet 256×256 and with SiT—generalization to other datasets, resolutions, and U-Net architectures is likely but not guaranteed; slight reconstruction fidelity trade-off versus some VAEs
- Training-efficiency and cost reduction for industrial generative model development
- Sectors: Software/AI platforms, Cloud/ML Ops, Policy (sustainability)
- Tools/Products/Workflow:
- Adopt Send-VAE to reach target FID with fewer training epochs; include attribute-probe F1 as an auxiliary convergence signal
- Incorporate into MLOps dashboards: track gFID, sFID, IS alongside attribute-probe metrics for early stopping/model selection
- Assumptions/Dependencies: Benefit size varies by architecture and dataset; compute savings must be validated in each stack; mapper depth and alignment loss weight require tuning
- Attribute-level editing and controllability in image generation
- Sectors: Photo/Video Editing, E-commerce/Fashion, Media/Advertising
- Tools/Products/Workflow:
- Train simple linear probes on the Send-VAE latent space for target attributes (e.g., color, texture, style); use probe outputs to guide sampling (e.g., classifier guidance, attribute steering) or to post-filter generated candidates
- Build UI “attribute sliders” leveraging linear separability to nudge outputs toward desired attributes
- Assumptions/Dependencies: Attribute definitions and datasets for probes (e.g., CelebA, DeepFashion) must reflect the deployment domain; probe accuracy depends on domain shift and label quality
- Automated attribute tagging for visual catalogs using frozen Send-VAE latents + linear probes
- Sectors: E-commerce, DAM/MAM systems, Media libraries
- Tools/Products/Workflow:
- Use the VAE encoder as a feature extractor; train lightweight linear classifiers on catalog labels for at-scale tagging
- Integrate into ingestion pipelines for rapid metadata enrichment
- Assumptions/Dependencies: Trained on natural images—domain adaptation may be required for product or long-tail categories; licensing and privacy compliance for datasets
- Safer generative pipelines via attribute monitoring and moderation hooks
- Sectors: Social Media, Content Platforms
- Tools/Products/Workflow:
- Attach attribute probes to check and block disallowed traits before content is emitted
- Log attribute distributions of generated outputs for bias drift and policy compliance audits
- Assumptions/Dependencies: Probes are fallible; policy-aligned attribute taxonomies and thresholding must be defined; requires human oversight for edge cases
- Model evaluation and selection metric: attribute linear probing as an early indicator of generative quality
- Sectors: Academia, Software/AI platforms
- Tools/Products/Workflow:
- Add standardized attribute-probe F1 benchmarks (e.g., CelebA, DeepFashion, AwA) to tokenizer evaluation suites
- Use the metric for VAE model selection, ablation comparison, and early stopping during VAE training
- Assumptions/Dependencies: Correlation demonstrated on tested settings; different domains/attributes may weaken the relationship; ensure no leakage between probe data and training data
- Synthetic data generation with finer attribute control for training vision models
- Sectors: Robotics/AV (perception), Manufacturing QA, Education
- Tools/Products/Workflow:
- Generate datasets with controlled attribute distributions (lighting, texture, color) to balance training sets; use attribute probes to enforce coverage
- Assumptions/Dependencies: Must validate sim-to-real transfer; requires domain-specific prompt engineering or conditional labels
- Pluginized “Send-VAE tokenizer pack” for common diffusion stacks
- Sectors: Software/AI platforms, Open-source ecosystems
- Tools/Products/Workflow:
- Package pretrained Send-VAE checkpoints and mapper configs for popular frameworks (PyTorch Diffusers, OpenXLA/TPU stacks)
- Provide recipes for integrating with Stable Diffusion-like pipelines, SiT/DiT, and REPA
- Assumptions/Dependencies: Version compatibility with frameworks; licensing of VFMs and pretrained weights
Long-Term Applications
The paper points to broader innovations that require additional research, scaling, or domain adaptation before production.
- Domain-specialized disentangled tokenizers (e.g., medical, remote sensing, document images)
- Sectors: Healthcare, Geospatial, Legal/Finance (document processing)
- Tools/Products/Workflow:
- Train Send-VAE variants aligned to domain VFMs (e.g., radiology self-supervised encoders, document-layout VFMs) for attribute-controllable generation and robust feature extraction
- Assumptions/Dependencies: High-quality domain VFMs and labeled attributes; strict regulatory validation in healthcare; privacy-preserving training
- Video and 3D generation with disentangled latent tokenizers
- Sectors: Media/Entertainment, Simulation, Robotics
- Tools/Products/Workflow:
- Extend mapper alignment to spatiotemporal VFMs for video (e.g., DINO-like video encoders) and to 3D (point-cloud/splatting encoders); enable control over motion, lighting, and geometry attributes
- Assumptions/Dependencies: Scalable video/3D VFMs; computational cost; designing temporal/geometry-aware mappers
- Programmatic attribute control during sampling (constraint-guided or RL-guided generation)
- Sectors: E-commerce, Design, Advertising, Game asset pipelines
- Tools/Products/Workflow:
- Integrate attribute probes into the sampler to enforce constraints (e.g., “generate 30% red items”); combine with guidance or reinforcement learning for target distributions
- Assumptions/Dependencies: Reliable, differentiable attribute signals; balancing constraints without harming diversity
- Bias, fairness, and safety auditing frameworks based on latent attribute separability
- Sectors: Policy/Regulation, Trust & Safety
- Tools/Products/Workflow:
- Standardize audits where latent attribute probes measure representational bias and detect sensitive attribute leakage; include metrics in model cards
- Assumptions/Dependencies: Ethical attribute definitions; risk of sensitive-attribute inference; requires stakeholder governance
- Energy- and cost-aware training standards for generative models
- Sectors: Policy/Regulation, Cloud Providers, Enterprises
- Tools/Products/Workflow:
- Codify practices (e.g., adopting tokenizer designs that reduce training epochs) into procurement and sustainability guidelines; tie to carbon disclosures
- Assumptions/Dependencies: Independent replication of efficiency gains across workloads; standardized measurement protocols
- Cross-modal disentanglement for multimodal generation (text–image–audio)
- Sectors: Creative tools, Education, Accessibility
- Tools/Products/Workflow:
- Generalize the mapper-alignment idea to align VAEs with multimodal foundation models so attributes (style, tempo, sentiment) are separable across modalities
- Assumptions/Dependencies: High-quality multimodal VFMs; careful handling of cross-modal attribute definitions
- On-device or small-footprint generative systems via tokenizer-centric efficiency
- Sectors: Mobile, Edge AI, AR/VR
- Tools/Products/Workflow:
- Leverage faster convergence and disentangled control to distill smaller diffusion models guided by Send-VAE latents; deploy attribute-limited editors on-device
- Assumptions/Dependencies: Further compression/distillation research; hardware acceleration; privacy constraints
- Workflow templates for controlled synthetic data governance
- Sectors: Enterprises, Public Sector
- Tools/Products/Workflow:
- End-to-end blueprints: define attribute schemas → train probes → generate → validate attribute distributions → document datasets for audits
- Assumptions/Dependencies: Organizational maturity for data governance; robust validation suites; legal review for synthetic data use
- Hybrid discrete–continuous tokenizers with disentanglement guarantees
- Sectors: Foundation Model Research, Creative AI
- Tools/Products/Workflow:
- Combine Send-VAE-style alignment with discrete tokenizers (VQ) to balance editability and compression; design mappers that preserve attribute axes in codebooks
- Assumptions/Dependencies: New training objectives and theory; scalability to high resolutions
Notes on feasibility across applications:
- Proven strengths: Faster convergence for transformer-based diffusion models, SOTA FID on ImageNet 256×256, robust gains using DINO-family VFMs, and strong correlation between attribute-probe performance and generative quality.
- Key dependencies: Choice and license of VFMs; mapper architecture/hyperparameters (1 ViT block often best); dataset domain and resolution; modest reconstruction trade-offs; reproducibility across stacks beyond SiT.
- Risk controls: Validate probe accuracy and fairness; monitor for domain shift; include human-in-the-loop checks for safety-critical deployments.
Glossary
- AdamW: An optimization algorithm that decouples weight decay from gradient updates to improve training stability. "AdamW Loshchilov & Hutter (2019) optimizer is adopted"
- Attribute prediction: Tasks that predict fine-grained properties (attributes) of images, used here to measure semantic disentanglement in latent spaces. "linear probing on attribute prediction tasks"
- AutoRegressive (AR): A generative modeling paradigm that predicts outputs sequentially using discrete tokenizers. "AutoRegressive (AR) generation models."
- AwA: A dataset (Animals with Attributes) used for evaluating attribute-based recognition and disentanglement. "AwA Lampert et al. (2013)"
- CelebA: A large-scale face attribute dataset used for attribute prediction benchmarking. "CelebA Liu et al. (2015)"
- Classifier-free guidance (CFG): A sampling technique that steers generative models toward desired outputs without a classifier, improving fidelity. "with and without classifier free guidance"
- CLIP: A vision-LLM whose representations are used as alignment targets for training tokenizers or diffusion models. "CLIP Radford et al. (2021)"
- Cosine similarity: A similarity measure between vectors based on the cosine of the angle; used for representation alignment loss. "patch-wise cosine similarity"
- DeepFashion: A clothing dataset with rich annotations used for attribute prediction evaluation. "DeepFashion Liu et al. (2016)"
- Denoising objective: The training goal in diffusion models that learns to remove noise progressively to synthesize images. "Denoising Objective"
- Diffusion Transformer (DiT): A transformer-based architecture for diffusion models that captures long-range dependencies. "Diffusion Transformer (DiT) framework."
- Diffusion models: Generative models that produce data by iteratively denoising Gaussian noise. "Diffusion models have emerged as a powerful class of generative models"
- DINOv2: A self-supervised vision foundation model used as an alignment target for representation learning. "DINOv2. Oquab et al. (2024)"
- DINOv3: A successor to DINOv2 offering object-centric features beneficial for alignment and disentanglement. "DINOv3"
- Euler-Maruyama sampler: A numerical solver for stochastic differential equations used in diffusion sampling. "SDE Euler-Maruyama sampler"
- Exponential moving average (EMA): A stabilization technique that maintains a smoothed version of model parameters during training. "exponential moving average (EMA) are applied"
- Flow-based transformers: Generative architectures based on normalizing flows implemented with transformer blocks. "flow-based transformers SiTs"
- Fréchet Inception Distance (gFID): A generation quality metric comparing statistics of generated and real images via Inception features. "Fréchet Inception Distance (gFID)"
- Gaussian mixture model (GMM): A probabilistic model used to assess latent space discrimination by fitting multiple Gaussian components. "fit a Gaussian mixture model (GMM)"
- gFID: FID computed on generated samples to measure image synthesis quality. "gFID score of 1.21 and 1.75"
- Gini coefficient: A statistical measure of distribution inequality used to quantify latent space uniformity. "Gini coefficients of data point distribution using kernel density estimation (KDE)"
- Gradient clipping: A training technique that limits gradient norms to prevent exploding gradients. "gradient clipping"
- I-JEPA: A self-supervised model (Joint-Embedding Predictive Architecture) used as a VFM alignment target. "I-JEPA Assran et al. (2023)"
- Inception Score (IS): A generation metric assessing image diversity and quality via a pretrained classifier’s predictions. "Inception Score (IS) Salimans et al. (2016)"
- IN-VAE: A VAE trained on ImageNet used as a baseline tokenizer. "IN-VAE Leng et al. (2025)"
- Kernel density estimation (KDE): A non-parametric method to estimate data distributions, used for computing Gini coefficients in latent space. "kernel density estimation (KDE)"
- KL divergence: A regularization term in VAEs that penalizes deviation from a prior distribution. "KL divergence loss"
- Latent Diffusion Models (LDMs): Diffusion models that operate in a compressed latent space for efficient high-resolution generation. "Latent Diffusion Models (LDMs) generate high-quality images"
- Latent space discrimination: An evaluation of how separable classes or attributes are within a model’s latent representations. "latent space discrimination"
- Latent space uniformity: A measure of how evenly latent representations are distributed across the space. "latent space uniformity"
- Linear probing: A technique that trains a linear classifier on frozen features to evaluate the separability of information. "linear probing on attribute prediction tasks"
- Mapper network: A learned non-linear module that transforms VAE latents to align with VFM representations. "a sophisticated non-linear mapper network"
- Masked image modeling: A self-supervised objective that reconstructs masked parts of images to learn robust features. "masked image modeling"
- Multilayer perceptron (MLP): A simple feedforward neural network used as a projector or mapper in alignment. "multilayer perceptron (MLP)"
- Number of function evaluations (NFE): The count of solver steps in diffusion sampling affecting speed and quality. "number of function evaluations (NFE) is set to 250"
- Patch embedding layer: A module that converts image patches into token embeddings for transformer processing. "a patch embedding layer"
- Precision and Recall (for generative models): Metrics assessing fidelity (precision) and coverage (recall) of generated data distributions. "Precision, and Recall Kynkäänniemi et al. (2019)"
- REPA: A training framework that aligns diffusion model representations with a frozen high-capacity encoder. "REPA Yu et al. (2025)"
- REPA-E: An end-to-end extension of REPA that backpropagates alignment loss into the VAE. "REPA-E Leng et al. (2025)"
- Reconstruction FID (rFID): FID computed between original and VAE-reconstructed images to assess reconstruction quality. "reconstruction FID (rFID)"
- Representation alignment: The practice of matching internal features of one model to another for guidance or regularization. "representation alignment loss"
- Representation gap: The mismatch between attribute-level VAE latents and high-level VFM semantics that must be bridged. "representation gap"
- SDE (stochastic differential equation): A formulation used in diffusion processes; solvers are employed for sampling. "SDE Euler-Maruyama solver"
- Semantic disentanglement: The property of encoding fine-grained, attribute-level information into separate, interpretable latent factors. "semantic disentanglement"
- Semantic gap: The difference in semantic richness between representations (e.g., model latents vs. VFM features). "semantic gap Yu et al. (2025)"
- Send-VAE: The proposed VAE optimized for semantic disentanglement via alignment to VFMs with a non-linear mapper. "Send-VAE"
- SiT: Scalable Interpolant Transformer, a flow/diffusion-based generative model architecture used downstream of the VAE. "SiT Ma et al. (2024)"
- Structural FID (sFID): A FID variant emphasizing structural aspects of images. "Structural FID (sFID) Nash et al. (2021)"
- Tokenizer (image tokenizer): A model (continuous or discrete) that maps images to compact latent or token spaces for generation. "image tokenizers such as Variational Autoencoders (VAEs)"
- Variational Autoencoder (VAE): A generative model that learns a latent distribution for reconstructing data with a KL-regularized objective. "Variational Autoencoders (VAEs)"
- Vision Foundation Models (VFMs): Large pretrained vision models whose representations guide alignment for VAEs or diffusion. "Vision Foundation Models (VFMs)"
- Vision Transformer (ViT): A transformer architecture for images that processes patch tokens with self-attention. "vision transformer (ViT)"
- VQGAN: A discrete tokenizer combining vector quantization with GAN training, often used in autoregressive generation. "VQGAN Esser et al. (2021)"
Collections
Sign up for free to add this paper to one or more collections.