Papers
Topics
Authors
Recent
Search
2000 character limit reached

Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning

Published 7 Oct 2026 in cs.CV and cs.AI | (2610.09450v1)

Abstract: Pixel-space diffusion models avoid the lossy VAE of latent models, which suggests an advantage on downstream tasks where fine-grained detail matters. We test this claim along both routes to a pixel-space backbone. We pretrain Iris-3B, a 3B-parameter pixel-space text-to-image transformer, from scratch through a 256→512→1024256\to512\to1024 curriculum, after first ablating the prediction target and representation alignment at $2562$ to decide what to scale. We also convert a pretrained latent model, FLUX.2 Klein base 4B, to pixel space. We fine-tune both families for monocular depth estimation and for image restoration/super-resolution. We find no significant improvement from using a pixel-space generative prior. Fine-tuned for depth with one matched direct-regression recipe, Iris-3B is level with the latent FLUX.2 Klein and the converted pixel FLUX.2 Klein falls behind it, and on 4×4\times DIV2K restoration neither pixel model beats a latent FLUX.2 Klein fine-tune, the converted one trailing it slightly. We document the recipes, the failure modes and the remaining confounds behind this negative result. Nevertheless, Iris-3B shows that pixel-space pretraining with the pixel-transformer (PiT) head of PixelDiT scales to 3B parameters and to text-to-image quality competitive with latent models, matching Qwen-Image on OneIG under the official evaluators at $10242$. We release its weights and training code in the hope that they help pave the way for further work on pixel-space generation.

Authors (2)

Summary

  • The paper investigates if removing the VAE bottleneck from latent diffusion models improves dense vision tasks but does not find significant downstream advantages.
  • Iris-3B, a 3B-parameter pixel-space text-to-image model, achieves competitive performance in image generation benchmarks with GenEval 0.798, DPG-Bench 86.5, and OneIG-English 0.540.
  • Downstream experiments using Iris-3B indicated that neither its comparison nor that of converted pixel-space models show a significant performance advantage over latent models for monocular depth estimation or image restoration tasks.

Research question and principal findings

The paper examines whether removing the VAE bottleneck from latent diffusion improves transfer to dense vision tasks whose outputs are spatially aligned with the input. The motivating premise is technically plausible: latent diffusion models compress images through an independently trained autoencoder, so their downstream representations and outputs are mediated by a lossy codec. Pixel-space models eliminate this interface and, in principle, permit joint optimization over the full-resolution image.

The study evaluates this premise through two routes. First, it trains Iris-3B, a 3B-parameter pixel-space text-to-image model from scratch. Second, it converts the 4B-parameter FLUX.2 Klein base model from latent space to pixel space. Both model families are then fine-tuned for monocular depth estimation and one-step image restoration. The central result is negative: the experiments do not show a significant downstream advantage for pixel-space generative priors. Iris-3B is approximately level with the latent FLUX.2 Klein model for depth, while the converted pixel model is consistently weaker than its latent counterpart. In 4×4\times restoration, neither pixel-space model outperforms the latent model.

The negative conclusion is appropriately qualified. Iris-3B has no latent twin with identical architecture, data, and compute, and the converted FLUX.2 model is not necessarily equivalent in generative quality to its fully pretrained latent parent. Each downstream comparison is also based on a single short fine-tuning run without statistical testing. Thus, the paper establishes that pixel-space pretraining is feasible and that the tested recipes do not yield an observable advantage; it does not establish that pixel-space representations are intrinsically inferior for dense prediction.

Iris-3B architecture and training

Iris-3B is a flow-matching transformer that operates directly on RGB pixels. At 102421024^2, the image is divided into 16×1616 \times 16 patches, producing a 64×6464 \times 64 token grid. The backbone uses a hybrid FLUX-style design consisting of eight dual-stream MMDiT blocks followed by sixteen single-stream blocks, with model width 2560. The architecture includes GQA, 2D axial RoPE, RMSNorm-based sandwich normalization, sigmoid output gating, and shared-bias adaptive layer normalization. The latter reduces parameterization relative to per-block modulation while preserving timestep conditioning.

A four-block Pixel Transformer (PiT) head converts patch-level trunk features into RGB pixels. The head applies per-pixel MLPs and performs cross-patch attention after compacting the pixels within each patch into a single token. This design is central to the paper’s scalability claim: it allows a large transformer trunk to reason over patches while retaining a pixel-level output interface. Text conditioning is provided by a frozen Qwen3-VL-4B-Instruct encoder, whose intermediate representations are fused through layerwise attention pooling and a two-block refiner.

Training proceeds through a 256→512→1024256 \rightarrow 512 \rightarrow 1024 curriculum, followed by supervised fine-tuning. The model uses rectified-flow training with vv-prediction and a shifted logit-normal noise schedule. The reported training budget is 375.8 million samples at 2562256^2, 166.9 million at $512$ with multiple aspect ratios, 48.6 million at $1024$, and 20.5 million samples for SFT. The $1024$ stage uses a reduced learning rate of 102421024^20 and global batch size 512.

The pretraining recipe combines Muon for hidden matrices with AdamW for embeddings, normalization parameters, and biases. A frozen DINOv2 teacher supplies representation-alignment supervision during the 102421024^21 stage, with the alignment loss removed at higher resolutions. The design therefore combines pixel-level flow matching, a large-patch transformer trunk, a specialized pixel decoder, and early representation alignment.

Ablation results at 102421024^22

The authors first conduct controlled ablations using a 1.3B PixelDiT reproduction at 102421024^23. All arms use the same data, model, batch size, sampling procedure, and evaluation protocol, with comparisons made at matched optimization steps. The ablations concern the prediction target and the representation-alignment objective.

The comparison between 102421024^24-prediction and 102421024^25-prediction does not reproduce the expected advantage of clean-image prediction. The two methods are tied on GenEval, while 102421024^26-prediction performs worse on DPG-Bench and FID. The result is not a perfectly isolated target comparison because the 102421024^27-prediction arm also uses JiT’s shifted timestep distribution, which emphasizes noisier examples. Nevertheless, the authors retain 102421024^28-prediction because it is stable and competitive under their setup.

Representation alignment produces a clearer result. REPA with a frozen DINOv2 teacher is the strongest overall configuration. It leads on GenEval at every reported milestone and remains competitive on DPG-Bench and CLIP score. DINOv3-based alignment reduces GenEval and worsens FID, while iREPA improves FID at intermediate checkpoints but trails the control on most semantic metrics. Self-Flow is substantially weaker throughout the experiment.

The implication for scaling is specific: the final Iris-3B recipe is not the result of a broad search over all pixel-space objectives, but of a constrained ablation in which DINOv2 REPA and 102421024^29-prediction were selected under a particular 1.3B, 16×1616 \times 160 regime. The transferability of this choice to other resolutions, model sizes, and datasets remains untested.

Figure 1

Figure 1: The 16×1616 \times 161- versus 16×1616 \times 162-prediction comparison at 16×1616 \times 163 under matched evaluation steps.

Figure 2

Figure 2: Representation-alignment ablations comparing REPA, iREPA, DINOv3 alignment, and Self-Flow.

Text-to-image performance

Iris-3B demonstrates that a pixel-space transformer can be scaled to 3B parameters and trained through one-megapixel generation without a VAE. Under official evaluators at 16×1616 \times 164, it obtains GenEval 0.798, DPG-Bench 86.52, LongText 0.857, and OneIG-English 0.540.

These results place Iris-3B above FLUX.1-dev on the reported metrics and approximately level with Qwen-Image on OneIG-English. It matches i1 on DPG-Bench but trails i1, Z-Image, and Qwen-Image on GenEval. The comparison is not fully uniform because i1 and Z-Image use prompt rewriting, whereas Iris-3B is evaluated on the original prompts. Iris-3B is strongest on single-object composition, two-object composition, and color binding, but weaker on spatial relations and counting. Long-form English text rendering is its clearest deficiency.

Model GenEval DPG-Bench LongText OneIG-English
FLUX.1-dev 0.660 83.8 0.607 0.434
i1 0.840 86.7 0.922 —
Z-Image 0.840 88.1 0.935 0.546
Qwen-Image 0.870 88.3 0.943 0.539
Iris-3B 0.798 86.52 0.857 0.540

The appropriate interpretation is not that pixel-space generation matches the strongest latent systems in every respect. Rather, Iris-3B provides evidence that the VAE can be removed without making high-resolution text-to-image synthesis uncompetitive at this scale. Its generation quality is sufficient to support the downstream investigation, although the paper explicitly notes that Iris-3B is not state of the art in raw generation quality.

Figure 3

Figure 3: One-megapixel samples from the PiT-converted FLUX.2 Klein model used in downstream experiments.

Latent-to-pixel conversion

The second experimental route converts FLUX.2 Klein base 4B by replacing its VAE with 16×1616 \times 165 RGB patch tokenization and a new pixel head. The conversion procedure exposes an important optimization issue: the latent transformer is trained on VAE embeddings, whereas raw pixel patches occupy a substantially different input distribution. A randomly initialized input projection therefore presents the pretrained trunk with an unfamiliar interface.

The authors report that, for FLUX.2 Klein, the best per-patch linear map from pixels explains only 14% of the variance of the corresponding latent embeddings. At a global batch size of eight, the conventional conversion recipe produces prompt-independent patch-grid noise even after 1,000 steps. Two interventions substantially improve early conversion:

  1. Gain-matched ridge initialization fits a linear projection from RGB patches to the parent model’s latent-token embeddings, with regularization selected to match the gain of the latent path rather than maximize held-out 16×1616 \times 166.
  2. Trunk learning-rate scaling reduces the FLUX trunk learning rate to 16×1616 \times 167 times the head rate, compensating for the trunk’s smaller weight RMS and preventing disproportionately large relative updates.

Either intervention reduces the 1,000-step loss, but neither alone produces coherent samples. Their combination produces photorealistic, prompt-aligned images within 1,000 steps. The authors do not test whether this advantage persists at the global batch size of 128 used in another conversion study, so the observed sensitivity may depend on small-batch optimization.

Figure 4

Figure 4: Early FLUX.2 Klein conversion probes after 1,000 steps under different input initializations and trunk learning rates.

The production conversion is trained for 30,000 steps with a global batch size of 112. The PiT-head version at 30K steps is used for downstream experiments. However, the paper does not report GenEval, DPG-Bench, FID, or another standardized generative comparison for the converted model. This omission matters: the downstream deficit of the converted model could reflect incomplete conversion rather than a disadvantage of pixel-space computation.

Monocular depth estimation

The depth experiment compares three models under one direct-regression recipe: latent FLUX.2 Klein, pixel-converted FLUX.2 Klein, and Iris-3B. Each model receives an RGB image and predicts relative log depth in one deterministic forward pass. The depth loss combines masked L1 regression with a gradient L1 term weighted by five. All models are fine-tuned for 10,000 steps using AdamW, batch size 32, and learning rate 16×1616 \times 168.

The evaluation covers NYUv2, KITTI, ETH3D, ScanNet, DIODE, and the soft edge error on Hypersim. The latent model has the best mean 16×1616 \times 169 score, while Iris-3B has the best mean AbsRel and the lowest Hypersim edge error. The converted pixel model is weaker on nearly every benchmark.

Parent Mean AbsRel 64×6464 \times 640 Mean 64×6464 \times 641 64×6464 \times 642 Hypersim SEE 64×6464 \times 643
FLUX.2 Klein latent 0.072 0.947 0.528
FLUX.2 Klein pixel 0.081 0.934 0.526
Iris-3B 0.071 0.946 0.496

Iris-3B performs best on KITTI and DIODE and has the strongest Hypersim edge score, but the latent model leads on NYUv2, ETH3D, and ScanNet. The aggregate metrics therefore support equivalence rather than superiority. The converted model falls behind the latent model on every benchmark except KITTI, where the two are approximately tied.

Figure 5

Figure 5: Qualitative depth predictions from latent FLUX.2 Klein, pixel-converted FLUX.2 Klein, and Iris-3B on Hypersim frames.

The strongest artifact-level finding concerns patch boundaries. The latent model has a patch-grid ratio close to one, indicating no measurable excess curvature near 64×6464 \times 644 boundaries. The pixel models exhibit elevated ratios: Iris-3B reaches 2.14 on Hypersim, 1.61 on KITTI, 2.21 on ETH3D, and 2.25 on DIODE; the converted model ranges from 1.31 to 1.44. The artifact follows directly from the PiT design: neighboring patches are not fully mixed at pixel resolution, so independently decoded patch regions can disagree at their borders.

Figure 6

Figure 6: Depth predictions on five zero-shot benchmarks after 10,000 steps of direct regression.

This result undermines a central motivation for pixel-space dense prediction. Eliminating the VAE does not automatically improve boundary fidelity when the pixel decoder itself imposes a non-overlapping patch structure. A latent decoder may lose information globally yet provide overlapping receptive fields that enforce spatial smoothness locally.

Restoration and super-resolution

The restoration experiment adapts each model to one-step 64×6464 \times 645 super-resolution using the HYPIR-style objective: pixelwise reconstruction, VGG-based LPIPS, and adversarial supervision from a frozen OpenCLIP ConvNeXt-XXL trunk. The FLUX.2 latent and pixel models receive matched training data, losses, and schedules for 30,000 steps. Iris-3B is trained separately and is therefore not a matched comparison.

On DIV2K validation images degraded with the native Real-ESRGAN pipeline, the latent FLUX.2 model is better than the converted pixel model on PSNR, SSIM, NIQE, MUSIQ, and DeQA, while the pixel model is essentially tied on LPIPS. Iris-3B achieves the highest MUSIQ score but is worse on the remaining metrics than the latent FLUX.2 model.

Model PSNR 64×6464 \times 646 SSIM 64×6464 \times 647 LPIPS 64×6464 \times 648 NIQE 64×6464 \times 649 MUSIQ 256→512→1024256 \rightarrow 512 \rightarrow 10240 DeQA 256→512→1024256 \rightarrow 512 \rightarrow 10241
FLUX.2 Klein latent 20.99 0.557 0.275 3.26 67.11 4.06
FLUX.2 Klein pixel 20.87 0.504 0.274 4.11 67.03 4.04
Iris-3B 20.68 0.481 0.292 3.94 68.01 3.87

The authors correctly avoid interpreting these differences as statistically reliable losses because each arm is a single run and no significance test is reported. The defensible conclusion is narrower: pixel-space adaptation provides no measured restoration advantage under this training protocol. The latent model’s advantage in SSIM and NIQE is notable because these metrics directly test structural fidelity and perceptual image quality, respectively, while Iris-3B’s MUSIQ lead indicates that no single metric establishes an unambiguous ranking.

The comparisons against external restoration systems show that the three adapted models occupy a competitive but heterogeneous region of the benchmark. SeeSR and Real-ESRGAN lead on PSNR and SSIM, while the diffusion-derived models perform better on some perceptual metrics. These comparisons primarily contextualize restoration quality; they do not isolate representation space because the external methods use different architectures, priors, and training procedures.

Figure 7

Figure 7: 256→512→1024256 \rightarrow 512 \rightarrow 10242 restoration on DIV2K inputs with native Real-ESRGAN degradations.

Figure 8

Figure 8: Additional DIV2K restoration examples from the same model family.

Figure 9

Figure 9: Qualitative restoration results on real-world low-quality images without ground truth.

Limitations and open questions

The principal limitation is incomplete experimental matching. Iris-3B differs from FLUX.2 Klein in parameter count, architecture, pretraining data, compute, and training recipe, so its comparison with the latent model cannot identify the effect of representation space alone. The matched FLUX.2 comparisons are cleaner, but they compare a fully pretrained latent parent against a pixel model obtained through a relatively short conversion. The converted model’s unmeasured generative quality leaves open whether its downstream deficit is caused by pixel-space modeling or by an incomplete transfer of the latent prior.

The downstream studies also use a single run per condition, short fine-tuning budgets, and no significance tests. The reported differences should therefore be treated as point estimates rather than robust estimates of population-level performance. Iris-3B’s restoration experiment is further confounded by differences in batch schedule, checkpointing, and inference-time processing, including wavelet color correction and tiling.

A second limitation concerns the pixel decoder. The paper’s central hypothesis concerns the information loss of the VAE, but the PiT head introduces a separate locality constraint that is visible as patch-grid artifacts in depth. Consequently, the experiments do not compare an unconstrained pixel representation with a latent representation; they compare a VAE-mediated latent decoder against a patch-local pixel decoder. Overlapping convolutional or PixelShuffle output heads could alter the dense-prediction result, but this remains an untested question within the paper.

The most important open experimental question is whether a matched latent and pixel pair, trained from scratch on identical data with identical compute and equivalent decoder capacity, would show the same lack of advantage. A second is whether conversion quality can be brought close enough to the latent parent that the representation-space comparison becomes valid. A third is whether pixel space is more beneficial for tasks requiring exact image preservation, such as reference-based editing, rather than for depth and restoration trained with task-specific output heads.

Conclusion

The paper establishes two results with different levels of strength. First, Iris-3B shows that a 3B-parameter PiT-based transformer can be pretrained directly on pixels through 256→512→1024256 \rightarrow 512 \rightarrow 10243 text-to-image generation and achieve competitive benchmark performance, including GenEval 0.798, DPG-Bench 86.52, and OneIG-English 0.540. Second, the downstream experiments provide no evidence that a pixel-space generative prior is intrinsically superior to a latent prior for monocular depth or one-step restoration under the tested recipes.

The converted FLUX.2 model underperforms its latent counterpart, while Iris-3B is approximately comparable to the latent model for depth and somewhat weaker for restoration. These findings are limited by conversion quality, unmatched model families, short single-run fine-tuning, and patch-grid artifacts. The paper’s main contribution is therefore not a definitive rejection of pixel-space transfer, but a careful negative result showing that removal of the VAE alone does not produce the expected downstream benefit.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

1. What is this paper about?

This paper studies two ways of making AI image generators:

  • Latent-space models, which first shrink an image into a smaller hidden version before working on it.
  • Pixel-space models, which work directly with the image’s pixels.

Latent models are usually faster and cheaper to train. However, shrinking an image can remove tiny details, such as thin lines, sharp edges, or small textures. The researchers wanted to know whether working directly with pixels would help AI perform tasks that need fine detail.

They created and tested a new pixel-space model called Iris-3B. They also changed an existing latent model, FLUX.2 Klein, into a pixel-space model so the two versions could be compared.

2. What questions did the researchers ask?

The paper focuses on several main questions:

  1. Can a large pixel-space model be trained successfully? In other words, can an AI learn to create good images while looking directly at pixels instead of using a compressed image representation?
  2. Does pixel-space training improve image generation? The researchers compared Iris-3B with other modern image generators.
  3. Do pixel-space models perform better on tasks requiring precise details? They tested:
    • Depth estimation: predicting how far away different parts of an image are.
    • Image restoration and super-resolution: improving blurry or damaged images.
  4. Is it better to convert an existing latent model into pixel space, or train a pixel-space model from the beginning?

3. How did the researchers conduct the study?

Training Iris-3B

The researchers trained Iris-3B, a model with about 3 billion adjustable numbers, called parameters. These parameters are like tiny settings that the AI changes while learning.

Iris-3B works directly with RGB pixels. To make training manageable, it divides an image into small square pieces, called patches. For example, a 1024×10241024 \times 1024 image is split into many 16×1616 \times 16 pixel patches.

The model was trained in stages:

  1. Images at 256×256256 \times 256 pixels
  2. Images at 512×512512 \times 512 pixels
  3. Images at 1024×10241024 \times 1024 pixels

This is similar to teaching someone to draw: first they practice on small, simple pictures, then gradually move to larger and more detailed ones.

During training, the researchers added noise to images and taught the model how to remove that noise. This is the basic idea behind diffusion models. A useful analogy is giving the AI a messy, partly erased picture and asking it to reconstruct the original.

Testing different training choices

Before training the full Iris-3B model, the researchers tested smaller versions to decide how the model should learn. They compared:

  • Different ways of predicting the clean image or the direction needed to remove noise.
  • Different methods for helping the model learn useful visual features.

One method, called REPA, encouraged the model’s internal features to resemble features learned by another image-understanding system. This is similar to having an experienced teacher guide a student’s learning.

Converting FLUX.2 Klein

The researchers also took the existing FLUX.2 Klein model, which normally works in latent space, and removed its image-compression system.

They replaced it with a system that reads pixel patches directly. However, this caused problems at first because the original model had been trained to understand compressed representations, not raw pixels.

To fix this, they:

  • Carefully initialized the new pixel-input layer.
  • Adjusted the learning speed of different parts of the model.

This helped the converted model produce reasonable images more quickly.

Testing other tasks

The researchers fine-tuned the models for two tasks.

Depth estimation means predicting the distance of objects in a photograph. For example, the model might decide that a nearby tree is closer than a mountain in the background.

Image restoration and super-resolution means turning a low-quality image into a clearer, larger one. The researchers used images that had been deliberately damaged or made blurry, then checked how well each model restored them.

They compared the pixel-space models with latent-space models using standard test datasets and measurements.

4. What did the researchers discover?

Iris-3B can generate good images

The first important finding was that pixel-space training is possible at a large scale.

Iris-3B produced images that were competitive with several strong image generators. It performed especially well on:

  • Creating one or two requested objects
  • Following color instructions
  • Producing images at 1024×10241024 \times 1024 resolution

Its results were similar to some larger models on the OneIG benchmark.

However, Iris-3B was weaker at:

  • Counting objects
  • Placing objects in exact positions
  • Generating long, accurate text inside images

So, pixel-space models can work well, but they are not automatically better at every part of image generation.

Pixel space did not clearly improve depth estimation

The researchers expected pixel-space models to be better at depth estimation because depth maps need accurate object boundaries and thin details.

That did not happen.

  • Iris-3B performed about as well as the latent FLUX.2 Klein model.
  • The converted pixel version of FLUX.2 Klein performed worse than the original latent version.

This means that simply using raw pixels did not provide a clear advantage.

Pixel space did not improve image restoration

The same pattern appeared in super-resolution and restoration.

The latent FLUX.2 Klein model was slightly better than its pixel-space converted version on most measurements. Iris-3B was competitive, but it did not clearly beat the latent model.

Therefore, the central idea of the paper was not confirmed: working directly with pixels did not automatically make the models better at tasks needing fine detail.

Pixel-space models created faint patch lines

The pixel models sometimes produced faint grid-like patterns. These patterns appeared because the models processed images as separate patches.

Imagine making a picture from many square tiles. If each tile is colored independently, tiny mismatches can appear along the borders. The same thing happened here: neighboring patches sometimes did not join perfectly.

These artifacts were not huge, but they were especially noticeable in detailed predictions such as depth maps.

Conversion may be the problem

The converted FLUX.2 Klein model performed worse than its latent version. The researchers think this may not prove that pixel space is bad.

Instead, the conversion process may have produced a weaker model. The original latent model had been trained for a long time, while the pixel version was created through a shorter adaptation process.

This makes the comparison imperfect.

5. Why are these results important?

The paper’s main result is a negative result: pixel-space models did not show a clear advantage over latent-space models for depth estimation or image restoration.

That is still useful. It tells researchers that removing the image-compression step is not enough by itself. Other parts of the model may matter more, including:

  • How the model divides images into patches
  • How neighboring patches communicate
  • How long the model is trained
  • The quality and size of the training data
  • The method used to convert a latent model into pixel space

The study also shows that pixel-space models can be made large and useful. Iris-3B demonstrates that a model can generate competitive images without using a separate image-compression system.

6. Possible future impact

The researchers released Iris-3B’s model weights and training code so other scientists can build on their work.

Future improvements could include:

  • Designing output layers that blend information across patch borders
  • Removing the faint grid patterns
  • Training pixel-space models with more data and computing power
  • Comparing pixel and latent models that are trained under exactly the same conditions
  • Testing pixel-space models for image editing, where preserving every unchanged pixel may be especially important

Overall, the paper suggests a balanced conclusion: pixel-space image generation is practical and promising, but it is not automatically better than latent-space generation. More careful designs and fairer experiments are needed to discover whether directly using pixels provides a real advantage.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

  • The causal effect of pixel-space representations remains unidentified. The matched FLUX.2 Klein comparison controls for the parent model but compares a fully pretrained latent model with a briefly converted pixel model, so it cannot determine whether pixel space itself helps or whether conversion quality is the limiting factor.
  • There is no matched latent counterpart for Iris-3B. Comparisons involving Iris-3B conflate representation space with differences in model architecture, parameter count, pretraining data, training compute, optimization, and generation quality.
  • The study does not establish whether stronger latent-to-pixel conversion can close the downstream gap. The conversion uses a 30K-step model, and the authors do not evaluate longer conversion schedules, larger conversion batches, alternative data mixtures, or more extensive adaptation of the trunk and input interface.
  • The quality of the converted FLUX.2 Klein prior is not measured quantitatively. GenEval, DPG-Bench, FID, and related metrics are not reported for the converted models, preventing assessment of how much generative capability was lost during conversion.
  • The effectiveness of gain-matched ridge initialization is not tested at the larger batch size used in prior work. The initialization experiments use a global batch of 8, whereas the production conversion uses 112 and related work uses 128; its robustness across batch sizes and training regimes is unresolved.
  • The prediction-target ablation does not isolate xx- versus vv-prediction. The xx-prediction model also uses a different logit-normal timestep distribution, so the reported performance difference cannot be attributed solely to the prediction target.
  • The pretraining ablations are limited to a 1.3B model at 2562256^2. It is unknown whether the conclusions about prediction targets and representation alignment transfer to Iris-3B, higher resolutions, different patch sizes, or the full 256-to-1024 curriculum.
  • The representation-alignment study lacks comparisons with alignment-free training at the final scale. REPA is retained for the 256px stage but removed later; the paper does not determine whether alignment during 512px or 1024px training affects downstream transfer or generation quality.
  • The effects of individual architectural choices are not disentangled. Iris-3B changes patch size, PiT head, attention structure, normalization, modulation, RoPE, and text conditioning relative to other models, but no systematic ablation identifies which components are responsible for quality, efficiency, or patch artifacts.
  • The training-data contribution is unclear. The dataset composition, source proportions, filtering, synthetic-image fraction, and rendered-text subset are not characterized sufficiently to determine how each affects generation quality or downstream transfer.
  • The influence of synthetic pretraining data is unresolved. Because the 1024px stage is dominated by non-photographic content and includes generated images, the study does not test whether this distribution improves or harms dense prediction, restoration, photorealism, or generalization.
  • The reported downstream results have limited statistical reliability. Each depth and restoration arm is a single run at a short budget, with no repeated seeds, confidence intervals, or significance tests, so small performance differences cannot be distinguished from optimization noise.
  • The downstream training budget may be insufficient for fair convergence comparisons. The models receive only 10K or 30K fine-tuning steps, and the paper does not provide learning curves or convergence studies showing whether pixel-space models require different optimization schedules or longer adaptation.
  • The restoration comparison is not fully matched for Iris-3B. Iris-3B uses a different batch schedule, checkpoint handling, EMA setting, wavelet color correction, and tiling procedure from the FLUX.2 Klein models, making its restoration results unsuitable for isolating representation-space effects.
  • Only one restoration degradation model and one scale factor are evaluated. The conclusions may not generalize beyond 4×4\times Real-ESRGAN degradations to real camera noise, blur, compression, denoising, deblurring, inpainting, or other super-resolution scales.
  • The depth evaluation is restricted to synthetic training data and a limited task formulation. The models are trained on Hypersim and Virtual KITTI 2 with relative log-depth targets, leaving open whether pixel-space backbones behave differently under real-image supervision, metric-depth prediction, domain adaptation, or alternative depth losses.
  • The study does not evaluate other dense prediction tasks. Semantic segmentation, surface normals, optical flow, correspondence, pose estimation, panoptic prediction, and fine-grained keypoint tasks could reveal advantages or disadvantages not captured by depth and restoration.
  • The claimed benefit for fine spatial detail is not directly tested with task-specific measurements. Although edge error and a patch-grid metric are reported for depth, there is no broad analysis of boundary localization, thin-structure recall, high-frequency preservation, or spatial frequency response across tasks.
  • The cause of patch-grid artifacts is not experimentally isolated. The paper attributes the artifacts to non-overlapping patch processing, but does not compare alternative patch sizes, overlapping tokenization, convolutional heads, PixelShuffle heads, or cross-patch pixel attention in a controlled downstream experiment.
  • It is unknown whether patch artifacts persist at different resolutions and aspect ratios. The seam analysis uses selected datasets and 60 images per benchmark; artifact behavior across image sizes, bucket shapes, sampling settings, guidance scales, and unseen resolutions is not established.
  • No evaluation measures the computational trade-off comprehensively. The paper does not report training FLOPs, memory use, latency, throughput, energy, or sampling cost for pixel and latent models under matched hardware and quality levels.
  • The comparison does not control for decoder architecture. The latent model benefits from an overlapping VAE decoder, while pixel models use a PiT head with limited cross-patch interaction; therefore, the experiments do not distinguish the representation space from the decoder’s spatial inductive bias.
  • The role of the frozen VAE in downstream latent models is not examined. It remains unclear whether replacing, fine-tuning, or bypassing the latent model’s VAE decoder would change the observed depth and restoration gap.
  • The model’s robustness to prompt and conditioning variations is unexplored. Downstream experiments use an empty prompt, while the generative evaluations use fixed CFG settings; the effect of text prompts, image references, classifier-free guidance, and multimodal conditioning on transfer remains unknown.
  • The generalization of Iris-3B beyond English text is untested. The training data and evaluations emphasize English, leaving multilingual text rendering, non-English prompts, and culturally diverse image concepts unresolved.
  • Long-text and spatial-relation weaknesses are not investigated mechanistically. Iris-3B performs relatively poorly on long-text rendering, counting, and positional relations, but the paper does not determine whether these failures arise from the text encoder, token fusion, data composition, resolution, or the pixel-space generator.
  • Reference-based generation and exact image editing remain unvalidated. The paper proposes editing as a setting where pixel space may have an advantage, but provides no experiments on reconstruction fidelity, masked editing, identity preservation, or reference-image conditioning.
  • The assumption that pixel space preserves details unavailable to latent models is not directly quantified. The paper does not measure which frequencies, textures, or structures are lost by the FLUX.2 VAE and whether those losses are actually relevant to the evaluated downstream tasks.
  • The relationship between generative quality and downstream transfer is unclear. The study does not test whether models with matched generation metrics yield different dense-prediction performance, or whether stronger text-to-image priors consistently produce better task-specific representations.
  • The conclusions may not generalize to larger models or alternative pixel-space architectures. Iris-3B is the only from-scratch pixel model evaluated, and the findings may change with larger parameter counts, different pixel decoders, smaller patches, or models trained with substantially more data and compute.
  • Reinforcement learning and few-step distillation are proposed but not evaluated. It remains unknown whether improving Iris-3B’s generation quality through post-training would also improve its usefulness as a downstream backbone or alter the comparison with latent models.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Los resultados permiten aplicaciones prácticas principalmente como modelo generativo abierto, infraestructura de investigación y punto de comparación frente a modelos latentes. El trabajo no demuestra una ventaja general del espacio de píxeles para tareas densas, por lo que las implementaciones actuales deben validarse frente a modelos latentes equivalentes.

  • Generación de imágenes de alta resolución para prototipos y productos creativos — software creativo, marketing, diseño y medios.
    • generación de conceptos para campañas publicitarias;
    • creación de storyboards y material visual preliminar;
    • diseño de interfaces, productos, personajes y escenarios;
    • generación de imágenes sintéticas para pruebas de aplicaciones visuales.
    • Dependencias: disponibilidad de GPU, latencia aceptable de inferencia, licencia y condiciones de uso de los datos de entrenamiento, y mitigación de sesgos o contenido no deseado. Su rendimiento es más débil en conteo, relaciones espaciales y texto largo, por lo que requiere revisión humana.
  • Despliegue local o privado de generación de imágenes — empresas, laboratorios y sectores regulados. La disponibilidad de los pesos y del código de entrenamiento permite construir servicios internos sin depender necesariamente de una API externa. Esto puede ser útil para organizaciones que no desean enviar imágenes o prompts a proveedores externos. Dependencias: Iris-3B sigue siendo un modelo grande; el despliegue en dispositivos de consumo o servidores con recursos limitados requerirá cuantización, poda, compilación o destilación. También deben revisarse los requisitos de seguridad, procedencia de datos y derechos de autor.
  • Base experimental para investigación en modelos generativos pixel-space — academia e I+D industrial.
    • entrenamiento progresivo de $256$ a $512$ y $1024$ píxeles;
    • flujo rectificado y vv-prediction;
    • alineamiento REPA con características DINOv2;
    • arquitecturas híbridas con bloques MMDiT y una cabeza PiT;
    • entrenamiento con múltiples relaciones de aspecto.
    • Dependencias: reproducir los resultados requiere grandes volúmenes de pares imagen-texto, infraestructura distribuida y una configuración de optimización cercana a la del artículo. Las ablations no aíslan perfectamente todos los factores; por ejemplo, la comparación entre xx-prediction y vv-prediction también cambia la distribución de tiempos de ruido.
  • Conversión acelerada de modelos latentes a modelos pixel-space — plataformas de modelos y compresión/adaptación de modelos. El procedimiento de conversión de FLUX.2 Klein muestra un flujo práctico para sustituir el VAE por tokenización de parches RGB. La inicialización mediante una regresión ridge con ganancia controlada y el ajuste de la tasa de aprendizaje del tronco pueden emplearse como receta inicial para conversiones similares. Dependencias: la conversión no es automática ni garantiza preservar la calidad del modelo original. La relación lineal entre parches RGB y embeddings latentes fue limitada —el modelo reporta solo 14 % de varianza explicada en una configuración— y el procedimiento necesita reentrenamiento. Además, el artículo no mide exhaustivamente la calidad generativa del modelo convertido.
  • Benchmarking y selección informada de backbones para visión densa — equipos de visión por computador y compras tecnológicas. Los resultados ofrecen una regla práctica: no debe elegirse un backbone pixel-space únicamente por la hipótesis de que conserva mejor el detalle. En profundidad monocular y restauración 4×4\times, los modelos pixel-space no superaron de forma consistente al modelo latente; Iris-3B fue comparable en profundidad y el modelo convertido quedó por detrás. Dependencias: las conclusiones se basan en ejecuciones únicas, presupuestos cortos y comparaciones parcialmente confusas. Antes de una decisión de producción se necesitan pruebas con el propio dominio, más semillas, presupuestos equivalentes y métricas de robustez.
  • Estimación monocular de profundidad para prototipos de robótica y realidad aumentada — robótica, vehículos, AR/VR y análisis 3D. Iris-3B puede adaptarse mediante regresión directa a partir de una imagen RGB y producir profundidad en una sola pasada. En los experimentos fue competitivo con FLUX.2 Klein latente y obtuvo el menor error de borde en Hypersim, además de mejores resultados relativos en KITTI y DIODE. Dependencias: el modelo no demostró superioridad estadísticamente significativa y fue entrenado principalmente con datos sintéticos de Hypersim y Virtual KITTI 2. Su transferencia a cámaras reales, condiciones nocturnas, lluvia, sensores específicos o escenas industriales debe validarse. Los artefactos de cuadrícula de 16×1616\times16 pueden perjudicar límites finos y mediciones geométricas.
  • Restauración y super-resolución como herramienta de mejora visual — fotografía, archivo digital, inspección visual y contenido multimedia.
    • ampliación de fotografías;
    • recuperación de imágenes comprimidas o desenfocadas;
    • mejora de material audiovisual antiguo;
    • preprocesamiento para sistemas de reconocimiento visual.
    • Dependencias: el estudio no muestra ventaja frente al FLUX.2 latente y las métricas dependen de la degradación sintética utilizada. En contextos forenses, médicos o históricos, la generación de detalles plausibles puede introducir información falsa; por tanto, debe distinguirse claramente entre restauración fiel y mejora perceptual.
  • Generación de datos sintéticos para entrenamiento visual — academia, robótica, inspección industrial y vehículos autónomos. Iris-3B puede producir imágenes condicionadas por texto para ampliar conjuntos de datos, crear variaciones de iluminación, composición y aspecto, o generar casos raros para pruebas preliminares. Dependencias: la calidad limitada en posición, conteo y texto largo reduce su utilidad para etiquetas geométricamente exactas. Las imágenes sintéticas deben filtrarse, verificarse y etiquetarse como sintéticas; no sustituyen automáticamente a datos reales representativos.
  • Flujos de trabajo educativos y de investigación reproducible — educación superior y laboratorios.
    • difusión y flow matching;
    • tokenización de imágenes;
    • entrenamiento progresivo por resolución;
    • alineamiento de representaciones;
    • evaluación de modelos generativos;
    • comparación entre representaciones latentes y píxel a píxel.
    • Dependencias: el coste computacional completo es elevado. Para cursos o laboratorios docentes convendría utilizar checkpoints reducidos, inferencia precomputada y experimentos de baja resolución.

Long-Term Applications

Las siguientes aplicaciones son plausibles, pero dependen de resolver limitaciones identificadas por el artículo: artefactos de cuadrícula, coste de inferencia, calidad desigual en relaciones espaciales, conversión incompleta desde modelos latentes y ausencia de comparaciones totalmente controladas.

  • Edición y generación basada en referencias sin VAE — software creativo, publicidad, diseño de producto y edición audiovisual. Un modelo pixel-space podría editar una imagen preservando con mayor exactitud las regiones no modificadas, intercambiar objetos, cambiar fondos o generar variaciones condicionadas por imágenes de referencia. Esta es una aplicación especialmente alineada con la motivación del trabajo: evitar la pérdida de información introducida por un autoencoder. Desarrollo necesario: añadir datos de edición, máscaras, pares antes-después y referencias limpias; entrenar mecanismos de condicionamiento de imagen; y comparar con un modelo latente entrenado con los mismos datos y presupuesto. También debe eliminarse la cuadrícula de parches para evitar discontinuidades visibles.
  • Modelos unificados de comprensión, generación y edición de imágenes y vídeo — agentes multimodales, robótica y herramientas creativas. La interfaz directa mediante parches RGB podría integrarse en sistemas que entiendan imágenes, generen contenido, describan escenas y realicen transformaciones visuales dentro de una arquitectura común. Desarrollo necesario: ampliar el entrenamiento a vídeo, comprensión visual, instrucciones de edición y memoria temporal. El coste cuadrático de procesar más píxeles y fotogramas es una dependencia crítica, al igual que la necesidad de controlar consistencia temporal.
  • Percepción visual de alta precisión para robótica y vehículos autónomos — robótica móvil, drones y conducción asistida. Si se solucionan los artefactos de borde y se mejora la robustez fuera de distribución, un backbone pixel-space podría utilizarse para profundidad, mapas de ocupación, segmentación, flujo óptico o mapas de puntos con mayor alineación espacial. Desarrollo necesario: entrenar directamente objetivos geométricos, incorporar datos reales y sensores multimodales, evaluar incertidumbre y garantizar tiempos de respuesta deterministas. La ausencia de una mejora clara sobre modelos latentes implica que la adopción debe justificarse mediante pruebas específicas, no por la representación pixel-space en sí misma.
  • Restauración profesional y reconstrucción científica de imágenes — patrimonio cultural, astronomía, microscopía y teledetección. Un modelo entrenado con objetivos y degradaciones específicas podría recuperar imágenes de baja resolución o dañadas sin pasar por un VAE. La posibilidad de operar directamente en píxeles podría ser valiosa cuando la preservación de estructuras pequeñas sea prioritaria. Desarrollo necesario: utilizar degradaciones físicas y datos calibrados del dominio, añadir restricciones de fidelidad y cuantificar incertidumbre. Para ciencia o patrimonio, no basta con mejorar métricas perceptuales: el sistema debe evitar inventar texturas y conservar mediciones reproducibles.
  • Restauración médica y apoyo al diagnóstico por imagen — salud. A largo plazo, la arquitectura podría adaptarse a super-resolución, eliminación de ruido o reconstrucción de imágenes médicas, donde los detalles finos son clínicamente relevantes. Dependencias críticas: validación clínica, datos anotados y regulados, preservación de señales patológicas, auditorías de sesgo y aprobación normativa. Los resultados actuales no justifican uso diagnóstico: fueron obtenidos en fotografías y benchmarks generales, y un modelo generativo puede crear detalles inexistentes.
  • Destilación a modelos de pocos pasos o de una sola pasada — aplicaciones móviles, edge computing y servicios interactivos. La generación directa en píxeles podría combinarse con destilación, cuantización y arquitecturas eficientes para producir herramientas de generación o restauración con baja latencia. El propio artículo identifica la destilación de pocos pasos como una dirección futura. Desarrollo necesario: preservar calidad a 102421024^2 con menos pasos, reducir memoria y energía, y medir la latencia en hardware real. El modelo actual usa hasta 100 pasos en evaluación generativa, lo que limita su uso interactivo inmediato.
  • Cabezas de salida con receptividad espacial solapada — infraestructura de visión densa y generación de imágenes. Sustituir la cabeza PiT por convoluciones, PixelShuffle u otros decodificadores que mezclen información entre parches puede reducir las discontinuidades de 16×1616\times16 observadas en profundidad y generación. Esto podría mejorar simultáneamente texturas, bordes, mapas de profundidad y super-resolución. Desarrollo necesario: medir el compromiso entre pérdida de entrenamiento, memoria, velocidad y calidad; evaluar diferentes tamaños de parche; e incluir métricas explícitas de continuidad espacial.
  • Preentrenamiento de backbones generales para múltiples tareas visuales — plataformas de IA empresarial y modelos fundacionales. Un Iris mejorado podría servir como backbone común para generación, profundidad, restauración, segmentación, estimación de puntos y edición, reduciendo el número de modelos especializados. Dependencias: se requieren datasets multitarea, objetivos de entrenamiento mejor calibrados y comparaciones con backbones visuales no generativos. El estudio muestra que el prior generativo por sí solo no garantiza mejores resultados downstream; la utilidad dependerá de cómo se incorporen las tareas y sus representaciones.
  • Políticas y estándares para evaluar modelos pixel-space — política tecnológica, auditoría y contratación pública.
    • comparaciones con modelos latentes de igual tamaño y presupuesto;
    • métricas de continuidad entre parches;
    • pruebas de robustez por dominio;
    • análisis de alucinación en restauración;
    • múltiples semillas y pruebas de significancia;
    • documentación de datos, degradaciones y condiciones de conversión.
    • Dependencias: consenso entre investigadores, disponibilidad de benchmarks controlados y métodos de auditoría reproducibles. Las conclusiones del artículo son prometedoras como guía metodológica, pero no constituyen por sí solas un estándar regulatorio.
  • Aplicaciones de preservación exacta de contenido en edición sensible — documentación legal, archivos, periodismo y flujos de aprobación. Si la edición pixel-space consigue mantener sin cambios las regiones no editadas, podría utilizarse para workflows donde se necesita comprobar qué partes de una imagen fueron modificadas. Desarrollo necesario: mecanismos de edición con garantías de identidad, marcas de procedencia, trazabilidad criptográfica y comparación píxel a píxel. La capacidad generativa actual de Iris-3B no ofrece todavía esas garantías, especialmente debido a los artefactos de parches y a la posible reconstrucción de detalles.
View Paper Prompt  View All Prompts 

Glossary

  • AdaLN: Adaptive layer normalization, in which normalization parameters are modulated conditionally, often per transformer block. “Timestep modulation replaces the per-block adaLN of DiT”
  • Adversarial loss: A loss produced by a discriminator or critic to encourage outputs that appear realistic. “the loss combines a pixel, a perceptual and an adversarial term”
  • Affine-invariant depth estimation: Depth prediction evaluated up to an unknown scale and shift, rather than requiring absolute metric depth. “Marigold~\cite{marigold} fine-tunes a latent text-to-image model into an affine-invariant depth estimator”
  • Axial RoPE: Rotary positional encoding applied separately along different spatial axes. “Image tokens use 2D axial RoPE”
  • Backbone: A pretrained core network whose representations are adapted to downstream tasks. “Pretrained generators are increasingly valued for exactly that reason, as backbones for other vision tasks”
  • CFG (classifier-free guidance): A sampling method that strengthens conditioning by combining conditional and unconditional model predictions without a separate classifier. “drop the caption for 10\% of samples to enable CFG”
  • Convolutional projector: A convolutional network that maps model features into another representation space, often while preserving spatial structure. “iREPA~\cite{irepa}, which adds a convolutional projector and spatial normalization”
  • Curriculum learning: A training strategy that presents progressively more difficult or higher-resolution data or tasks. “We train in three resolution stages”
  • DINOv2: A self-supervised vision transformer used to provide pretrained visual feature representations. “The control is REPA~\cite{repa} with a frozen DINOv2 teacher”
  • Distillation: Training a model to reproduce the behavior of another model, usually to reduce inference cost. “OSEDiff~\cite{osediff} distills restoration into a single step”
  • Dual-stream transformer: A transformer architecture in which different modalities retain separate processing streams or parameters before interaction. “8 dual-stream MMDiT blocks”
  • EMA (exponential moving average): A smoothed version of model parameters obtained by exponentially averaging weights during training. “We keep an EMA of the weights with decay 0.9999”
  • Flow matching: A generative-training framework that learns a velocity field transporting noise into data distributions. “Iris-3B is a flow-matching transformer”
  • FlowDPM++: A diffusion or flow-based sampling procedure used to generate model outputs in a specified number of steps. “25-step FlowDPM++ sampling”
  • Generative prior: A pretrained generative model or representation that supplies useful structural information for another task. “We find no significant improvement from a pixel-space generative prior”
  • GQA (grouped-query attention): An attention variant in which multiple query heads share a smaller number of key and value heads. “Attention uses GQA with 20 Q and 5 KV heads”
  • Gradient clipping: Limiting gradient magnitudes during optimization to improve numerical stability. “with weight decay 0 and gradient clipping at 0.5”
  • Latent diffusion: Diffusion modeling performed in a compressed representation space rather than directly on image pixels. “Latent diffusion~\cite{ldm} has become the dominant paradigm for high-resolution image synthesis”
  • Latent twin: A latent-space model corresponding to a pixel-space model, typically derived from the same parent model for comparison. “A clean test of the hypothesis needs a latent twin of each parent”
  • Logit-normal distribution: A probability distribution formed by applying the logistic transform to a normally distributed variable. “The noise level is drawn from a shifted logit-normal distribution”
  • MMDiT: A multimodal diffusion transformer architecture that uses modality-specific streams and attention mechanisms. “8 dual-stream MMDiT blocks”
  • Monocular depth estimation: Predicting scene depth from a single image. “We fine-tune both families for monocular depth estimation”
  • Neural field: A continuous function, usually represented by a neural network, that maps coordinates or features to visual quantities. “PixNerd~\cite{pixnerd} replace the per-patch linear decoder with a lightweight pixel decoder and a neural field”
  • PixelShuffle: An operation that rearranges channel values into spatial dimensions to increase image resolution. “with PixelShuffle heads giving the best trade-off against training loss”
  • Pixel-unshuffle: The inverse spatial rearrangement that converts local spatial neighborhoods into feature channels. “The input is cut into non-overlapping patches by a pixel-unshuffle”
  • Procrustes initialization: An initialization method that aligns two representations using a fitted transformation, often preserving geometric structure. “Like the Procrustes initialization of AsymFlow~\cite{asymflow}, this fits the new projection to the parent's latent interface”
  • Rectified flow: A flow-based generative formulation that interpolates between data and noise along a relatively straight path. “we interpolate along the rectified-flow path”
  • Representation alignment: Training an internal representation to match features from a separate pretrained or target encoder. “We then compare four alignment objectives”
  • REPA: Representation Alignment for Generation, an auxiliary objective that aligns intermediate generative-model features with pretrained visual features. “The total loss adds REPA”
  • Ridge regression: Regularized linear regression that penalizes large parameter values to improve conditioning and generalization. “We fit, by ridge regression, a linear map from each 16×1616{\times}16 RGB patch”
  • RMSNorm: Root-mean-square normalization, a normalization technique that rescales activations without subtracting their mean. “sandwich normalization~\cite{cogview} with RMSNorm~\cite{rmsnorm}”
  • RoPE (rotary positional embedding): A positional encoding that applies position-dependent rotations to query and key representations. “RoPE uses isotropic coordinates for non-square images”
  • Score distillation: A method that uses a diffusion model’s estimated score function to optimize another model or representation. “variational score distillation”
  • Self-supervised objective: A training objective whose targets are derived from the input data rather than manually supplied labels. “Self-Flow~\cite{selfflow} replaces the external teacher with a self-supervised objective”
  • Sinkhorn-based optimization: An iterative optimization procedure commonly used to compute entropy-regularized optimal transport or align distributions. “adds representation alignment against ground-truth features and a Sinkhorn-based second stage”
  • Super-resolution: The reconstruction of a higher-resolution image from a lower-resolution input. “image restoration with super-resolution”
  • Timestep modulation: Conditioning a diffusion or flow model on the noise or diffusion timestep by modifying intermediate activations. “Timestep modulation replaces the per-block adaLN of DiT”
  • Tokenization: Dividing an input, such as an image, into discrete units that can be processed as tokens. “replaced by 16×1616{\times}16 pixel patches”
  • Variational autoencoder (VAE): A probabilistic encoder–decoder model that maps data into a learned lower-dimensional latent representation and reconstructs it. “By running the generative process in the compressed space of a pretrained variational autoencoder”
  • Variational score distillation: A score-distillation technique derived from a variational objective for transferring diffusion-model information. “distills restoration into a single step with variational score distillation”
  • Vector-quantized latent representation: A discrete latent representation in which continuous encoder outputs are mapped to entries in a learned codebook. “any detail the codec discards lies beyond the generator's reach”
  • Zero-shot benchmark: An evaluation performed on a task or dataset without task-specific training on its test distribution. “zero-shot on five benchmarks”
View Paper Prompt  View All Prompts 

Tweets

Sign up for free to view the 3 tweets with 140 likes about this paper.

Sign Up for Free