--- title: Iris-3B - Text-to-Image Generation url: https://www.emergentmind.com/topics/iris-3b type: topic --- # Iris-3B - Text-to-Image Generation Iris-3B is a 3-billion-parameter pixel-space text-to-image diffusion transformer developed to investigate whether eliminating the lossy VAE used by latent diffusion improves fine-grained visual generation and downstream dense-vision tasks. It operates directly on RGB pixels, uses a Pixel Transformer (PiT) output head, and is pretrained through a $256^2 \rightarrow 512^2 \rightarrow 1024^2$ resolution curriculum. Its reported results show competitive text-to-image synthesis without a VAE, but no significant advantage over latent models for monocular depth estimation or image restoration under the tested fine-tuning protocols. The system and its training code and weights are released in “Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning” [2610.09450]. ## 1. Motivation and research scope Iris-3B examines two routes to pixel-space generation. The first is pretraining a model from scratch directly on RGB pixels. The second is converting a pretrained latent model, FLUX.2 Klein base 4B, to pixel space by removing its VAE and replacing the latent interface with pixel patches and a pixel output head. The motivating hypothesis is that latent diffusion may lose fine-grained spatial information because its output must ultimately pass through a lossy VAE decoder. Pixel-space diffusion avoids that bottleneck and could therefore benefit tasks such as monocular depth estimation, image restoration, and super-resolution. The experiments do not establish such an advantage. Iris-3B is competitive with latent models for text-to-image synthesis, but it does not outperform latent FLUX.2 Klein on the tested downstream tasks. The converted pixel FLUX.2 Klein is generally weaker than its latent counterpart. The authors characterize the negative downstream result as non-definitive because the comparisons remain confounded by model size, data, compute, conversion quality, training budget, and single-run evaluation. Iris-3B has no latent twin with identical architecture and pretraining conditions, while the converted FLUX.2 Klein provides a more direct but still imperfect latent-to-pixel comparison. The designation “Iris-3B” is specific to the pixel-space diffusion model described in the 2026 paper. A separate 2014 paper on the Thirty Meter Telescope’s InfraRed Imaging Spectrograph uses the name IRIS for an astronomical instrument and does not mention Iris-3B or identify it as an instrument version, detector, probe-arm assembly, optical prescription, or mechanical configuration [1407.2999]. ## 2. Pixel-space architecture Iris-3B does not use a VAE. Its input and output are RGB pixels divided into non-overlapping $16\times16$ patches. The representation therefore contains a $16\times16$ patch grid at $256^2$, a $32\times32$ grid at $512^2$, and a $64\times64$ grid at $1024^2$. A $1024^2$ image is represented by 4096 patch tokens, while each token retains a $16\times16$ pixel footprint. The use of $16\times16$ patches is important at the highest resolution. The paper cites PixelUMM’s observation that $32\times32$ patches produce higher loss even when trained on four times as many images. Pixel-space operation therefore does not imply fully pixel-local computation throughout the network: Iris-3B retains patch-level tokenization and uses a specialized pixel transformer head to recover pixel-level outputs. ### Transformer trunk The main transformer follows a hybrid FLUX.1-style design comprising: - 8 dual-stream MMDiT blocks, in which text and image tokens retain separate parameter streams; - 16 single-stream blocks, in which the modalities are jointly processed; - hidden width 2560; - generalized-query attention with 20 query heads and 5 key/value heads; - a sigmoid attention output gate; - sandwich normalization using RMSNorm. The model uses shared-bias adaptive layer normalization. There is one shared modulation projection per stream and per-block learned biases, replacing conventional per-block adaLN. At equal FLOPs, the authors state that this removes approximately 28% of the parameters relative to a comparable per-block-adaLN model. Image tokens use parameter-free 2D axial RoPE. From the $512$-pixel stage onward, non-square images use isotropic coordinates. Because the positional encoding is not tied to a fixed grid size, the same model parameters can be used throughout the resolution curriculum. ### Pixel Transformer head The final trunk output is converted to pixels by a four-block Pixel Transformer head based on PixelDiT. The PiT head maintains patch-level computational efficiency while performing diffusion directly over RGB values: 1. Each patch token modulates the pixels belonging to that patch. 2. A per-pixel MLP processes pixel features. 3. Pixels within each patch are compacted into a single 1280-dimensional token for attention. 4. The compacted patch tokens attend across the patch grid. 5. The attended representations are expanded back to the pixels within each patch. This design creates a structural distinction between pixel-space representation and unrestricted pixel-resolution interaction. Cross-patch communication occurs through compacted patch-level tokens rather than fully overlapping pixel-resolution operations. The authors associate this limitation with the $16\times16$ patch-grid artifacts observed in downstream predictions. ### Text conditioning Text conditioning is provided by a frozen Qwen3-VL-4B-Instruct encoder. Iris-3B reads 12 Qwen3-VL layers, combines them with layerwise attention pooling, and applies a two-block refiner. Captions are dropped for 10% of training samples to enable classifier-free guidance. ## 3. Flow matching, ablations, and pretraining Iris-3B uses rectified-flow-style flow matching. Given a clean image $x_0$, Gaussian noise $\epsilon\sim\mathcal N(0,I)$, and caption $c$, the interpolated state is $$ x_\sigma=(1-\sigma)x_0+\sigma\epsilon. $$ The velocity target is $$ v=\epsilon-x_0. $$ The primary flow-matching objective is $$ \mathcal L_{\mathrm{flow}} = \mathbb E_{x_0,\epsilon,\sigma} \left[ \left\| v_\theta(x_\sigma,\sigma,c)-(\epsilon-x_0) \right\|_2^2 \right]. $$ The total pretraining loss adds a representation-alignment term: $$ \mathcal L = \mathcal L_{\mathrm{flow}} + \lambda\mathcal L_{\mathrm{REPA}}. $$ REPA is based on negative cosine similarity between projected intermediate trunk tokens and frozen DINOv2-B/14 patch features. Its coefficient is $\lambda=0.5$ during the $256^2$ stage and zero thereafter. Noise levels use a shifted logit-normal parameterization, $$ \sigma=\frac{s\,u}{1+(s-1)u}, \qquad u\sim\operatorname{logit-normal}(0,1), $$ with $s=2$ at $256^2$, $s=3$ at $512^2$, and $s=4$ at $1024^2$. ### Prediction-target ablation At $256^2$, a 1.3B dual-stream PixelDiT reproduction compared $x$-prediction with $v$-prediction under fixed data, matched training steps, EMA weights, 25-step FlowDPM++, CFG 4.5, GenEval, DPG-Bench, FID-10K, and CLIP score. The $v$-prediction formulation was stable, and the two targets were tied on GenEval at the reported milestones. $x$-prediction was worse on DPG and FID. The comparison was not a perfectly isolated target ablation because the $x$-prediction arm also used JiT’s noisier timestep distribution, a logit-normal $(0.8,0.8)$ setting rather than the default $(0.0,1.0)$. Iris-3B therefore retained $v$-prediction. ### Representation-alignment ablation Four alignment strategies were compared at $256^2$: 1. REPA with a DINOv2 teacher; 2. iREPA, which adds a convolutional projector and spatial normalization; 3. REPA with a DINOv3 teacher; 4. Self-Flow, which replaces the external teacher with a self-supervised flow objective. DINOv2 REPA was best overall, leading GenEval at every milestone and tying the best arm on DPG and CLIP. DINOv3 REPA produced lower GenEval and worse FID throughout. iREPA produced a clear improvement on FID at 150K and 200K steps, but its advantage had almost disappeared by 250K and it trailed on GenEval and CLIP at nearly every milestone. Self-Flow remained well below REPA on all four metrics. Iris-3B consequently uses DINOv2 REPA only during the initial $256^2$ stage. ## 4. Resolution curriculum, data, and optimization Iris-3B was trained through the curriculum $$ 256^2\rightarrow512^2\rightarrow1024^2 $$ followed by supervised fine-tuning at $1024^2$. | Stage | Steps | Global batch | Samples | |---|---:|---:|---:| | $256$ | 0–367K | 1024 | 375.8M | | $512$, multi-aspect-ratio | 367K–530K | 1024 | 166.9M | | $1024$, multi-aspect-ratio | 530K–625K | 512 | 48.6M | | SFT | 625K–665K | 512 | 20.5M | The total duration is 665K steps: 625K pretraining steps and 40K supervised fine-tuning steps. The $256^2$ stage uses centered square crops. At $512$ and $1024$, images are assigned to one of 21 shared aspect-ratio buckets spanning approximately $1:4$ to $4:1$, with near-equal token counts. Images must have a short edge at least as large as the current training resolution. Rare source and bucket combinations are merged to ensure at least one global batch per cell. Because fewer photographs satisfy the resolution threshold, the $1024^2$ stage contains proportionally more non-photographic material. Pretraining uses a large English image-text collection containing real photographs, synthetic images generated by recent image generators, and rendered-text images intended to improve text rendering. Supervised fine-tuning uses curated high-quality real and synthetic images with captions filtered for quality and prompt alignment. The exact dataset size, source-by-source composition, and compute hardware are not provided. ### Optimization The optimizer is hybrid Muon. Muon is used for hidden weight matrices with momentum $0.95$, Nesterov momentum, and RMS-matched learning rate. AdamW is used for embeddings, normalization parameters, and biases with $\beta_1=0.9$ and $\beta_2=0.95$. Other settings include: - weight decay $0$; - gradient clipping $0.5$; - constant learning rate $10^{-4}$; - linear warmup for 2K steps; - EMA decay $0.9999$; - caption dropout 10%. At $1024^2$, the learning rate is halved to $5\times10^{-5}$ and the global batch is reduced to 512. Supervised fine-tuning uses a constant learning rate of $4\times10^{-5}$. ## 5. Text-to-image performance The released Iris-3B checkpoint is the EMA model after supervised fine-tuning at step 665K. Evaluation uses $1024^2$ resolution, official unmodified evaluators, official prompts without rewriting, CFG 3, 100 sampling steps, and four samples per prompt. Every sample is scored; no best-of-$N$ selection is used. | Model | GenEval | DPG-Bench | LongText-Bench | OneIG-Bench | |---|---:|---:|---:|---:| | FLUX.1-dev, 12B | 0.66 | 83.8 | 0.607 | 0.434 | | i1, 3B* | 0.84 | 86.7 | 0.922 | — | | Z-Image, 6B* | 0.84 | 88.1 | 0.935 | 0.546 | | Qwen-Image, 20B | 0.87 | 88.3 | 0.943 | 0.539 | | **Iris-3B** | **0.798** | **86.52** | **0.857** | **0.540** | \* Prompt rewriting is used by the cited reference systems. Iris-3B clearly exceeds FLUX.1-dev and is approximately level with i1 on DPG-Bench. It trails i1, Z-Image, and Qwen-Image on GenEval and is below the strongest systems on LongText-Bench. On OneIG-Bench, it scores 0.540, matching Qwen-Image’s 0.539 and remaining just below Z-Image’s 0.546. The OneIG result demonstrates that a 3B-parameter pixel-space model can achieve text-to-image quality competitive with a much larger latent model under the official $1024^2$ evaluator. It does not demonstrate that pixel space is intrinsically superior. Iris-3B is strongest on single objects, two objects, and colors, while its weakest GenEval categories are position and counting. Its clearest generation weakness is long English text rendering. ## 6. Latent-to-pixel conversion and downstream evaluation ### Conversion of FLUX.2 Klein The second pixel-space route begins with FLUX.2 Klein base 4B. The VAE is removed and replaced with $16\times16$ RGB pixel patches, a new input projection, and a new pixel output head. The patch footprint is chosen to match the parent model’s latent-token footprint. Two output heads are tested: DiP and a four-block PiT. The PiT version, designated PiT 30K, is used for downstream comparisons. It is trained at $1024^2$ on a 1:1 mixture of real photographs and images generated by step-distilled FLUX.2 Klein 4B, which is a sibling model rather than the exact latent parent. A randomly initialized input projection creates a severe interface mismatch because the FLUX trunk was trained on VAE-latent embeddings rather than raw pixel-patch embeddings. The best per-patch linear map from pixels to corresponding parent latent-token embeddings explains only 14% of the variance of the original latent-token embeddings. Under the unmodified conversion recipe, a 1K-step probe produces prompt-independent patch-grid noise, and more than ten times as many steps are required to approach the loss achieved by the improved recipe. Two interventions address the instability: - **Gain-matched ridge initialization**: a linear map is fitted from each $16\times16$ RGB patch to the corresponding parent latent-token embedding, with regularization chosen to pass noise with the same gain as the original latent path rather than maximize held-out $R^2$. - **Scaled trunk learning rate**: for FLUX, the trunk learning rate is set to $0.12\times$ the head’s rate because FLUX trunk weights are much smaller and an unscaled Adam update is large relative to their parameter magnitude. In global-batch-8, 1K-step probes, either intervention improves the loss substantially, but neither alone produces coherent images. Together they produce photorealistic, prompt-aligned images within 1K steps. Gain-matched ridge initialization also removes a color cast seen with the $R^2$-optimized fit. Production conversion uses global batch 112, caption dropout 0.1, a 1K-step trunk warmup, and the PiT conversion after 30K steps. The converted models produce photorealistic images and short correctly rendered words. However, GenEval, DPG-Bench, and FID are not reported for the converted model, so its generation quality relative to latent FLUX.2 Klein remains unknown. ### Monocular depth estimation Depth fine-tuning compares latent FLUX.2 Klein base 4B, converted pixel FLUX.2 Klein PiT 30K, and Iris-3B. All receive the same nominal direct-regression recipe, although Iris-3B has no latent twin and is not representation-matched. Depth is predicted deterministically in one forward pass from an RGB image, an empty text prompt, the final timestep, and zero noise. For pixel models, a three-to-one convolution reads depth from the RGB output. For the latent model, the latent output is first decoded through the frozen FLUX.2 VAE, after which the same convolution predicts depth. The loss is masked over valid pixels and combines absolute depth error with a gradient loss: $$ \mathcal L_{\mathrm{depth}} = \frac{1}{|M|} \sum_{p\in M} \left( |\hat d_p-d_p| + 5\left\| \nabla\hat d_p-\nabla d_p \right\|_1 \right). $$ Training uses AdamW, $\beta=(0.9,0.999)$, weight decay $0.01$, learning rate $3\times10^{-5}$, no warmup, gradient clipping 1.0, global batch 32, and 10K steps. Raw non-EMA weights are evaluated. Training data consist only of Hypersim and Virtual KITTI 2, mixed 9:1 per batch. | Parent | Mean AbsRel | Mean $\delta_1$ | Hypersim SEE | |---|---:|---:|---:| | Klein latent | .072 | .947 | .528 | | Klein pixel | .081 | .934 | .526 | | Iris-3B | **.071** | .946 | **.496** | Iris-3B is effectively level with latent FLUX.2 Klein. It performs better on KITTI and DIODE and has lower Hypersim soft edge error, but it is worse on NYUv2, ETH3D, and ScanNet. Converted FLUX.2 Klein is behind the latent model on every benchmark except KITTI, where it is close. Raw pixel-space depth predictions contain faint $16\times16$ patch-grid artifacts. The measured grid ratio is defined as the mean absolute second difference adjacent to patch borders divided by the mean absolute second difference elsewhere; a model without a grid should score approximately 1. | Parent | Hypersim | KITTI | ETH3D | DIODE | |---|---:|---:|---:|---:| | Klein latent | .97 | .99 | .99 | .98 | | Klein pixel | 1.31 | 1.44 | 1.36 | 1.34 | | Iris-3B | 2.14 | 1.61 | 2.21 | 2.25 | The proposed explanation is that non-overlapping pixel patches are decoded primarily from their own trunk tokens, allowing neighboring patches to disagree at boundaries. The latent VAE decoder has overlapping receptive fields and naturally smooths across token boundaries. Convolutional or PixelShuffle output heads may reduce these artifacts. ### DIV2K restoration and super-resolution Restoration follows HYPIR’s one-step design. A degraded low-resolution image is upsampled to the target resolution and passed through the model once with an empty prompt. Training uses $1024^2$ crops, horizontal flips, on-the-fly second-order Real-ESRGAN degradations, $4\times$ degradation including sharpening, and the loss $$ \mathcal L_{\mathrm{SR}} = \|\hat y-y\|_2^2 + 5\mathcal L_{\mathrm{LPIPS}} + 0.5\mathcal L_{\mathrm{adv}}. $$ Latent and converted pixel FLUX.2 Klein use the same data, losses, schedule, global batch 8, 500 warmup steps, 10K steps, and continuation to 30K with learning-rate reset. Iris-3B uses a different, unmatched setup involving constant learning rate, global batch 40 for the first 5K steps and 10 thereafter, EMA decay 0.999, EMA-weight inference, wavelet color correction, and tiling. | Model | PSNR | SSIM | LPIPS | NIQE | MUSIQ | DeQA | |---|---:|---:|---:|---:|---:|---:| | Klein latent | **20.99** | **.557** | .275 | **3.26** | 67.11 | **4.06** | | Klein pixel | 20.87 | .504 | **.274** | 4.11 | 67.03 | 4.04 | | Iris-3B | 20.68 | .481 | .292 | 3.94 | **68.01** | 3.87 | The converted pixel model is worse on most metrics, particularly SSIM and NIQE, and is essentially tied on LPIPS. Iris-3B is slightly below both FLUX.2 Klein models on PSNR, SSIM, LPIPS, and DeQA, between them on NIQE, and best on MUSIQ. Because the experiments use single runs and no significance testing, the paper interprets these findings as showing no advantage rather than establishing a statistically reliable loss. ## 7. Limitations and significance The reported failure modes include patch-grid artifacts in both pixel models, weak long-text rendering, weaknesses in position and counting, conversion instability, and a conversion color cast under $R^2$-tuned ridge initialization. The converted FLUX.2 Klein lacks reported GenEval, DPG-Bench, and FID scores, preventing a complete generation comparison with its latent parent. The downstream conclusions are also limited by unmatched conditions. Iris-3B differs from latent FLUX.2 Klein in architecture, parameter count, data, compute, and pretraining recipe. Restoration comparisons use different batch sizes, schedules, parent checkpoints, EMA procedures, and inference pipelines. Each downstream arm is a single run with short training budgets, and no significance tests are reported. The experiments support several bounded conclusions. Pixel-space diffusion can scale to a 3B-parameter text-to-image transformer trained through $1024^2$. A PiT head is a viable architecture for direct RGB diffusion. Pixel-space models can be competitive with latent models for text-to-image generation, and careful input-interface initialization and learning-rate scaling are required for latent-to-pixel conversion. They do not establish that pixel-space priors are inherently inferior or inherently better at preserving fine detail. Iris-3B’s lower Hypersim soft edge error indicates a possible advantage on one depth metric, but its overall depth performance is approximately level with latent FLUX.2 Klein and its restoration results do not show a consistent fidelity improvement. A plausible implication is that the potential benefit of avoiding VAE reconstruction may be offset by patch-local decoding, conversion quality, or differences in training scale and data. The paper identifies matched latent and pixel pretraining with the same data and compute, together with improved overlapping pixel heads, as the experiments needed to resolve whether pixel-space representations provide a genuine advantage for dense visual prediction and restoration. Source: https://www.emergentmind.com/topics/iris-3b