Pixel-Space Diffusion Transformer

• Pixel-space diffusion transformers are generative models that operate directly on patchified RGB images using transformer backbones to predict clean outputs without a latent encoding stage.
• They address challenges of VAE-induced reconstruction bottlenecks by directly modeling high-dimensional pixel manifolds, preserving both global semantics and fine details.
• Recent variants integrate architectural innovations such as U-shaped decoders, global–local decomposition, and semantic guidance to enhance performance and efficiency.

Searching arXiv for papers on pixel-space diffusion transformers and closely related methods. Pixel-space diffusion transformers are diffusion or flow-matching generative models whose denoising backbone is a transformer and whose generative process is defined directly on raw pixels, rather than on a compressed latent space produced by a variational autoencoder. In this regime, the model patchifies RGB images, applies transformer blocks over those patch tokens, and predicts clean images, noise, or velocities in pixel space. Recent work frames pixel-space modeling as a way to avoid the reconstruction bottleneck, artifacts, and training complexity associated with VAE-based latent diffusion, while confronting a separate optimization problem: direct modeling of high-dimensional pixel manifolds that entangle global semantics with high-frequency detail (Ma et al., 2 Feb 2026). The resulting research area includes plain DiT-style backbones in RGB space, U-shaped transformer variants, hybrid global–local decoders, semantic waypoint guidance, neural-field decoders, and task-specific extensions to text-to-image generation, geometry estimation, medical imaging, image editing, and novel-view synthesis (Guo et al., 26 Jun 2026).

1. Definition and scope

A pixel-space diffusion transformer is a transformer-based denoiser or flow network that operates on patchified images in the original signal domain, with no VAE encoder, no latent representation, and no auxiliary decoder stage for reconstruction (Ma et al., 2 Feb 2026). In the canonical image-generation setting, an image $x \in \mathbb{R}^{H \times W \times 3}$ is corrupted by a continuous-time interpolation with Gaussian noise, patchified into tokens, processed by a DiT-style backbone, and mapped back to an RGB prediction. PixelGen states this definition explicitly: a “Pixel-Space Diffusion Transformer” is “a DiT-style transformer backbone trained as a diffusion model directly in RGB pixel space, augmented with strong perceptual supervision” (Ma et al., 2 Feb 2026).

This differs from latent diffusion, where an image is first encoded into a latent $z$, denoising is performed in latent space, and a VAE decoder reconstructs the final image. The data describe several drawbacks of the latent pipeline: a reconstruction bottleneck, low-level artifacts, instability in joint VAE–diffusion optimization, and sensitivity to latent-distribution design (Ma et al., 2 Feb 2026). Pixel-space transformers are presented as a direct alternative: they remain end-to-end, avoid autoencoder-induced degradation, and remove dependence on a separately trained tokenizer or decoder (Chen et al., 10 Apr 2025).

The term also extends beyond unconditional or class-conditional image synthesis. PointDiT formulates dense monocular geometry estimation as diffusion directly over 3D point maps $x \in \mathbb{R}^{H \times W \times 3}$, conditioned on DINOv3 image tokens (Xu et al., 2 Jul 2026). “Pixel-Perfect Depth” models normalized depth maps in pixel space using a semantics-prompted DiT and a cascade token schedule (Xu et al., 8 Oct 2025). LazyDiffusion localizes diffusion to only masked spatial tokens for interactive editing, although it operates in the latent space of a pretrained VAE and is thus only “pixel-space” in a looser, spatially aligned patch-token sense (Nitzan et al., 2024). FTIR virtual staining uses a DiT-based Brownian bridge process directly on pixel-space RGB outputs from spectroscopic inputs (Tian et al., 9 Mar 2026). Novel-view synthesis adopts a pixel-space diffusion backbone with transformer-style joint attention inside a U-Net rather than a pure ViT, but still illustrates the broader shift toward pixel-space generative modeling when local fidelity is critical (Elata et al., 2024).

2. Core formulation and prediction targets

A recurring formulation in the recent literature is continuous-time flow matching with direct prediction of the clean signal. PixelGen, PixelU, PointDiT, HyperDiT, and Latent Forcing all use or analyze the linear interpolation

$x_t = t x + (1-t)\epsilon,$

or an equivalent notation $z_t$, with $t \in [0,1]$, $x$ the clean data, and $\epsilon \sim \mathcal{N}(0,I)$ (Ma et al., 2 Feb 2026, Guo et al., 26 Jun 2026, Xu et al., 2 Jul 2026, He et al., 15 May 2026, Baade et al., 11 Feb 2026). The network predicts the clean signal $\hat{x}$, which is converted into a velocity prediction

$\hat{v} = \frac{\hat{x} - x_t}{1-t},$

yielding the flow-matching objective

$z$0

in the image or task domain (Ma et al., 2 Feb 2026, Guo et al., 26 Jun 2026).

The choice of prediction target is central. PixelU argues that complex pixel decoders in prior work largely compensate for the optimization difficulty of direct $z$1-prediction in raw pixel space, and that under clean-data $z$2-prediction those decoders become largely redundant (Guo et al., 26 Jun 2026). Its frequency-domain analysis distinguishes natural images, whose high-frequency power decays, from white-noise components that remain flat across the spectrum. Under direct $z$3-prediction, the target contains full-strength high-frequency Gaussian noise; under $z$4-prediction, the target is the clean image itself, whose high-frequency energy diminishes with frequency (Guo et al., 26 Jun 2026). PointDiT reports the same pattern for geometry: direct $z$5-prediction “fails badly,” whereas $z$6-prediction is crucial for stable monocular point-map diffusion (Xu et al., 2 Jul 2026).

This pattern is not universal across all architectures. HyperDiT compares $z$7-prediction and $z$8-prediction in its multi-stream cross-scale design and reports that $z$9-prediction yields better final FID because the $x \in \mathbb{R}^{H \times W \times 3}$0 weighting in $x \in \mathbb{R}^{H \times W \times 3}$1-prediction destabilizes its cross-attention-heavy system near $x \in \mathbb{R}^{H \times W \times 3}$2 (He et al., 15 May 2026). This suggests that prediction-target preference depends not only on the data domain but also on how semantic and pixel streams are coupled.

Sampling typically uses explicit ODE solvers. PixelGen uses Euler, Heun, or Adams-2nd after converting $x \in \mathbb{R}^{H \times W \times 3}$3 to $x \in \mathbb{R}^{H \times W \times 3}$4 (Ma et al., 2 Feb 2026). PixelU uses 50 Heun steps (Guo et al., 26 Jun 2026). PointDiT uses Euler with only 1–4 steps for geometry estimation (Xu et al., 2 Jul 2026). Text-to-image PixelGen uses Adams-2nd with 25 steps and classifier-free guidance $x \in \mathbb{R}^{H \times W \times 3}$5 (Ma et al., 2 Feb 2026). These details indicate that pixel-space transformers are often developed in the rectified-flow or probability-flow-ODE setting rather than the original discrete DDPM parameterization.

3. Architectural patterns

The simplest pixel-space diffusion transformer is a flat DiT operating on pixel patches. PixelGen instantiates this directly: 16×16 RGB patches are linearly projected, enriched with RoPE2d positional embeddings, processed by transformer blocks with multi-head self-attention, RMSNorm, and SwiGLU MLPs, and projected back to per-pixel RGB space (Ma et al., 2 Feb 2026). Models include DiT-L/16, DiT-XL/16, and DiT-XXL/16 at 256×256 and 512×512 (Ma et al., 2 Feb 2026). JiT, DeCo, PixelFlow, PixNerd, and DiP all share this broad DiT-style patch-token paradigm, differing mainly in how they decode or refine high-frequency content (Ma et al., 2 Feb 2026, Chen et al., 24 Nov 2025, Chen et al., 10 Apr 2025, Wang et al., 31 Jul 2025).

A second pattern is the U-shaped pixel transformer. PixelU proposes a single-stage encoder–bottleneck–decoder transformer with one spatial down-sampling, one up-sampling, constant channel width, and zero-cost skip connections (Guo et al., 26 Jun 2026). Its central mechanism is “frequency decoupling”: down-sampling and the bottleneck form a compact low-frequency semantic manifold, while long skip connections preserve and restore high-frequency details (Guo et al., 26 Jun 2026). Empirically, down-sampling alone harms fidelity because details are lost, skip connections alone help but keep frequencies entangled, and the combination yields the best FID at lower GFLOPs (Guo et al., 26 Jun 2026).

A third pattern is global–local decomposition. DiP uses a large-patch DiT backbone for efficient global structure construction and a lightweight patch detailer head—a small convolutional U-Net instantiated per patch—to restore local detail (Chen et al., 24 Nov 2025). The backbone sees only 16×16 pixel patches, keeping token count comparable to latent-space DiTs, while the local head injects convolutional inductive bias and outputs per-pixel predictions with only a 0.3% parameter increase (Chen et al., 24 Nov 2025). FrequencyBooster follows a related but distinct logic: it uses a standard DiT backbone for low-frequency semantics and a separate high-capacity FB-Decoder with expanded hidden dimension $x \in \mathbb{R}^{H \times W \times 3}$6 and global attention over the same patch tokens, arguing that prior decoders suppress high-frequency information through token compression or local-only refinement (Ma et al., 18 May 2026).

A fourth pattern is cross-scale semantic guidance. HyperDiT addresses what it calls the “granularity dilemma”: large patches favor global semantics but blur details, while small patches favor fine fidelity but lack stable semantic anchors (He et al., 15 May 2026). It therefore introduces a dual-stream architecture with a large-patch “Semantics Flow,” a small-patch “Fine-grained Flow,” Hyper-Connectors that perform cross-attention from fine tokens to semantic anchors, Scale-Aware RoPE to align different patch scales, and non-spatial register tokens aligned with DINOv2 features (He et al., 15 May 2026). This suggests a broader architectural principle: semantic and pixel manifolds can be bridged by explicit cross-scale interaction rather than by forcing a single token stream to solve both simultaneously.

A fifth pattern is alternative output parameterization. PixNerd maintains latent-like token counts by using large 16×16 patches but replaces the usual linear per-patch decoder with a patch-wise neural field: for each patch token, the network predicts the weights of a small MLP that maps local pixel coordinates and noisy pixel values to per-pixel velocities (Wang et al., 31 Jul 2025). The aim is to resolve the “large-patch detail problem” without cascades or VAEs (Wang et al., 31 Jul 2025). This is a different answer to the same tension addressed by DiP, PixelU, and FrequencyBooster.

4. Guidance, supervision, and representation structure

A major line of work argues that pixel-space transformers are not limited by expressivity alone but by supervision mismatch. PixelGen’s central claim is that pixel diffusion should target a “perceptual manifold” rather than the full image manifold, which includes perceptually irrelevant high-frequency variation (Ma et al., 2 Feb 2026). It adds two explicit perceptual losses on the x-prediction: $x \in \mathbb{R}^{H \times W \times 3}$7 LPIPS encourages local texture and edge fidelity through frozen VGG features, while patchwise DINOv2 cosine loss strengthens global semantics and structure (Ma et al., 2 Feb 2026). PixelGen further finds that perceptual losses should not be applied at the earliest, highest-noise timesteps, because that hurts recall and diversity; its “noise-gating” activates LPIPS and P-DINO only in the last 70% of timesteps (Ma et al., 2 Feb 2026).

Several papers instead or additionally inject semantics explicitly. Pixel-Perfect Depth introduces semantics-prompted DiT (SP-DiT), where dense features from a frozen vision foundation model are spatially aligned to the DiT token grid, concatenated with DiT tokens, and fused with an MLP (Xu et al., 8 Oct 2025). It reports that this semantic prompting is “essential” for making high-resolution pixel-space DiT viable for depth estimation, with strong improvements over a bare pixel-space DiT baseline (Xu et al., 8 Oct 2025). PointDiT uses frozen DINOv3 image tokens from four uniformly spaced layers, concatenated patch-wise with point-map tokens and projected back into the model dimension, without cross-attention (Xu et al., 2 Jul 2026). This suggests that spatially aligned conditioning by foundation-model features can substitute for architectural complexity in some dense-prediction settings.

WiT tackles a different problem: trajectory conflict in pixel-space flow matching. It argues that pixel space is not a semantically continuous manifold, so class-conditional trajectories overlap in noisy regions, forcing the model toward averaged velocities (Wang et al., 16 Mar 2026). WiT factors the vector field through intermediate semantic waypoints derived from DINOv3 features and conditions the main pixel-space transformer with a “Just-Pixel AdaLN” mechanism that provides spatially varying modulation parameters per token (Wang et al., 16 Mar 2026). Its variance decomposition formalizes the intuition that conditioning on a semantic waypoint reduces $x \in \mathbb{R}^{H \times W \times 3}$8 by conditioning away ambiguity (Wang et al., 16 Mar 2026).

A related but architecturally lighter perspective is provided by “Registers Matter for Pixel-Space Diffusion Transformers.” That work shows that pixel-space DiTs do not exhibit the classic ViT patch-token outlier problem, but nevertheless benefit strongly from register tokens (Starodubcev et al., 15 May 2026). Registers improve convergence and FID, reduce patch-token norms, and produce cleaner feature maps at high-noise timesteps, precisely where pixel-space optimization is hardest (Starodubcev et al., 15 May 2026). The same paper argues that in-context class tokens in JiT and text tokens in large text-to-image DiTs already behave as implicit registers, functioning as norm sinks and semantic carriers (Starodubcev et al., 15 May 2026). This suggests that some benefits previously attributed solely to richer conditioning may partly arise from the architectural availability of loss-free or quasi-loss-free global token slots.

Latent Forcing generalizes the idea of internal semantic structure. It jointly diffuses raw pixels and aligned latent features from a pretrained vision model under separate but coupled time schedules, so that latents denoise first and act as a “scratchpad” for semantic structure before pixel denoising begins in earnest (Baade et al., 11 Feb 2026). This is still a pixel-space diffusion transformer because the final objective is the pixel distribution and the latent outputs are discarded at inference, but it reorders the denoising trajectory to recover some of the efficiency and semantic organization of latent diffusion without introducing a VAE bottleneck (Baade et al., 11 Feb 2026).

5. Empirical performance and scaling

Recent work establishes that pixel-space transformers are competitive with, and in some settings superior to, latent diffusion baselines. PixelGen reports FID 5.11 on ImageNet-256 without classifier-free guidance using only 80 training epochs, and FID 1.83 with CFG for PixelGen-XL/16 at 160 epochs and 50 Heun steps (Ma et al., 2 Feb 2026). Under matched 200K-step ImageNet-256 comparisons with the same DiT-L backbone, PixelGen-L/16 reaches FID 7.53 versus JiT-L/16 at 23.67 and latent DDT-L/2 at 10.00 (Ma et al., 2 Feb 2026). In text-to-image generation, PixelGen-XXL/16 achieves a GenEval score of 0.79 at 512×512, exceeding the listed scores for PixNerd-XXL/16, FLUX.1-dev, SD3, and DALL·E 3 (Ma et al., 2 Feb 2026).

PixelU pushes class-conditional ImageNet further. At 256×256, PixelU-H/16 reaches FID 1.63, sFID 5.04, IS 305.88, precision 0.79, and recall 0.64, while using about one third of JiT-G’s computation cost and fewer parameters (Guo et al., 26 Jun 2026). At 512×512, PixelU-H/32 achieves FID 1.92 and IS 322.10, surpassing listed pixel baselines such as PixNerd-XL/16, DeCo-XL/16, VDM++, SimpleDiffusion, and ADM-G (Guo et al., 26 Jun 2026).

HyperDiT reports FID 1.63 for HyperDiT-XL and 1.56 for HyperDiT-H on ImageNet 256×256 directly in pixel space, with precision 0.80 and recall up to 0.64 (He et al., 15 May 2026). FrequencyBooster reports FID 1.60 at 256×256 within 320 epochs and FID 1.69 at 512×512, describing these as state-of-the-art among the listed pixel-space methods (Ma et al., 18 May 2026). DiP reaches FID 1.90 on ImageNet 256×256 and 2.31 at 512×512, while emphasizing efficiency comparable to latent models and up to 10× faster inference than earlier pixel-space approaches under comparable quality (Chen et al., 24 Nov 2025). PixNerd reaches FID 2.15 at 256×256 and 2.84 at 512×512 without cascades or VAEs, and additionally posts GenEval 0.73 and DPG 80.9 in text-to-image settings (Wang et al., 31 Jul 2025). PixelFlow, a cascade flow model in pixel space, achieves FID 1.98 at 256×256 and is framed as a proof that pixel-space end-to-end modeling can be affordable when multiscale scheduling is used (Chen et al., 10 Apr 2025).

These numbers indicate both rapid progress and methodological diversity. The strongest reported 256×256 results in the provided data cluster around 1.56–1.63 for HyperDiT, PixelU, and FrequencyBooster (He et al., 15 May 2026, Guo et al., 26 Jun 2026, Ma et al., 18 May 2026). Some latent-space models listed alongside them still attain lower FID under certain settings, such as RAE-XL/2 at 1.13 or DDT-XL/2 at 1.26 (He et al., 15 May 2026, Ma et al., 18 May 2026). A plausible implication is that pixel-space transformers have largely overcome the older claim that direct pixel diffusion is categorically inferior, but the absolute frontier still depends on compute budget, architectural bias, guidance, and evaluation regime.

6. Domain-specific variants and open questions

The concept now extends well beyond class-conditional ImageNet. PointDiT shows that direct pixel-space diffusion over point maps can outperform complex latent-based geometry models and deterministic regressors on monocular 3D reconstruction metrics, while producing sharper boundaries and better handling transparent objects (Xu et al., 2 Jul 2026). Pixel-Perfect Depth applies a semantics-prompted pixel-space DiT to monocular depth estimation and attributes its edge-clean point-cloud quality to the removal of VAE-induced “flying pixels” (Xu et al., 8 Oct 2025). These works suggest that dense geometric fields are especially sensitive to latent compression and therefore especially suitable for pixel-space transformers.

In scientific and medical imaging, the FTIR virtual-staining work uses a Brownian bridge diffusion transformer to translate low-resolution multispectral tissue measurements into high-resolution H&E images in pixel space, achieving 4× pixel-level super-resolution and roughly fourfold speedup over a U-Net diffusion baseline through large-patch transformer processing (Tian et al., 9 Mar 2026). Novel-view synthesis with pixel-space diffusion models argues that pixel-space operation is important when local correspondence fidelity matters, though its backbone remains a U-Net with heavy attention rather than a pure DiT (Elata et al., 2024).

Interactive editing raises a different issue: pixel-space fidelity versus localized computation. LazyDiffusion processes only masked spatial tokens in the diffusion decoder while using a single full-image encoder pass for global context, obtaining about a 10× speedup for typical 10% masks relative to full-image regeneration (Nitzan et al., 2024). Because it still relies on a Stable Diffusion VAE, it is not a pure pixel-space transformer in the strict sense used by PixelGen or PixelU, but it demonstrates that spatial sparsity is a major direction for making transformer-based generative models interactive (Nitzan et al., 2024).

Several open questions recur across the literature. One concerns the best mechanism for semantic guidance: perceptual losses (Ma et al., 2 Feb 2026), frozen-feature prompting (Xu et al., 8 Oct 2025, Xu et al., 2 Jul 2026), explicit semantic waypoints (Tian et al., 9 Mar 2026), cross-scale semantic anchors with registers (He et al., 15 May 2026), or reordered joint latent–pixel trajectories (Baade et al., 11 Feb 2026). Another concerns the best way to balance global structure and high-frequency detail: U-shaped frequency decoupling (Guo et al., 26 Jun 2026), global–local patch detailers (Chen et al., 24 Nov 2025), neural-field patch decoders (Wang et al., 31 Jul 2025), or full-frequency wide decoders (Ma et al., 18 May 2026). A third concerns scaling and efficiency: pixel-space models remain more memory- and FLOP-intensive than latent-space ones at very high resolutions, even when they simplify the training pipeline by removing the VAE (Guo et al., 26 Jun 2026, Chen et al., 24 Nov 2025).

This suggests that “Pixel-Space Diffusion Transformer” is no longer a single architecture class but a research program. At minimum, it denotes a transformer-driven generative model that diffuses directly in the native signal domain. In current practice, it also implies an active attempt to recover semantic organization, preserve high-frequency fidelity, and manage the computational burden of raw-pixel modeling without reintroducing the bottlenecks of latent diffusion.