DecFormer: Latent Compositing Transformer

• DecFormer is a transformer-based latent-space compositor designed for pixel-equivalent alpha compositing, eliminating color shifts, halo artifacts, and masking inaccuracies.
• It leverages per-channel, spatially-varying blend weights and off-manifold residual corrections across hierarchical transformer modules to accurately replicate pixel-space editing.
• Integrable into diffusion pipelines, DecFormer offers improved metrics (SSIM, PSNR, LPIPS) and efficient inpainting with only ≈3.5% FLOP overhead.

DecFormer is a transformer-based latent-space compositor for masked image editing within diffusion models, designed to realize compositing operations in the latent space that are strictly equivalent to pixel-space alpha compositing after VAE decoding. Conventional approaches perform naive linear interpolation of latents using downsampled masks, which leads to significant color shifts, halo artifacts, and masking inaccuracies due to the globally nonlinear and nonlocal structure of modern VAE decoders. DecFormer provides pixel-equivalent latent compositing (PELC) by learning per-channel, spatially-variant blend weights and residual corrections that guarantee the decoded result precisely matches pixel-space compositing, thus enabling sharp boundaries and high-fidelity edits under soft masks with minimal computational overhead (Bradbury et al., 4 Dec 2025).

1. Pixel-Equivalent Latent Compositing (PELC) Principle

Standard latent blending methods in diffusion pipelines operate on two VAE-encoded images, $z_A=E(x_A)$ and $z_B=E(x_B)$, and a mask $M$ by downsampling $M$ to latent-space resolution ($m$) and linearly blending: $z_{\mathrm{interp}} = (1-m)\odot z_A + m\odot z_B$. While this heuristic is computationally efficient, it can only be pixel-equivalent if the VAE decoder $D$ is channel-wise linear and spatially local, neither of which hold for modern architectures. This loss of pixel equivalence is manifested as boundary leakage, color drift, and global artifacts far from the editing region.

PELC is defined by the requirement that latent-space compositing, via an operator $C_F(z_A, z_B, M)$, satisfies: $$D(C_F(z_A, z_B, M)) = (1-M)\odot D(z_A) + M\odot D(z_B)$$ and, in reverse,

$$C_F(E(x_A), E(x_B), M) = E((1-M)\odot x_A + M \odot x_B)$$

for frozen encoder/decoder pairs. DecFormer is the first model to achieve this property by learning a mapping that compensates for the nonlinearity and spatial entanglement of $D$ (Bradbury et al., 4 Dec 2025).

2. DecFormer Architecture and Mechanisms

DecFormer comprises 7.7 million parameters and introduces a lightweight yet expressive transformer architecture for latent compositing. For each latent voxel $(i,j)$ and channel $c$, it predicts:

• a per-channel, spatially-varying blend weight $\alpha_c(i,j) \in [0,1]$
• an off-manifold residual correction $s_c(i,j) \in \mathbb{R}$

The composed latent is computed as: $$\hat z = (1-\alpha) \odot z_A + \alpha \odot z_B + s$$

The pipeline consists of four hierarchical "patch-and-attend" stages using patch sizes $\{4,2,1,1\}$, which provide efficient aggregation of both global context and local spatial information. Each stage reconstitutes the latent as an image, applies FiLM conditioning via a mask embedding and "halo" maps focusing on boundary pixels, and refines local structures with convolutional residual blocks. In the final two (patch size 1) stages, cross-attention to mask-token embeddings is applied for additional spatial grounding. A parallel 0.7M parameter CNN processes the full-resolution mask (augmented by Fourier features) to yield an initial $\alpha_0$ prior and produces spatial features for mask-aware cross-attention. Transformer blocks use 8-head self-attention (hidden dimension $\approx 256$), 2-layer MLPs, LayerNorm, and residual connections.

3. Training Objective and Loss Formulation

Supervision is synthesized from pixel-level alpha compositing. Given random image pairs $(x_A, x_B)$ and mask $M$:

• Target latent: $z_T = E((1-M)\odot x_A + M\odot x_B)$
• Predicted latent: $\hat z = \mathcal D_\phi(E(x_A), E(x_B), M)$
• Decoded images: $x_T = D(z_T)$, $\hat x = D(\hat z)$

The PELC loss is: $$\mathcal{L}_{\mathrm{PELC}} = \lambda_E\,\mathbb{E}\|\hat z - z_T\|_2^2 + \mathbb{E}\left[\mathrm{LPIPS}(\hat x, x_T)\right] + \lambda_H\,\mathbb{E}\|\hat x - x_T\|_{1,\mathrm{halo}}$$ where the $L_2$ regularizes encoder-equivalence in latent space, LPIPS focuses on perceptual similarity, and the "halo-weighted" $L_1$ loss targets accuracy at mask boundaries by upweighting errors near the editing seam. Training proceeds in stages: initially optimizing only $\alpha$ with $s$ disabled, then activating $s$ and emphasizing halo fidelity as training progresses. Empirical choices: $\lambda_E=1$, $\lambda_H\approx0.5$.

4. Integration into Diffusion Sampling Pipelines

DecFormer is fully modular and plug-compatible with any diffusion autoencoder that employs latent-space inpainting or compositing. It operates independently of the diffusion backbone, requiring no finetuning. During each sampling step ($z_t \to z_{t'}$) in the denoising chain, a $z_0$-retargeting strategy is employed:

• Predict the denoised latent $z_0^\theta$ from the current $z_t$.
• Use DecFormer to blend $z_0^\theta$ (predicted current) and $z_0^{\mathrm{ref}}$ (reference/completion) with $M$ to obtain $z_0^*$:

$$z_0^* = (1-\alpha)\odot z_0^\theta + \alpha\odot z_0^{\mathrm{ref}} + s$$

• Update the velocity as $v^* = (z_t - z_0^*)/t$ and proceed to $z_{t'} = z_t + (t' - t)v^*$.

DecFormer imposes only ≈3.5% FLOP overhead (80 GFLOPs per 1024×1024 step, compared to the backbone’s 2000 GFLOPs). The mask processing CNN is called once per generation (28 GFLOPs).

5. Quantitative Performance and Analysis

DecFormer's efficacy versus the traditional latent blending baseline is demonstrated on COCO 2017 val with Compositions-1k masks, at 1024 px, summarized here:

Mask Type	SSIM	PSNR (dB)	LPIPS	Halo-L1
Soft ($\sigma=21$)	0.985±0.003 vs 0.941±0.010	41.3 vs 32.9	0.027 vs 0.088	0.018 vs 0.050 (≈64% reduction)
Binary	0.964 vs 0.913	35.7 vs 28.4	0.045 vs 0.110	0.060 vs 0.141 (≈57% reduction)

For thin-edge masks, error reduction at the boundaries reaches up to 53%. DecFormer consistently produces boundary-accurate, color-stable, and artifact-free composites, as visually corroborated by edge-focused evaluations (see Figure 1 in the source).

Ablation studies show that naive channel-agnostic blending, or omitting the residual shift $s$, leads to substantial degradation along mask seams and even away from the boundaries, confirming the benefit of both design elements.

6. Applications: Inpainting and Latent-Space Editing

In the context of diffusion inpainting, DecFormer directly enhances the FLUX.1-Dev baseline (without backbone modification):

• SSIM increases from 0.643 to 0.682
• PSNR increases from 13.58 to 13.94 dB
• LPIPS decreases from 0.354 to 0.314
• FID decreases from 23.51 to 20.56

Training a low-rank LoRA (16 rank, 1M iterations) for this composite baseline closes the gap to the fully finetuned FLUX.1-Fill inpainting model (12B parameters).

DecFormer’s PELC training generalizes to other pixel-space operators, as demonstrated in parametric color-correction tasks (gamma, contrast, brightness). A FiLM-conditioned DecFormer variant trained to approximate $F(x;\gamma,c,b)=(x^{1/\gamma}-0.5)c+0.5 + b$ in latent space achieves LPIPS=0.0875 and PSNR=27.3 dB (naive formula yields LPIPS≈0.50), supporting PELC as a generic recipe for latent editing (Bradbury et al., 4 Dec 2025).

7. Significance, Limitations, and Extensions

DecFormer is the first latent-space compositor to satisfy strict pixel-equivalence constraints with a frozen diffusion VAE, resolving longstanding issues of boundary artifacts and color fidelity in mask-based editing. Its architectural efficiency (7.7M parameters, 3.5% computational overhead) and modularity allow immediate integration into any VAE-based latent diffusion pipeline. Potential limitations include inherited VAE artifacts and scalability bounds set by latent spatial resolution (typically 1/8 × input size). A plausible implication is that extending the DecFormer/PELC paradigm to multi-modal or higher-resolution latent spaces, or to non-VAE latent autoencoders, may require further architectural tuning. Its design offers a foundation for future latent-space editing tools requiring guaranteed fidelity to pixel-domain semantics (Bradbury et al., 4 Dec 2025).