Representation Forcing for Unified Multimodal Models

• Representation Forcing (RF) is a technique that forces a unified multimodal model to predict discrete visual representation tokens as structural scaffolds for image generation.
• RF replaces frozen, VAE-based pathways by integrating discrete representation prediction directly in the decoder, thus enabling end-to-end training.
• RF employs autoregressive token prediction followed by pixel-space denoising, effectively bridging high-level semantic structure with low-level rendering.

Searching arXiv for papers on “Representation Forcing” and closely related multimodal generation work. Representation Forcing (RF) is a training and inference technique for unified multimodal models (UMMs) in which the model is forced to predict discrete visual representations as intermediate tokens before generating pixels. In the formulation introduced for bottleneck-free UMMs, RF addresses a specific architectural problem: many ostensibly unified systems still rely on a frozen, separately pretrained VAE for image generation, so generation remains dependent on an external latent space rather than being fully end-to-end within the multimodal backbone (Wang et al., 29 May 2026). RF replaces that dependency by making representation prediction a native capability of the decoder: an EMA copy of the jointly trained image encoder extracts patch-wise features from ground-truth images, those features are discretized into representation tokens, and the decoder autoregressively predicts those tokens before pixel-space diffusion, while keeping them in context so that pixel generation can attend to them through shared self-attention (Wang et al., 29 May 2026). In a distinct and unrelated use of the same abbreviation, the term “RF” also appears in bounded-arithmetic forcing as a specific typical forcing with partial structures; that usage belongs to model theory and NP search complexity rather than multimodal learning (Müller, 2020).

1. Conceptual definition and scope

In contemporary multimodal modeling, RF denotes a mechanism for converting perception-side visual representations into explicit generation targets. The core claim is that the model’s own understanding encoder already learns structured features for understanding, such as object identity, spatial layout, and composition, and that these features can be repurposed to scaffold image generation inside the same backbone (Wang et al., 29 May 2026). Rather than treating visual features as auxiliary signals or external conditioning sources, RF inserts them into the model’s native token sequence.

The motivating problem is the structural bottleneck created by frozen VAE-based generation pathways in unified models. In that setup, the latent space is optimized for reconstruction rather than for the UMM’s joint multimodal objectives, the compression is lossy, and the VAE remains an external component, so image generation is not end-to-end within the UMM (Wang et al., 29 May 2026). A natural alternative is to remove the VAE and diffuse directly in pixel space, but the cited work reports that naively doing so creates a quality gap because the model must learn both high-level semantic structure and low-level rendering from raw pixels and rich conditioning; without a structural scaffold, it tends to produce distorted shapes and incoherent layouts (Wang et al., 29 May 2026).

RF is therefore defined by a particular internal factorization of generation. The decoder first predicts a sequence of representation tokens that encode high-level structure, and then performs pixel-space denoising conditioned on both the original conditioning context and those predicted tokens. This suggests that RF is not merely a regularizer but an explicit decomposition of multimodal image synthesis into structure prediction followed by rendering (Wang et al., 29 May 2026).

2. Architectural mechanism

The RF architecture operates on a unified sequence ordered as text tokens, representation tokens, and pixel patches: $[T, R, P]$ (Wang et al., 29 May 2026). Text tokens follow standard causal autoregression. Representation tokens are also predicted autoregressively under the same causal mask and are placed after text tokens in the sequence. Pixel patches, by contrast, use bidirectional self-attention among themselves to implement diffusion or flow, while attending causally to all preceding text and representation tokens (Wang et al., 29 May 2026). This mask design is central because it allows representation tokens to serve as in-context conditioning without adding a separate cross-attention module.

The underlying backbone is described as Mixture-of-Transformers style: all tokens share multi-head self-attention layers, while FFN blocks are routed to modality-specific expert pools for understanding, representation prediction, and pixel generation (Wang et al., 29 May 2026). The image encoder is DINOv3 ViT-H+/16 with NaViT-style variable-resolution support and is jointly trained; an EMA copy provides targets for the representation sequence $R$ (Wang et al., 29 May 2026). The shared transformer outputs next-token logits at text and representation-token positions and an $x$-prediction head at pixel positions for flow matching (Wang et al., 29 May 2026).

A defining property of RF is that the representation tokens remain in sequence during image generation. Pixel patches can therefore attend directly to them through the shared self-attention trunk. In the terminology of the source paper, this implements “in-context conditioning” by $R$ without a separate cross-attention module or external latent (Wang et al., 29 May 2026). Relative to prior VAE-based UMMs, the practical implication is that the generative pathway no longer depends on an external tokenizer or fixed latent decoder.

3. Mathematical formulation

RF induces a two-stage factorization inside one backbone:

$$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$

with representation prediction modeled autoregressively as

$$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$

(Wang et al., 29 May 2026).

The notation in the paper is as follows. $C$ denotes the conditioning context, such as text tokens; $X$ is the ground-truth image, decomposed into pixel patches $P$; $f_{\mathrm{enc}}$ is the jointly trained understanding image encoder; $R$0 is an exponential moving average copy of $R$1 used to extract slow-moving representation targets during training; $R$2 is the discrete representation token sequence obtained by quantizing $R$3’s patch features; and $R$4 denotes the parameters of the shared generative backbone (Wang et al., 29 May 2026).

Training uses three losses. The representation-prediction loss is autoregressive cross-entropy:

$R$5

If text next-token prediction is included, the language-modeling loss is

$R$6

For pixels, the model uses flow matching with $R$7-prediction and a velocity loss. Given clean pixel patches $R$8, Gaussian noise $R$9, and $x$0,

$x$1

$x$2

and

$x$3

The combined objective is

$x$4

(Wang et al., 29 May 2026).

Classifier-free guidance training is supported by independently dropping $x$5 and/or the $x$6 sequence with probability $x$7 when computing $x$8 and $x$9, enabling unconditional and partially conditional branches for guidance at inference (Wang et al., 29 May 2026). This suggests that RF is integrated not only into the model’s representation pipeline but also into its guidance strategy.

4. Representation tokens and quantization

Representation tokens in RF are derived from patch-level features of the EMA encoder. These last-layer patch features are extracted before final norm and normalized to unit vectors (Wang et al., 29 May 2026). A learnable codebook $R$0 is maintained, with each patch feature assigned to its nearest prototype by cosine similarity. To stabilize assignments as the encoder evolves, the codebook uses momentum updates of prototypes in a SwAV-style manner and applies one Sinkhorn–Knopp iteration per batch for usage balancing to prevent codebook collapse (Wang et al., 29 May 2026). The codebook is updated online with momentum $R$1, such as $R$2, and temperature $R$3, such as $R$4, and the discrete indices form the token sequence $R$5 (Wang et al., 29 May 2026).

Spatial structure is preserved by allocating one representation token per pooled spatial patch. The default vocabulary size is $R$6, with $R$7 also tested (Wang et al., 29 May 2026). Spatial pooling is $R$8 on encoder features before quantization; with $R$9 pixel patching, pooling yields $$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$0 representation tokens for every $$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$1 pixel patches over the same spatial layout (Wang et al., 29 May 2026). Each representation token is embedded as the sum of a learned $$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$2D spatial position embedding and a learned code embedding from a $$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$3-entry table (Wang et al., 29 May 2026).

A notable design decision is the use of discrete rather than continuous representation targets. The reported ablation states that discretization, trained by cross-entropy, far outperforms continuous regression of features on GenEval, with $$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$4 versus $$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$5 (Wang et al., 29 May 2026). The source attributes this to robustness to sequential error accumulation and to the fact that discrete prediction encourages encoding high-level structure while discarding fine detail, thereby separating structure from rendering (Wang et al., 29 May 2026). A plausible implication is that RF depends on abstraction rather than exact feature reconstruction: the representation sequence should preserve compositional and spatial information while omitting low-level image detail that the pixel denoiser is intended to supply.

5. Training and inference workflow

The reported training procedure uses a mixture of text-only corpora and large-scale text–image pairs that include both understanding tasks, such as VQA, documents, and spatial reasoning, and generation tasks such as text-to-image (Wang et al., 29 May 2026). Resolution handling follows NaViT-style dynamic resolution sampling and variable-resolution batching within per-stage limits (Wang et al., 29 May 2026).

Training proceeds in three stages. Stage 1 freezes the backbone and encoder and trains an MLP connector for $$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$6K iterations. Stage 2 unfreezes all components and trains on text and text–image pairs up to $$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$7px resolution for $$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$8K iterations. Stage 3 scales to $$p_\theta(R, P \mid C) = p_\theta(R \mid C)\cdot p_\theta(P \mid R, C),$$9px resolution for $$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$0K iterations (Wang et al., 29 May 2026). Optimization uses AdamW with $$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$1, $$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$2, $$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$3, weight decay $$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$4, and gradient clipping $$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$5; the learning rate uses linear warmup then constant scheduling, with $$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$6 in Stages 1–2 and $$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$7 in Stage 3, and generation-related newly initialized parameters use a $$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$8 learning-rate multiplier while the LLM backbone uses the base learning rate (Wang et al., 29 May 2026).

During training, the ground-truth representation sequence $$p_\theta(R \mid C)=\prod_{i=1}^{L_R} p_\theta(r_i \mid C, r_{<i})$$9 from $C$0 is provided at representation-token positions by teacher forcing and is also the target for $C$1; pixel patches are noised per step for $C$2 (Wang et al., 29 May 2026). At inference, neither $C$3 nor teacher forcing is used. Instead, the model first predicts $C$4 autoregressively from the text prompt and then runs pixel-space rectified-flow denoising conditioned on both the text and the predicted representation sequence (Wang et al., 29 May 2026).

The paper’s inference configuration uses the EMA of model parameters with decay $C$5, classifier-free guidance for both stages, and approximately $C$6 for representation-token sampling and $C$7 for pixel denoising. It uses $C$8 flow-matching steps with dynamic timestep shifting (Wang et al., 29 May 2026). The representation stage samples tokens autoregressively using top-$C$9 over guided logits, and the pixel stage integrates the rectified-flow ODE with Euler updates (Wang et al., 29 May 2026).

6. Empirical results and ablations

The reported experiments initialize the backbone from Qwen3-A3B with $X$0B active parameters and use a MoT architecture with three expert pools for understanding, representation prediction, and pixel generation. The image encoder is DINOv3 ViT-H+/16, jointly trained, with an EMA copy used for representation targets. The main RF model operates in pixel space, while controlled VAE-based variants use WanX-2.1 VAE latents with all else kept identical (Wang et al., 29 May 2026).

On text-to-image generation, RF-Pixel achieves GenEval overall $X$1 without rewriter, slightly above BAGEL at $X$2, and with rewriter reaches $X$3, matching state-of-the-art unified models. On DPG-Bench, RF-Pixel scores $X$4, described as comparable to VAE-based UMMs (Wang et al., 29 May 2026). On image understanding, four variants are compared under the same pretraining baseline and budget: VAE, VAE+RF, Pixel, and Pixel+RF. RF improves understanding in both generation pathways, and Pixel+RF shows gains on $X$5 benchmarks, including MMMU $X$6, MME $X$7, BLINK $X$8, and AI2D $X$9, while showing small reductions on DocVQA ($P$0) and ChartQA ($P$1); Pixel+RF outperforms VAE+RF on $P$2 benchmarks (Wang et al., 29 May 2026).

The most direct ablation concerns whether RF is necessary in pixel space. Without RF, pixel-space GenEval is $P$3; with RF, it rises to $P$4, matching VAE+RF at $P$5. VAE-based generation also benefits from RF, improving from $P$6 to $P$7 (Wang et al., 29 May 2026). Another ablation compares RF with REPA: in pixel space, REPA gives $P$8 whereas RF yields $P$9 (Wang et al., 29 May 2026). This supports the interpretation that direct in-context conditioning via predicted representation tokens is more effective than auxiliary feature alignment.

Additional ablations indicate that codebook size is not highly sensitive in the tested range, with $f_{\mathrm{enc}}$0 and $f_{\mathrm{enc}}$1 giving $f_{\mathrm{enc}}$2 and $f_{\mathrm{enc}}$3 respectively on GenEval, and that DINOv3 outperforms SigLIP2 on most understanding benchmarks, with better spatial and structural fidelity helping RF (Wang et al., 29 May 2026). These findings suggest that RF depends more critically on the existence of a structurally informative representation sequence than on fine-grained codebook scaling within the examined range.

7. Relation to adjacent methods, limitations, and alternate usage

RF is positioned against several neighboring lines of work. VAE-dependent UMMs such as Transfusion, Show-o or Show-o2, the Janus series, BAGEL, JanusFlow, and Emu3 unify generation and understanding in one backbone but retain a frozen VAE or an external diffusion module; RF removes that pretrained generative bottleneck and keeps everything end-to-end (Wang et al., 29 May 2026). Standalone pixel-space generators such as JiT, SiD2, PixelFlow, and PixNerd show that pixel-space generation is feasible, but in UMMs naive pixel-space diffusion lags in quality; RF is presented as supplying the missing structural scaffold from within the model (Wang et al., 29 May 2026). Compared with representation-guided approaches such as REPA and RAE, RF differs in turning the model’s own perception outputs into explicit generation targets predicted autoregressively and kept in context, rather than using external or frozen representations as auxiliary alignment signals (Wang et al., 29 May 2026).

The reported benefits are threefold: removal of the VAE bottleneck, provision of a structural scaffold that lets diffusion focus on low-level rendering, and creation of a shared representation space in which understanding and generation co-train the same representation (Wang et al., 29 May 2026). The trade-offs are equally explicit. Pixel-space diffusion is heavier in compute and memory than latent diffusion, and adding representation tokens increases sequence length (Wang et al., 29 May 2026). Sensitivity to representation design is substantial: discrete tokenization and pooling choices matter, and poor codebook dynamics can cause collapse, which the method mitigates via EMA targets and Sinkhorn balancing (Wang et al., 29 May 2026). Small drops on document-centric tasks such as DocVQA and ChartQA suggest that the representation sequence is tuned to structural semantics more than to fine OCR or layout fidelity (Wang et al., 29 May 2026). The reported training budget also relies on initialization from a strong LLM and a practical three-stage schedule; fully from-scratch pretraining is identified as future work (Wang et al., 29 May 2026).

In a different literature, “Representation Forcing” or “RF” denotes a forcing construction in bounded arithmetic. There, RF is described as a specific typical forcing “with partial structures” designed to force representation properties of finite structures coded by a generic oracle, and it is used to obtain independence results for total NP search problems over the universal variant of relativized $f_{\mathrm{enc}}$4 (Müller, 2020). That meaning is unrelated to multimodal generation, despite the shared acronym. The coexistence of these two uses makes disambiguation necessary in cross-disciplinary contexts.