UnifiedVisual: Unifying Visual Representations

• UnifiedVisual is a design principle that replaces fragmented, modality-specific approaches with a single, unified visual substrate to standardize representation, model families, and task objectives.
• It employs diverse representations—such as painted text images, discrete tokens, and diagram code—to handle tasks ranging from image segmentation and captioning to visual generation and control.
• Empirical results indicate that unified representations can enhance efficiency, improve robustness to variations, and foster emergent cross-modal behaviors without relying on separate task-specific modules.

“UnifiedVisual” (Editor’s term) denotes a family of research programs that seek to replace modality-specific, task-specific, or pipeline-specific fragmentation with a single visual or visually grounded substrate. In recent arXiv work, that substrate has appeared as learned discrete guiding codes, discrete visual tokens, painted text images, visual specification pages, unified view features, or structured diagram code. The common premise is that a model becomes more genuinely unified when representation, task formulation, and objective are standardized rather than merely sharing a backbone (Kolesnikov et al., 2022, Wu et al., 2024, Zhang et al., 21 Nov 2025).

1. Conceptual foundations

The unification agenda in vision has been articulated in several distinct but convergent forms. UViM proposes “the same functional form for all tasks” and applies one two-component recipe to panoptic segmentation, depth prediction, and image colorization (Kolesnikov et al., 2022). VILA-U integrates “Video, Image, Language understanding and generation” within a single autoregressive next-token prediction framework rather than coupling a perception model to a separate diffusion model (Wu et al., 2024). UniModel makes the strongest representation-level claim: multimodal learning can be reformulated as a fully vision-native problem once both text and images are expressed as RGB pixels, so that captioning and text-to-image generation become two directions of one shared pixel-to-pixel process (Zhang et al., 21 Nov 2025).

This suggests that UnifiedVisual is best understood not as one architecture class, but as a design principle with three recurring ambitions: one representation, one model family, and one task algebra. Some papers unify dense prediction tasks; others unify understanding and generation; others unify control, editing, or multimodal reasoning. What remains stable is the attempt to move from ad hoc cross-modal interfaces toward a common substrate.

System	Shared representation or interface	Unified scope
UViM	learned discrete guiding code	panoptic segmentation, depth prediction, image colorization
VILA-U	discrete visual tokens with next-token prediction	Video, Image, Language understanding and generation
UniModel	RGB image $\longleftrightarrow$ painted text image	visual understanding and visual generation
UniVL	textual instruction rendered onto the spatial mask	spatially grounded contextual image generation
VCG-Bench	mxGraph XML	Vision-to-Code generation and Code-to-Code editing

These systems illustrate different answers to the same question: what representation should be shared if a model is to cross task boundaries without being reduced to a loose multi-head bundle (Wang et al., 20 May 2026, Su et al., 15 May 2026).

2. Representation unification

Representation design is the most decisive axis of recent unified visual systems. UniModel removes the text–image modality gap by rendering “textual prompts, captions, questions, and answers” onto a blank $512 \times 512 \times 3$ canvas using a fixed font and line spacing. The resulting “painted text image” is processed by the same preprocessing pipeline and the same VAE encoder as a natural image, yielding the slogan

$$\text{RGB image} \;\longleftrightarrow\; \text{painted text image}.$$

In this formulation, there is “no language tokenizer, no text embedding table, and no separate language decoder” (Zhang et al., 21 Nov 2025).

UniVL adopts a related but spatially grounded variant. Instead of pairing a mask-conditioned image encoder with a separate text encoder, it renders the text label directly inside each mask region, creating one contextual condition image $C_I$. The UniVL encoder, adapted from an OCR-pretrained backbone, reads this composite image and produces a single embedding $f_{\text{VL}}$ that fuses spatial context and semantic intent. The fusion rule is explicitly mask-aware: $$f_{\text{VL}} = f_{\text{VAE}} \odot (1 - M) + f_{\text{CLIP}} \odot M.$$ This removes the standalone text encoder at inference time and makes “what should appear where” a single optical input (Wang et al., 20 May 2026).

A third variant appears in “visual-to-visual (V2V) generation.” There the conditioning input is a “visual specification page” containing rendered text, color chips, inline thumbnails, sketches, poses, style references, or layout cues. V2V-Zero is training-free: it replaces text-only conditioning with final-layer hidden states extracted from such visual pages, exploiting the fact that a frozen VLM already maps both text and images into the generator’s conditioning space (Liu et al., 12 May 2026). The paper’s mechanistic analysis further reports that the default reasoning path is primarily visually routed, with 95.0% of conditioning-token attention mass on visual-page hidden states (Liu et al., 12 May 2026).

Not all unified representations remain purely pixel-based. VCG-Bench argues that professional diagram workflows are better served by a “Diagram-as-Code” paradigm using mxGraph XML rather than pixel synthesis. In that setting, nodes, edges, geometry, and styles are explicit symbolic objects, and both generation and editing are evaluated over the same executable representation (Su et al., 15 May 2026). DuetSVG makes a similar move for vector graphics, but keeps image tokens and SVG tokens in one autoregressive model so that code generation is visually guided during decoding (Zhang et al., 11 Dec 2025).

In robotics, representation unification can be physical rather than symbolic. RoboUniView learns a “unified 3D space view” from multi-perspective images using UVFormer, with UniView queries tied to a discretized workspace where $L=B=20$ and each pillar cell represents $0.05^2$ meters in the ground plane (Liu et al., 2024). Here the shared space is not language-like or code-like; it is a camera-normalized spatial field aligned with manipulation.

3. Unified models and objectives

Once a common representation is chosen, unified visual systems typically reduce heterogeneous tasks to one training objective or one generative formalism. UniModel uses a single Unified Diffusion Transformer trained with rectified flow in VAE latent space. With target latent $\mathbf{z}_0$, Gaussian noise $\mathbf{z}_1$, and interpolation

$512 \times 512 \times 3$0

the model learns the velocity field

$512 \times 512 \times 3$1

The same loss supervises both RGB$512 \times 512 \times 3$2text and text$512 \times 512 \times 3$3RGB because source and target are randomly swapped with equal probability during training (Zhang et al., 21 Nov 2025).

Autoregressive tokenization is the other dominant strategy. VILA-U treats text and visual codes as one multimodal sequence and trains standard next-token prediction over both. Its unified vision tower is pretrained with both reconstruction and contrastive alignment, because the paper argues that discrete visual tokens must be semantically aligned with text rather than optimized only for reconstruction (Wu et al., 2024). UniVLA extends the same logic to action: vision, language, and action are all discrete tokens in one shared sequence space, with

$512 \times 512 \times 3$4

and task-specific supervision applied only to the relevant token types. The paper’s central architectural move is autoregressive interleaving of $512 \times 512 \times 3$5, $512 \times 512 \times 3$6, and $512 \times 512 \times 3$7, so that the model learns both policy outputs and causal video dynamics (Wang et al., 24 Jun 2025).

UViM takes a two-stage route. A feed-forward base model predicts raw vision outputs conditioned on a short learned discrete code, while an autoregressive LLM predicts that code from the image: $512 \times 512 \times 3$8 The resulting division of labor is explicit: the LLM handles structured interdependent data, while the base model handles high-dimensional outputs efficiently (Kolesnikov et al., 2022).

DuetSVG combines image-token generation and SVG-token generation in one causal transformer: $512 \times 512 \times 3$9 Because image tokens are generated before SVG tokens, the SVG decoder can attend to the model’s own visual prediction as “internal visual guidance,” which the paper identifies as the central reason for improved geometric coherence and semantic fidelity (Zhang et al., 11 Dec 2025).

4. Task domains and application regimes

Unified visual formulations have been applied to a wide range of tasks that were previously separated by output type or modality interface. In UniModel, image captioning, VQA, and related understanding tasks become “image $$\text{RGB image} \;\longleftrightarrow\; \text{painted text image}.$$0 painted text image,” while text-to-image generation becomes “painted text image $$\text{RGB image} \;\longleftrightarrow\; \text{painted text image}.$$1 RGB image.” The paper emphasizes that the same backbone is used bidirectionally, with only lightweight task embeddings distinguishing understanding from generation (Zhang et al., 21 Nov 2025).

VILA-U broadens the scope to joint understanding and generation over both images and videos. It reports competitive performance on VQAv2, GQA, TextVQA, POPE, MME, SEED, MM-Vet, MSVD-QA, MSRVTT-QA, TGIF-QA, and ActivityNet-QA, while also reporting MJHQ-30K FID of 12.81 at $$\text{RGB image} \;\longleftrightarrow\; \text{painted text image}.$$2 resolution and 7.69 at $$\text{RGB image} \;\longleftrightarrow\; \text{painted text image}.$$3 resolution (Wu et al., 2024). The stated significance is that autoregressive image generation can be practically competitive when the visual token space is aligned and the data are sufficiently high quality (Wu et al., 2024).

In controllable image generation, UniVL reframes conditioning around spatially grounded contextual image generation. On UniVL-ImgGen, it reports FID 11.13, PSNR 19.61, SSIM 0.754, and MUSIQ 73.36, improving over OminiControl without a text prompt, while also eliminating the text encoder and reducing inference TFLOPs by up to 51.8% and runtime by up to 44.2% (Wang et al., 20 May 2026). V2V-Zero addresses a related conditioning problem from the interface side: it allows a user to prompt a generator with a visual page rather than a sentence, and on GenEval reaches 0.85 with a frozen Qwen-Image backbone (Liu et al., 12 May 2026).

In structured graphics, VCG-Bench unifies diagram generation and editing around mxGraph XML. It defines Vision-to-Code generation and Code-to-Code editing in one benchmark, with 1,449 curated diagrams spanning 6 domains and 15 sub-domains and a tailored metric suite including Execution Success Rate, Style Consistency Score, CodeXQA, SigLIP2 similarity, and XDRFR (Su et al., 15 May 2026). DuetSVG analogously treats vector graphics as both code and image, and supports T2SVG, I2SVG, SVG completion, and semantic SVG editing in one multimodal model (Zhang et al., 11 Dec 2025).

In robotics, the same unification principle appears in two forms. RoboUniView decouples view construction from action learning and reports CALVIN improvements from 93.0% to 96.2% in the $$\text{RGB image} \;\longleftrightarrow\; \text{painted text image}.$$4 setting and from 92.2% to 94.2% in the $$\text{RGB image} \;\longleftrightarrow\; \text{painted text image}.$$5 setting, while also remaining stable under unseen camera parameters (Liu et al., 2024). UniVLA instead tokenizes vision, language, and action jointly and uses world-model post-training on 622K robot-centric videos. It reports 95.5% average success on LIBERO, surpassing pi0-FAST’s 85.5%, and shows transfer to ALOHA manipulation and autonomous driving (Wang et al., 24 Jun 2025).

Unified formulations also extend to tracking and audio-visual understanding. UVLTrack uses the same parameters for BBOX, NL, and NL+BBOX tracking and reports state-of-the-art results across seven visual tracking benchmarks, three vision-language tracking benchmarks, and three visual grounding datasets (Ma et al., 2024). AV-Unified standardizes event localization, parsing, sound source localization, segmentation, and question answering into a shared tokenized interface, reaching 78.7 on AVE, 39.16 CIoU and 41.24 AUC on VGG-SS, and 76.42 overall on MUSIC-AVQA (Li et al., 6 Mar 2026).

5. Empirical regularities and evaluation patterns

A notable empirical pattern is that successful unification often produces behaviors not explicitly optimized for. UniModel reports emergent cycle inference

$$\text{RGB image} \;\longleftrightarrow\; \text{painted text image}.$$6

and states that this cycle-consistent behavior emerges without any explicit cycle-consistency loss. The reconstructed image preserves key semantics even when the intermediate painted text contains distorted glyphs or incomplete words, which the paper interprets as evidence of a genuinely shared visual semantic space (Zhang et al., 21 Nov 2025).

Another recurring pattern is that representation unification can improve robustness to nuisance variation. RoboUniView shows that camera changes that reduce RoboFlamingo success from 86.3% to 80.8% can be countered by a camera-normalized unified view representation, with performance in $$\text{RGB image} \;\longleftrightarrow\; \text{painted text image}.$$7 remaining close to the standard setting (Liu et al., 2024). UVLTrack reports that its multi-modal contrastive loss improves AUC by +1.2 in BBOX, +2.2 in NL, and +2.3 in NL+BBOX, indicating that explicit alignment can help even when one modality is absent at test time (Ma et al., 2024).

Benchmarks designed around unified outputs also reveal sharp asymmetries between related abilities. VCG-Bench finds that Vision-to-Code generation is substantially harder than Code-to-Code editing: ESR is often near-saturated for editing, while generation remains bottlenecked by executability, structured fidelity, and counting (Su et al., 15 May 2026). V2V-Zero similarly reports a clear capability hierarchy on Simple-V2V Bench: attribute binding is strong, content generation is unreliable, and structural control remains hard even for commercial systems (Liu et al., 12 May 2026).

Efficiency is another prominent evaluation axis. UniVLR replaces interleaved text chain-of-thought plus visual latent tokens with a unified visual latent reasoning path in which textual reasoning traces and auxiliary evidence are rendered into one canvas and compressed into latent tokens. It reports overall scores of 82.7 on V*, 73.3 on HRBench4K, 68.8 on HRBench8K, and 50.7 on MME-RealWorld-Lite, while using only 12 latent tokens at inference and no generated intermediate text. The paper characterizes this as about a 15.2× reduction in generated reasoning tokens with an average accuracy improvement of 5.4% (Jiang et al., 12 May 2026).

6. Limitations, misconceptions, and open directions

A recurring misconception is that unification is equivalent to parameter sharing. Several papers explicitly reject this weaker interpretation. UniModel argues that many systems are only “superficially unified” because they still keep text and images in different representations, optimize different objectives, and use different heads; its claim is that backbone sharing alone is not enough (Zhang et al., 21 Nov 2025). AV-Unified makes a related point from a multi-task perspective: prompt conditioning and specialized temporal and spatial modules remain necessary even within a shared framework, because different tasks stress different cues (Li et al., 6 Mar 2026).

Another misconception is that a single shared representation automatically dominates specialized systems. The evidence is more mixed. AV-Unified reports that some task-specific baselines still do better on single tasks and that some easier subtasks can lose performance under joint training (Li et al., 6 Mar 2026). UniVL shows that adding T5 back to form a UniVL+T5 hybrid improves some alignment metrics, but this sacrifices the efficiency gains that motivated the unified visual-language pathway (Wang et al., 20 May 2026). VILA-U notes that contrastive alignment improves understanding but can slightly hurt pure generation fidelity relative to a reconstruction-only tokenizer (Wu et al., 2024).

Representation choices also impose hard limits. VCG-Bench shows that current VLMs remain weak in structured fidelity, instruction compliance, topology recovery, and executable diagram generation (Su et al., 15 May 2026). V2V-Zero finds that pose control and sketch reference remain particularly difficult (Liu et al., 12 May 2026). UniVLR notes that latent reasoning tokens are less human-interpretable than textual chain-of-thought and are not intended to replace external tools for tasks requiring exact measurement or exhaustive search (Jiang et al., 12 May 2026). UniVLA states that reinforcement learning integration, larger video datasets, and broader post-training scalability remain open directions (Wang et al., 24 Jun 2025).

Taken together, these results suggest that UnifiedVisual is less a claim that all problems should collapse into pixels, tokens, or code, than a claim that the boundary between understanding, generation, control, and reasoning can often be redrawn around a shared substrate. The current literature indicates several viable substrates—painted text images, discrete visual tokens, rendered spatial instructions, visual specification pages, mxGraph XML, unified 3D workspace features, and multimodal token sequences—but it also shows that the choice of substrate determines the remaining bottlenecks. The field’s central technical question is therefore not whether to unify, but which representation and objective preserve enough structure to make unification computationally efficient, semantically aligned, and task-general at the same time.