TBAC-UniImage: Unified Multimodal Generation

• TBAC-UniImage is a unified multimodal model that integrates a frozen pre-trained MLLM with a diffusion transformer via layered conditioning.
• The model employs Ladder-Side Diffusion Tuning to inject intermediate MLLM representations into successive DiT layers, enhancing semantic depth.
• It uses a parameter-efficient training pipeline with Flow Matching and staged pretraining, achieving competitive results on GenEval, DPG-Bench, and TIIF-Bench.

Searching arXiv for the target paper and closely related unified multimodal generation work. TBAC-UniImage is a unified multimodal understanding-and-generation model that couples a frozen pre-trained Multimodal LLM (MLLM) with a fine-tuned Diffusion Transformer (DiT) through “Ladder-Side Diffusion Tuning,” a layer-wise conditioning scheme in which multiple intermediate MLLM representations guide generation rather than only the final hidden state. The model is presented as a response to two limitations in prior diffusion-based unified models: shallow final-state conditioning and the computational cost of training unified generative architectures from scratch. Its reported implementation uses Qwen2.5-VL-3B-Instruct as the understanding backbone, SANA-1600M-512px as the generator, learnable query tokens, and a lightweight two-layer connector, with empirical evaluation on GenEval, DPG-Bench, and TIIF-Bench (Xu et al., 11 Aug 2025).

1. Concept and positioning

TBAC-UniImage is defined as a unified model for multimodal understanding and generation. Its central claim is that unification should be deeper than passing a single semantic summary from an MLLM into a generator. Instead, the model exposes the diffusion generator to a hierarchy of semantic representations drawn from multiple depths of the MLLM. This design is described as “Ladder-Side Diffusion Tuning,” with the diffusion model treated as a generative ladder that receives guidance from diverse layers of the MLLM (Xu et al., 11 Aug 2025).

The paper situates TBAC-UniImage against three broader approaches to unified image generation. One approach discretizes images and predicts image tokens autoregressively; the paper criticizes this for violating the 2-D structure of images. A second approach adds a diffusion head to an off-the-shelf LLM; the paper argues that this tends to treat generation as a bolt-on tool rather than a deeply integrated component. A third approach trains a unified model from scratch; this is described as computationally expensive and prohibitive for many researchers. TBAC-UniImage is positioned as a middle ground: more deeply unified than final-state conditioning, but more training-friendly than full from-scratch pretraining.

A plausible implication is that the model’s notion of unification is architectural rather than merely interface-level. Understanding is treated as a layered process inside the MLLM, and generation is aligned to that process rather than to a single terminal representation.

2. Architectural composition

The architecture combines four named components: a pre-trained MLLM, a pre-trained DiT, a set of learnable query tokens, and a lightweight connector. The MLLM is Qwen2.5-VL-3B-Instruct, and the DiT is SANA-1600M-512px. The MLLM serves as the understanding backbone, while the DiT serves as the generator. The two are not trained as a single monolithic architecture from scratch; instead, they are integrated through layer-wise alignment (Xu et al., 11 Aug 2025).

The learnable queries are defined as

$\mathcal{Q} \in \mathbb{R}^{N \times D}.$

These queries are injected alongside the prompt input into the MLLM. As they pass through the MLLM, their hidden states are extracted at multiple layers, denoted

$Q^{(l)}.$

If the MLLM has $m$ layers and the DiT has $n$ layers, with the assumption $m \ge n$, the selected MLLM query states are progressively mapped into the DiT so that $Q^{(m-n+1)}$ conditions DiT layer 1, $Q^{(m-n+2)}$ conditions DiT layer 2, and so on until $Q^{(m)}$ conditions DiT layer $n$.

Each selected MLLM hidden state is processed by a lightweight two-layer connector $f$, yielding conditioning signals of the form

$Q^{(l)}.$0

The paper states that this connector has only about 10M trainable parameters. The MLLM itself is frozen, and training focuses on the learnable queries, the DiT parameters, and the connector. This is described as a parameter-efficient alignment strategy.

This design suggests that TBAC-UniImage attempts to preserve semantic granularity across the full generative process. Rather than providing the diffusion model with one compressed summary vector, it provides a sequence of layer-specific signals aligned to the DiT depth.

3. Ladder-Side Diffusion Tuning

“Ladder-Side Diffusion Tuning” is the paper’s defining mechanism. The name is explicitly inspired by Ladder Side-Tuning from LLM transfer learning, but adapted to diffusion. In this formulation, the diffusion model is the side module attached to the MLLM, and the attachment occurs at multiple depths rather than at a single final layer (Xu et al., 11 Aug 2025).

For an MLLM with $Q^{(l)}.$1 layers and a DiT with $Q^{(l)}.$2 layers, the mapping is:

$Q^{(l)}.$3

The model therefore uses multiple intermediate MLLM layers as generative conditions. The paper contrasts this with prior methods that rely only on the MLLM’s final hidden state. According to the paper’s framing, final-layer-only conditioning creates a shallow connection because the generator is isolated from the rich hierarchical representations encoded in the MLLM’s intermediate layers.

The intended benefit of the ladder-side design is deeper and more fine-grained unification of understanding and generation. The paper attributes to this design the ability to expose the generator to low-level grounding, mid-level compositional information, high-level instruction semantics, and multimodal understanding signals from different depths. No explicit ablation table quantifying final-layer-only versus multi-layer conditioning is included in the supplied material, so the strength of this claim is primarily conceptual rather than numerically isolated in the available evidence.

This suggests that TBAC-UniImage’s core novelty lies less in inventing a new base generator or backbone and more in specifying how semantic structure should be transferred from an MLLM into a diffusion model.

4. Training procedure and optimization

The model uses Flow Matching as its denoising objective. The paper does not restate the full flow-matching formula in the supplied material, but it explicitly identifies Flow Matching as the objective. The conditioning notation centers on the learnable queries $Q^{(l)}.$4, their layer-wise hidden states $Q^{(l)}.$5, and the connector output $Q^{(l)}.$6 that feeds each DiT layer (Xu et al., 11 Aug 2025).

Training follows a three-stage pipeline:

Stage	Dataset(s)	Configuration
Stage 1: Text-to-image pretraining	BLIP3o-Long-Caption, BLIP3o-JourneyDB	30M data, batch size 512, 150K steps
Stage 2: Image-text-to-image pretraining	GPT-Image-Edit-1.5M	1.5M data, batch size 256, 60K steps
Stage 3: Fine-tuning	BLIP3o-60k, ShareGPT-4o-Image	150K data, batch size 256, 60K steps

The stated goals of these stages are, respectively, to align the diffusion latent space with high-level semantic representations from the MLLM, to improve multimodal conditioning and image-text-to-image generation, and to improve instruction following and generation quality.

The optimizer is AdamW, the scheduler is Cosine, the maximum learning rate is $Q^{(l)}.$7, the minimum learning rate is $Q^{(l)}.$8, warm-up lasts 5,000 steps, and the number of learnable queries is 64. The paper also reports early training instability in the text-to-image phase, specifically gradient norm spikes. To address this, it adopts MetaQuery-style monitoring: gradient norm and loss are continuously monitored, and if a spike crosses a threshold, the gradients are discarded and the update is skipped by zeroing gradients.

The training recipe indicates that the model is not merely fine-tuned on downstream tasks. Instead, it undergoes staged alignment from text-to-image pretraining through multimodal pretraining to instruction-sensitive fine-tuning.

5. Empirical evaluation

TBAC-UniImage is evaluated on GenEval, DPG-Bench, and TIIF-Bench. On GenEval, which measures object-focused text-to-image alignment across single object, two objects, counting, colors, position, and attribute binding, the reported TBAC-UniImage-3B scores are 0.99, 0.94, 0.77, 0.92, 0.83, and 0.75, respectively, with an overall score of 0.87 (Xu et al., 11 Aug 2025).

The paper states that this GenEval overall score equals Qwen-Image at 0.87 and exceeds several unified models listed in the supplied material, including Janus-Pro-7B at 0.80, BLIP3-o-8B at 0.83, MetaQuery-L at 0.78, Emu3-Gen at 0.54, and Show-o at 0.53. In the paper’s interpretation, this indicates strong compositional alignment.

On DPG-Bench, which uses 1,000 complex prompts and reports Global, Entity, Attribute, Relation, Other, and Overall, TBAC-UniImage-3B obtains 83.52, 87.94, 87.80, 87.17, 87.02, and 80.97. The supplied material notes that MetaQuery-L shows an overall 81.10, so TBAC-UniImage is slightly below that raw number, while remaining above BLIP3-o-8B at 80.73 and OpenUni-B-512 at 80.29. It trails specialized text-to-image systems such as Qwen-Image at 88.32, Seedream 3.0 at 88.27, and Lumina-Image 2.0 at 87.20. The evidence therefore supports the narrower statement that it is competitive among unified models rather than dominant across all generative systems.

On TIIF-Bench, TBAC-UniImage reports Overall short 62.37 and Overall long 61.18. In Basic Following, the averages are 75.92 short and 73.97 long, with Attribute 76.50 and 71.75, Relation 81.44 and 79.64, and Reasoning 69.84 and 70.52. In Advanced Following, the averages are 62.91 short and 65.03 long, with Attribute+Relation 74.65 and 71.61, Attribute+Reasoning 57.47 and 63.65, Relation+Reasoning 61.99 and 67.59, Style 80.00 and 66.67, Text 2.71 and 1.36, and Real-World 56.72 and 57.84.

The TIIF-Bench results support two points emphasized in the supplied material: the model is especially competitive in Advanced Following for an open-source model of its size, and its in-image text rendering is a clear weakness, as reflected by the very low scores in the Text subset.

6. Strengths, limitations, and relation to adjacent work

The paper attributes several strengths to TBAC-UniImage: a new unified architecture coupling a frozen MLLM with a pre-trained DiT, the Ladder-Side Diffusion Tuning mechanism, parameter-efficient training through frozen MLLM weights and a small connector, strong compositional generation on GenEval, and strong advanced instruction-following on TIIF-Bench (Xu et al., 11 Aug 2025).

Its explicitly stated limitations are threefold. First, fine-grained comprehension of complex, dense prompts still needs improvement. Second, image-text-to-image consistency can be improved. Third, in-image text rendering remains weak. The TIIF-Bench Text scores strongly reinforce the third limitation.

The supplied material also places TBAC-UniImage within a broader line of work on unified or multimodal image generation. “UNIC-Adapter” proposes a unified image-instruction adapter for controllable image generation across pixel-level spatial control, subject-driven generation, and style-image-based synthesis (Duan et al., 2024). “UniMIC” addresses universal multi-modality perceptual image compression through a codec repository, multi-grained textual coding, and a Stable-Diffusion-based universal perception compensator (Gao et al., 2024). These systems address different problems, but they provide useful context for the term “unified”: in adjacent work, unification may refer to control modalities or codec families, whereas in TBAC-UniImage it refers specifically to deeper integration of multimodal understanding and generation through layer-wise coupling.

A common misconception would be to interpret TBAC-UniImage as merely an MLLM with a diffusion head. The supplied material argues against that characterization. The model’s defining feature is not the presence of a diffusion component alone, but the multi-layer transfer of intermediate MLLM semantics into successive DiT layers. Another possible misconception would be to treat its benchmark results as evidence that unified models have closed the gap to the best specialized text-to-image systems across the board. The reported DPG-Bench numbers do not support that broader claim; they support competitiveness among unified models and strong compositional and instruction-following behavior, while also showing remaining deficits in some settings.

In that sense, TBAC-UniImage can be understood as a layer-aligned unification scheme: understanding is represented as a hierarchy inside a frozen MLLM, and generation is tuned to consume that hierarchy progressively. This suggests a research direction in which multimodal unification is achieved not by collapsing modalities into one terminal representation, but by preserving intermediate structure across the full generative stack.