Nonunified Score in Multimodal Models

• Nonunified score is an intrinsic metric that quantifies the self-contradiction between a model’s generated image and its own understanding of the prompt.
• It operationalizes the generation–understanding gap by evaluating binary decisions without reliance on external labels.
• Empirical results show scores typically in the 10–40% range, highlighting its use in diagnosing generation weaknesses and guiding self-improvement.

Searching arXiv for the cited work and closely related multimodal unification papers. The Nonunified score is an intrinsic metric for unified multimodal LLMs (MLLMs) that quantifies self-contradiction between image generation and multimodal understanding. It measures how often a model’s own understanding branch judges the model’s own generated image as not aligned with the input prompt. Introduced in the study of unified MLLMs such as Janus-Pro, the metric operationalizes the generation–understanding gap: a model can generate from text and answer image-text alignment questions, yet still disagree with itself about whether its generated outputs satisfy the prompt. In that setting, a high Nonunified score indicates substantial internal inconsistency, whereas a low score indicates stronger behavioral unification between branches (Han et al., 22 Jul 2025).

1. Conceptual basis

Unified MLLMs are designed to perform both text-to-image generation and multimodal understanding within a single model. The Nonunified score was proposed to make internal inconsistency in such systems observable. In a fully unified model, if the generation branch produces an image for a prompt $\mathbf{y}$, the understanding branch should endorse that image as matching $\mathbf{y}$. The metric records violations of that expectation and thereby turns self-contradiction into a measurable property (Han et al., 22 Jul 2025).

The score is explicitly intrinsic. Its definition depends only on the model’s own generation branch, its own understanding branch, and a benchmark prompt set such as T2I-CompBench subsets. No external labels or rewards are used in the definition itself. This makes the metric suitable for diagnosing whether the two capabilities behave consistently, but it also means that it evaluates agreement rather than external correctness.

The motivating empirical observation is a capability asymmetry: understanding tends to be stronger than generation. This asymmetry gives the metric a dual role. First, it diagnoses unification failure. Second, it reveals a possible training signal: if the understanding branch reliably rejects prompt-misaligned generations, then self-contradiction can be used as internal supervision to improve the weaker generation branch.

2. Formal definition

Let $\pi_\theta$ be a unified MLLM, let $\mathcal{Y}$ denote a prompt set, and let $\mathbf{y} \in \mathcal{Y}$ be a prompt. The generation branch $\pi^{\text{gen}}_\theta$ produces an image

$\mathbf{x} = \pi^{\text{gen}}_\theta(\mathbf{y}).$

The corresponding understanding query is the yes/no question

$$q(\mathbf{y}) := \text{``Does this image describe } \mathbf{y}\text{?''}$$

which is evaluated by the understanding branch $\pi^{\text{und}}_\theta$. The understanding branch outputs a binary decision: $1$ if it judges the image as aligned with the prompt, and $\mathbf{y}$0 otherwise.

The per-example self-contradiction indicator is

$\mathbf{y}$1

The Nonunified score is the empirical mean of this indicator over the prompt set:

$\mathbf{y}$2

At the sample level, $\mathbf{y}$3 records whether a particular prompt-image pair is nonunified. At the dataset level, the score lies in $\mathbf{y}$4 and is often reported as a percentage. There is no threshold beyond the binary decision itself: a sample is unified if the understanding branch outputs $\mathbf{y}$5, and nonunified otherwise (Han et al., 22 Jul 2025).

This definition isolates a specific notion of contradiction. It does not ask whether the generated image is objectively correct; it asks whether the model later endorses its own output as correct. That distinction is central to both the usefulness and the limitations of the metric.

3. Measurement protocol and empirical characterization

Computing the score proceeds by sampling prompts from a benchmark such as T2I-CompBench or T2I-CompBench++, generating images with the model’s generation branch, constructing the yes/no alignment query for each prompt, obtaining the understanding branch’s binary decision, and averaging the resulting indicator over prompts. The paper notes that the exact decoding rule is less important than using the same rule consistently across prompts (Han et al., 22 Jul 2025).

In the main analysis of Janus-Pro-7B, the score is nonzero across all evaluated tasks. Reported values are typically in the 10–40% range, with the Texture subset reaching > 43%. This means that, for a substantial fraction of prompts, the model’s own understanding branch rejects the image produced by its own generation branch.

To determine why a sample is nonunified, the study uses Qwen2.5-VL-72B-Instruct as an external judge on the subset where Janus’ understanding outputs $\mathbf{y}$6. On those nonunified samples, over 85% are attributed to weak generation, meaning that Janus’ understanding is correct and the image is genuinely misaligned. Less than 15% are attributed to weak understanding, meaning that the understanding branch incorrectly rejects a good image. This decomposition supports the interpretation that the Nonunified score primarily reflects generation weakness rather than misunderstanding.

Setting	3D Spatial metrics	Brief pattern
Baseline Janus-Pro-7B	Gen 27.37; Non. 40.33	substantial gap
+DPO	Gen 30.50; Und 52.63; Non. 33.00	lower contradiction
+SFT	Gen 32.50; Und 53.57; Non. 24.33	largest Non. drop
+CLO-DPO	Gen 31.35; Und 52.17; Non. 33.00	curriculum variant
+CLO-SFT	Gen 34.91; Und 65.22; Non. 28.00	strongest Und.

These results establish the metric as a diagnostic of unification and as a tracker of post-training progress. On 3D Spatial, direct self-improvement training raises generation quality, measured by UniDet, by about 5% and reduces the Nonunified score by roughly 10 percentage points. Similar trends are reported on OOD subsets such as color, shape, texture, and complexity, where post-training lowers the score relative to the baseline (Han et al., 22 Jul 2025).

A further empirical finding in the abstract is that the observed gains stem from better detection of false positives that were previously incorrectly identified as prompt-aligned. This links lower Nonunified scores not merely to fewer bad generations, but also to stricter rejection of misaligned outputs by the understanding branch.

4. Function within self-improvement

The metric is central to the paper’s self-improvement framework. A high Nonunified score indicates that many prompt-image pairs are being rejected by the model’s own understanding branch, and the external analysis shows that these rejections usually correspond to genuinely weak generations. This suggests using the stronger understanding branch as a teacher or internal reward model for the weaker generation branch (Han et al., 22 Jul 2025).

The training pipeline generates $\mathbf{y}$7 candidate images $\mathbf{y}$8 for each prompt $\mathbf{y}$9. The understanding branch scores each candidate using the question “Does this image describe $\pi_\theta$0?” Candidates endorsed by the understanding branch are treated as chosen images; candidates rejected by it are treated as rejected images. These model-internal labels are then converted into standard post-training data:

• SFT uses only chosen images.
• DPO uses preference pairs $\pi_\theta$1.

Only the generation branch parameters are explicitly fine-tuned. However, because generation and understanding share a backbone, the understanding branch can change implicitly as well. The resulting empirical pattern is a co-improvement effect: generation improves, understanding improves, and the Nonunified score decreases, even though only the generation branch is directly optimized.

The Nonunified score is not itself a training objective. The actual optimization targets are SFT losses and DPO objectives on the generation branch using pseudo-labels or preferences provided by the understanding branch. The score functions instead as an intrinsic diagnostic that justifies the self-improvement setup, tracks whether the generation–understanding gap is narrowing, and helps reveal failure modes.

5. Theoretical interpretation and the possibility of co-degradation

The theoretical analysis studies the learning dynamics of the generation distribution $\pi_\theta$2 and the understanding distribution $\pi_\theta$3. For a fine-tuning pair $\pi_\theta$4 and loss $\pi_\theta$5, the paper defines one-step changes

$\pi_\theta$6

and

$\pi_\theta$7

These updates are related to the empirical NTK and residual terms. The core theoretical claim is that, under self-improvement training, the update directions of generation and understanding tend to be aligned. When $\pi_\theta$8 is similar to $\pi_\theta$9 and the alignment term dominates, the signs of $\mathcal{Y}$0 and $\mathcal{Y}$1 tend to match. If training increases the probability of the correct image given the prompt, it also tends to increase the probability of the correct prompt given the image; if it decreases the probability of a misaligned pair, both branches tend to become better at rejecting that pair (Han et al., 22 Jul 2025).

This theoretical picture explains the empirical co-improvement effect. Reduced prompt-misaligned generations also improve mismatch detection in the understanding branch, even when only the generation branch is explicitly fine-tuned. It also clarifies why lowering the Nonunified score is usually associated with better generation and better understanding under good supervision.

The same analysis predicts the opposite phenomenon under poor supervision. If fine-tuning pairs are corrupted, the residual alignment term becomes negative for true target pairs, producing co-degradation: both generation and understanding move in harmful directions. This possibility is not hypothetical. In experiments with Janus-Pro-1B, the training labels are corrupted by making every training sample target the same fixed image token, token ID 624, irrespective of prompt. Under this incorrect objective, generation performance measured by UniDet drops, understanding performance measured by WinRate vs Qwen drops, yet unification improves in the intrinsic sense that $\mathcal{Y}$2 increases. Janus-Pro-7B shows a milder version: corrupted labels mostly harm generation, understanding is more stable, but unification can still improve while actual generation degrades (Han et al., 22 Jul 2025).

6. Limits, curriculum learning, and broader implications

A central limitation of the Nonunified score is that it is blind to whether the model is jointly improving or jointly degrading. Because it measures agreement between branches rather than correctness against external ground truth, it cannot distinguish between two cases: both branches becoming more correct, and both branches becoming wrong in the same way. A low or decreasing score therefore indicates stronger internal consistency, not necessarily better task performance. This directly motivates the paper’s insistence on data quality checks and external evaluations, such as WinRate against Qwen-based judgments (Han et al., 22 Jul 2025).

The curriculum-based extension, CLO (Curriculum Learning-based Online) post-training, is proposed in response to the observed structure of the training data. On 3D Spatial with pretrained Janus-Pro-7B, only 49.7% of prompts fall into the Ideal Set, meaning that generated candidates include both aligned and misaligned images according to the understanding branch and can therefore form useful DPO pairs. The remaining prompts fall into FR (Fully Rejected) or FC (Fully Chosen). After SFT, improved generation and understanding cause 9.7% of previously FR/FC prompts to become usable. The resulting curriculum begins with the Ideal set, then periodically re-generates and re-scores FR/FC prompts with the improved model so that harder prompts are gradually introduced as the model becomes competent enough to extract signal from them.

CLO does not explicitly target high-Nonunified-score samples. Rather, it stages training by prompt complexity, starting from prompts for which the model can already generate both good and bad candidates and then expanding outward. Empirically, CLO-DPO and CLO-SFT further improve generation and understanding and reduce Nonunified scores across ID and OOD evaluations. On 3D Spatial, CLO-SFT yields Nonunified 28.00 relative to the baseline 40.33, while also producing stronger understanding than standard SFT in the reported table.

The broader implication is that the Nonunified score is best understood as a unification diagnostic for models that combine generation and understanding within a shared architecture. The paper states that the definition is modality-agnostic: for any unified model with generation and understanding branches, one can analogously define a Nonunified score as the rate at which the understanding module rejects its own generations. This suggests a general template for studying internal consistency in multimodal systems, while also underscoring the metric’s boundary: internal agreement is not equivalent to external correctness (Han et al., 22 Jul 2025).