---
title: Nonunified Score in Multimodal Models
url: https://www.emergentmind.com/topics/nonunified-score
type: topic
---

# Nonunified Score in Multimodal Models

Searching arXiv for the cited work and closely related multimodal unification papers.
The **Nonunified score** is an intrinsic metric for unified multimodal large language models (MLLMs) that quantifies self-contradiction between image generation and multimodal understanding. It measures how often a model’s own understanding branch judges the model’s own generated image as not aligned with the input prompt. Introduced in the study of unified MLLMs such as Janus-Pro, the metric operationalizes the **generation–understanding gap**: a model can generate from text and answer image-text alignment questions, yet still disagree with itself about whether its generated outputs satisfy the prompt. In that setting, a high Nonunified score indicates substantial internal inconsistency, whereas a low score indicates stronger behavioral unification between branches [2507.16663].

## 1. Conceptual basis

Unified MLLMs are designed to perform both text-to-image generation and multimodal understanding within a single model. The Nonunified score was proposed to make internal inconsistency in such systems observable. In a fully unified model, if the generation branch produces an image for a prompt $\mathbf{y}$, the understanding branch should endorse that image as matching $\mathbf{y}$. The metric records violations of that expectation and thereby turns self-contradiction into a measurable property [2507.16663].

The score is explicitly **intrinsic**. Its definition depends only on the model’s own generation branch, its own understanding branch, and a benchmark prompt set such as T2I-CompBench subsets. No external labels or rewards are used in the definition itself. This makes the metric suitable for diagnosing whether the two capabilities behave consistently, but it also means that it evaluates agreement rather than external correctness.

The motivating empirical observation is a capability asymmetry: understanding tends to be stronger than generation. This asymmetry gives the metric a dual role. First, it diagnoses unification failure. Second, it reveals a possible training signal: if the understanding branch reliably rejects prompt-misaligned generations, then self-contradiction can be used as internal supervision to improve the weaker generation branch.

## 2. Formal definition

Let $\pi_\theta$ be a unified MLLM, let $\mathcal{Y}$ denote a prompt set, and let $\mathbf{y} \in \mathcal{Y}$ be a prompt. The generation branch $\pi^{\text{gen}}_\theta$ produces an image
$$
\mathbf{x} = \pi^{\text{gen}}_\theta(\mathbf{y}).
$$

The corresponding understanding query is the yes/no question
$$
q(\mathbf{y}) := \text{``Does this image describe } \mathbf{y}\text{?''}
$$
which is evaluated by the understanding branch $\pi^{\text{und}}_\theta$. The understanding branch outputs a binary decision: $1$ if it judges the image as aligned with the prompt, and $0$ otherwise.

The per-example self-contradiction indicator is
$$
s_\theta(\mathbf{x},\mathbf{y}) := \mathbb{I}\!\left[\pi^{\text{und}}_\theta(\mathbf{x}, q(\mathbf{y})) \neq 1\right].
$$

The **Nonunified score** is the empirical mean of this indicator over the prompt set:
$$
\text{Nonunified score}
:= \frac{1}{|\mathcal{Y}|}\sum_{\mathbf{y}\in\mathcal{Y}}
\mathbb{I}\!\left[\pi^{\text{und}}_\theta(\mathbf{x}, q(\mathbf{y})) \neq 1\right],
\qquad
\mathbf{x} = \pi^{\text{gen}}_\theta(\mathbf{y}).
$$

At the sample level, $s_\theta(\mathbf{x},\mathbf{y}) \in \{0,1\}$ records whether a particular prompt-image pair is nonunified. At the dataset level, the score lies in $[0,1]$ and is often reported as a percentage. There is no threshold beyond the binary decision itself: a sample is unified if the understanding branch outputs $1$, and nonunified otherwise [2507.16663].

This definition isolates a specific notion of contradiction. It does not ask whether the generated image is objectively correct; it asks whether the model later endorses its own output as correct. That distinction is central to both the usefulness and the limitations of the metric.

## 3. Measurement protocol and empirical characterization

Computing the score proceeds by sampling prompts from a benchmark such as T2I-CompBench or T2I-CompBench++, generating images with the model’s generation branch, constructing the yes/no alignment query for each prompt, obtaining the understanding branch’s binary decision, and averaging the resulting indicator over prompts. The paper notes that the exact decoding rule is less important than using the same rule consistently across prompts [2507.16663].

In the main analysis of Janus-Pro-7B, the score is nonzero across all evaluated tasks. Reported values are typically in the **10–40%** range, with the **Texture** subset reaching **> 43%**. This means that, for a substantial fraction of prompts, the model’s own understanding branch rejects the image produced by its own generation branch.

To determine why a sample is nonunified, the study uses Qwen2.5-VL-72B-Instruct as an external judge on the subset where Janus’ understanding outputs $0$. On those nonunified samples, over **85%** are attributed to **weak generation**, meaning that Janus’ understanding is correct and the image is genuinely misaligned. Less than **15%** are attributed to **weak understanding**, meaning that the understanding branch incorrectly rejects a good image. This decomposition supports the interpretation that the Nonunified score primarily reflects generation weakness rather than misunderstanding.

| Setting | 3D Spatial metrics | Brief pattern |
|---|---|---|
| Baseline Janus-Pro-7B | Gen 27.37; Non. 40.33 | substantial gap |
| +DPO | Gen 30.50; Und 52.63; Non. 33.00 | lower contradiction |
| +SFT | Gen 32.50; Und 53.57; Non. 24.33 | largest Non. drop |
| +CLO-DPO | Gen 31.35; Und 52.17; Non. 33.00 | curriculum variant |
| +CLO-SFT | Gen 34.91; Und 65.22; Non. 28.00 | strongest Und. |

These results establish the metric as a diagnostic of unification and as a tracker of post-training progress. On 3D Spatial, direct self-improvement training raises generation quality, measured by UniDet, by about **5%** and reduces the Nonunified score by roughly **10 percentage points**. Similar trends are reported on OOD subsets such as color, shape, texture, and complexity, where post-training lowers the score relative to the baseline [2507.16663].

A further empirical finding in the abstract is that the observed gains stem from better detection of **false positives** that were previously incorrectly identified as prompt-aligned. This links lower Nonunified scores not merely to fewer bad generations, but also to stricter rejection of misaligned outputs by the understanding branch.

## 4. Function within self-improvement

The metric is central to the paper’s self-improvement framework. A high Nonunified score indicates that many prompt-image pairs are being rejected by the model’s own understanding branch, and the external analysis shows that these rejections usually correspond to genuinely weak generations. This suggests using the stronger understanding branch as a **teacher** or **internal reward model** for the weaker generation branch [2507.16663].

The training pipeline generates $N$ candidate images $\{\mathbf{x}_i\}_{i=1}^N$ for each prompt $\mathbf{y}$. The understanding branch scores each candidate using the question “Does this image describe $\mathbf{y}$?” Candidates endorsed by the understanding branch are treated as **chosen** images; candidates rejected by it are treated as **rejected** images. These model-internal labels are then converted into standard post-training data:

- **SFT** uses only chosen images.
- **DPO** uses preference pairs $(\mathbf{y}, \mathbf{x}_{\text{chosen}}, \mathbf{x}_{\text{rejected}})$.

Only the **generation branch parameters** are explicitly fine-tuned. However, because generation and understanding share a backbone, the understanding branch can change implicitly as well. The resulting empirical pattern is a **co-improvement effect**: generation improves, understanding improves, and the Nonunified score decreases, even though only the generation branch is directly optimized.

The Nonunified score is not itself a training objective. The actual optimization targets are SFT losses and DPO objectives on the generation branch using pseudo-labels or preferences provided by the understanding branch. The score functions instead as an intrinsic diagnostic that justifies the self-improvement setup, tracks whether the generation–understanding gap is narrowing, and helps reveal failure modes.

## 5. Theoretical interpretation and the possibility of co-degradation

The theoretical analysis studies the learning dynamics of the generation distribution $\pi_\theta(\mathbf{x}\mid\mathbf{y})$ and the understanding distribution $\pi_\theta(\mathbf{y}\mid\mathbf{x})$. For a fine-tuning pair $(\mathbf{x}_u,\mathbf{y}_u)$ and loss $\mathcal{L}(\theta; \mathbf{x}_u,\mathbf{y}_u)$, the paper defines one-step changes
$$
\Delta G_t(\mathbf{x}\mid\mathbf{y})
:= \log \pi_{\theta_{t+1}}(\mathbf{x}\mid\mathbf{y})
- \log \pi_{\theta_t}(\mathbf{x}\mid\mathbf{y}),
$$
and
$$
\Delta U_t(\mathbf{y}\mid\mathbf{x})
:= \log \pi_{\theta_{t+1}}(\mathbf{y}\mid\mathbf{x})
- \log \pi_{\theta_t}(\mathbf{y}\mid\mathbf{x}).
$$

These updates are related to the empirical NTK and residual terms. The core theoretical claim is that, under self-improvement training, the update directions of generation and understanding tend to be aligned. When $\mathbf{y}$ is similar to $\mathbf{y}_u}$ and the alignment term dominates, the signs of $\Delta G_t(\mathbf{x}\mid\mathbf{y})$ and $\Delta U_t(\mathbf{y}\mid\mathbf{x})$ tend to match. If training increases the probability of the correct image given the prompt, it also tends to increase the probability of the correct prompt given the image; if it decreases the probability of a misaligned pair, both branches tend to become better at rejecting that pair [2507.16663].

This theoretical picture explains the empirical co-improvement effect. Reduced prompt-misaligned generations also improve mismatch detection in the understanding branch, even when only the generation branch is explicitly fine-tuned. It also clarifies why lowering the Nonunified score is usually associated with better generation and better understanding under good supervision.

The same analysis predicts the opposite phenomenon under poor supervision. If fine-tuning pairs are corrupted, the residual alignment term becomes negative for true target pairs, producing **co-degradation**: both generation and understanding move in harmful directions. This possibility is not hypothetical. In experiments with Janus-Pro-1B, the training labels are corrupted by making every training sample target the same fixed image token, **token ID 624**, irrespective of prompt. Under this incorrect objective, generation performance measured by UniDet drops, understanding performance measured by WinRate vs Qwen drops, yet unification improves in the intrinsic sense that $1 - \text{Nonunified score}$ increases. Janus-Pro-7B shows a milder version: corrupted labels mostly harm generation, understanding is more stable, but unification can still improve while actual generation degrades [2507.16663].

## 6. Limits, curriculum learning, and broader implications

A central limitation of the Nonunified score is that it is **blind to whether the model is jointly improving or jointly degrading**. Because it measures agreement between branches rather than correctness against external ground truth, it cannot distinguish between two cases: both branches becoming more correct, and both branches becoming wrong in the same way. A low or decreasing score therefore indicates stronger internal consistency, not necessarily better task performance. This directly motivates the paper’s insistence on **data quality checks** and **external evaluations**, such as WinRate against Qwen-based judgments [2507.16663].

The curriculum-based extension, **CLO (Curriculum Learning-based Online) post-training**, is proposed in response to the observed structure of the training data. On 3D Spatial with pretrained Janus-Pro-7B, only **49.7%** of prompts fall into the **Ideal Set**, meaning that generated candidates include both aligned and misaligned images according to the understanding branch and can therefore form useful DPO pairs. The remaining prompts fall into **FR (Fully Rejected)** or **FC (Fully Chosen)**. After SFT, improved generation and understanding cause **9.7%** of previously FR/FC prompts to become usable. The resulting curriculum begins with the Ideal set, then periodically re-generates and re-scores FR/FC prompts with the improved model so that harder prompts are gradually introduced as the model becomes competent enough to extract signal from them.

CLO does not explicitly target high-Nonunified-score samples. Rather, it stages training by prompt complexity, starting from prompts for which the model can already generate both good and bad candidates and then expanding outward. Empirically, CLO-DPO and CLO-SFT further improve generation and understanding and reduce Nonunified scores across ID and OOD evaluations. On 3D Spatial, CLO-SFT yields **Nonunified 28.00** relative to the baseline **40.33**, while also producing stronger understanding than standard SFT in the reported table.

The broader implication is that the Nonunified score is best understood as a unification diagnostic for models that combine generation and understanding within a shared architecture. The paper states that the definition is **modality-agnostic**: for any unified model with generation and understanding branches, one can analogously define a Nonunified score as the rate at which the understanding module rejects its own generations. This suggests a general template for studying internal consistency in multimodal systems, while also underscoring the metric’s boundary: internal agreement is not equivalent to external correctness [2507.16663].

Source: https://www.emergentmind.com/topics/nonunified-score