Disentangling to Re-couple: Resolving the Similarity-Controllability Paradox in Subject-Driven Text-to-Image Generation

Abstract: Subject-Driven Text-to-Image (T2I) Generation aims to preserve a subject's identity while editing its context based on a text prompt. A core challenge in this task is the "similarity-controllability paradox", where enhancing textual control often degrades the subject's fidelity, and vice-versa. We argue this paradox stems from the ambiguous role of text prompts, which are often tasked with describing both the subject and the desired modifications, leading to conflicting signals for the model. To resolve this, we propose DisCo, a novel framework that first Disntangles and then re-Couples visual and textual information. First, our textual-visual decoupling module isolates the sources of information: subject identity is extracted exclusively from the reference image with the entity word of the subject, while the text prompt is simplified to contain only the modification command, where the subject refers to general pronouns, eliminating descriptive ambiguity. However, this strict separation can lead to unnatural compositions between the subject and its contexts. We address this by designing a dedicated reward signal and using reinforcement learning to seamlessly recouple the visually-defined subject and the textually-generated context. Our approach effectively resolves the paradox, enabling simultaneous high-fidelity subject preservation and precise textual control. Extensive experiments demonstrate that our method achieves state-of-the-art performance, producing highly realistic and coherent images.

• The paper introduces DisCo, a new method that resolves the similarity-controllability paradox in text-to-image generation, improving performance by 3% across multiple metrics.
• this methodology involves a two-stage process to process the text and visual input to maintain subjects' identity while adhering to prompt-specific details.
• Experiments show that DisCo outperform existing methods, achieving an 80% win rate against state-of-the-art models.

Motivation and problem statement

Subject-driven text-to-image (T2I) generation requires preserving a reference subject's identity while re-rendering it in a context specified by a text prompt. The field has long been constrained by a "similarity-controllability paradox": methods that improve subject fidelity tend to degrade prompt adherence, and vice versa. Prior approaches—encoder-based injection (IP-Adapter, RealCustom++, SSR-Encoder) and unified-sequence DiT conditioning (OminiControl, OmniGen)—mitigate but do not eliminate this dual-optimum trade-off.

The paper's central diagnosis is that the paradox originates from role entanglement in the text prompt. Standard prompts such as "a duck toy in the jungle" describe both the subject and the modification. Because textual descriptions are discrete and abstract, they cannot capture continuous attributes like color, texture, and style; instead, they activate the model's pretrained priors, which override the visual reference. The authors demonstrate this with FLUX Kontext [dev]: prompting with "a duck toy" produces a generic duck whose beak and wing details deviate from the reference, whereas replacing the entity word with a generic pronoun ("this item", "it") preserves fidelity. A candle example shows the same effect—mentioning "candle" causes the model to default to generating a lit candle regardless of the reference.

Method

DisCo operates in two stages: decoupling, then re-coupling, built on FLUX with 3D RoPE and multi-modal attention over concatenated image, text, and reference-image tokens.

Textual-visual decoupling module (TVD)

TVD enforces the principle that identity comes exclusively from the reference image while the prompt dictates only modifications:

• Prompt simplification: Qwen2.5-VL 72B identifies the subject's entity word in the prompt given the reference image, then rewrites the prompt replacing the entity word and its descriptions with a generic placeholder ("this item" / "it").
• Visual grounding: GroundingDINO localizes the subject in the reference image using the original entity word, bridging the generic pronoun to specific visual features without altering the reference.
• Attention-map analysis supports the mechanism: with entangled prompts, the entity word ("a colorful sneaker") exerts strong attention on the generated subject region; after decoupling, attention shifts to the grounded reference subject, confirming that control of identity migrates from text to the visual modality.

Re-coupling via GRPO with a task-specific reward model

Strict separation creates a compositional gap: the model may place the visually-defined subject implausibly (e.g., a candle floating mid-air), since the prompt no longer conveys contextual information about the subject. DisCo addresses this with Group Relative Policy Optimization (GRPO), following Flow-GRPO and DanceGRPO adaptations of GRPO to flow-based models.

A key contribution is a dedicated reward model based on Qwen3-VL-30B, trained on 25k synthetic preference pairs. A VLM generates editing instructions that deliberately corrupt either subject identity or subject-context interaction; applying these instructions yields negative images, forming preference pairs against originals. The reward model is trained by negative log-likelihood on pairwise preferences. During GRPO, each sample's reward is the sum of its win probabilities against all other samples in its group, normalized within the group to form advantages, optimized with a clipped PPO-style objective plus KL regularization to a reference policy.

Experimental results

On DreamBench (30 subjects × 25 prompts = 750 cases), with SAM-based segmentation before computing similarity metrics, DisCo achieves state-of-the-art results across all metrics:

Method	CLIP-B-I	CLIP-L-I	DINO-I	CLIP-B-T	CLIP-L-T	ImageReward
RealCustom++	0.801	0.772	0.749	0.315	0.258	1.251
OminiControl	0.857	0.867	0.713	0.320	0.263	1.254
DreamO	0.899	0.901	0.813	0.322	0.267	1.186
UNO	0.899	0.907	0.827	0.311	0.255	0.854
FLUX.1 Kontext [dev]	0.910	0.911	0.839	0.321	0.268	1.276
DisCo	0.928	0.937	0.903	0.329	0.273	1.339

Notably, DisCo attains the highest scores on both similarity and controllability simultaneously—directly contradicting the trade-off pattern visible in prior baselines (e.g., SSR-Encoder vs. MS-Diffusion, DreamO vs. UNO). Supplementary results extend this to DreamBench++ (CLIP-B-I 0.801, DINO-I 0.610, IR 1.291, all best) and additional baselines including RPO and Diptych Prompting. A pairwise human study on 100 samples shows win rates of 80% vs. UNO, 82% vs. DreamO, 71% vs. OminiControl, 66% vs. ACE++, and 51% (with 24% losses) vs. FLUX Kontext. Qualitative comparisons against general editing systems (Nano Banana, Qwen-Image-Edit, Seedream 4.0) indicate competitive or superior balance of fidelity, integration, and quality.

Ablations

Removing TVD drops both CLIP-I (0.915) and CLIP-T (0.319). Removing GRPO preserves high similarity (CLIP-I 0.922) but yields the lowest prompt adherence (CLIP-T 0.319) and sharply degraded ImageReward (1.189), confirming that GRPO is essential for compositional re-coupling rather than merely cosmetic improvement. Reward-model substitution is informative: CLIP as reward degrades overall performance due to its inability to assess compositional harmony; ImageReward improves IR (1.404) but significantly reduces subject similarity (0.914); a pretrained VLM without task-specific training underperforms across metrics. These results substantiate the claim that subject-driven generation requires rewards conditioned on the reference image—a point the ablation table makes plainly, though the IR-as-reward variant achieving higher IR than the full model suggests the aggregate metric does not fully capture the fidelity-control balance.

Limitations and open questions

Several constraints are acknowledged or implicit. The pipeline depends on external components—Qwen2.5-VL for entity identification and rewriting, GroundingDINO for localization—so errors in subject identification propagate through both stages; robustness to ambiguous or multi-subject prompts is not quantified. The reward model is trained on synthetically corrupted negatives produced by the generator itself, which risks distributional bias toward the model's own failure modes. Evaluation relies primarily on DreamBench-style benchmarks with CLIP/DINO similarity proxies, which may not fully capture perceptual fidelity. The supplementary tables show that applying decoupling naively to other backbones (DreamO-decouple, UNO-decouple) degrades performance substantially, indicating the framework's gains depend on the joint training recipe rather than prompt rewriting alone—an assumption the paper does not fully dissect. IC-LoRA comparisons were deferred due to workflow incompatibilities. Open questions include whether TVD generalizes to multi-subject customization and how sensitive the GRPO stage is to reward-model scale and preference-pair construction.

Conclusion

DisCo reframes the similarity-controllability paradox as a symptom of overloaded prompts rather than an inherent capacity limitation of T2I models. By sourcing identity exclusively from the grounded reference image, reducing the prompt to a modification command, and restoring subject-context coherence through GRPO with a purpose-built VLM reward model, it achieves simultaneous state-of-the-art subject similarity and text controllability on DreamBench and DreamBench++. The evidence indicates that careful allocation of information across modalities, combined with targeted reinforcement learning, can resolve a trade-off previously treated as fundamental.