Decomposing Subject-Driven Image Generation via Intermediate Structural Prediction

Abstract: Subject-driven text-to-image generation still struggles to preserve high-frequency identity details such as logos, patterns, and text. Existing methods typically operate directly in RGB space, which often leads to detail degradation under substantial edits. We propose a two-stage framework that decouples structure from appearance by first predicting a Canny map and then rendering the final image conditioned on both the source appearance and the predicted structure. To improve text handling, we further introduce a fully automatic pipeline that constructs a 100k-pair text-aware dataset with cross-view textual consistency. Experiments, including GPT-4.1-based evaluation and a knowledge distillation study, show clear gains over selected baselines and suggest that intermediate structural prediction is an effective route for high-fidelity subject-driven generation. Our dataset and code will be made publicly available.

• The paper introduces a novel two-stage framework that decouples structural prediction from appearance rendering to enhance subject identity preservation.
• It leverages Canny edge maps and lightweight LoRA modules to improve text and logo fidelity while reducing training overhead (<0.5%).
• Quantitative results show a 16.5% improvement in subject consistency alongside robust prompt adherence and perceptual quality.

Decomposing Subject-Driven Image Generation via Intermediate Structural Prediction

Introduction

The paper "Decomposing Subject-Driven Image Generation via Intermediate Structural Prediction" (2605.20807) introduces a two-stage framework targeting the persistent challenge of high-frequency identity preservation in subject-driven text-to-image (T2I) generation. Standard approaches, predominantly direct end-to-end mappings in RGB space, regularly degrade critical details such as text, logos, and intricate patterns during substantial edits. The proposed methodology reframes the generation process, decoupling structure and appearance via an intermediate structural representation, specifically leveraging Canny edge maps to retain fine identity fidelity.

Two-Stage Structural Decomposition Framework

The method's key premise is to decompose the ill-posed mapping from subject and text prompt to output image into two sequential sub-tasks: structural prediction and appearance rendering.

• Stage 1 (Structure Prediction): Given a source image and prompt, the model predicts a target Canny map, which encodes object geometry, pose, and textual layout.
• Stage 2 (Appearance Rendering): Conditioned on the original image, prompt, and the generated Canny map, the model synthesizes the final image, ensuring subject identity and prompt adherence.

Both stages utilize the same frozen FLUX.1-dev backbone, adapted via lightweight, task-specific LoRA modules, yielding high parameter efficiency (<0.5% overhead). The Canny map acts as a low-dimensional, information-rich structural blueprint, providing explicit guidance unattainable through holistic feature injections.

Figure 1: Overview of the two-stage pipeline, unified architecture with task-specific LoRA modules, and text-aware data generation leveraging Bagel for novel viewpoints and strict OCR filtering.

Data Pipeline and TextingSubject100k: Specialized Dataset Construction

Preservation of typographical fidelity demands robust cross-view, text-consistent training pairs. The paper introduces a fully automated pipeline to construct the 100k-sample TextingSubject100k dataset:

1. Seedream 3.0 generates text-embedded object images.
2. Bagel synthesizes novel rotated views for each object.
3. OCR filtering enforces textual consistency and legibility across views.

This dataset addresses a major bottleneck in subject-driven T2I: the lack of paired data for text-on-object customization. It is combined with the general-purpose Subject200k dataset in an 80/20 ratio, producing a balanced training set that enhances both generalization and text specialization.

Figure 2: Examples from Subject200k, augmented for robust viewpoint consistency.

Figure 3: Triplets from TextingSubject100k, demonstrating rigorous OCR-enforced textual coherence.

Architectural and Learning Paradigm

The generation stages employ conditional flow matching, minimizing a vector field-driven loss which is stable and well-suited for diffusion transformers. Conditioning encompasses multimodal tokens from image/text/Canny encoders, fused via MM-Attn blocks augmented by LoRA. The architectural duality permits domain specialization without redundant backbone training.

LoRA rank ablation demonstrates optimality at rank 16, balancing capacity and parameter count; lower ranks incur degraded performance in structural and appearance fidelity.

Quantitative and Qualitative Results

GPT-4.1-based automated evaluation on the DreamBooth benchmark produces strong subject consistency (SC), prompt adherence (PA), and perceptual quality (PQ) metrics for the proposed method, outperforming IP-Adapter, SSR-Encoder, and OminiControl baselines. The structure decomposition yields a 16.5% relative SC improvement over OminiControl.

• Main numerical results: SC 7.05 (+16.5%), PA 7.15, PQ 6.93, FID 78.12, CLIP 0.820.

OCR-based evaluation confirms the effectiveness of TextingSubject100k: text rendering accuracy improved from 65.2% (generic dataset) to 85.7% with the specialized dataset.

Qualitative comparisons indicate superior preservation of text, logos, and high-frequency details even under diverse prompts, substantial stylistic edits, and background transformations.

Figure 4: Side-by-side outputs against baselines, showing prompt-consistent subject identity and textual fidelity.

Figure 5: Generated scenes for text-free objects, indicating stable identity retention across environments.

Figure 6: Results for objects with text; legibility and layout preserved under dramatic context shifts.

Ablation and Distillation Studies

Data composition ablation reveals critical trade-offs:

• Pure Subject200k yields genericity; pure TextingSubject100k produces "prompt specialists" with reduced subject generalization.
• 80/20 mix optimizes both generalization and text specificity.

Knowledge distillation from synthetic Seedream4 data shows the intermediate structural prediction (ISP) variant leverages supervision more effectively than vanilla FLUX.Kontext fine-tuning, underscoring the generality of the decomposition paradigm.

Practical and Theoretical Implications

The methodology advances T2I subject-driven generation by formalizing structural prediction as an explicit intermediary, mitigating information entanglement and loss. Practical benefits include lightweight adaptation, editability-preserving fidelity, and robust text rendering. Theoretically, this curriculum introduces a modular learning route, simplifying sub-task optimization and fostering architectural extensibility.

The implications extend to broader domains: in-context editing, subject-driven video synthesis, real-time personalized diffusion, and potential expansion to alternative structural guides (depth, segmentation). The explicit structural decoupling is expected to remain vital for identity-sensitive generative model control.

Future Directions

The framework is currently validated on FLUX.1-dev and FLUX.Kontext; broader backbone generalization, latency reduction, and exploration of structural alternatives (e.g., depth maps, semantic segmentation) are critical next steps. Integrated perceptual metrics and multi-modal similarity evaluation (e.g., DINO, DreamSim) may better capture subjective fidelity constraints.

Conclusion

The paper provides a parameter-efficient, modular solution to high-fidelity subject-driven image generation, leveraging structural decomposition via Canny maps and specialized dataset curation. Strong numerical and qualitative performance substantiates the intermediate prediction paradigm. The approach delineates a clear direction for future generative model development, emphasizing explicit structure and targeted data strategies to achieve consistent identity and text preservation.

Figure 7: Stage 1 Canny predictor output demonstrates accurate preservation of fine textual details using TextingSubject100k.