DanceOPD: On-Policy Generative Field Distillation

Abstract: Modern image generation demands a single model that unifies diverse capabilities, including text-to-image (T2I), local editing, and global editing. However, these capabilities are rarely naturally aligned and often conflict. For instance, editing tends to degrade T2I performance, while global and local editing interfere with each other. Consequently, effectively composing these capabilities has become a central challenge for image generation model training. To tackle this, we introduce DanceOPD, an on-policy generative field distillation framework for flow-matching models that routes each sample to one capability field, queries one low-noise student-induced state, and trains with a simple velocity MSE objective. With each capability source defined as a velocity field over the shared flow state space, the student learns from fields queried on its own rollout states to compose expert capabilities. This formulation also absorbs operator-defined fields such as classifier-free guidance. Comprehensive experiments on T2I, editing, realism-field absorption, and CFG absorption show that our approach improves multi-capability composition, strengthening target capabilities while preserving anchor generation quality. We believe this work establishes a practical route for generative field distillation in flow-matching models.

• The paper introduces a novel on-policy generative field distillation method that integrates distinct teacher models for T2I and editing tasks.
• It leverages hard routing and on-policy field querying to align supervision signals, addressing gradient conflicts and state-distribution mismatches.
• Experimental results show significant gains in compositional performance and field absorption, outperforming prior baselines on multiple metrics.

DanceOPD: On-Policy Generative Field Distillation for Multi-Capability Image Synthesis

Problem Setting and Motivation

The increasing demand for unified image generation models capable of text-to-image (T2I), local, and global editing has exposed fundamental incompatibilities in compositional learning of diverse generative capabilities. Text-to-image models optimize for open-ended visual diversity and prompt compliance, while editing models, especially local and global variants, encode content preservation or large visual transformations, respectively. Attempts to jointly optimize these capabilities with standard multitask, parameter-merging, or inference-time composition approaches often result in mutual degradation due to gradient conflict, diluted supervision, or lack of semantic coherence.

The core technical challenge is: How can a single student model be trained to effectively compose and internalize the distinct generative capabilities of multiple teacher models, such that it integrates them without compromising individual task fidelity? DanceOPD reframes this as a generative field distillation problem within the flow-matching paradigm, reconciling field-level supervision, sample routing, and on-policy coverage.

Methodology

Field-Based Formulation

Building on flow-matching and diffusion-based generative modeling, DanceOPD formalizes each capability source (e.g., T2I, editing, style transfer) as a frozen velocity field over a shared latent state space. This abstraction enables treating the generative composition problem as a field-query problem—the student must be trained to route queries to appropriate expert fields, at the right states and timesteps, and ingest supervision in a way that preserves the semantic meaning of each capability.

Alignment Challenges and Solutions

1. Target-Field Ambiguity

• Problem: Combining multiple teacher outputs via weighted averaging for a single sample loses semantic clarity, especially when teacher fields encode incompatible generative intents.
• Solution: Hard-routed sample-wise field matching. Each sample is explicitly routed to one capability source (T2I, Edit, etc.), ensuring each supervision signal is semantically aligned to a meaningful generative operation.

2. State-Distribution Mismatch

• Problem: Teacher signals collected on fixed data states or teacher-rolled trajectories may not match the distribution of states visited by the evolving student, leading to covariate shift at inference.
• Solution: On-policy field querying. Teacher fields are queried on the student's own rollout states, with stop-gradient detachment to ensure gradients are propagated only to the local velocity predictor.

3. Trajectory-Query Correlation

• Problem: Supervising on multiple correlated states from the same rollout (dense querying) can amplify redundant gradient directions and obscure capability-specific learning.
• Solution: Semantic-side single query. A single, high-information state is sampled from the low-noise ("semantic") side of the student trajectory—where semantic content is densest—reducing redundancy and focusing learning on the most informative points.

Training Objective

The default loss is a mean squared error (MSE) between the student and routed teacher velocity predictions at the queried state:

$$L_\text{DanceOPD} = \mathbb{E}_{m, (x,c), Z_T, s} \left[ \left\| v_\theta(z^\prime, t, c) - v_m(z^\prime, t, c) \right\|^2 \right]$$

where $v_\theta$ is the student, $v_m$ is the routed teacher field, and $z'$ is the student-rolled stop-gradient state.

Theoretical justification is provided for this choice: under local Gaussian transition kernels, velocity MSE and KL divergence become equivalent up to weighting. The framework naturally subsumes operator-defined fields, such as classifier-free guidance (CFG), by treating them as additional velocity fields.

Experimental Results

Multi-Capability Composition

Experiments are performed on Z-Image and SD3.5-M backbones, evaluating both:

• Capability composition: (T2I + Editing, Local + Global Editing) — where the student must execute both prompt following and edits, or reconcile local preservation with global transformation.
• Field absorption: (Realism, CFG) — where the student absorbs a specific field (e.g., photorealism reward, inference-time guidance) while preserving base generation.

Quantitative Highlights

• T2I + Editing: DanceOPD surpasses the best on-policy distillation (OPD) baseline on GEditBench by 8.1% and the edit teacher by 8.5%, while exceeding the T2I teacher on GenEval by 2.0%.
• Local + Global Editing: DanceOPD improves over the best prior baseline by 16.1% on GEditBench, and over the local edit teacher by 7.9%, with superior overall T2I generalization.
• Realism Absorption: Achieves a 9.9% realism reward increase versus off-policy distillation, closing 85.3% of the student-teacher gap, simultaneously maintaining T2I metrics.
• CFG Absorption: Integrated guidance outperforms train-only and eval-only baselines, with improvements of 7.6% and 1.4%, respectively, but demonstrates explicit over-guidance failure when training and inference scales are multiplied.

Ablations and Diagnostics

• Hard Routing vs. Soft Mixing: Hard routing is essential, yielding 15.2% and 10.6% gains over soft (mixed) teacher targets under MSE and KL losses, respectively.
• Timestep Selection: Querying on low-noise (semantic) timesteps yields 23.7% and 19.5% higher GEditBench scores over median- and high-noise queries.
• Single vs. Dense Querying: A single semantic-side query outperforms multi-query strategies by up to 16.6%, due to avoidance of trajectory correlation.
• Objective Choice: Plain velocity MSE is more stable and effective (up to 4.5% better) than timestep-weighted or KL-weighted alternatives.
• Initialization: Performance is maximized when the student is initialized from the closest relevant capability checkpoint (e.g., local edit), instead of merged or generic initialization.

Theoretical Analysis

DanceOPD is justified via probabilistic and field-theoretic analysis:

• KL-MSE Equivalence: Shows why velocity MSE is a theoretically sound objective when student and teacher induce local Gaussians with shared covariance.
• Smoothness and Local Validity: Demonstrates that querying off-policy states can bias the target if the teacher is not proximate in state space.
• Field-Conflict Bias and Correlation: Analysis of sample routing and dense queries reveals algebraic and statistical sources of interference and justifies core design choices.

Advantages over Prior Art

DanceOPD directly addresses limitations in classical and OPD-based distillation for generative flows:

• Avoids capability dilution inherent to joint training or soft-task aggregation.
• Internalizes inference-time operators like CFG, enabling efficient single-pass sampling for enhanced generations.
• General framework: Extensive ablations show robust applicability across editing, stylization, and reward-field absorption.
• Computational efficiency: Training cost is moderate—a single rollout per step, with a single query per sample—outperforming dense-query approaches in both sample efficiency and wall-clock time.

Practical and Theoretical Implications

DanceOPD establishes a scalable, modular paradigm for multi-capability image generation models by explicitly decoupling capability learning through routing and on-policy supervision. This method unlocks new strategies for:

• Unified, composable image generators that do not trade off anchor task quality for additional abilities.
• Efficient model post-training, enabling practical field absorption (realism, personalized guidance) and adaptation without retraining from scratch.
• Structured field-based distillation, paving the way for research on dynamically routed or behaviorally inferred sample-to-field assignments, reward-driven or verifier-based supervision, and further analysis of field compatibility and task disentanglement in generative backbones.

Conclusion

DanceOPD introduces a principled on-policy generative field distillation approach for flow-matching image generation models, reconciling capability-specific field routing, on-policy distribution matching, and anti-correlation in rollout querying. The resulting methodology achieves superior compositional performance on multi-task image generation, preserving anchor fidelity while robustly integrating new editing and realism capabilities. This framework establishes a foundation for continued advances in compositional generative learning, field absorption, and post-hoc capability integration into unified generative models.

---

Citation:

Wei Zhou et al., "DanceOPD: On-Policy Generative Field Distillation" (2606.27377)