ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control
Abstract: Reinforcement Learning (RL) post-training has become the standard for aligning generative models with human preferences, yet most methods rely on a single scalar reward. When multiple criteria matter, the prevailing practice of ``early scalarization'' collapses rewards into a fixed weighted sum. This commits the model to a single trade-off point at training time, providing no inference-time control over inherently conflicting goals -- such as prompt adherence versus source fidelity in image editing. We introduce ParetoSlider, a multi-objective RL (MORL) framework that trains a single diffusion model to approximate the entire Pareto front. By training the model with continuously varying preference weights as a conditioning signal, we enable users to navigate optimal trade-offs at inference time without retraining or maintaining multiple checkpoints. We evaluate ParetoSlider across three state-of-the-art flow-matching backbones: SD3.5, FluxKontext, and LTX-2. Our single preference-conditioned model matches or exceeds the performance of baselines trained separately for fixed reward trade-offs, while uniquely providing fine-grained control over competing generative goals.
Summary
- The paper introduces a novel framework that enables continuous, user-controlled trade-offs in diffusion models through a preference-conditioned policy.
- It employs a late scalarization mechanism to aggregate per-reward advantages, ensuring faithful multi-objective optimization with improved computational efficiency.
- Experiments across text-to-image, image editing, and text-to-video tasks demonstrate smoother Pareto frontier transitions and superior performance over fixed-weight baselines.
ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control
Motivation and Problem Formulation
Existing reinforcement learning (RL)-based fine-tuning approaches for generative diffusion models typically reduce multi-objective reward supervision to scalarization, optimizing a fixed linear combination of user-aligned reward models. This leads to inherent inflexibility: once trained for a given trade-off between competing objectives (e.g., image realism versus stylistic faithfulness, or edit adherence versus input preservation in image manipulation), the inference-time model is rigidly anchored to its scalarization setting, precluding post-hoc adjustment by the user. Alternative approaches—such as model interpolation or per-sample guidance scaling—increase computational and storage costs, and still fail to create a unified policy that can smoothly and optimally traverse the Pareto frontier of trade-offs for multiple objectives.
"ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control" (2604.20816) introduces a scalable online multi-objective RL framework for diffusion alignment, enabling a single fine-tuned model to expose inference-time controllability over reward trade-offs by conditioning on a continuous user-specified preference vector . The method achieves Pareto-optimal alignment within the same parameter set, eliminating the need for training multiple models or resorting to costly per-step optimization.
Methodology
ParetoSlider augments diffusion model RL post-training via three main advances:
- Preference-Conditioned Policy: The model is fine-tuned with additional input describing the user's desired trade-off between competing reward models as a simplex-constrained preference vector . The diffusion transformation backbone is explicitly conditioned on this vector by careful integration pathways (e.g., AdaLN modulation, residual MLP injection), ensuring parameter efficiency and stable learning. See (Figure 1).
Figure 1: The ParetoSlider training pipeline integrates preference conditioning, per-objective reward evaluation, and late scalarization using DiffusionNFT loss aggregation.
- Late Scalarization of Per-Reward Advantages: To mitigate reward imbalance and ensure faithful -dependent behavior, the method eschews early reward scalarization. Instead, each reward function computes normalized, group-relative advantages independently; these per-reward advantages are only aggregated downstream via , prior to applying the DiffusionNFT loss for policy gradient updates. This late scalarization prevents "reward hijacking" and enables nuanced, preference-consistent learning.
- Efficient Online RL with DiffusionNFT: Building on the DiffusionNFT policy optimization framework, training proceeds with group-based advantage normalization and EMA-based target velocity steering, without explicit value network estimation. Combined with preference-conditioning, this mechanism yields rapid, stable convergence (3–25× efficiency improvements over FlowGRPO), and supports continuous exploration and coverage of the reward Pareto surface.
The inference protocol becomes trivial: at runtime, the user specifies their trade-off , and the single trained model deterministically traverses the resultant analytical operating point on the Pareto front—offering smooth transitions and intermediate interpolations not possible with scalarized or model-interpolated baselines.
Empirical Evaluation
The proposed approach is exhaustively validated across three domains: text-to-image synthesis (Stable Diffusion 3.5), instruction-based image editing (FluxKontext), and text-to-video generation (LTX-2). For all tasks, ParetoSlider conditions the backbone on a simplex-valued preference vector and is compared against the following:
- Fixed-Weights Baselines: Standard single-scalarization RL, requiring separate model for each trade-off
- FlowMulti: GRPO-based static Pareto-based policy selection
- Model Interpolation: Post-training checkpoint blending
- Prompt Rewriting: LLM-based task reformulation
- Classifier-Free Guidance (CFG): Inference-time guidance scale sweeps
Text-to-Image: Style and Realism Control
ParetoSlider enables seamless sliding between realism, sketch, vector art, and other stylistic axes for a fixed prompt. (Figure 2) demonstrates coverage of the Pareto surface across photorealism/sketch, anime, and multiple styles, with smooth, high-fidelity transitions between user-desired configurations.
Figure 2: Multi-objective style interpolation; each triplet demonstrates model outputs for distinct traversals, showcasing trajectory continuity between reward optima.
Comparisons demonstrate that ParetoSlider matches or exceeds the best performance of all baselines at their static operating points, while enabling continuous preference-based control not possible with any alternative method. The Pareto front traced by ParetoSlider consistently dominates all baselines on relevant evaluation metrics (so-called hypervolume, non-dominated point count). See (Figure 3).
Figure 3: Quantitative Pareto front traces for SD3.5, comparing ParetoSlider with baselines on the photorealism-sketch axis. ParetoSlider yields a smoother, broader Pareto surface.
Instruction-Based Editing: Preservation vs. Adherence
For edit instructions (e.g., "make this portrait into a pixel-art style"), the method exposes precise tuning between exact input image preservation and full instruction compliance—again exceeding the operating point quality of FixedWeights and surpassing Text/Image classifier-free guidance-based control (which results in suboptimal or artifact-prone edits at either extreme). The continuous slider provides stable and interpretable interpolation. (Figure 4), (Figure 5).
Figure 4: The method enables precise control over the trade-off between instruction adherence and source preservation in editing tasks.
Text-to-Video: Animation vs. Photorealism
The single ParetoSlider-trained LTX-2 model produces video outputs spanning animation and photorealistic renderings, mediating style at inference-time per the user-set , as evidenced by frame samples (Figure 6).
Figure 6: For each prompt, a single model produces stylized or photorealistic videos—highlighting continuous user steerability at inference.
Ablation Studies
Extensive ablations dissect the contributions of different preference-conditioning modules (shared residual, per-block, token injection, hybrid), and various scalarization strategies (late, early, Smooth Tchebycheff). Both qualitative and quantitative analysis underline that shared or per-block modulation architectures, combined with late scalarization, are necessary and sufficient for faithful, robust, and monotonic control over reward trade-offs—while variants show reduced controllability, Pareto surface collapse, or non-uniform spacing across preference settings. See (Figure 7) and (Figure 8).
Figure 7: Conditioning ablation reveals that shared and per-block modulation yield monotonic, well-spread interpolation from photorealism to sketch, while others do not.
Figure 8: Scalarization ablation: late scalarization achieves more uniform, preference-faithful control. Early and STCH alternative scalarizations are prone to premature collapse.
Implications and Future Directions
ParetoSlider realizes a scalable, extensible paradigm for amortized Pareto-optimal alignment in high-dimensional generative models. By decomposing preference incorporation into explicit conditioning and judicious scalarization, it removes the need for expensive checkpoint multiplexing or inference-time optimization, and uniquely amortizes the reward surface across the model parameters.
This approach has several key practical implications:
- Unified Deployment: Model maintainers need only a single checkpoint to provide any mix of reward-aligned outputs, greatly simplifying workflows.
- User-Facing Control: Supports the construction of inference-time UI sliders for arbitrary, interpretable reward axes (e.g., fidelity vs. creativity; realism vs. stylization).
- Efficient Exploration: Online RL training guarantees exploration beyond the convex hull of pre-collected datasets, improving generalization in under-constrained objectives.
Theoretically, the demonstrated superiority of late scalarization and conditioning for continuous MORL settings motivates deeper investigation into optimality criteria and non-linear preference maps for high-dimensional reward spaces, as well as more expressive conditioning in very high-capacity backbones.
Conclusion
ParetoSlider systematically addresses continuous preference control in diffusion model post-training, introducing both the methodological innovations and empirical rigor required for robust, scalable, and user-controllable multi-objective generative modeling. Its design generalizes across visual generation domains, realizing Pareto frontier coverage within a single model that is both parameter and inference efficient. The framework opens new research directions for conditional amortized optimization and scalable MORL in complex, user-facing AI systems.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What this paper is about
This paper introduces ParetoSlider, a way to train image and video generators so you can smoothly move a “slider” between competing goals—like making a picture look more realistic versus more like a sketch—without retraining the model each time. Instead of locking the model into one fixed balance during training, ParetoSlider teaches a single model to cover the whole range of good trade‑offs (called the Pareto front) and to follow whatever balance you ask for at generation time.
The main questions the authors ask
The authors set out to find out:
- Can one model be trained to handle many different, often conflicting goals at once and let users smoothly pick their favorite balance later?
- How should we train such a model so it really listens to the user’s preferences and doesn’t get “hijacked” by one goal dominating the others?
- Does this work across different tasks—making images from text, editing existing images with instructions, and making videos from text?
How the method works (in everyday language)
Think of making hot chocolate with two preferences: “more chocolatey” vs “more milky.” If you pick one recipe during training (say, 70% chocolate, 30% milk), you’re stuck with that taste forever. ParetoSlider does something different:
- It gives the model a “preference vector” (like a recipe card) that says how much to care about each goal, for example 60% realism, 40% sketchiness. This vector is shown to the model while it learns and also when it generates images or videos.
- During training, the model tries lots of preference settings and learns how to respond to each one. This lets it learn the whole curve of best possible trade‑offs, known as the Pareto front. On that front, you can’t improve one goal without hurting another.
To make this efficient and stable, they build on a fast reinforcement learning (RL) post‑training method for diffusion/flow models called DiffusionNFT. Here’s the training recipe in simple steps:
- Make small batches (“groups”) of samples for the same prompt and the same preference vector. For example, generate K different images for “a cat on a couch” with “70% realism, 30% sketch.”
- Score each sample with several reward models—one per goal (e.g., realism score, sketch score, prompt matching score, etc.).
- Normalize each reward separately within the group. This is like grading each subject (math, English, science) on its own curve so one subject doesn’t drown out the others. The paper calls this “late scalarization”: you keep rewards separate and only mix them using the user’s preference weights at the very end. This avoids “reward hijacking,” where a goal with naturally bigger numbers overwhelms the rest.
- Update the model using a normal diffusion training loss that is steered by these per‑goal, per‑group signals, and then weighted by the chosen preference vector.
How does the model “see” your preferences? The authors add tiny “conditioning” modules—small neural layers that take the preference vector and nudge the model’s internal activations. Different backbones get slightly different nudges:
- Text‑to‑image (SD3.5): inject the preference both into a global timing/conditioning vector and as a small shared residual added to the image stream in each transformer block.
- Image editing (FluxKontext): modulate the text/context stream (where instructions live) by adding small shifts and scales based on the preference.
- Text‑to‑video (LTX‑2): add a small shared residual to the video stream, similar to the image case.
Analogy: The preference vector is like a set of dials the user can turn. The conditioning modules connect those dials to the right parts of the model so the style shifts smoothly when you move them.
What they found and why it matters
The authors test ParetoSlider on three tasks and backbones—SD3.5 (text‑to‑image), FluxKontext (image editing), and LTX‑2 (text‑to‑video)—and compare it to several baselines that either:
- Collapse all goals into one fixed score during training (so you get one operating point), or
- Require training and storing several separate models and blending them later, or
- Do heavy, slow guidance at inference time.
Main takeaways:
- A single ParetoSlider model matches or beats separately trained models at their chosen trade‑offs, while also letting you smoothly move between trade‑offs at inference time.
- The transitions are smooth and predictable. For example:
- Text‑to‑image: you can slide between photorealistic and sketch/anime/watercolor styles.
- Image editing: you can slide between preserving the original image and strongly following the edit instruction.
- Text‑to‑video: you can slide between animated and photorealistic looks for the same prompt.
- Late scalarization (normalize each reward first, then weight by preferences) is crucial. It makes the model actually follow the requested balance instead of being dominated by one goal.
- Efficient and practical: You only need one checkpoint to cover many preferences, you don’t need to store multiple models, and you don’t need slow per‑step guidance at generation time.
Why this is important
Many creative and practical tasks have conflicting goals. For instance, you may want an edit to stay true to the original photo but also follow a bold instruction, or you might want a video that’s realistic but still stylized. In real life, there’s no single perfect setting—different users prefer different balances.
ParetoSlider shows a way to:
- Give users real‑time control with a simple “slider” over those trade‑offs.
- Keep quality high without training many separate models.
- Generalize across tasks (images, edits, videos).
- Provide a template for multi‑objective alignment in other generative systems, not just vision.
In short, this work makes generative models more flexible, efficient, and user‑friendly: one model learns the whole spectrum of “best possible” trade‑offs, and you get to pick the point you like—whenever you want.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, consolidated list of gaps that remain unresolved and could guide future research:
- Lack of theoretical guarantees: No formal proof that preference-conditioned training with linear (late) scalarization recovers the full Pareto front; coverage, optimality, and convergence properties are not characterized (e.g., ε-Pareto optimality bounds, sample complexity).
- Linear scalarization limits non-convex fronts: Weighted-sum aggregation only recovers the convex portion of the Pareto frontier; feasibility of reaching non-convex regions (e.g., via Tchebycheff, ε-constraint, or Chebyshev scalarizations) is not explored.
- Preference distribution design: The distribution used to sample preference vectors ω over the simplex during training is unspecified; its impact on coverage, bias toward certain trade-offs, and generalization to rare ω configurations is unstudied.
- Calibration and monotonicity of control: No quantitative assessment of whether changes in ω produce predictable, monotonic, and calibrated shifts in objective values; mapping ω to achieved rewards and inverse-mapping (finding ω that hits target constraints) is missing.
- Scaling to many objectives: Practical and algorithmic behavior when M grows (e.g., M > 4) is untested—training stability, compute overhead from scoring M rewards per sample, interference between objectives, and UI/UX for higher-dimensional control remain unclear.
- Robustness to reward mis-specification: Sensitivity to noisy, biased, or drifting reward models (e.g., VLM-based judges) is not evaluated; strategies for robust training under imperfect or contradictory rewards are needed.
- Impact of correlated or redundant rewards: The method treats rewards independently before aggregation, but how correlated rewards affect learning (e.g., redundant channels or conflicting gradients) is unexamined.
- Advantage normalization design: Only per-reward group-relative standardization is considered; alternative stabilizers (e.g., quantile/robust normalization, learned baselines, per-reward clipping schedules) and their effects on fairness across rewards are not analyzed.
- Group size K sensitivity: The effect of prompt-group size on variance, stability, bias, and sample efficiency (including degenerate small-K or large-K regimes) is not reported.
- KL regularization tuning: How the KL term influences retention of base capabilities, avoidance of catastrophic forgetting across ω extremes, and the trade-off with alignment strength is not quantified.
- Conditioning architecture generality: Preference injection is tailored to each backbone; broader comparisons (e.g., cross-attention, FiLM, prompt-level conditioning, feature-wise modulation depth) and guidelines for selecting conditioning locations are limited.
- Time-varying ω during generation: The method fixes ω over the denoising trajectory; potential benefits or pitfalls of dynamically varying ω across timesteps (e.g., coarse-to-fine control, stage-wise objectives) are unexplored.
- Multi-step/temporal rewards: Only terminal rewards are used; for video, step-wise (frame/sequence) rewards for motion smoothness, temporal consistency, or content safety and their integration into the training pipeline are not addressed.
- Sample diversity and mode coverage: Effects on diversity (e.g., avoidance of mode collapse as ω varies) and whether preference conditioning reduces variability within each trade-off setting are not measured.
- Generalization across backbones: Results are shown for SD3.5, FluxKontext, and LTX-2; extension to standard DDPMs, latent diffusion, and non-flow-matching architectures (and to discrete-token generators) is untested.
- Comparative evaluation breadth: Head-to-head quantitative comparisons against interpolation-based baselines (Rewarded Soups, Diffusion Blend) and training-free steering methods (e.g., per-step guidance) are insufficiently detailed beyond qualitative or limited scenarios.
- Human preference validation: Alignment claims rely on automated reward models; user studies assessing perceived control, satisfaction across ω, and fidelity to intended trade-offs are absent.
- Safety, bias, and misuse: How multi-objective control interacts with safety constraints (e.g., toxicity, fairness), potential reward hacking, and unintended emergent behaviors is not evaluated; incorporating safety rewards or constraints remains open.
- Compute and memory profiling: Training efficiency and scalability claims (especially as M increases) are not backed by detailed wall-clock, memory, and inference-time latency measurements; per-reward scoring cost is not benchmarked.
- Dataset coverage and OOD behavior: Robustness of Pareto control under out-of-distribution prompts, sources, or styles is not assessed; how limited training distributions bias reachable trade-offs is unclear.
- Hypervolume and Pareto metrics reporting: Hypervolume and frontier coverage results are deferred to supplementary material; standardized metrics and protocols for evaluating Pareto approximation quality are not established in the main text.
- Reward interaction audits: Systematic audits showing how improvements in one objective degrade others (beyond stylized examples) and identifying regions of steep trade-off (cliffs) on the frontier are missing.
- Transferability of ω embeddings: Whether preference-conditioning modules trained on one set of rewards can be reused or adapted for new reward sets without full retraining is unknown.
- Constraint-based control: The framework only supports weighted trade-offs; support for hard constraints (e.g., “maximize A subject to B ≥ b”) or goal-attainment functions is not explored.
Practical Applications
Immediate Applications
The following opportunities can be deployed now with existing diffusion backbones, reward models, and standard MLOps:
- Creative tools with live “trade‑off sliders” for generation and editing
- Sector: media/entertainment, design software, consumer apps
- Use cases:
- Text‑to‑image: interactively slide between photorealism and stylization (e.g., sketch, watercolor, anime), preserving a single checkpoint while exploring styles at inference time
- Image editing: user‑controlled balance between instruction adherence and source preservation (edit “strength” vs. fidelity)
- Text‑to‑video: dial between animated vs. photorealistic aesthetics for pre‑viz and social content
- Tools/products/workflows: plugins for Photoshop/After Effects/Premiere/Figma; mobile photo/video apps (e.g., Lightricks, Canva); web UIs with a single “style vs realism”/“edit intensity” slider backed by one preference‑conditioned model
- Assumptions/dependencies: reward models for each objective (e.g., photorealism classifiers, VLM‑based style scorers) are available and reasonably calibrated; base models support lightweight conditioning (e.g., AdaLN modulation); users accept VLM/CLIP‑based subjective scoring
- One‑checkpoint deployment instead of multiple tuned models
- Sector: software/MLOps, cloud inference, enterprise GenAI platforms
- Use cases: replace model “zoos” (one checkpoint per reward weight) with a single preference‑conditioned checkpoint; reduce storage, deployment complexity, and model selection logic
- Tools/products/workflows: unified API endpoint (generate(input, omega)), model registries storing preference metadata rather than fleets of checkpoints
- Assumptions/dependencies: LoRA or small adapters for preference injection are supported; late‑scalarization training (DiffusionNFT‑style) is run once with representative preference sampling
- Variant sweeps for A/B testing and creative review
- Sector: marketing/advertising, e‑commerce, media production
- Use cases: automatically generate a small Pareto sweep per brief (e.g., five points along the front) for human selection; align creative with engagement vs. brand identity trade‑offs
- Tools/products/workflows: “Pareto sweep” button in creative pipelines; batch generation with preset omega grid and reporting of reward scores
- Assumptions/dependencies: online or offline reward measurement available (e.g., aesthetic score, brand safety score); compute budget for small sweeps
- Content safety and policy tuning at inference time
- Sector: platform operations, content moderation, policy
- Use cases: adjust strictness trade‑offs (e.g., vividness vs. safety) for different surfaces or jurisdictions without retraining
- Tools/products/workflows: policy presets mapped to omega vectors; moderation dashboards tracing Pareto fronts between “utility” and “safety” rewards
- Assumptions/dependencies: reliable content safety reward models; governance over who can adjust omega and audit trails for settings
- Controlled synthetic data generation for ML
- Sector: academia/ML, enterprise data ops
- Use cases: generate datasets balancing realism vs. stylization to test robustness; produce controlled edits balancing label fidelity vs. appearance changes for augmentation
- Tools/products/workflows: data pipelines that emit images/videos along a defined omega grid with reward logs; HV (hypervolume) dashboards to track coverage
- Assumptions/dependencies: reward‑aligned to downstream metrics; adequate prompt diversity; controls for dataset bias introduced by reward models
- Evaluation and analysis tooling for multi‑objective alignment
- Sector: academia/ML research
- Use cases: use ParetoSlider to study/visualize trade‑offs, compare scalarization strategies (early vs. late), and monitor hypervolume improvements during training
- Tools/products/workflows: training dashboards that plot Pareto fronts and HV; ablation kits for conditioning strategies
- Assumptions/dependencies: standardized reward interfaces and logging; reproducible reward scales per experiment
Long‑Term Applications
These opportunities require additional research, domain‑specific reward design, scaling, or integration with broader systems:
- Multi‑stakeholder and jurisdiction‑aware alignment
- Sector: policy/regulation, platform governance
- Use cases: balance objectives such as safety, fairness, engagement, and diversity via transparent omega presets per region or demographic goal
- Tools/products/workflows: policy libraries mapping governance choices to omega; audit tools showing movement along fronts when policy changes
- Assumptions/dependencies: validated reward models for fairness/harms; processes for public accountability; mitigation of reward gaming
- Personalized preference control with hybrid conditioning
- Sector: consumer/enterprise apps
- Use cases: combine explicit omega weights with learned user embeddings to tailor trade‑offs to individuals or teams (e.g., brand stylization vs. realism for a specific studio)
- Tools/products/workflows: few‑shot preference capture plus explicit sliders; user/profile‑conditioned control surfaces
- Assumptions/dependencies: stable online learning without catastrophic forgetting; privacy‑preserving preference storage; mechanisms to prevent “reward hijacking” by idiosyncratic users
- Cross‑modal expansion (audio, speech, and music generation)
- Sector: audio tech, accessibility, entertainment
- Use cases: TTS trade‑offs (naturalness vs. intelligibility vs. latency); music generation (originality vs. prompt adherence vs. genre fidelity) with a single preference‑conditioned model
- Tools/products/workflows: omega‑controlled TTS/musical generators; UI sliders for clarity‑naturalness‑speed
- Assumptions/dependencies: differentiable or proxy rewards for audio attributes; adaptation of late‑scalarization to non‑diffusion policies if needed
- Energy/latency‑aware generation on edge and mobile
- Sector: energy/edge computing, mobile apps
- Use cases: include compute/latency as an explicit “cost” reward to expose a quality‑vs‑speed slider for on‑device generation
- Tools/products/workflows: dynamic inference that adjusts steps/samplers via omega; user‑facing “eco” vs. “quality” modes
- Assumptions/dependencies: measurable and stable mapping from quality to compute; training with cost rewards that generalize across devices
- Synthetic medical or scientific imagery with constrained trade‑offs
- Sector: healthcare, scientific research
- Use cases: control realism vs. privacy vs. label clarity; balance pathology visibility vs. anonymity in synthetic datasets
- Tools/products/workflows: regulated pipelines producing omega‑tagged outputs; QA steps with human oversight and domain‑specific rewards
- Assumptions/dependencies: clinically validated reward models; rigorous bias and safety evaluations; regulatory approvals
- Closed‑loop, metric‑driven creative optimization
- Sector: advertising, social platforms, streaming
- Use cases: real‑time adjustment of omega based on live metrics (CTR, watch time) while respecting safety/brand constraints; automated content iteration along Pareto fronts
- Tools/products/workflows: feedback controllers that tune omega; experiment platforms that monitor movement along fronts and guardrail thresholds
- Assumptions/dependencies: reliable online metrics; safeguards against optimizing to undesirable shortcuts; traffic‑aware experimentation ethics
- Robotics and planning with MORL principles from ParetoSlider
- Sector: robotics, autonomous systems
- Use cases: extend late‑scalarization and preference conditioning to policies balancing safety, speed, and comfort; generate perceptual inputs with controlled characteristics for sim‑to‑real
- Tools/products/workflows: MORL training wrappers with per‑objective advantage normalization; shared policy with omega control for deployment
- Assumptions/dependencies: adaptation from diffusion to control policies; reward engineering for real‑world safety; sample‑efficient training
- Interoperable “preference vectors” as a standard interface
- Sector: software/infrastructure
- Use cases: define portable omega schemas so different models and vendors accept consistent preference inputs (e.g., “safety”, “realism”, “brand style” axes)
- Tools/products/workflows: SDKs and API standards; model cards publishing supported reward axes
- Assumptions/dependencies: consensus on reward definitions and calibration; governance around naming, measurement, and comparability
Notes on feasibility across all applications:
- Reward availability and validity are the primary bottlenecks; the approach inherits the biases and limitations of the reward models (VLMs, CLIP‑based scores, domain classifiers).
- Stable training relies on group‑based advantage normalization and sufficient sample diversity per prompt; compute and data budgets must support online RL post‑training.
- Conditioning mechanisms assume diffusion/flow‑matching backbones with modules (e.g., AdaLN) that can be extended via lightweight adapters; other architectures may need alternative injection points.
- For safety‑critical or regulated domains, human oversight and rigorous validation remain necessary despite improved controllability.
Glossary
- AdaLN: Adaptive Layer Normalization; a conditioning mechanism that modulates transformer block activations via learned scale/shift (and often gating) parameters. "each modulated by AdaLN parameters derived from a shared timestep embedding temb."
- classifier-free guidance: An inference-time technique that adjusts generations by mixing conditional and unconditional predictions to control fidelity vs. creativity. "such as classifier-free guidance scales or prompt engineering"
- CLIPScore: A metric that uses CLIP embeddings to measure image–text alignment. "and CLIPScore~\cite{hessel2021clipscore, radford2021learning}"
- DiffusionNFT: An online RL fine-tuning framework for diffusion/flow-matching models that optimizes via the forward process with a flow-matching loss instead of trajectory policy gradients. "DiffusionNFT addresses these limitations by reformulating the policy optimization on the forward process rather than the reverse denoising process."
- DPO: Direct Preference Optimization; an offline alignment method that learns from pairwise human preference data without explicit reward modeling. "first adopted DPO for diffusion models"
- EMA: Exponential Moving Average; a smoothed copy of model parameters used as a stable target or teacher during training. "An exponential moving average (EMA) of the policy"
- early scalarization: Collapsing multiple rewards into a single weighted sum before optimization, which commits training to a fixed trade-off. "the prevailing practice of ``early scalarization'' collapses rewards into a fixed weighted sum."
- FlowGRPO: A variant of GRPO adapted to diffusion/flow models that performs group-relative policy optimization over reverse-time trajectories. "FlowGRPO optimization is carried out over a multi-step reverse-time trajectory."
- flow-matching: A generative modeling paradigm that learns a time-dependent velocity field to transport noise to data, enabling continuous-time sampling. "GRPO was adapted to flow-matching models"
- flow-matching loss: A supervised objective that trains the model to predict target velocities along the forward (noise-adding) process. "and a flow-matching loss"
- GRPO: Grouped Relative Policy Optimization; a policy-gradient method that normalizes rewards within groups to reduce variance without a learned value function. "More recently, GRPO~\cite{shao2024deepseekmathpushinglimitsmathematical} was adapted to flow-matching models"
- group-relative advantage: An advantage computed by standardizing rewards within a prompt group to create a relative training signal. "DiffusionNFT computes a group-relative advantage by normalizing rewards within each prompt group."
- hypervolume (HV) indicator: A Pareto-front quality metric measuring the volume dominated by a set of solutions with respect to a reference point. "using the hypervolume (HV) indicator, where our method consistently dominates."
- implicit velocity steering: A training trick that nudges the model’s velocity predictions toward higher-reward samples via EMA-based positive/negative targets and an interpolation weight. "using implicit velocity steering (§\ref{sec:prelim})"
- KL-divergence: A regularization term that keeps the fine-tuned policy close to a reference model by penalizing distributional drift. "A KL-divergence term regularizes the policy toward the pretrained reference model"
- LoRA: Low-Rank Adaptation; a parameter-efficient fine-tuning method that inserts low-rank adapters into weight matrices. "trained jointly with LoRA adapters."
- Markov decision process (MDP): A formalism for sequential decision-making defined by states, actions, transition dynamics, and rewards. "as a sequential Markov decision process (MDP)"
- MORL: Multi-Objective Reinforcement Learning; learning policies that balance multiple (often conflicting) objectives. "a multi-objective RL (MORL) framework"
- multi-objective optimization (MOO): Optimization over several competing objectives, typically yielding a set of trade-off solutions rather than a single optimum. "This conflict is a hallmark of multi-objective optimization (MOO)"
- Pareto dominance: A partial order where one solution is at least as good in all objectives and strictly better in at least one. "Pareto Dominance."
- Pareto front: The set of all non-dominated (Pareto-optimal) trade-offs where improving one objective would worsen another. "approximate the entire Pareto front."
- Pareto optimality: The property of a solution for which no objective can be improved without degrading another; non-dominated. "Pareto Optimality."
- PickScore: A learned human-preference model used as a reward/metric for aesthetic or alignment quality. "including PickScore~\cite{kirstain2023pickapicopendatasetuser}"
- preference vector: A nonnegative weight vector specifying the desired trade-off among objectives and used to condition the model. "we introduce a preference vector "
- probability simplex: The set of nonnegative vectors that sum to one, often used to represent mixtures or preferences. "where lies on the probability simplex "
- REINFORCE: A classic Monte Carlo policy-gradient algorithm using sampled returns to estimate gradients. "applied REINFORCE-style policy gradients"
- sampling scheduler: The rule that determines time-step progression and noise transitions during diffusion/flow sampling. "which in a flow-matching model corresponds to the sampling scheduler."
- timestep embedding: A learned embedding of the diffusion/flow time variable used to modulate the network across denoising steps. "timestep embedding temb"
- Vision-LLM (VLM): A model jointly trained on images and text, used here to define or compute reward signals for stylistic or semantic objectives. "we use VLM-based reward models"
Collections
Sign up for free to add this paper to one or more collections.
