RewardFlow: Generate Images by Optimizing What You Reward

Abstract: We introduce RewardFlow, an inversion-free framework that steers pretrained diffusion and flow-matching models at inference time through multi-reward Langevin dynamics. RewardFlow unifies complementary differentiable rewards for semantic alignment, perceptual fidelity, localized grounding, object consistency, and human preference, and further introduces a differentiable VQA-based reward that provides fine-grained semantic supervision through language-vision reasoning. To coordinate these heterogeneous objectives, we design a prompt-aware adaptive policy that extracts semantic primitives from the instruction, infers edit intent, and dynamically modulates reward weights and step sizes throughout sampling. Across several image editing and compositional generation benchmarks, RewardFlow delivers state-of-the-art edit fidelity and compositional alignment.

• The paper introduces a unified, training-free approach that fuses global, regional, and language-based rewards using multi-reward Langevin dynamics.
• It employs a prompt-aware adaptive policy to dynamically modulate reward influence, enhancing semantic fidelity and spatial precision in image edits.
• Experimental results demonstrate significant gains in compositional accuracy, editing fidelity, and efficiency through reduced sampling complexity.

RewardFlow: Unified Inference-Time Multi-Reward Langevin Guidance for Controllable Image Generation and Editing

Introduction

The RewardFlow framework addresses the limitations of current inference-time, training-free methods for text-guided image generation and editing, which often lack fine-grained controllability, semantic faithfulness, and spatial precision. Diffusion and flow-matching models have established new paradigms in generative vision, but despite significant progress, methods without explicit image inversion suffer from drift, weak identity preservation, semantic leakage, and limited local control. Prior reward-guided optimization approaches improve global alignment but do not enable precise spatial or object-level constraint enforcement due to coarse, non-adaptive reward formulations.

RewardFlow introduces a general-purpose, training-free, inversion-free paradigm for controllable image transformation on pre-trained diffusion/flow-matching backbones. Leveraging multi-reward Langevin dynamics, it unifies global, spatially localized, and language-based objectives into a fused differentiable reward function. This is modulated at inference by a prompt-aware, adaptive fusion policy that parses semantic primitives, infers editing intent, and allocates reward influence dynamically across the sampling trajectory.

Methodology

Multi-Reward Langevin Dynamics

RewardFlow operates by iteratively steering the latent denoising trajectory of a pre-trained model at inference. At each step, the system computes gradients of several heterogeneous, differentiable rewards in image space, including semantic alignment, perceptual similarity, spatial grounding, object consistency, human preference alignment, and a novel VQA-based reward for fine-grained supervision. These are fused adaptively and projected back to latent space via the denoiser and decoder Jacobians.

A closed-loop controller predicts both reward weights and sampling step sizes as functions of the prompt structure, the parsed intention (add, remove, modify), the evolving reward profiles, and the denoising schedule. This achieves coarse-to-fine guidance: global alignment rewards dominate early, while local and object-specific objectives are strengthened as the image approaches the target region and state.

A KL-divergence-based regularizer tethers the clean latent back towards the encoded source image, preventing identity drift and excessive content modification.

Figure 2: Overview of the RewardFlow framework, highlighting adaptive reward modulation and the multi-reward diffusion guidance pipeline.

Hierarchical Reward Toolkit

The reward regime is explicitly diversified:

• Global and perceptual rewards: Vision-LLMs (SigLIP, Perception Encoder) assess overall semantic and pixel-level prompt alignment.
• Region-level grounding: RegionCLIP-based localization rewards promote spatial precision for edit focus.
• Object consistency: Text-guided SAM2 is leveraged to compute soft masks and enforce edit constraints strictly within object boundaries, penalizing leakage.
• Human preference: HPS-based scores measure aesthetics and holistic prompt adherence.
• VQA-based reward: A differentiable vision-LLM (Qwen-2.5-VL) evaluates the answer to contextually generated, instruction-driven QAs, enabling granular attribute and relationship enforcement.

  Figure 3: Gradient localization of the fused differentiable rewards—gradients are sharply localized to the target semantic regions, supporting high-precision control.

Prompt-Aware Adaptive Policy

Reward weights and Langevin step sizes are determined by a lightweight policy that parses prompt "semantic primitives" (atomic actionable intentions) using an LLM. The framework additionally infers edit direction (add vs. remove), intent class, and dynamically tunes reward prioritization and exploration-refinement ratio over time.

This modular, compositional policy prevents cross-objective interference, limits semantic drift, and allows RewardFlow to support complex, multi-instruction prompts with robust disentanglement.

Figure 5: Gradient localization with various reward compositions—enabling all components eliminates gradient leakage and concentrates edits at object boundaries.

Experimental Results

Image Editing

On PIE-Bench, RewardFlow achieves new state-of-the-art (SOTA) for zero-shot, training-free editing fidelity and compositional alignment under controlled backbone settings (e.g., using Flux):

• Quantitative improvements include:
  • Distance reduction by 7.3% over the best prior control baseline (7.78 vs. 8.39),
  • PSNR and SSIM gains of 5.3% and 2.6%,
  • Whole and Edited accuracy gains up to 8.6%,
  • All improvements are achieved with significantly reduced sampling complexity.

Additionally, in few-step settings, RewardFlow with Flux and Qwen Image reduces error metrics by up to 44.4% over baselines, with proportional gains in semantic edit accuracy.

Figure 4: Qualitative editing results—RewardFlow generates both semantically correct and highly localized edits, better preserving background, structure, and identity than competitive methods.

Figure 7: Qualitative results with Flux + RewardFlow, showing precise, fine-grained, and diverse edits achieved without over-editing or content destruction.

Text-to-Image Generation

On T2I-CompBench and GenEval, RewardFlow systematically enhances compositional attribute binding and relational correctness performance on both moderate and strong backbones (PixArt-$\alpha$, Flux, Qwen):

• Overall compositional accuracy (Flux backbone, GenEval): 0.81 (RewardFlow), vs. 0.72 (reward-guided baseline ReNO), and 0.64 (backbone).
• Particularly strong improvements are observed in complex, multi-object, relational, and attribute-specific categories.

  Figure 8: Text-to-image generation—RewardFlow exhibits superior global and localized compositional alignment, aesthetic quality, and attribute binding across diverse prompts.

  

  Figure 1: Comparison of generations with and without RewardFlow across three modes—full multi-reward guidance delivers improved compositional adherence and fidelity.

Ablation and Analysis

Ablation studies confirm the necessity of every component:

• Exclusion of the KL tether, dynamic weighting, reward-aware step sizing, or semantic primitives each leads to measurable performance and edit alignment degradation, as verified both quantitatively and qualitatively.
• Progressive inclusion of region, object, and VQA rewards refines gradient focus, supporting edits that are restricted, instruction-faithful, and non-intrusive.

  Figure 10: Stepwise removal of reward components—each differentiable reward contributes uniquely to localization and semantic precision.

  

  Figure 11: Architectural ablations—jointly adaptive control is required for visual consistency and prompt satisfaction.

Reward progression curves exhibit smooth, monotonic reward optimization under all objectives, confirming the stability and efficacy of multi-reward Langevin dynamics.

Figure 6: Reward signal progression during sampling, showing all components trend upward, with the adaptive policy enabling coordinated, multi-objective optimization.

Theoretical Justification

The update rule realized by RewardFlow is theoretically derived as an Euler–Maruyama discretization of a Langevin SDE on a prompt-tilted energy landscape. The framework provably converges to the distribution maximizing total (weighted) reward and semantic consistency, with the KL term ensuring bounded deviation from the latent source.

Implications and Future Directions

RewardFlow reframes controllable generation as modular, reward-guided test-time optimization, supporting plug-and-play extensibility. All reward modules are differentiable, interpretable, and replaceable. This compositionality allows for the potential integration of stronger or domain-specific reward networks (e.g., for photogrammetry, medical imaging, or factual alignment) and eventual extension to higher-order modalities such as video. The generalized adaptive control policy supports robust generalization to out-of-distribution instructions and multi-stage editing workflows.

Crucially, embedding vision-language reasoning directly into the reward function (via differentiable VQA supervision) represents a shift towards reward models with richer semantic structure. Limitations due to visual-LLM accuracy (e.g., counting limitations in current architectures) indicate an avenue where advances in VLMs or external supervision will translate directly to improved controllable generation and editing fidelity.

Conclusion

RewardFlow provides a unified, robust, and efficient methodology for fine-grained, instruction-faithful image editing and generation, outperforming previous zero-shot, training-free frameworks across multiple metrics and backbones. By leveraging prompt-aware adaptive control and a diversified suite of differentiable rewards—including language-based and region-conditional objectives—it achieves unprecedented edit locality, semantic fidelity, and visual consistency in both editing and generative regimes. The flexibility of the approach points towards future advances in multi-modal controllable synthesis and deeper integration of vision-language reasoning as an explicit optimization target.