Reinforcement Learning from Denoising Feedback

• Reinforcement Learning from Denoising Feedback (RLDF) is a family of methods that derive localized feedback from denoising processes rather than relying on a single scalar reward.
• In diffusion language models, RLDF optimizes clean-state prediction from noisy states using token clipping and weighted timestep sampling to balance efficiency and accuracy.
• In image and semantic domains, RLDF variants employ pixel-wise rewards and natural language critiques to enable precise credit assignment and enhance recovery from corrupted states.

Searching arXiv for relevant RLDF and denoising-feedback papers to ground the article. Reinforcement Learning from Denoising Feedback (RLDF) denotes a class of reinforcement-learning formulations in which the learning signal is derived from denoising-related structure rather than from a single undifferentiated terminal score. In the literature, the term is used in multiple non-identical ways. In diffusion LLMs, RLDF is introduced as a training paradigm for policy loss estimation from intermediate noisy states and clipped clean-state targets (He et al., 25 May 2026). In image generation, RLDF has also been used to describe Q-learning over a grammar-structured semantic state space with rewards extracted from diffusion outputs (Marathe, 2023). Closely related work extends scalar-reward diffusion RL to pixel-level feedback, recasts sparse temporal-difference learning in terms of a learned denoising-feedback operator, converts natural-language critiques into token-level gradients, or trains policies to recover from corrupted reasoning prefixes (Kordzanganeh et al., 2024, Liu et al., 2018, Wang et al., 28 May 2025, Xu et al., 27 May 2026). Accordingly, RLDF is best understood as a technically heterogeneous but conceptually coherent family centered on localized, denoising-like, or recovery-oriented feedback.

1. Terminological scope and historical usage

The most explicit contemporary formulation appears in "Reinforcement Learning from Denoising Feedback" (He et al., 25 May 2026), where RLDF addresses policy loss estimation for diffusion LLMs. That paper treats rollout trajectories and intermediate noisy states as the source of feedback, and optimizes the model toward a clipped clean state $\hat{x}_0$ from sampled noisy states $x_t$. Its stated objective is to balance computational efficiency and estimation effectiveness through clean-state prediction and weighted timestep sampling.

The same acronym had already been used differently in "Reinforcement Learning from Diffusion Feedback: Q* for Image Search" (Marathe, 2023). There, RLDF is not a policy-loss estimator for diffusion LLMs, but a semantic reinforcement-learning procedure for image generation and image search. The method performs Q-learning or SARSA over a finite grammar-based semantic encoding space, with diffusion-derived rewards rather than human feedback or direct model fine-tuning.

Several other papers do not present a canonical RLDF algorithm but are explicitly described as RLDF-style or naturally connected to an RLDF interpretation. "Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback" (Kordzanganeh et al., 2024) replaces a single scalar image reward with a pixel-wise heatmap, thereby reducing reward sparsity and cross-talk. "Dantzig Selector with an Approximately Optimal Denoising Matrix and its Application to Reinforcement Learning" (Liu et al., 2018) extends sparse recovery to TD learning through a learned denoising matrix $Q$, which transforms Bellman residuals before optimization. "Text2Grad: Reinforcement Learning from Natural Language Feedback" (Wang et al., 28 May 2025) turns critiques into span-level gradients, and "DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes" (Xu et al., 27 May 2026) trains policies to recover from incorrect reasoning states. This suggests that RLDF functions both as a specific method name and as a broader descriptor for RL systems in which the feedback signal identifies, filters, or repairs structured corruption.

2. RLDF for diffusion LLMs

In diffusion LLMs, RLDF is motivated by the difficulty of policy estimation under iterative denoising with bidirectional attention and changing masked-token configurations (He et al., 25 May 2026). The paper distinguishes three imperfect alternatives: no-masking or full-sequence approximations, random-masking or ELBO-style estimators, and exact stepwise loss at every denoising step. The first two induce training-inference mismatch; the third is faithful but computationally expensive.

RLDF therefore estimates loss from actual rollout contexts while subsampling the most informative denoising steps. The model generates trajectories

$o_T, o_{T-1}, \ldots, o_0,$

where $o_T$ is the fully masked initial state, $o_0$ is the final clean response, and $o_t$ denotes an intermediate denoising state. Rather than predicting $x_{t-1}$ from $x_t$, RLDF predicts the clean state $x_0$ from $x_t$0. The theoretical relation is stated as

$x_t$1

The method directly optimizes $x_t$2 and then clips the clean target to $x_t$3, retaining only sufficiently confident tokens. The implementation threshold is stated to be around $x_t$4 for token clipping.

Timestep selection is driven by rollout-derived uncertainty. For each denoising step,

$x_t$5

where $x_t$6 is the set of positions unmasked at step $x_t$7. These scores define sampling weights

$x_t$8

and the paper samples $x_t$9 timesteps per training example without replacement. The policy objective is REINFORCE-style for a single inner iteration and PPO-style with clipping for multiple inner updates; a KL term regularizes against a reference policy. The final objective aggregates policy and KL losses over sampled timesteps and responses.

The reported empirical outcome is consistent improvement across LLaDA-8B-Instruct and Dream-7B-Instruct on MATH500, GSM8K, AMC23, HumanEval, and MBPP (He et al., 25 May 2026). On LLaDA, RLDF improves over the base model by up to about $Q$0 on MATH500 and $Q$1 on MBPP; on Dream, the reported gains reach about $Q$2 on MATH500, $Q$3 on GSM8K, $Q$4 on AMC23, $Q$5 on HumanEval, and $Q$6 on MBPP. Ablations further state that $Q$7-prediction is preferable to $Q$8-prediction, that token clipping is necessary for stability, and that token-level normalization converges slightly faster.

3. Localized denoising feedback in diffusion image models

A closely related image-domain line replaces scalar reward with localized feedback. "Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback" (Kordzanganeh et al., 2024) starts from DDPO, which models the denoising process as a Markov chain with a final scalar reward $Q$9, and argues that such reward is sparse because each denoising step is credited by the same image-level score. PXPO instead introduces a pixel-wise reward map $o_T, o_{T-1}, \ldots, o_0,$0 and defines the total reward as

$o_T, o_{T-1}, \ldots, o_0,$1

Under the factorization

$o_T, o_{T-1}, \ldots, o_0,$2

the PXPO gradient becomes

$o_T, o_{T-1}, \ldots, o_0,$3

The paper presents this as a mechanism for reducing ambiguity, improving credit assignment, and preventing unwanted interaction between unrelated pixels. Its explicit conceptual contrast is that DDPO induces global coupling, whereas PXPO weights each pixel gradient by that pixel’s own reward.

Implementation is shaped by latent diffusion. Feedback is produced in pixel space but must be aligned with latent-space denoising states, so the heatmap is downsampled by interpolation to latent resolution, analogously to inpainting mask alignment. The training loop samples latent noise from $o_T, o_{T-1}, \ldots, o_0,$4, runs iterative denoising while retaining gradients of pixel-wise log probabilities, obtains a black-box feedback heatmap, downsamples it, multiplies it element-wise with pixel-wise log-likelihood gradients, and averages over time and pixel dimensions for the update.

The experiments show three feedback regimes. With dense blue-channel penalty on the prompt “red taxi in traffic,” mean reward improves from $o_T, o_{T-1}, \ldots, o_0,$5 to $o_T, o_{T-1}, \ldots, o_0,$6 using $o_T, o_{T-1}, \ldots, o_0,$7 samples from Stable Diffusion v1.4. With sparse AI feedback on “portrait of a man on the beach,” using SegFormer-based hair reduction, $o_T, o_{T-1}, \ldots, o_0,$8 images, $o_T, o_{T-1}, \ldots, o_0,$9 epochs, LoRA, and a single Nvidia A10, reward improves from $o_T$0 to $o_T$1. In the single-image human-feedback setting for “nature landscape,” the same generated image is aligned in two dramatically different ways from one starting point over $o_T$2 epochs. The paper states that PXPO is more sample-efficient than scalar-reward RL on diffusion models, can learn from a small number of images, does not require a learned reward model, and supports localized edits rather than only global ranking (Kordzanganeh et al., 2024).

4. RLDF as semantic reinforcement learning from diffusion feedback

The 2023 RLDF formulation for images treats the diffusion model as an environment and learns a policy over a semantic encoding space rather than over pixels or prompts (Marathe, 2023). The setup is a finite discounted MDP

$o_T$3

initialized at a random noise encoding state. The agent observes encoded state $o_T$4, selects action $o_T$5, receives reward $o_T$6, and transitions to $o_T$7. The paper describes this as semantic goal-conditional RL.

The state is derived from a context-free grammar

$o_T$8

whose nonterminals and terminals encode object, action, scene, and related attributes. The raw grammar state is compressed into a vector $o_T$9, producing a many-to-one semantic representation in which visually similar concepts are close in the encoding space. Reward can take one of three forms: Multi-Semantic Reward, Partial-Semantic Reward, or CLIP Reward, with the latter given explicitly as cosine similarity,

$o_0$0

Learning proceeds through standard value-based RL. The paper states both the Bellman optimality equations and the Q-learning update

$o_0$1

and also includes a SARSA branch. The practical algorithm encodes the input image into a terminal semantic state, initializes $o_0$2, chooses a reward function and exploration rate $o_0$3, and iteratively updates the value function from diffusion-generated image feedback.

A defining claim of this RLDF variant is that it requires no text input, no text guidance, and no model fine-tuning (Marathe, 2023). It is described as prior-preserving reward function guidance: the diffusion model remains unchanged, while the RL agent searches semantic space and exploits the diffusion model’s generative prior. The method supports image search, image generation from a single input image, and diversity control by moving beyond the exact goal along axes such as object, action, scene, weather, location, and artistic style.

The reported quantitative results include generation of $o_0$4M synthetic images for $o_0$5 classes, with splits $o_0$6. A ResNet-18 trained on synthetic RLDF ImageNet-100 data achieves $o_0$7 Acc@1 on the paper’s validation set, compared with $o_0$8 on an ImageNet100 random sample, $o_0$9 on the ImageNet100 validation set, and $o_t$0 for a generated IN100 baseline. Reported FID values for the RLDF clone are $o_t$1 on ImageNet, $o_t$2 on Sketch, $o_t$3 on ImageNet-R, $o_t$4 on ImageNet-A, and $o_t$5 on ImageNet-O, with KID values around $o_t$6–$o_t$7. The paper also states that $o_t$8 is often a good exploration-exploitation balance, that Partial-Semantic Reward is more gradual and stable than CLIP reward in some settings, and that CLIP reward offers finer guidance but can get stuck in local optima.

5. RLDF-style formulations beyond diffusion image generation

Several papers instantiate the same feedback principle outside the exact named RLDF setting. In sparse TD learning, "Dantzig Selector with an Approximately Optimal Denoising Matrix and its Application to Reinforcement Learning" (Liu et al., 2018) generalizes the Dantzig Selector by replacing the fixed denoising matrix $o_t$9 with a learned matrix $x_{t-1}$0. The generalized formulation is

$x_{t-1}$1

The paper’s denoising-feedback interpretation is that $x_{t-1}$2 feeds the residual into a denoised space, and the same idea is transferred to temporal-difference learning through the empirical Bellman residual $x_{t-1}$3:

$x_{t-1}$4

Here $x_{t-1}$5 is learned from the surrogate $x_{t-1}$6 subject to column constraints. The paper reports that, in a 20-state corrupted chain with 200 off-policy samples and many noisy Gaussian features, ODDS-TD produces value functions much closer to the true topology than DS-TD, and in the harder 606-feature setting preserves structure and yields the correct policy.

"Text2Grad: Reinforcement Learning from Natural Language Feedback" (Wang et al., 28 May 2025) is explicitly described as RLDF-like rather than a canonical RLDF instantiation. It begins from the claim that scalar RLHF rewards are too coarse and instead converts critiques into token-level pseudo-rewards. Given span annotations $x_{t-1}$7, tokens receive $x_{t-1}$8, and the central update is the Natural Language Gradient,

$x_{t-1}$9

A PPO objective is then applied with token-level advantages. Across SLF5K summarization, KodCode code generation, and UltraFeedback alignment, the paper reports that Text2Grad surpasses scalar-reward RL and prompt-only baselines, with, for example, ROUGE-1 $x_t$0, BLEU $x_t$1, and BERTScore F1 $x_t$2 on summarization, plus a $x_t$3 BLEU improvement over PPO and a GPT-4 judge win-rate improvement of $x_t$4.

"DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes" (Xu et al., 27 May 2026) uses denoising in a different operational sense. Wrong prefixes $x_t$5 are harvested from incorrect weak-model traces, with

$x_t$6

and the policy is trained to continue from this corrupted context and still reach the correct answer. The joint PPO-style objective mixes standard on-policy rollouts with denoise rollouts. On Qwen3-4B-Base, GRPO averages $x_t$7 while DenoiseRL-GRPO averages $x_t$8; DAPO averages $x_t$9 while DenoiseRL-DAPO averages $x_0$0. On Qwen3-8B-Base, GRPO averages $x_0$1, DenoiseRL-GRPO $x_0$2, DAPO $x_0$3, and DenoiseRL-DAPO $x_0$4. The paper states that denoise rollouts broaden the training state distribution and induce repair behavior rather than imitation.

A further antecedent is "Deep Reinforcement Learning Autoencoder with Noisy Feedback" (Goutay et al., 2018), where the transmitter is trained with a policy-gradient estimator weighted by scalar losses computed at the receiver:

$x_0$5

The feedback link is noisy, $x_0$6, but the estimator remains unbiased and only its variance increases. The paper then learns a separate real-number communication system to transmit these losses. On both AWGN and Rayleigh block-fading channels, the communication system trained with the learned feedback system achieves the same BLER as training with perfect feedback.

6. Technical themes, limitations, and common points of confusion

Across these formulations, the recurring technical theme is fine-grained credit assignment. In diffusion language-model RLDF, the informative unit is a sampled denoising timestep and a clipped subset of clean tokens (He et al., 25 May 2026). In PXPO, it is the pixel and latent position (Kordzanganeh et al., 2024). In Text2Grad, it is the token span annotated by critique (Wang et al., 28 May 2025). In DenoiseRL, it is the continuation conditioned on a corrupted reasoning prefix (Xu et al., 27 May 2026). In ODDS-TD, it is the denoised residual transformed by a learned operator $x_0$7 (Liu et al., 2018). This suggests that RLDF-style methods are less about a single reward formalism than about replacing globally compressed supervision with structured feedback tied to the locus of error or uncertainty.

A common point of confusion is that RLDF is not used uniformly across the literature. The 2026 dLLM paper defines RLDF as a specific policy-loss estimation method for diffusion LLMs (He et al., 25 May 2026). The 2023 image-search paper uses the same acronym for semantic Q-learning over diffusion feedback (Marathe, 2023). Other papers are only said to be RLDF-like or naturally connected to an RLDF interpretation. It is therefore inaccurate to treat all such methods as instances of a single standardized algorithm.

The limitations also differ but exhibit a family resemblance. The dLLM RLDF paper notes that compute remains significant, that Dream is more unstable than LLaDA and requires stronger KL regularization, and that the approach depends on good confidence estimation (He et al., 25 May 2026). PXPO assumes pixel-wise independence, depends on the quality of the heatmap source, and is presented as a proof-of-concept with limited comparative benchmarking; it also demonstrates that the model can exploit weaknesses in the feedback model (Kordzanganeh et al., 2024). The semantic RLDF image-search framework notes sensitivity to reward design, random initialization, and world-size scaling, while partial semantic reward and CLIP reward expose different stability-precision trade-offs (Marathe, 2023). DenoiseRL reports overthinking and verbosity for longer corrupted prefixes, instability if off-policy prefixes are updated directly, and reliance on a verifier (Xu et al., 27 May 2026). Text2Grad depends on critique quality and span alignment (Wang et al., 28 May 2025). ODDS-TD relies on a surrogate because the true optimal denoising matrix is NP-hard to compute (Liu et al., 2018).

Taken together, these works establish RLDF as a research direction in which denoising feedback is not merely auxiliary metadata but the operative signal shaping policy updates. Depending on the domain, that signal may be an informative timestep in a denoising trajectory, a pixel heatmap, a semantic reward over grammar states, a learned residual transform, a token-level critique map, or a recovery signal from a corrupted prefix. The unifying principle is that reinforcement learning is driven by feedback that identifies where and how a partially corrupted generation or estimate should be repaired, rather than by a single scalar evaluation applied uniformly to the entire trajectory.