---
title: Dual-Feedback Actor in Reinforcement Learning
url: https://www.emergentmind.com/topics/dual-feedback-actor-dfa
type: topic
---

# Dual-Feedback Actor in Reinforcement Learning

Dual-Feedback Actor (DFA) is a reinforcement-learning algorithm that fuses both individual rewards and pairwise preferences into a single policy-improvement framework, while avoiding a separate reward-modeling stage. In the formulation introduced in "Fusing Rewards and Preferences in Reinforcement Learning" [2508.11363], numeric rewards update the critic as in an off-policy method such as Soft Actor-Critic (SAC), whereas preferences—whether state-level, trajectory-level, or synthesized online from stored \(Q\)-values—update the policy directly through a likelihood defined on the policy’s own log-probabilities. Under a Bradley–Terry model, minimizing DFA’s preference loss recovers the entropy-regularized SAC policy [2508.11363].

## 1. Conceptual formulation

DFA is defined on a finite-horizon Markov decision process
\[
\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,
\]
with policy \(\pi_\theta(a\mid s)\), discount \(\gamma\in(0,1)\), and trajectories
\[
\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).
\]
Its motivation is to remove the intermediate reward-model step that is typical in reinforcement learning from human feedback (RLHF), while still retaining the strengths of entropy-regularized off-policy RL [2508.11363].

The two supervisory channels in DFA are scalar rewards and pairwise preferences. The paper’s central claim is that these need not be handled by separate learning subsystems. Instead, rewards inform \(Q\)-learning, while preferences act directly on the actor through a preference likelihood over policy outputs. This makes DFA a bridge between off-policy RL and direct preference optimization in stochastic control settings [2508.11363].

The method is particularly motivated by settings in which rewards are sparse, noisy, or unavailable, but preferences are easier to obtain. The reported drawbacks of the standard two-stage RLHF pipeline are reward-model misspecification, overfitting, distribution shift, and additional data and engineering cost [2508.11363]. DFA addresses these by directly optimizing the policy from preferences, optionally while also using numeric rewards.

## 2. Preference modeling and policy loss

For state-wise comparisons, the preference dataset is
\[
D=\{(s_k,a_k^+,a_k^-)\}_{k=1}^K, \qquad a_k^+\succ a_k^-.
\]
DFA defines the probability that the policy prefers \(a^+\) to \(a^-\) at state \(s\) directly from policy probabilities:
\[
P_\theta(a^+\succ a^-\mid s) =
\frac{\pi_\theta(a^+\mid s)^\alpha}
{\pi_\theta(a^+\mid s)^\alpha+\pi_\theta(a^-\mid s)^\alpha},
\qquad \alpha>0.
\]
Equivalently,
\[
P_\theta(a^+\succ a^-\mid s)
=
\sigma\!\left(\alpha\bigl[\log\pi_\theta(a^+\mid s)-\log\pi_\theta(a^-\mid s)\bigr]\right),
\]
where
\[
\sigma(z)=\frac{1}{1+e^{-z}}.
\]
This is the defining modeling move of DFA: the policy’s own log-probabilities serve as the score differences that explain observed preferences [2508.11363].

The state-wise preference loss is the negative log-likelihood
\[
\mathcal{L}_{\mathrm{pref}}(\theta)
=
-\mathbb{E}_{(s,a^+,a^-)\sim D}
\left[
\log P_\theta(a^+\succ a^-\mid s)
\right],
\]
or, in logistic form,
\[
\mathcal{L}_{\mathrm{pref}}(\theta)
=
-\mathbb{E}_{(s,a^+,a^-)\sim D}
\left[
\log \sigma\!\left(
\alpha\bigl(\log\pi_\theta(a^+\mid s)-\log\pi_\theta(a^-\mid s)\bigr)
\right)
\right].
\]

DFA also supports trajectory-level comparisons. For
\[
D^{\mathrm{traj}}=\{(\tau_k^+,\tau_k^-)\}_{k=1}^K,
\]
the policy’s trajectory likelihood is
\[
\pi_\theta(\tau)=\prod_{t=1}^T \pi_\theta(a_t\mid s_t),
\]
and the corresponding preference probability is
\[
P_\theta^{\mathrm{traj}}(\tau^+\succ\tau^-)
=
\frac{\pi_\theta(\tau^+)^\alpha}
{\pi_\theta(\tau^+)^\alpha+\pi_\theta(\tau^-)^\alpha}
=
\sigma\!\left(
\alpha\bigl(\log\pi_\theta(\tau^+)-\log\pi_\theta(\tau^-)\bigr)
\right).
\]
The trajectory loss is then
\[
\mathcal{L}_{\mathrm{traj}}(\theta)
=
-\mathbb{E}_{(\tau^+,\tau^-)\sim D^{\mathrm{traj}}}
\left[
\log P_\theta^{\mathrm{traj}}(\tau^+\succ\tau^-)
\right].
\]

The temperature \(\alpha\) controls the sharpness of the induced comparisons. The paper reports that \(\alpha\to 0\) yields very soft, near-uniform comparisons, whereas \(\alpha\to\infty\) approaches deterministic winner-take-all behavior [2508.11363].

## 3. Fusion of rewards and preferences in off-policy learning

DFA combines reward-driven critic learning with preference-driven actor updates. When numeric rewards are available, the critic is updated as in an off-policy RL method such as SAC; preferences are then used for policy improvement through \(\mathcal{L}_{\mathrm{pref}}\) or \(\mathcal{L}_{\mathrm{traj}}\). The source description is explicit that the method does not present a single combined scalar formula of the form
\[
\mathcal{L}_{\text{total}}=\mathcal{L}_Q+\mathcal{L}_{\text{pref}},
\]
but the algorithmic fusion is exactly critic learning from rewards plus actor learning from preferences [2508.11363].

A notable feature is synthesized online preference generation from replay and \(Q\)-values. For a batch of states \(s_i\), the method takes the stored action \(a_i\), finds a nearby state \(s_i'\) in the replay buffer and its action \(a_i'\), and compares the two using the critic:
\[
\text{If } Q(s_i,a_i)>Q(s_i,a_i') \text{ then } (a_i^+,a_i^-)=(a_i,a_i'),
\]
and otherwise the pair is swapped. This yields synthetic comparisons
\[
D^{\mathrm{Syn}}=\{(s_i,a_i^+,a_i^-)\}_{i=1}^N,
\]
with loss
\[
\mathcal{L}^{\mathrm{Syn}}(\theta)
=
-\mathbb{E}_{(s_i,a_i^+,a_i^-)\sim D^{\mathrm{Syn}}}
\left[
\log \sigma\!\left(
\alpha\bigl(\log\pi_\theta(a_i^+\mid s_i)-\log\pi_\theta(a_i^-\mid s_i)\bigr)
\right)
\right].
\]

The paper describes three feedback modes within one framework: state-level human preferences, trajectory-level human preferences, and synthesized online preferences from replay and critic values [2508.11363]. This makes DFA applicable when true preferences are available from annotators and also when reward signals are available but preferences are not.

A plausible implication is that DFA reframes the actor update as a comparison-learning problem without removing the role of reward-based value estimation. The critic remains reward-grounded, while the actor is trained through comparative evidence.

## 4. Theoretical relation to entropy-regularized control

The central theoretical result in DFA is stated under a Bradley–Terry preference model. The true preference probability is assumed to satisfy
\[
P^\star(a\succ b\mid s)
=
\sigma\!\bigl(\beta[Q^\star(s,a)-Q^\star(s,b)]\bigr),
\]
where \(Q^\star\) is the soft-optimal action-value function
\[
Q^\star(s,a)
=
\max_\pi
\mathbb{E}\!\left[
\sum_{t=0}^\infty \gamma^t
\bigl(r(s_t,a_t)+\lambda\,\mathcal H(\pi(\cdot\mid s_t))\bigr)
\Bigm| s_0=s,a_0=a
\right].
\]

If a tabular policy is parameterized by \(\ell_a=\log\pi(a\mid s)\) and one minimizes
\[
\mathcal{L}(\boldsymbol\ell)
=
-\frac{1}{|\mathcal A|^2}
\sum_{a,b\in\mathcal A}
P^\star(a\succ b\mid s)
\log \sigma\!\bigl(\alpha(\ell_a-\ell_b)\bigr),
\]
then the unique minimizer is
\[
\pi_\star(a\mid s)
=
\frac{\exp\!\bigl(\frac{\beta}{\alpha}Q^\star(s,a)\bigr)}
{\sum_{a'}\exp\!\bigl(\frac{\beta}{\alpha}Q^\star(s,a')\bigr)}.
\]
This is a Gibbs policy over \(Q^\star\), and it matches the SAC policy-improvement solution
\[
\pi_{\mathrm{SAC}}(a\mid s)\propto
\exp\!\left(\frac{Q^\star(s,a)}{\lambda}\right)
\]
when
\[
\lambda=\frac{\alpha}{\beta}.
\]
The paper therefore proves that minimizing DFA’s preference loss recovers the SAC policy under Bradley–Terry preferences [2508.11363].

The proof intuition given in the source has four parts: the preference loss is strictly convex in the full-support tabular log-policy variables; its gradient vanishes when modeled preference probabilities equal true Bradley–Terry probabilities; this equality implies
\[
\ell_a-\ell_b=\frac{\beta}{\alpha}\bigl(Q^\star(s,a)-Q^\star(s,b)\bigr);
\]
and normalization yields the Gibbs form. The Hessian is described as a weighted graph Laplacian over actions, positive definite on the feasible tangent space, which establishes uniqueness [2508.11363].

This result situates DFA as a preference-driven view of SAC-style policy improvement. When comparisons are generated according to Bradley–Terry from the soft-optimal \(Q^\star\), DFA and SAC agree at the policy level.

## 5. Empirical evaluation

The empirical study reported for DFA covers two distinct regimes: continuous-control experiments with synthetic preferences and a stochastic GridWorld with human-like preference comparisons [2508.11363].

In six MuJoCo control tasks—Walker2d, Hopper, Swimmer, Humanoid, MountainCarContinuous, and Pendulum—the setup uses the same replay buffer as SAC, 10 million environment interactions, 5 random seeds, batch size 256, and synthetic preference pairs generated from rewards or \(Q\)-values at each gradient step. DFA is reported to match or exceed SAC on Walker2d, Hopper, Swimmer, and Humanoid, to solve MountainCarContinuous where SAC struggled under the tested hyperparameters, and to exhibit smoother training curves, which the authors attribute to the denoising effect of synthesized comparisons [2508.11363].

In the stochastic GridWorld experiments, the main setting is \(5\times 5\), with larger variants in the appendix, stochastic action reversal with probability \(0.4\), random cell rewards, horizon \(20\), and a tabular softmax policy. The baselines are RM+PPO, ZPG, OnlineDPO, and Oracle-PPO. DFA is reported to outperform RM+PPO, approach Oracle-PPO, and outperform or be more stable than ZPG and OnlineDPO in the reported setting [2508.11363].

The paper also reports hyperparameter sensitivity for \(\alpha\). If \(\alpha\) is too large, the learning signal becomes weak; if it is too small, the policy becomes overly stochastic. The best GridWorld run used a small but nonzero \(\alpha\), around \(10^{-3}\) [2508.11363].

| Setting | Reported outcome |
|---|---|
| MuJoCo synthetic preferences | DFA matches or exceeds SAC on Walker2d, Hopper, Swimmer, and Humanoid |
| MountainCarContinuous | DFA solves the task where SAC struggled under the tested hyperparameters |
| Stochastic GridWorld | DFA outperforms RM+PPO and approaches Oracle-PPO |

The reported empirical interpretation is not that DFA improves predictive modeling or value estimation in isolation, but that comparison-based policy improvement can be competitive with or superior to reward-only actor updates while remaining stable in off-policy training.

## 6. Scope, distinctions, and related formulations

The name Dual-Feedback Actor refers specifically to the 2025 reinforcement-learning method that fuses rewards and pairwise preferences in policy improvement [2508.11363]. It is distinct from several nearby formulations in the literature.

First, DFA should not be conflated with Direct Feedback Alignment, for which the same abbreviation is widely used in deep learning. In that line of work, DFA denotes a non-backpropagation training rule in which fixed random matrices transmit output error directly to hidden layers, as in "Training DNNs in O(1) memory with MEM-DFA using Random Matrices" [2012.11745], "Differentially Private Deep Learning with Direct Feedback Alignment" [2010.03701], and "DFA-GNN: Forward Learning of Graph Neural Networks by Direct Feedback Alignment" [2406.02040]. Those methods address gradient transport, memory, privacy, or graph learning; they are unrelated to preference-based reinforcement learning in the sense used by Dual-Feedback Actor.

Second, earlier actor-critic and interactive-RL papers contain dual-channel or dual-critic ideas that are related in motivation but different in mechanism. "Actor-Critic Reinforcement Learning with Simultaneous Human Control and Feedback" [1703.01274] studies concurrent human control and reward-shaping feedback for an actor-critic learner. The source description explicitly characterizes that work as a precursor and conceptual ancestor of DFA-style interactive learning, but not as a Dual-Feedback Actor algorithm. Its dual aspect is at the human-interface and reward-shaping level, not an explicit policy update from pairwise preferences [1703.01274].

Third, dual-critic approaches are also distinct. "Actor-Dual-Critic Dynamics for Zero-sum and Identical-Interest Stochastic Games" [2602.00606] uses an actor with a fast critic and a slow critic in a three-timescale, payoff-based stochastic-game framework. "Boosting the Actor with Dual Critic" [1712.10282] derives Dual-AC from the Lagrangian dual form of the Bellman optimality equation and interprets critic feedback as Bellman residual or constraint-violation information. These are dual-critic or dual-objective actor-critic methods, not reward-plus-preference fusion in the sense of DFA [2602.00606; 1712.10282].

The most important practical distinction is therefore between three notions that share overlapping vocabulary: direct feedback alignment in neural-network training, dual-critic actor-critic schemes in RL, and Dual-Feedback Actor as a preference-based off-policy RL algorithm. In the 2025 usage, the “dual feedback” refers to the coexistence of scalar rewards and pairwise preferences within a single policy-learning framework [2508.11363].

A common misconception is that DFA is simply reward modeling without an explicit reward network. The defining claim of the method is stronger: preferences are modeled directly through policy log-probabilities, and the reward-modeling stage is omitted entirely [2508.11363]. Another common misconception is that DFA replaces critics. It does not; rewards still update \(Q\)-networks, and the actor is trained from preference losses rather than from a separate learned reward model.

Within preference-based reinforcement learning, DFA’s reported advantages are a single preference-based update rule, dual supervision, off-policy compatibility, entropy-preserving exploration, a theoretical bridge to SAC, and empirically smoother learning curves. Its reported limitations are the Bradley–Terry assumption, possible synthetic preference noise when the critic is inaccurate, nearest-state search cost in replay, sensitivity to \(\alpha\), and limited real-human evaluation, which is semi-synthetic rather than large-scale human-in-the-loop deployment [2508.11363].

Source: https://www.emergentmind.com/topics/dual-feedback-actor-dfa