Dual-Feedback Actor in Reinforcement Learning

• Dual-Feedback Actor (DFA) is a reinforcement learning algorithm that fuses individual rewards and pairwise preferences to update policies without a separate reward-modeling stage.
• It uses scalar rewards to update the critic as in off-policy methods while employing a Bradley–Terry model for direct policy improvement via preference likelihoods.
• Empirical results in MuJoCo and GridWorld demonstrate that DFA offers smoother training and superior performance compared to traditional reward-only updates.

Dual-Feedback Actor (DFA) is a reinforcement-learning algorithm that fuses both individual rewards and pairwise preferences into a single policy-improvement framework, while avoiding a separate reward-modeling stage. In the formulation introduced in "Fusing Rewards and Preferences in Reinforcement Learning" (Khorasani et al., 15 Aug 2025), numeric rewards update the critic as in an off-policy method such as Soft Actor-Critic (SAC), whereas preferences—whether state-level, trajectory-level, or synthesized online from stored $Q$-values—update the policy directly through a likelihood defined on the policy’s own log-probabilities. Under a Bradley–Terry model, minimizing DFA’s preference loss recovers the entropy-regularized SAC policy (Khorasani et al., 15 Aug 2025).

1. Conceptual formulation

DFA is defined on a finite-horizon Markov decision process

$$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$

with policy $\pi_\theta(a\mid s)$, discount $\gamma\in(0,1)$, and trajectories

$\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$

Its motivation is to remove the intermediate reward-model step that is typical in reinforcement learning from human feedback (RLHF), while still retaining the strengths of entropy-regularized off-policy RL (Khorasani et al., 15 Aug 2025).

The two supervisory channels in DFA are scalar rewards and pairwise preferences. The paper’s central claim is that these need not be handled by separate learning subsystems. Instead, rewards inform $Q$-learning, while preferences act directly on the actor through a preference likelihood over policy outputs. This makes DFA a bridge between off-policy RL and direct preference optimization in stochastic control settings (Khorasani et al., 15 Aug 2025).

The method is particularly motivated by settings in which rewards are sparse, noisy, or unavailable, but preferences are easier to obtain. The reported drawbacks of the standard two-stage RLHF pipeline are reward-model misspecification, overfitting, distribution shift, and additional data and engineering cost (Khorasani et al., 15 Aug 2025). DFA addresses these by directly optimizing the policy from preferences, optionally while also using numeric rewards.

2. Preference modeling and policy loss

For state-wise comparisons, the preference dataset is

$$D=\{(s_k,a_k^+,a_k^-)\}_{k=1}^K, \qquad a_k^+\succ a_k^-.$$

DFA defines the probability that the policy prefers $a^+$ to $a^-$ at state $s$ directly from policy probabilities: $$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$0 Equivalently,

$$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$1

where

$$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$2

This is the defining modeling move of DFA: the policy’s own log-probabilities serve as the score differences that explain observed preferences (Khorasani et al., 15 Aug 2025).

The state-wise preference loss is the negative log-likelihood

$$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$3

or, in logistic form,

$$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$4

DFA also supports trajectory-level comparisons. For

$$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$5

the policy’s trajectory likelihood is

$$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$6

and the corresponding preference probability is

$$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$7

The trajectory loss is then

$$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$8

The temperature $$\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\gamma,p_0\rangle,$$9 controls the sharpness of the induced comparisons. The paper reports that $\pi_\theta(a\mid s)$0 yields very soft, near-uniform comparisons, whereas $\pi_\theta(a\mid s)$1 approaches deterministic winner-take-all behavior (Khorasani et al., 15 Aug 2025).

3. Fusion of rewards and preferences in off-policy learning

DFA combines reward-driven critic learning with preference-driven actor updates. When numeric rewards are available, the critic is updated as in an off-policy RL method such as SAC; preferences are then used for policy improvement through $\pi_\theta(a\mid s)$2 or $\pi_\theta(a\mid s)$3. The source description is explicit that the method does not present a single combined scalar formula of the form

$\pi_\theta(a\mid s)$4

but the algorithmic fusion is exactly critic learning from rewards plus actor learning from preferences (Khorasani et al., 15 Aug 2025).

A notable feature is synthesized online preference generation from replay and $\pi_\theta(a\mid s)$5-values. For a batch of states $\pi_\theta(a\mid s)$6, the method takes the stored action $\pi_\theta(a\mid s)$7, finds a nearby state $\pi_\theta(a\mid s)$8 in the replay buffer and its action $\pi_\theta(a\mid s)$9, and compares the two using the critic: $\gamma\in(0,1)$0 and otherwise the pair is swapped. This yields synthetic comparisons

$\gamma\in(0,1)$1

with loss

$\gamma\in(0,1)$2

The paper describes three feedback modes within one framework: state-level human preferences, trajectory-level human preferences, and synthesized online preferences from replay and critic values (Khorasani et al., 15 Aug 2025). This makes DFA applicable when true preferences are available from annotators and also when reward signals are available but preferences are not.

A plausible implication is that DFA reframes the actor update as a comparison-learning problem without removing the role of reward-based value estimation. The critic remains reward-grounded, while the actor is trained through comparative evidence.

4. Theoretical relation to entropy-regularized control

The central theoretical result in DFA is stated under a Bradley–Terry preference model. The true preference probability is assumed to satisfy

$\gamma\in(0,1)$3

where $\gamma\in(0,1)$4 is the soft-optimal action-value function

$\gamma\in(0,1)$5

If a tabular policy is parameterized by $\gamma\in(0,1)$6 and one minimizes

$\gamma\in(0,1)$7

then the unique minimizer is

$\gamma\in(0,1)$8

This is a Gibbs policy over $\gamma\in(0,1)$9, and it matches the SAC policy-improvement solution

$\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$0

when

$\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$1

The paper therefore proves that minimizing DFA’s preference loss recovers the SAC policy under Bradley–Terry preferences (Khorasani et al., 15 Aug 2025).

The proof intuition given in the source has four parts: the preference loss is strictly convex in the full-support tabular log-policy variables; its gradient vanishes when modeled preference probabilities equal true Bradley–Terry probabilities; this equality implies

$\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$2

and normalization yields the Gibbs form. The Hessian is described as a weighted graph Laplacian over actions, positive definite on the feasible tangent space, which establishes uniqueness (Khorasani et al., 15 Aug 2025).

This result situates DFA as a preference-driven view of SAC-style policy improvement. When comparisons are generated according to Bradley–Terry from the soft-optimal $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$3, DFA and SAC agree at the policy level.

5. Empirical evaluation

The empirical study reported for DFA covers two distinct regimes: continuous-control experiments with synthetic preferences and a stochastic GridWorld with human-like preference comparisons (Khorasani et al., 15 Aug 2025).

In six MuJoCo control tasks—Walker2d, Hopper, Swimmer, Humanoid, MountainCarContinuous, and Pendulum—the setup uses the same replay buffer as SAC, 10 million environment interactions, 5 random seeds, batch size 256, and synthetic preference pairs generated from rewards or $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$4-values at each gradient step. DFA is reported to match or exceed SAC on Walker2d, Hopper, Swimmer, and Humanoid, to solve MountainCarContinuous where SAC struggled under the tested hyperparameters, and to exhibit smoother training curves, which the authors attribute to the denoising effect of synthesized comparisons (Khorasani et al., 15 Aug 2025).

In the stochastic GridWorld experiments, the main setting is $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$5, with larger variants in the appendix, stochastic action reversal with probability $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$6, random cell rewards, horizon $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$7, and a tabular softmax policy. The baselines are RM+PPO, ZPG, OnlineDPO, and Oracle-PPO. DFA is reported to outperform RM+PPO, approach Oracle-PPO, and outperform or be more stable than ZPG and OnlineDPO in the reported setting (Khorasani et al., 15 Aug 2025).

The paper also reports hyperparameter sensitivity for $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$8. If $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1}).$9 is too large, the learning signal becomes weak; if it is too small, the policy becomes overly stochastic. The best GridWorld run used a small but nonzero $Q$0, around $Q$1 (Khorasani et al., 15 Aug 2025).

Setting	Reported outcome
MuJoCo synthetic preferences	DFA matches or exceeds SAC on Walker2d, Hopper, Swimmer, and Humanoid
MountainCarContinuous	DFA solves the task where SAC struggled under the tested hyperparameters
Stochastic GridWorld	DFA outperforms RM+PPO and approaches Oracle-PPO

The reported empirical interpretation is not that DFA improves predictive modeling or value estimation in isolation, but that comparison-based policy improvement can be competitive with or superior to reward-only actor updates while remaining stable in off-policy training.

6. Scope, distinctions, and related formulations

The name Dual-Feedback Actor refers specifically to the 2025 reinforcement-learning method that fuses rewards and pairwise preferences in policy improvement (Khorasani et al., 15 Aug 2025). It is distinct from several nearby formulations in the literature.

First, DFA should not be conflated with Direct Feedback Alignment, for which the same abbreviation is widely used in deep learning. In that line of work, DFA denotes a non-backpropagation training rule in which fixed random matrices transmit output error directly to hidden layers, as in "Training DNNs in O(1) memory with MEM-DFA using Random Matrices" (Chu et al., 2020), "Differentially Private Deep Learning with Direct Feedback Alignment" (Lee et al., 2020), and "DFA-GNN: Forward Learning of Graph Neural Networks by Direct Feedback Alignment" (Zhao et al., 2024). Those methods address gradient transport, memory, privacy, or graph learning; they are unrelated to preference-based reinforcement learning in the sense used by Dual-Feedback Actor.

Second, earlier actor-critic and interactive-RL papers contain dual-channel or dual-critic ideas that are related in motivation but different in mechanism. "Actor-Critic Reinforcement Learning with Simultaneous Human Control and Feedback" (Mathewson et al., 2017) studies concurrent human control and reward-shaping feedback for an actor-critic learner. The source description explicitly characterizes that work as a precursor and conceptual ancestor of DFA-style interactive learning, but not as a Dual-Feedback Actor algorithm. Its dual aspect is at the human-interface and reward-shaping level, not an explicit policy update from pairwise preferences (Mathewson et al., 2017).

Third, dual-critic approaches are also distinct. "Actor-Dual-Critic Dynamics for Zero-sum and Identical-Interest Stochastic Games" (Donmez et al., 31 Jan 2026) uses an actor with a fast critic and a slow critic in a three-timescale, payoff-based stochastic-game framework. "Boosting the Actor with Dual Critic" (Dai et al., 2017) derives Dual-AC from the Lagrangian dual form of the Bellman optimality equation and interprets critic feedback as Bellman residual or constraint-violation information. These are dual-critic or dual-objective actor-critic methods, not reward-plus-preference fusion in the sense of DFA (Donmez et al., 31 Jan 2026, Dai et al., 2017).

The most important practical distinction is therefore between three notions that share overlapping vocabulary: direct feedback alignment in neural-network training, dual-critic actor-critic schemes in RL, and Dual-Feedback Actor as a preference-based off-policy RL algorithm. In the 2025 usage, the “dual feedback” refers to the coexistence of scalar rewards and pairwise preferences within a single policy-learning framework (Khorasani et al., 15 Aug 2025).

A common misconception is that DFA is simply reward modeling without an explicit reward network. The defining claim of the method is stronger: preferences are modeled directly through policy log-probabilities, and the reward-modeling stage is omitted entirely (Khorasani et al., 15 Aug 2025). Another common misconception is that DFA replaces critics. It does not; rewards still update $Q$2-networks, and the actor is trained from preference losses rather than from a separate learned reward model.

Within preference-based reinforcement learning, DFA’s reported advantages are a single preference-based update rule, dual supervision, off-policy compatibility, entropy-preserving exploration, a theoretical bridge to SAC, and empirically smoother learning curves. Its reported limitations are the Bradley–Terry assumption, possible synthetic preference noise when the critic is inaccurate, nearest-state search cost in replay, sensitivity to $Q$3, and limited real-human evaluation, which is semi-synthetic rather than large-scale human-in-the-loop deployment (Khorasani et al., 15 Aug 2025).