FMR: Feedback Manipulation Regularization

• Feedback Manipulation Regularization (FMR) is a framework that integrates evaluative feedback into offline imitation learning to directly reshape policy distributions for human-aligned behavior.
• It employs per-action temperature adjustments and a reverse-KL regularizer to modulate policy entropy based on feedback polarity, ensuring effective alignment from mixed-quality demonstrations.
• FMR operates in a single offline training stage without environment interaction, achieving significant misalignment reductions in safety-critical benchmarks while preserving performance.

Feedback Manipulation Regularization (FMR) is a regularization framework for offline imitation learning that uses offline evaluative feedback to directly reshape an imitation-learned policy toward human-aligned behavior in fully sequential Markov decision processes. It is formulated for settings with no environment interaction and no true reward during training, where the available supervision consists of mixed-quality demonstrations $D^U = D^E \cup D^I$ and a human feedback function $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$ that labels state–action pairs as good, bad, or neutral. Its defining mechanism is a feedback-tempered surrogate policy $\pi_\tau$, constructed by modifying action probabilities with per-action temperatures, together with a reverse-KL-based regularizer that encourages the learned policy $\pi_\theta$ to match that surrogate in a single offline training stage (Poole et al., 8 Jul 2026).

1. Problem formulation and alignment objective

FMR is defined in a modified MDP with feedback,

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$

but its training regime is explicitly offline imitation: the reward $r$ is not used, and learning proceeds only from demonstrations and evaluative feedback. The demonstration corpus contains aligned expert trajectories $D^E$, imperfect trajectories $D^I$, and their union $D^U = D^E \cup D^I$. The target of alignment is a latent human-aligned policy $\pi_h$, while the learned policy is $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$0. A standard coverage assumption is imposed: for any $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$1, if $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$2 then $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$3. This ensures that aligned actions remain in the support of the learned policy (Poole et al., 8 Jul 2026).

Within this formulation, feedback is absolute and local rather than comparative. The signal $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$4 designates a specific state–action pair as desirable, undesirable, or neutral. This differs from preference-learning setups that only express relative judgments between trajectories or segments. FMR therefore treats demonstrations as coverage over feasible behavior and feedback as local discrimination over which parts of that behavior should be amplified or suppressed. The result is a single-stage offline alignment objective in which demonstrations and feedback are optimized jointly rather than through separate pretraining, reward-modeling, and reinforcement-learning phases (Poole et al., 8 Jul 2026).

A central misconception addressed by this formulation is that evaluative feedback must be converted into reward. FMR does not create or modify rewards; instead, it reshapes the action distribution directly. This places the method closer to direct policy regularization than to offline RL with learned reward surrogates.

2. Temperature-based manipulation of the policy distribution

The core device in FMR is a per-action temperature $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$5, abbreviated $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$6, derived from evaluative feedback. Positive feedback corresponds to $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$7 in effect, encouraging the action by sharpening the distribution; negative feedback corresponds to $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$8, discouraging the action by flattening the distribution; neutral feedback leaves the action unchanged with $h : \mathcal{S}\times\mathcal{A} \to \{-1,0,1\}$9. The discrete-action mapping is

$\pi_\tau$0

with hyperparameter $\pi_\tau$1. For negative feedback on the chosen action, the selected action receives higher temperature and is discouraged. For positive feedback, the formulation raises the temperature of the other actions instead, which is equivalent in a renormalized sense to lowering the temperature of the selected action while avoiding numerical issues associated with $\pi_\tau$2 (Poole et al., 8 Jul 2026).

Given $\pi_\tau$3 and $\pi_\tau$4, FMR defines a feedback-tempered surrogate policy

$\pi_\tau$5

This construction operates directly in probability space rather than on logits. Actions with lower temperature are up-weighted, and actions with higher temperature are down-weighted. Conceptually, $\pi_\tau$6 serves as an approximation to the latent aligned policy $\pi_\tau$7, obtained by correcting the current policy with feedback (Poole et al., 8 Jul 2026).

Alignment is then enforced through the reverse KL

$\pi_\tau$8

For a fixed state $\pi_\tau$9, the decomposition

$\pi_\theta$0

yields the regularization term

$\pi_\theta$1

For a base imitation-learning objective $\pi_\theta$2, FMR forms

$\pi_\theta$3

with $\pi_\theta$4 controlling alignment strength (Poole et al., 8 Jul 2026).

This formulation admits an entropy interpretation. If $\pi_\theta$5, then $\pi_\theta$6, so the regularizer reduces to negative entropy. More generally, FMR can be interpreted as human-controlled entropy regularization: negative feedback increases entropy around undesirable actions, whereas positive feedback decreases entropy around desirable actions (Poole et al., 8 Jul 2026).

3. Integration with imitation learning and offline data

FMR is algorithm-agnostic. It requires only a policy that induces a distribution $\pi_\theta$7, and can therefore be attached to Behavior Cloning, IQ-Learn, DemoDICE, ReCOIL, and related imitation-learning or Learning from Noisy Demonstrations methods. In actor-only settings, the regularizer is applied directly to the policy. In critic-only settings, Q-values are converted into a distribution, for example by softmax over actions, and FMR is applied to that distribution. In actor–critic settings, it can be applied to both actor and critic distributions (Poole et al., 8 Jul 2026).

For Behavior Cloning, the base loss is

$\pi_\theta$8

and the combined objective is

$\pi_\theta$9

For DemoDICE, ReCOIL, and IQ-Learn, the same additive regularization principle is retained, even though the underlying base losses differ in whether they are framed as occupancy matching, actor–critic learning, or soft-Q optimization (Poole et al., 8 Jul 2026).

The interaction between demonstrations and feedback is structurally important. Demonstrations determine which states and actions are available to learn from, including aligned fragments within otherwise imperfect trajectories. Feedback then manipulates the policy’s uncertainty locally. For state–action pairs in $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$0, negative labels increase temperature and reallocate probability mass away from the labeled action, whereas positive labels reduce relative temperature and concentrate probability on the labeled action. FMR therefore re-weights the policy’s action distribution rather than reweighting the dataset itself. It does not edit labels or trajectories; its effect is realized through gradient updates to the policy (Poole et al., 8 Jul 2026).

The data-collection protocol also matters. In navigation environments, feedback is provided by humans through a replay tool and aligned to states with a 600 ms credit assignment window. In velocity-limited locomotion, the feedback is an oracle proxy derived from cost signals and is mostly negative only. A plausible implication is that the formalism is agnostic to the origin of the evaluative signal so long as it can be represented as offline labels over state–action pairs.

4. Experimental evaluation in Safety Gymnasium

FMR is evaluated in adapted Safety Gymnasium environments designed to separate aptitude from alignment. Rewards are not used for training. Alignment is measured by a misalignment score,

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$1

where $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$2 is the environment cost, and aptitude is measured independently: success rate for navigation and normalized return relative to $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$3 for velocity tasks (Poole et al., 8 Jul 2026).

The navigation benchmark uses a red spherical agent with lidar observations that must reach a green goal while avoiding unobservable hazards, including floor hazards and hazard walls. Two aligned policies are defined. PathM follows a diagonal corridor through the center and has high overlap with imperfect demonstrations. PathBB follows the right boundary and then the bottom boundary into the goal and has low overlap. The locomotion benchmark discretizes the action space of Swimmer, Hopper, and Walker2D for demonstration collection and defines alignment through maximum $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$4-velocity constraints: SlowSwim with $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$5, SlowHop with $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$6, and SlowWalk with $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$7 (Poole et al., 8 Jul 2026).

Across Behavior Cloning, IQ-Learn, DemoDICE, and ReCOIL, baseline imitation-learning methods exhibit substantial misalignment, especially when expert data are scarce and imperfect demonstrations dominate. Adding FMR consistently reduces misalignment, often while preserving or improving aptitude. On PathM, baseline misalignment is approximately $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$8–$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, p, r, h, \gamma),$$9 for BC and IQL and about $r$0 for ReCOIL at the 10–50 ratio, whereas ReCOIL+FMR reduces misalignment to around $r$1–$r$2, corresponding to up to $r$3 reduction. On PathBB, where overlap is low, ReCOIL+FMR achieves up to $r$4 reduction in misalignment while improving success rates. On SlowSwim, FMR variants retain return near $r$5 while cutting misalignment by up to $r$6, and ReCOIL+FMR produces misalignment effectively near zero at all ratios. On SlowHop, FMR reduces misalignment by roughly $r$7 across algorithms and data ratios while maintaining competitive returns. On SlowWalk, with positive feedback on $r$8 and $r$9, ReCOIL+FMR attains the best return–alignment tradeoff and up to $D^E$0 reduction in misalignment (Poole et al., 8 Jul 2026).

The ablation results clarify how the method uses feedback. Treating feedback as reward through DVL often preserves high return but leaves misalignment high or worsens it in low-expert settings. CPL, which uses feedback-summed segment scores in a preference-learning style, performs reasonably on some tasks but degrades in more complex or high-dimensional action spaces such as SlowWalk and still underperforms ReCOIL+FMR in alignment. Experiments with only negative feedback show little degradation on most tasks, indicating that FMR can operate effectively with negative-only supervision. Feedback sparsity experiments, in which only $D^E$1 or $D^E$2 of trajectories carry feedback, show graceful degradation and usually remain superior to baselines without FMR, especially when the retained feedback is targeted rather than randomly thinned (Poole et al., 8 Jul 2026).

5. Relation to adjacent methods and conceptual lineage

FMR occupies a distinct position relative to standard RLHF, learning-from-preferences, and evaluative-feedback RL methods. In prevailing RLHF pipelines for LLMs, demonstrations and human judgments are typically separated into multiple stages: supervised learning on demonstrations, reward-model fitting from pairwise preferences, and RL fine-tuning against the learned reward. Those methods are usually framed as contextual bandits rather than fully sequential MDPs. By contrast, FMR performs single-stage, fully offline integration of demonstrations and feedback in a sequential setting, and the feedback signal is evaluative and absolute rather than pairwise and relative (Poole et al., 8 Jul 2026).

This distinction has methodological consequences. Pairwise preference methods only determine which of two segments is better; they do not establish that either segment is aligned. FMR instead provides local, state–action-level control over where probability mass should contract or disperse. Online evaluative-feedback methods such as TAMER and COACH use human feedback as pseudo-reward with credit assignment, but they require online interaction and do not formulate feedback as a direct distribution-shaping regularizer in offline imitation learning. FMR therefore should not be understood as offline reward learning in disguise; its objective acts directly on the policy distribution (Poole et al., 8 Jul 2026).

A related but terminologically distinct use of feedback manipulation appears in recommender systems. “DPR: An Algorithm Mitigate Bias Accumulation in Recommendation feedback loops” does not use the label FMR, but it describes an analogous strategy in which training feedback is explicitly manipulated, re-weighted, or regularized so that the model does not inherit and amplify exposure bias under feedback loops. In that setting, Dynamic Personalized Ranking acts as a feedback re-weighting or regularization scheme, and Universal Anti-False Negative acts as an additional regularization term targeted at false negatives in implicit feedback (Xu et al., 2023). This suggests a broader abstraction of “feedback manipulation regularization” as direct correction of feedback-distorted learning signals, even though the formal objects differ sharply between recommendation and sequential policy learning.

6. Practical considerations, limitations, and prospective extensions

Several implementation details are emphasized. A default choice of $D^E$3 works well across tasks, while $D^E$4 is a good default regularization strength; $D^E$5 is reported to help in Walker2D, where the action space is large. Larger $D^E$6 can make temperatures too extreme, and very large $D^E$7 can over-regularize and hurt aptitude. In navigation, a 600 ms credit assignment window is used by default and has only modest sensitivity. When expert demonstrations are scarce, oversampling $D^E$8 to roughly match the size of $D^E$9 stabilizes training across algorithms. In actor–critic settings such as DemoDICE+FMR and ReCOIL+FMR, applying FMR to both actor and critic distributions is reported as beneficial (Poole et al., 8 Jul 2026).

The main limitation is dependence on overlap and feedback coverage. FMR is most effective when imperfect demonstrations contain aligned sub-trajectories that feedback can identify and amplify. When $D^I$0 is almost entirely misaligned or irrelevant, the method has little aligned behavior to exploit. Manual evaluative feedback also remains expensive, even though targeted and partial annotation can still be effective. Experimental validation is currently limited to discrete action spaces; a generalized continuous-action variant,

$D^I$1

is proposed and behaves similarly in discrete environments, but high-dimensional continuous control remains unvalidated. The framework also lacks formal convergence or regret bounds, and its notion of alignment is environment-specific, being defined through safety costs in Safety Gymnasium (Poole et al., 8 Jul 2026).

The broader implications are nevertheless clear within the stated scope. FMR demonstrates that offline data and evaluative feedback can substantially align policies without online RL fine-tuning, including in limited-data regimes with scarce aligned demonstrations and noisy mixed-quality data. Potential extensions identified for LLMs, robotics, autonomous driving, and healthcare decision support all retain the same conceptual move: using feedback as a direct distribution-shaping regularizer rather than as a learned reward. This suggests that FMR is best understood as a general recipe for human-guided local entropy modulation in sequential decision-making, with current evidence concentrated in safety-oriented offline imitation learning (Poole et al., 8 Jul 2026).