Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning

Abstract: Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98\% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations.

• The paper introduces FMR, a novel regularization strategy that leverages fine-grained evaluative feedback to suppress misaligned actions in offline imitation learning.
• It demonstrates FMR's effectiveness in Safety Gymnasium environments, achieving up to 98% reduction in misalignment under imbalanced data regimes.
• The method employs adaptive temperature scaling, enabling seamless integration with diverse IL pipelines and enhancing robustness in noisy demonstration settings.

Feedback Manipulation Regularization for Offline Agent Alignment in Imitation Learning

Overview

The paper “Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning” (2607.07859) introduces Feedback Manipulation Regularization (FMR), an algorithm-agnostic regularization strategy for imitation learning. The method directly leverages fine-grained evaluative feedback to realign policies, enabling robust agent alignment—even in heavily imbalanced or noisy offline demonstration datasets. The study provides both theoretical analysis and systematic empirical evaluation in Safety Gymnasium environments, establishing FMR’s advantages over prior techniques and analyzing its integration into existing imitation learning pipelines.

Motivation and Theoretical Framework

Imitation learning (IL) in realistic settings frequently operates under data scarcity and heterogeneity, with a limited pool of expert demonstrations ($D^E$) and a potentially much larger, less reliable set of imperfect demonstrations ($D^I$). Classical IL approaches, such as Behavioral Cloning (BC) and recent robust Learning-from-Noisy-Demonstrations (LfND) algorithms like DemoDICE and ReCOIL, effectively model aptitude but are empirically shown to produce high rates of behavioral misalignment with human intent, especially when $|D^E| \ll |D^I|$.

Existing preference-based offline RLHF methods, originally designed for language modeling, use pairwise human preferences as a training signal. However, such relative feedback is too coarse-grained for fine-grained behavioral alignment in sequential decision-making tasks, and fails to precisely distinguish aligned from unsafe policy executions.

FMR instead utilizes evaluative feedback $h(s, a)$—a signal indicating positive, neutral, or negative assessment at the state-action pair level. This enables direct suppression of misaligned behaviors and reinforcement of desired sub-trajectories by reparametrizing policy probabilities through adaptive temperature scaling. The induced “feedback-tempered” policy is defined as

$$\pi_\tau(a_j|s) = \frac{\pi_\theta(a_j | s) \tau_j^{-1}}{Z_\theta(s)},$$

where $\tau_j$ is a per-action temperature parameter dependent on the feedback, and $Z_\theta(s)$ is a normalization term. The learning objective is formulated as reverse KL divergence minimization between the original and feedback-modulated policies.

This yields a model-agnostic regularizer that can be introduced into any likelihood-based IL objective, enabling single-stage, fully offline agent alignment.

Experimental Design

Environments and Tasks

Experiments use adapted Safety Gymnasium [ji_safety-gymnasium_2023] environments with navigation and velocity-constrained locomotion tasks. The navigation environment includes two reference policies (PathM and PathBB), distinguished by how their aligned trajectories overlap with imperfect data. Locomotion environments (e.g., Swimmer, Hopper, Walker2D) impose velocity constraints that induce low-return, highly specific aligned gaits distinct from optimal control behaviors. Costs serve as an evaluation-only metric to quantify misalignment (i.e., frequency of unsafe actions); no cost information is available during training.

Methodology and Baselines

Human-collected or proxy expert/imperfect demonstrations are supplemented with per-step evaluative feedback for $D^I$ (and optionally $D^E$ in large action spaces), which is mapped to prior trajectory points via credit assignment. The study benchmarks FMR-augmented variants of BC, IQL, DemoDICE, and ReCOIL under varying $|D^E|:|D^I|$ data regimes.

Comparisons are performed against strong alternatives, including:

• DVL: interprets feedback as a replacement for reward in offline RL,
• CPL: utilizes summation of evaluative feedback as pairwise preference scores,
• Purely demonstration-driven (expert-only) IL baselines.

Results are reported for task completion (success rate/normalized return) and alignment (cost-based misalignment).

Empirical Results

Alignment Gains under Data Imbalance

Figure 1: FMR achieves large and consistent reductions in misalignment over baselines on PathM with 10–50 (1:5) expert:imperfect demonstration ratio.

Across both navigation and velocity-constrained settings, FMR achieves up to 98% reduction in misalignment with little to no degradation in task success or normalized return, even when $D^I$0 constitutes as little as 1/6 of the training data. The improvement is particularly marked when imperfect demonstrations include aligned sub-trajectories (i.e., data overlap).

Figure 2: FMR consistently decreases misalignment and bolsters success across data ratios and IL baselines for both PathM and PathBB.

Robustness to Data Regimes and Task Structure

• High-overlap tasks (e.g., PathM, SlowSwim): FMR leverages feedback to upweight relevant segments of $D^I$1, ameliorating the scarcity of $D^I$2.
• Low-overlap tasks (e.g., PathBB, SlowHop): FMR outperforms baselines but alignment performance deteriorates as $D^I$3 shrinks, reflecting limits of feedback in the near-absence of supportive imperfect demonstrations.

Figure 3: FMR outperforms DVL (reward substitution) and CPL (feedback-to-preference) on misalignment and task success across data regimes.

Comparative Analysis with RLHF and Preference Optimization

FMR performs strictly better or on par with DVL and CPL, both in alignment (misalignment reduction) and aptitude (return/success rate), across all data splits and environments. DVL degrades in low-data regimes, and CPL’s relative-signal forms are shown insufficient for enforcement of fine-grained policy correction. FMR’s absolute evaluative signal anchors the policy closer to true human preference.

Analysis and Theoretical Implications

• Negative Feedback Dominance: Experiments demonstrate that negative feedback alone suffices for strong policy realignment. Positive feedback for aligned actions contributes only marginal improvements, except in high-dimensional action scenarios.
• Noisy/Irrelevant Demonstration Management: FMR enables selective utilization of noisy or imperfect demonstrations, provided they contain structurally relevant behaviors; otherwise, performance degrades gracefully to the “reliable demonstration only” regime.
• Feedback Scalability: Reducing feedback coverage leads to a gradual decline in performance, highlighting the necessity of strategically targeted or generalizable feedback mechanisms in practical systems.
• Extension to Continuous Actions: Generalizing the feedback-adaptive temperature formulation allows seamless application of FMR beyond discrete settings, enabling alignment in continuous-control domains.

Implications and Future Directions

Practically, FMR provides a drop-in, model-agnostic regularization strategy that robustly improves agent alignment without requiring environment interaction, on-policy learning, or staged pipelines. This is theoretically relevant for constructing offline RLHF methods that need fine-grained, absolute, and directly human-controllable correction channels beyond what preference modeling can deliver.

Future work will involve scalable generalization of fine-grained feedback—using learned feedback-propagation heuristics or leveraging behaviorally similar state-action analogs—to further minimize human annotation overhead. Extending FMR to continuous action spaces, and formalizing its relationship to entropy regularization and reward modeling frameworks (e.g., inverse RL), are essential next steps for standardized RLHF pipelines.

Conclusion

Feedback Manipulation Regularization provides a principled, highly effective mechanism for enforcing alignment in offline imitation learning by directly modulating policy entropy in response to targeted evaluative feedback. Through rigorous benchmarking, the method surpasses traditional and preference-based pipelines, offering strong improvements in alignment under limited and partially noisy expert supervision, and setting a new standard for single-stage agent alignment workflows in sequential RL domains.