Argumentative Reward Learning Insights

• Argumentative Reward Learning is a neuro-symbolic framework that applies preference-based argumentation to enhance RLHF with richer training signals and improved interpretability.
• It systematically constructs abstract argumentation frameworks from roll-out trajectories, reducing human feedback needs by inferring additional pseudo-labels.
• Experimental results in maze-solving benchmarks demonstrate that ARL achieves superior policy performance and robust reward model generalisation compared to standard RLHF.

Argumentative Reward Learning (ARL) is a neuro-symbolic framework that augments reinforcement learning from human feedback (RLHF) with preference-based argumentation. ARL addresses significant limitations of conventional RLHF pipelines—namely data inefficiency, poor generalisation of reward models, and lack of explainability—by operationalizing trajectories as arguments within an abstract argumentation framework (AAF). Via argumentation semantics, ARL non-monotonically generalises sparse human-labelled preferences, producing a richer, more coherent set of training signals for reward learning. This methodological innovation reduces user effort, enhances model robustness, and yields more interpretable reward functions (Ward et al., 2022).

1. Background and Problem Motivation

In standard RLHF, a learning agent collects roll-out trajectories under its current policy and queries a human to label numerous trajectory pairs with binary preferences (“≻”). A neural reward model is then trained to predict these comparisons, serving as the reward signal for reinforcement learning updates. Two key issues arise: neural models trained in this way tend to overfit correlations present in the labelled data—often attending to spurious cues—failing to generalise to novel states. Further, human feedback collection is costly and often fails to scale, as hundreds or thousands of pairwise annotations are typically required to train robust reward models.

ARL aims to mitigate these deficiencies. It re-interprets trajectories as arguments and systematically embeds preference-based argumentation into the RLHF learning loop. Within this framework, trajectories that are “dissimilar” attack each other; the human annotator resolves some of these attacks through explicit feedback. Argumentation semantics then generalise these base preferences—by, for example, considering conflict-freeness, admissibility, and extension ordering—to infer a much broader set of trajectory rankings, thus creating significantly more training data with fewer queries.

2. Formal Framework and Key Definitions

ARL leverages the machinery of abstract argumentation frameworks (AAF) and strict partial orderings to generalise human preference information.

• Abstract Argumentation Framework (AAF): Defined as a pair $AF = (\mathit{Args},\;\mathrm{attacks})$, where $\mathit{Args}$ is a finite set of arguments (corresponding to trajectories $\tau$), and $$\mathrm{attacks} \subseteq \mathit{Args} \times \mathit{Args}$$ specifies a binary attack relation.
• Preference-based AAF (PAF): Builds upon AAF by introducing a strict partial order $\succ \subseteq \mathit{Args} \times \mathit{Args}$ (transitive and asymmetric), representing human preferences between arguments ($A \succ B$ denotes "A is preferred to B"). Any attack $(B,A)$ is dropped if $A \succ B$ when reducing to the underlying AAF.
• Preferred Semantics: Used to extract maximally admissible, conflict-free sets of arguments (preferred extensions). Extensions are then ordered (for instance, by aggregated returns or by human label tallies).

The reward model $\hat r_\theta:S\times A \to \mathbb{R}$ is neural and parameterised by $\theta$, with return $$\hat R(\tau)=\sum_{(s,a)\in\tau} \hat r_\theta(s,a)$$. The likelihood that the model assigns $\tau_1 \succ \tau_2$ is

$$\Pr_\theta(\tau_1\succ\tau_2) = \frac{\exp(\hat R(\tau_1))}{\exp(\hat R(\tau_1))+\exp(\hat R(\tau_2))}.$$

Given an expanded set of generalised preferences $\mathcal D$, the binary cross-entropy loss is minimised: $$\mathcal L(\theta) = -\sum_{(\tau_1,\tau_2)\in\mathcal D} \Big[\mathbf 1_{\tau_1\succ\tau_2}\log\Pr_\theta(\tau_1\succ\tau_2) + \mathbf 1_{\tau_2\succ\tau_1}\log\Pr_\theta(\tau_2\succ\tau_1)\Big].$$ Stochastic gradient descent is used for parameter updates.

3. End-to-End Algorithmic Workflow

Each ARL iteration proceeds through the following stages:

1. Trajectory Collection: Sample a batch of trajectories $T=\{\tau_1, \ldots, \tau_N\}$ by rolling out the current policy $\pi$ in the environment MDP.
2. AAF Construction: Define arguments as $T$. Specify attacks: $(\tau_i, \tau_j) \in \mathrm{attacks}$ iff $\max_t \|s_i(t) - s_j(t)\| > \delta$, thereby only connecting trajectories that are sufficiently dissimilar.
3. Human Queries: Select a small subset $Q \subset \mathrm{attacks}$ (e.g., 100 pairs). Solicit human preference between trajectories in $Q$ and record the labelled subset $H$.
4. Generalisation via PAF:
   • Form the PAF by injecting $H$ into $(\mathit{Args},\mathrm{attacks})$.
   • Reduce to $AAF_H$ by dropping attacks invalidated by known preferences.
   • Compute all preferred extensions $\{E_1,\ldots,E_k\}$ of $AAF_H$.
   • Order extensions, e.g., using sum of returns or count of human-favoured arguments.
   • Declare $\tau_1\succ\tau_2$ whenever $\tau_1$ resides in an extension ranked stronger than $\tau_2$, generating the generalised label set $\mathcal D$.
5. Reward-Model Update: Minimise the cross-entropy loss over $\mathcal D$ via gradient descent.
6. Policy Update: Treat $r\leftarrow\hat r_\theta$, and update the policy $\pi$ using any standard RL method (e.g., DQN).

This cycle repeats until convergence or computational budget is exhausted.

4. Experimental Results and Empirical Findings

ARL has been validated using a continuous 2D maze-solving benchmark, where the agent navigates in $[0,1]^2$ toward a goal, with randomly generated walls and four-way motion. The true reward function is hand-engineered but withheld from the learner.

• Two feedback regimes were assessed: synthetic preference labels (using access to the hidden $r$) and real human-labeled comparisons (100 or 200 pairs).
• The baseline is Christiano et al.'s RLHF without argumentation; ARL uses the same RL pipeline, with added argumentation-based preference generalisation.

Key performance metrics:

Metric	Baseline (100 human)	ARL (100 human)	Baseline (200 human)	ARL (200 human)	Synthetic Baseline	Synthetic ARL
Mean Preference-Prediction Accuracy	0.798	0.847	0.896	0.893	0.895	0.954
Normalised Distance to Goal (policy)	0.97	0.63	(not stated)	(converges faster, more robustly)	-	-
Pseudo-labels Synthesised	100	~4,115	-	-	-	-

Reward-heatmaps for ARL display spatial coherence (higher values near the goal and consistent action recommendations), while baseline models often overfit to initial regions without generalising effectively.

5. Analysis: Strengths and Limitations

ARL demonstrates several strengths:

• Data efficiency: By generalising each human label into many coherent pseudo-labels, ARL achieves robust learning with fewer queries.
• Generalisation: Argumentation semantics enable explainable, non-monotonic inference, reducing vulnerability to spurious correlations.
• User burden: Query count is substantially reduced, supported by the use of binary insertion sort for extension ordering.

Limitations include:

• Heuristic extension ordering: Current approaches rely on ad hoc measures, such as aggregated returns or counts of human labels, which may limit generalisation quality.
• Computational cost: Enumerating all preferred extensions is expensive when the number of trajectories or attacks is large.
• Iterative ARL performance: In early empirical results, the iterated ARL loop sometimes underperforms due to stochastic RL updates and evolving policy distributions affecting argumentation reliability.

6. Potential Extensions and Future Research Directions

Anticipated directions for advancing ARL include:

• Richer Argumentation Semantics: Exploration of alternative semantics (e.g., grounded or stable) and more expressive frameworks (such as bipolar argumentation) to capture nuanced preference structures.
• Active Preference Elicitation: Incorporation of uncertainty measures over arguments to select maximally informative trajectory queries.
• Dialogue Protocols: Introducing multi-step clarification with humans to refine argument strengths iteratively.
• Scalability: Development of approximate or sampling-based enumeration algorithms to handle large trajectory and attack sets efficiently.

A plausible implication is that as preference-based argumentation for RLHF becomes scalable and expressive, neuro-symbolic models like ARL will offer more robust, interpretable, and data-efficient solutions for learning from limited human feedback (Ward et al., 2022).