Reward Interfaces in RL

• Reward interfaces are mechanisms that translate human feedback and uncertain signals into precise learning rewards for reinforcement learning agents.
• They employ formal abstractions, interactive query systems, and uncertainty-aware models to map feedback to trajectory utilities.
• They integrate programmatic structures and safety measures to repair, audit, and mitigate reward hacking, ensuring robust policy alignment.

Reward interfaces are mechanisms that mediate between human-preferred outcomes and the objective functions an RL agent actually optimizes. They translate uncertain, noisy, incomplete, or heterogeneous supervisory signals into learning signals that an optimizer can use, and in RLHF/RLAIF they also define what feedback is requested, which evaluator or reward model is queried, and how partial signals are turned into scalar rewards, preference likelihoods, or structured supervision (Singha, 29 Apr 2026, Wu et al., 3 Oct 2025). Across recent work, the term spans interactive reward specification, preference-query systems, semantic and programmatic reward-function interfaces, uncertainty-aware modulation, generative reward models, routing over multiple reward models, and mechanistic tools for inspecting reward computation itself (Mindermann et al., 2018, Wang et al., 27 May 2026, Nadaf, 28 Apr 2026).

1. Formal abstractions and mathematical role

A recurring formalization treats the reward interface as an explicit mapping from feedback to utility-bearing consequences. In reward-rational implicit choice, an interface $I$ induces a choice set $Y_I$ and a grounding function $g_I$ that maps a feedback outcome to trajectories or distributions over trajectories. With trajectory return $R_\theta(\tau)$, the interface-specific utility is

$$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$

and the observation model is

$$P(y \mid \theta, I) = \frac{\exp(\beta u_I(y; \theta))}{Z_I(\theta)}.$$

This gives a single likelihood template for demonstrations, comparisons, corrections, language, shutdown events, proxy rewards, and even the choice of feedback type itself (Jeon et al., 2020).

A second formalization makes the interface itself part of the induced task. LIMEN defines an RL task interface as a pair of executable programs

$$I = (\phi : S \to O,\; r : S \times A \times S \to \mathbb{R}),$$

where $\phi$ maps raw simulator state to observations and $r$ computes per-step scalar rewards from transitions. Policy performance is then evaluated under the induced MDP, while outer-loop search optimizes interfaces against a trajectory-level success metric rather than internal reward return (Jaswal et al., 5 May 2026). This makes reward interface design a bilevel problem: the interface shapes the learning problem, and policy optimization responds to that induced problem.

Reward Machines provide a related but more structured abstraction. They represent reward functions as finite-state machines over high-level events, either as full Moore-style reward machines

$R_{P,S,A} = (U, u_0, F, \delta_u, \delta_r)$

or simple Mealy-style machines

$Y_I$0

By augmenting the environment state with the reward-machine state, they convert non-Markovian rewards into Markovian rewards over a cross-product MDP, exposing loops, sequences, conditionals, and regular-language reward structure to the agent (Icarte et al., 2020).

A third abstraction treats the interface as an internal learning signal distinct from the environment reward. SORS learns a dense per-step score $Y_I$1 and uses

$Y_I$2

as the learner’s internal reward. Its theoretical guarantee is phrased in terms of total-order equivalence over trajectory returns: in deterministic MDPs, if the learned dense reward preserves the total order induced by the sparse reward, optimal policies are unchanged (Memarian et al., 2021). This places reward interfaces at the boundary between semantics and optimization: they need not equal the external reward, but they must preserve the right decision structure.

2. Interactive specification and preference-query interfaces

Active Inverse Reward Design reframes reward design as an interactive query process over reward functions. Instead of observing only a final proxy reward, the interface presents the designer with small, actively selected query sets $Y_I$3, and the designer chooses the best reward for the training environment. The likelihood of selecting $Y_I$4 is

$Y_I$5

and the posterior update is

$Y_I$6

AIRD selects queries by maximizing mutual information, and its key interface property is that it elicits preferences about suboptimal behaviors, which vanilla IRD cannot observe when it only sees the final proxy reward (Mindermann et al., 2018).

The same design logic appears in interactive demonstration interfaces. An extensible interface for agent design constructs a library of primitive policies, previews their short rollouts, and lets the user switch between them at fixed intervals to demonstrate more complex behaviors. Those demonstrations induce pairwise preferences between chosen and non-chosen rollouts, from which a reward model is learned. In Lunar Lander, about 400 labeled examples per goal were used to train goal discriminators, only eight demonstration episodes were collected in about 15 minutes of human time, PPO training thereafter took about 90 minutes, and the learned policy landed successfully in over 90% of episodes. Under the same human-time budget, DRLHP succeeded only in 1/3 runs, while DAgger was more robust than DRLHP but did not achieve high performance (Rahtz et al., 2019).

The statistical model used for preferences is itself a reward-interface choice. A standard partial-return model assumes

$Y_I$7

whereas a regret model uses

$Y_I$8

The regret preference model is identifiable up to the reward–discount equivalence class that preserves optimal policies, while the partial-return model lacks identifiability in multiple contexts, including variable-horizon tasks and some stochastic settings. Empirically, regret better predicts real human preferences and yields reward functions whose downstream policies are better human-aligned (Knox et al., 2022). This establishes that a reward interface is not only a UI or API choice; it is also an assumption about what a preference means.

3. Structured, semantic, and domain-specific interfaces

Programmatic structure can itself be the interface. Reward Machines expose reward-function code to the agent and support automated reward shaping, task decomposition via options, and counterfactual reasoning with off-policy learning. CRM and HRM improve sample efficiency over black-box baselines across Office, Craft, Water World, and HalfCheetah domains; HRM often learns fastest initially but may converge to suboptimal policies, while CRM typically achieves the best asymptotic performance (Icarte et al., 2020).

LIMEN extends programmatic design from rewards alone to full task interfaces. It jointly evolves observation mappings and reward functions from raw state, and its results show that co-design is not a cosmetic refinement. In 10-seed evaluation, joint interface discovery reached success rates of 99% on Easy, 99% on Medium, 85% on Hard, 45% on Panda, and 48% on Go1. Observation-only optimization fails on Panda at 0%, reward-only optimization fails on Hard at 1%, and every single-component ablation fails catastrophically on at least one domain. The paper’s interpretation is that observation and reward components often benefit from co-design because either component alone can leave the induced learning problem unlearnable (Jaswal et al., 5 May 2026).

Other domains instantiate reward interfaces as structured aggregations of measurable signals. In adaptive UI, the state includes UI configuration, context of use, user behavior telemetry, and historical feedback; the reward can be an HCI-only weighted sum of normalized interaction metrics, or an HCI+HF fusion

$Y_I$9

where $g_I$0 can be built from QUIS, UES, in-situ Likert prompts, or pairwise preference losses over UI snippets. The system uses MCTS because dependencies across sequences of UI changes matter and rewards can be delayed (Gaspar-Figueiredo et al., 2023).

In brain-computer interfaces, the interface is often a bandit: the system chooses among stimuli, decoding strategies, or task configurations and receives sparse, noisy rewards such as accuracy gains, target-containment indicators in a P300 speller, or user comfort proxies. The finite-horizon objective is cumulative reward

$g_I$1

with regret

$g_I$2

This casts the reward interface as online exploration–exploitation under partial feedback (Heskebeck et al., 2022).

4. Uncertainty, repair, and anti-hacking interfaces

A central recent direction treats reward interfaces as safety layers against over-optimization. UARD models both epistemic uncertainty in value estimation and uncertainty in human preferences. With an ensemble mean $g_I$3, model uncertainty $g_I$4, preference uncertainty $g_I$5, and weights $g_I$6, it defines

$g_I$7

and confidence

$g_I$8

The agent uses $g_I$9 for action selection and discounts reward by confidence via $R_\theta(\tau)$0. On grid configurations and continuous control, this reduced reward-hacking behavior by about 93.7%, cut the alignment gap from about 77.6 to $R_\theta(\tau)$1, achieved a 95.9% reduction in alignment gap, and remained robust under up to 30% supervisory noise, albeit with a trade-off in peak observed reward. The interface also supports abstention when the risk signal exceeds $R_\theta(\tau)$2 (Singha, 29 Apr 2026).

Preference-Based Reward Repair starts from a human-specified proxy reward and learns an additive transition-dependent correction term from preferences:

$R_\theta(\tau)$3

Its objective regularizes corrections to remain near zero on already aligned trajectory pairs and, on misclassified pairs, preferentially reduces reward on undesirable trajectories rather than inflating reward on preferred ones. In tabular settings it matches the cumulative-regret rate of prior preference-based RL methods up to constants, and on reward-hacking benchmarks it consistently outperforms baselines that learn rewards from scratch or modify the proxy in other ways, while requiring substantially fewer preferences to learn high-performing policies (Hatgis-Kessell et al., 14 Oct 2025).

For sparse structured RL with LLM-generated reward code, the interface itself can be the main failure surface. Diagnostic-driven refinement studies semantic reward-function interfaces that expose fields such as agent_pos, carrying, event_text, step_count, and max_steps, and asks an LLM to write a reward_fn(obs, action, reward, terminated, truncated, info, state). The audit identifies reward flooding and semantic/API misunderstanding as dominant one-shot failures. Iterative refinement improves DoorKey-8x8 from 2.3% to 97.6% and KeyCorridor from 31.2% to 86.7%; metrics-only re-prompting drops these to 68.6% and 11.5%; and event_text can help, hurt, or be neutral depending on the environment (Wang et al., 27 May 2026).

In RLAIF for portable query generation, the reward interface combines a rubric-based LLM judge, a deterministic rule-based reward floor, and a KL anchor to an SFT reference. The deterministic floor detects verbatim copying via 6-gram overlap and lifted date ranges, clamps the reward to $R_\theta(\tau)$4, and skips the grader. Under this adversarial reward surface, GRPO is disproportionately susceptible to spurious reward signals, while RLOO and REINFORCE++ are more resistant. Adding the deterministic floor yields a $R_\theta(\tau)$5 improvement on an independent cross-family evaluation judge, and the paper finds that the training-time reward model inflates performance gains by $R_\theta(\tau)$6 (Liu et al., 25 Jun 2026).

5. Foundation models, reward models, and generative evaluators

Foundation-model pipelines increasingly treat the reward interface as a composition of pretrained models. FoMo Rewards maps a trajectory of visual observations $R_\theta(\tau)$7 and a natural-language instruction $R_\theta(\tau)$8 into a reward by encoding frames with a vision model, projecting them into an LLM’s embedding space, and using the LLM’s log-likelihood of the instruction as the reward:

$R_\theta(\tau)$9

It also defines online shaping by log-likelihood differences over partial trajectories. Contrastive and success-detection training yield strong discrimination between correct and perturbed behaviors; for example, in T1 under contrastive training, Correct is $$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$0 versus PT_Inc $$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$1, PT_Rev $$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$2, PT_Rep $$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$3, and PT_Len $$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$4 (Lubana et al., 2023).

CRew turns an LLM’s token-level confidence on the final answer into a reward model. For a generated solution $$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$5 with answer tokens $$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$6, it defines

$$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$7

and uses $$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$8 as the reward. On MATH500 with Qwen2.5-7B-Instruct, SC+Reward reaches 79.2 for Confidence, compared with 79.0 for Perplexity, 78.8 for Generative Verifier, and 78.2 for LLM-as-a-Judge. On RewardMATH, CRew obtains 27.12, 32.71, 40.17, and 38.51 for Qwen2.5-7B, 14B, 32B, and 72B, substantially exceeding other training-free rewards; across eight models, its evaluator performance correlates with reasoning ability at 0.83. CRew-DPO then improves RewardMATH to 36.02 and 38.72 over two iterations for Qwen2.5-7B-Instruct (Du et al., 15 Oct 2025).

When multiple reward models are available, routing becomes the interface. BayesianRouter learns offline per-RM reliability embeddings with Bradley–Terry and classification heads, uses them as Gaussian priors, and performs per-query Thompson sampling with an $$u_I(y; \theta) = \mathbb{E}_{\tau \sim g_I(y)}[R_\theta(\tau)],$$9 RM-call budget. On AlpacaEval-2 it reaches 63.23 versus 61.86 for the best single RM; on MT-Bench, 58.75 versus 57.50; on Chat-Arena-Hard, 66.20 versus 64.80; on GSM8K, 75.66 versus 74.68; and on MMLU, 57.39 versus 57.03 (Wu et al., 3 Oct 2025).

PaTaRM pushes the interface further toward generative reward modeling. It outputs global and instance-specific rubrics, a rationale, and a strict score tag such as <answer\>8</answer>, while learning from pairwise data via Preference-Aware Reward. Its rollout-level reward uses

$$P(y \mid \theta, I) = \frac{\exp(\beta u_I(y; \theta))}{Z_I(\theta)}.$$0

This yields an average relative improvement of 4.7% on RewardBench and RMBench across Qwen3-8B and Qwen3-14B, and an average downstream RLHF improvement of 13.6% across IFEval and InFoBench (Jian et al., 28 Oct 2025).

6. Interpretability, evaluation, and boundary conditions

As reward interfaces become more complex, inspecting them becomes a first-class technical problem. reward-lens re-derives mechanistic interpretability tools around the reward-head direction $$P(y \mid \theta, I) = \frac{\exp(\beta u_I(y; \theta))}{Z_I(\theta)}.$$1, using the Reward Lens

$$P(y \mid \theta, I) = \frac{\exp(\beta u_I(y; \theta))}{Z_I(\theta)}.$$2

as the analogue of the logit lens, along with component attribution, three-mode activation patching, reward-hacking probes, SAE feature attribution, and concept-vector interventions. Its central empirical finding is negative: linear attribution does not predict causal patching effects, with mean Spearman $$P(y \mid \theta, I) = \frac{\exp(\beta u_I(y; \theta))}{Z_I(\theta)}.$$3 on Skywork and $$P(y \mid \theta, I) = \frac{\exp(\beta u_I(y; \theta))}{Z_I(\theta)}.$$4 on ArmoRM. Late MLPs dominate observational attributions, while early and mid components dominate causal effects in patching, so observational and causal analyses must be kept first-class and directly comparable (Nadaf, 28 Apr 2026).

Several limits recur across the literature. AIRD still lacks user studies, and features absent from the training environment cannot be inferred, which directly limits generalization (Mindermann et al., 2018). LIMEN requires access to raw, structured simulator state and a reliable trajectory-level success metric; if either is unavailable or misspecified, interface discovery becomes difficult or unsafe (Jaswal et al., 5 May 2026). Diagnostic-driven refinement is explicitly bounded to sparse structured tasks with reliable interfaces under PPO, and in dense locomotion success-based diagnostics can misfire (Wang et al., 27 May 2026). UARD incurs approximately $$P(y \mid \theta, I) = \frac{\exp(\beta u_I(y; \theta))}{Z_I(\theta)}.$$5–$$P(y \mid \theta, I) = \frac{\exp(\beta u_I(y; \theta))}{Z_I(\theta)}.$$6 training cost relative to single-head baselines because of ensembles and may lower peak observed reward through conservative discounting (Singha, 29 Apr 2026). In adaptive UI, cognitive load, privacy, latency of post-task feedback, and the validity of QUIS/UES or predictive HCI metrics remain operational constraints (Gaspar-Figueiredo et al., 2023).

Taken together, these results suggest that reward interfaces are no longer secondary plumbing around a fixed scalar reward. They are the locus where semantics, uncertainty, human interaction, evaluator choice, safety constraints, and interpretability are explicitly encoded. A plausible implication is that future progress in alignment and reliable RL will depend at least as much on interface design—what is exposed, queried, discounted, routed, repaired, and audited—as on policy optimization alone.