Regret-Based Objectives in Control and Learning

• Regret-based objectives are a framework that quantifies the performance gap between a candidate policy and a benchmark (e.g., clairvoyant or best fixed in hindsight) in control and learning settings.
• They enable nuanced trade-offs between mean performance, tail risk, and adversarial scenarios by interpolating between risk-neutral and worst-case formulations.
• Applications span optimal control, reinforcement learning, online planning, and multi-objective optimization, often employing dynamic programming, Riccati equations, and operator-theoretic methods.

Regret-based objectives constitute a fundamental analytic and algorithmic framework across optimal control, reinforcement learning, online decision-making, multi-objective optimization, and robust operations research. Unlike traditional risk-neutral or worst-case (robust) formulations, regret-based approaches explicitly compare the performance of a candidate policy or action sequence to that of a powerful benchmark, such as an offline/clairvoyant solution, a best fixed policy in hindsight, or a finely specified set of alternative actions. This direct benchmark-relative comparison leads to nuanced trade-offs between mean, tail, and adversarial scenarios across various domains.

1. Core Definitions and Theoretical Formulation

Regret in its most general form is defined as the performance gap between a decision-maker's realized outcome and the best possible outcome achievable by a given comparator scheme:

$$\mathrm{Regret} = \text{Performance of Comparator} - \text{Performance of Candidate (Causal) Policy}$$

The benchmark—often non-causal, non-adaptive, or "oracle"—is domain-dependent:

• Control: Regret quantifies the cost gap to a noncausal ("clairvoyant") controller with access to future disturbances, as in regret-optimal LQR (Sabag et al., 2021).
• Online learning/MDP: Regret is the cumulative or simple loss versus the best fixed action or policy in hindsight or the best possible policy for each realized environment (Feldman et al., 2012).
• Reinforcement Learning with Uncertainty: Regret is the shortfall to the best achievable reward under all admissible reward functions or environmental parameters, including risk-sensitive distributions and preference-induced ambiguity (Wu et al., 2020, Regan et al., 2012, Bastani et al., 2022, Shi et al., 20 Mar 2026).

Static regret contrasts a sequence of candidate actions to the best fixed action in hindsight. Dynamic regret (or pathwise regret) compares each action to the best action for the observed context or disturbance at each time, a distinction crucial in adaptive control and nonstationary environments (Gibson et al., 8 Jan 2025, Goel et al., 2021, Polatov et al., 26 Apr 2026).

In multi-agent and multi-objective contexts, regret can be defined elementwise or via set-level comparators, such as in the selection of bounded-size action sets under imprecise probabilities (Nakharutai et al., 2024) or multi-criteria bandits (Davoodi et al., 16 Jun 2025):

$$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$

2. Regret-Optimal Control and Estimation

The regret-optimal control paradigm, as formalized in "Regret-Optimal LQR Control" (Sabag et al., 2021) and extended in (Goel et al., 2021, Polatov et al., 26 Apr 2026), seeks causal controllers whose worst-case cost over bounded disturbances is as close as possible to that of the offline optimal (clairvoyant) controller. The canonical objective reads:

$$\mathrm{Regret}^* = \min_{\pi \in \text{Causal}}\max_{w:\|w\|_2 \leq 1} \left[\operatorname{Cost}(\pi;w) - \min_{\pi' \in \text{Noncausal}} \operatorname{Cost}(\pi';w)\right]$$

Key theoretical structure:

• Operator-theoretic reduction: The gap to the noncausal benchmark is recast as a Nehari extension problem in functional analysis, solvable via best causal approximation in operator norm (i.e., optimal approximation to a strictly anti-causal operator) (Sabag et al., 2021).
• Explicit state-space realization: The regret-optimal controller combines the classic $H_2$ (linear-quadratic regulator) law and an $n$th-order compensator function of disturbance histories, requiring solution of a Riccati equation and two Lyapunov equations.
• Interpolation property: The regret-optimal controller smoothly interpolates between $H_2$ (mean-optimal) and $H_\infty$ (worst-case-optimal) control, guaranteeing that both average and tail costs remain close to their optima without knowledge of disturbance statistics.

In finite-state and time-varying systems, the regret-optimal policy is computed via a nested dynamic programming recursion, using a regret-Bellman operator that tracks both current and lookahead benchmark states (Polatov et al., 26 Apr 2026).

3. Regret-Based Objectives in Online and Reinforcement Learning

3.1. Online Planning and Simple Regret

In online MDP planning, simple regret measures the expected loss of the recommended action at an interrupt with respect to the true best action, as opposed to cumulative regret, which aggregates over the entire learning horizon. Algorithms such as BRUE (based on a two-phase Monte-Carlo tree search) achieve exponential-rate reduction in simple regret, a theoretical advance over polynomial-rate cumulative regret reduction in classical UCT (Feldman et al., 2012).

3.2. Minimax Regret and Robustness

In robust or ambiguity-averse RL and MDPs, the minimax regret criterion seeks a policy that minimizes the greatest possible regret over all admissible reward or model parameters:

$$\min_\pi\,\max_{r \in \mathcal{R}}[V^*(r) - V^\pi(r)]$$

This bi-level optimization (outer: over policies, inner: over reward/model) is solved via constraint-generation schemes; practical performance hinges on efficiently eliciting only those reward components critical to regret reduction (Regan et al., 2012).

3.3. Best-of-Both-Regimes Regret in RLHF

In reinforcement learning from human feedback, multiple feedback sources may provide inconsistent or biased labels. The regret framework models a "cumulative imperfection budget" for each source and derives algorithms whose regret is

$\tilde{O}(\sqrt{K/M} + \omega)$

where $K$ is the number of episodes, $$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$0 the number of sources, and $$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$1 the imperfection budget—explicitly quantifying the trade-off between statistical gain from redundancy and robustness to bias (Shi et al., 20 Mar 2026).

4. Multi-Objective Regret and Regret in Bandits

Multi-objective optimization and bandit problems introduce vector-valued regret metrics to enforce guarantees across Pareto-optimal sets rather than in a single scalarized objective.

4.1. Multi-Objective Regret Definitions

• Coverage-regret: For each Pareto-optimal arm $$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$2 and each objective $$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$3, ensure

$$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$4

• Cumulative adjustment-regret: The sum over rounds of the minimal additive shift needed for each action played to weakly dominate some Pareto-optimal action remains sublinear (Davoodi et al., 16 Jun 2025).

4.2. Efficient Pareto-Optimality

Efficient Pareto-optimal arms are those not dominated by any convex combination of other Pareto arms, corresponding to the convex hull of the front. Regret control focuses on these as the relevant set for balanced learning.

4.3. Regret Bounds

Sublinear bounds $$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$5 are established for both coverage and adjustment-regret, both for the Pareto set and its efficient subset (Davoodi et al., 16 Jun 2025). In adversarial preference scenarios for multi-objective RL, nearly minimax-optimal regret rates scaling as $$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$6 are shown, with preference-free exploration scaling parameterically with objective dimension $$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$7 (Wu et al., 2020).

5. Regret Objectives in Robust Optimization and Decision Analysis

5.1. Budgeted Set-Valued Decisions under Imprecision

In severe uncertainty, regret-based rules for set-valued recommendations enforce a cardinality constraint $$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$8, optimizing minimax or maximin set-level regret over convex sets of priors. Consistency and coverage properties are characterized in terms of weak and strong inclusion of maximal elements, with complexity differing between minimax (polynomial) and maximin (NP-complete) variants (Nakharutai et al., 2024).

5.2. Regret in Combinatorial Optimization

In multi-period facility location (nested $$\max_{a^* \in \mathcal{A}^{*}}\min_{a \in \mathcal{A}}\mathrm{Regret}(a,a^*)$$9-center), both sum-absolute and max-relative regret objectives quantify the loss due to enforced consistency under facility nesting constraints compared to periodwise optimality. MILP-based algorithms incorporate these regret objectives efficiently, and empirical results show small cost overhead for full multi-temporal consistency (Brandstetter et al., 2024).

6. Regret-Based Objectives in Curriculum and Adversarial Environment Design

In automated environment design for robust policy learning, regret-based teacher-student paradigms pose a two-player game: the teacher adversarially seeks to maximize the agent's regret (i.e., the gap to task-specific optimality), while the agent minimizes worst-case regret over the environment space. Algorithms such as PLR and ACCEL maintain theoretical minimax-regret guarantees and produce increasingly challenging curricula with strong empirical performance (Parker-Holder et al., 2022, Sadek et al., 3 Jul 2025).

7. Practical Implications, Limitations, and Algorithmic Properties

Regret-based objectives offer several practical and theoretical advantages:

• Explicitly quantify robustness to model/belief misspecification by focusing on the best possible alternative.
• Interpolate smoothly between expected-case and worst-case paradigms, e.g., $$\mathrm{Regret}^* = \min_{\pi \in \text{Causal}}\max_{w:\|w\|_2 \leq 1} \left[\operatorname{Cost}(\pi;w) - \min_{\pi' \in \text{Noncausal}} \operatorname{Cost}(\pi';w)\right]$$0 versus $$\mathrm{Regret}^* = \min_{\pi \in \text{Causal}}\max_{w:\|w\|_2 \leq 1} \left[\operatorname{Cost}(\pi;w) - \min_{\pi' \in \text{Noncausal}} \operatorname{Cost}(\pi';w)\right]$$1 control (Sabag et al., 2021).
• Support tractable optimization in linear and finite-state systems via reduction to Riccati/Lyapunov equations or dynamic programming recursions (Goel et al., 2021, Polatov et al., 26 Apr 2026).
• Enable principled set-based recommendations under imprecise models, balancing coverage and computational complexity (Nakharutai et al., 2024).
• Outperform strictly equilibrium- or risk-neutral-based methods in behavioral econometrics, preference elicitation, and learning from imperfect feedback through robust bias correction and adaptivity (Nisan et al., 2016, Shi et al., 20 Mar 2026).

However, regret-based designs can incur computational overhead, particularly in high-dimensional or combinatorial settings, and may be sensitive to benchmark specification. Algorithmic strategies—such as operator-theoretic reductions, constraint generation, and hybrid evolutionary schedules—are essential for scaling regret minimization to realistic tasks.

---

References

• "Regret-Optimal LQR Control" (Sabag et al., 2021)
• "Regret-optimal Estimation and Control" (Goel et al., 2021)
• "Regret-Optimal Control for Finite-State Systems" (Polatov et al., 26 Apr 2026)
• "Simple Regret Optimization in Online Planning for Markov Decision Processes" (Feldman et al., 2012)
• "Reinforcement Learning from Multi-Source Imperfect Preferences: Best-of-Both-Regimes Regret" (Shi et al., 20 Mar 2026)
• "Adaptive Incentive Design with Regret Minimization" (Vasileiou et al., 7 Apr 2026)
• "An Experimental Evaluation of Regret-Based Econometrics" (Nisan et al., 2016)
• "Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learning" (Wu et al., 2020)
• "Regret-based budgeted decision rules under severe uncertainty" (Nakharutai et al., 2024)
• "Stochastic Multi-Objective Multi-Armed Bandits: Regret Definition and Algorithm" (Davoodi et al., 16 Jun 2025)
• "Time-Varying Multi-Objective Optimization: Tradeoff Regret Bounds" (Shafiei et al., 2022)
• "Evolving Curricula with Regret-Based Environment Design" (Parker-Holder et al., 2022)
• "Direct Regret Optimization in Bayesian Optimization" (Zhang et al., 9 Jul 2025)
• "Mixed-integer linear programming approaches for nested $$\mathrm{Regret}^* = \min_{\pi \in \text{Causal}}\max_{w:\|w\|_2 \leq 1} \left[\operatorname{Cost}(\pi;w) - \min_{\pi' \in \text{Noncausal}} \operatorname{Cost}(\pi';w)\right]$$2-center problems with absolute and relative regret objectives" (Brandstetter et al., 2024)
• "Mitigating Goal Misgeneralization with Minimax Regret" (Sadek et al., 3 Jul 2025)
• "Regret Bounds for Risk-Sensitive Reinforcement Learning" (Bastani et al., 2022)
• "Regret-based Reward Elicitation for Markov Decision Processes" (Regan et al., 2012)
• "Regret Analysis: a control perspective" (Gibson et al., 8 Jan 2025)
• "Regret-Based Defense in Adversarial Reinforcement Learning" (Belaire et al., 2023)