Envelope Q-Learning

• Envelope Q-Learning is a reinforcement learning technique that enforces explicit upper/lower bounds or convex envelopes on Q-value updates to mitigate overestimation bias.
• It achieves efficient multi-objective and preference-agnostic policy optimization by training a universal Q-function valid across various linear reward scalarizations.
• Empirical results show that variants like LBQL and envelope-DDQN enhance convergence speed and robustness in stochastic, multi-agent, and high-dimensional control tasks.

Envelope Q-Learning is a class of reinforcement learning (RL) algorithms in which the value-function updates are explicitly constrained or regularized using envelopes—upper and/or lower bounds, or convex hulls—over the space of state-action pairs, preferences, or reward objectives. These methods have emerged to address key limitations of classical Q-learning, particularly in multi-objective, preference-agnostic, and stochastic settings, and share a rigorous theoretical foundation rooted in contraction mappings and stochastic approximation.

1. Foundations and Conceptual Motivation

Envelope Q-Learning generalizes tabular and function-approximation Q-learning by explicitly maintaining bounding or coverage structures on value functions. The most common instantiations include:

• Bounds-based envelopes: Using upper and lower (lookahead) bounds to constrain Q-iterates, ensuring that updates remain within estimated feasible intervals.
• Convex envelopes over preference vectors: Constructing the convex envelope of Q-functions for all linear scalarizations of vector-valued rewards, producing a universal parametric Q-function valid for any user-specified preference.
• Envelope-based convexification for optimal stopping or function approximation: Employing RL-based methods to approximate convex envelopes in stochastic control and PDE contexts.

The principal aims of envelope-based Q-learning are to increase sample efficiency, reduce overestimation bias, enable robust multi-objective trade-off policies, and allow generalization across preference configurations without retraining.

2. Envelopes in Single-Objective Q-Learning: Lookahead-Bounded Q-Learning

Lookahead-Bounded Q-Learning (LBQL) is a provably convergent Q-learning variant that employs sampled upper and lower bounds derived from information relaxation duals (Shar et al., 2020). The envelope is constructed as follows:

• At each iteration, after a standard Q-learning step, the algorithm solves two sampled subproblems: a perfect-information relaxation yields an upper bound, while a fixed-policy evaluation yields a lower bound. Both involve recursively solving a deterministic dynamic program along sampled transitions, using current Q-iterate values as penalties.
• The upper bound $\widehat Q^U_0(s,a)$ and lower bound $\widehat Q^L_0(s,a)$ are tracked for each (s, a) pair using stochastic approximation with step size $\beta_n$:

$$U_{n+1}(s,a) = \Pi_{[-\rho,+\infty)}\{U_n(s,a)+\beta_n[\widehat Q^U_0(s,a)-U_n(s,a)]\}$$

$$L_{n+1}(s,a) = \Pi_{(-\infty,+\rho]}\{L_n(s,a)+\beta_n[\widehat Q^L_0(s,a)-L_n(s,a)]\}$$

where $\rho = R_{\max}/(1-\gamma)$.

• The Q-learning TD update is then projected into this [L, U] envelope:

$$Q'_{n+1}(s,a) = \Pi_{[L_{n+1}(s,a), U_{n+1}(s,a)]}[Q_{n+1}(s,a)]$$

• Theoretical analysis establishes almost-sure convergence of all Q-iterates to $Q^*$, uniform boundedness, and rapid/robust contraction properties. The projection step is key to ensuring stability and mitigating overestimation bias, particularly when transitions are expensive to sample or the environment is highly stochastic.

LBQL exhibits accelerated convergence and reduced sensitivity to hyperparameters relative to Q-learning and its contemporaries. This is attributed to the role of approximate planning (via lookahead bounds) and regularization of Q-iterates within tight envelopes (Shar et al., 2020).

3. Envelope Q-Learning for Multi-Objective and Preference-Agnostic RL

Envelope Q-Learning has become central to multi-objective RL (MORL), in which the agent seeks to optimize policies with respect to multiple, often conflicting, scalar reward components under arbitrary user-specified linear preferences. Approaches include:

• Learning a parametric family $Q_{env}(s,a;\omega)$ for all weight vectors $\omega$ (preferences) over objectives, enforcing convexity in $\widehat Q^L_0(s,a)$0 via envelope updates (Yan et al., 2024, Lin et al., 2024).
• Defining an envelope Bellman operator as

$\widehat Q^L_0(s,a)$1

This operator iteratively takes the supremum over next-step preferences, effectively constructing the convex envelope of attainable Q-values.

• After each batch of TD updates (across a discretized grid of $\widehat Q^L_0(s,a)$2), re-projecting the Q-table or network onto the convex envelope:

$\widehat Q^L_0(s,a)$3

• A key property is that the envelope update guarantees the learned $\widehat Q^L_0(s,a)$4 is simultaneously valid for all linear scalarizations, obviating the need for training multiple specialized policies and enabling on-demand adaptation at test time.

In mean-field multi-agent domains, such as decentralized moving agents in formation (MAiF), envelope Q-learning is further combined with mean-field approximations to handle the combinatorial state explosion and partial observability, yielding scalable and preference-transferable solutions (Lin et al., 2024).

4. Mathematical and Algorithmic Structure

Envelope Q-Learning algorithms share a common family structure, but implementation varies:

• For vector-valued reward MOMDPs, the algorithm maintains a Q-table or network indexed by (s, a, $\widehat Q^L_0(s,a)$5) and samples over a set of discrete preferences $\widehat Q^L_0(s,a)$6.
• At each episode/timestep:
  • A preference vector $\widehat Q^L_0(s,a)$7 is sampled.
  • Actions are selected greedily with respect to $\widehat Q^L_0(s,a)$8.
  • Observed rewards and next-states update $\widehat Q^L_0(s,a)$9 using TD learning, and the envelope projection is performed as above, ensuring convexity in $\beta_n$0.
• In deep RL variants, the envelope is enforced via batched forward passes and re-projection (e.g., network outputs are projected onto the convex upper-hull in the $\beta_n$1-dimension).

Pseudocode for basic envelope Q-learning, as used in vehicular networks (Yan et al., 2024), follows this pattern:

• For each (s, a, $\beta_n$2): update $\beta_n$3
• For all $\beta_n$4 and $\beta_n$5: $\beta_n$6

In mean-field multi-agent settings, the updates further account for the mean-action of neighbors and communications constraints (Lin et al., 2024).

5. Theoretical Properties and Convergence Guarantees

All major envelope Q-learning variants establish strong theoretical properties:

• The envelope Bellman operator is a $\beta_n$7-contraction in the sup-norm over the augmented space $\beta_n$8, guaranteeing the existence and uniqueness of a fixed point $\beta_n$9 representing the convex coverage set (Yan et al., 2024, Lin et al., 2024).
• Stochastic-approximation frameworks and TD-error decompositions ensure almost sure convergence to the unique fixed point provided all state-action-preference triples are visited infinitely often, step-size schedules follow the Robbins–Monro conditions, and rewards are bounded.
• Boundedness is enforced throughout the learning process via envelope or projection steps, which ensures numerical stability and eliminates divergence.

In single-objective bounded Q-learning, envelopes ensure the iterates never violate the derived information relaxation bounds on $$U_{n+1}(s,a) = \Pi_{[-\rho,+\infty)}\{U_n(s,a)+\beta_n[\widehat Q^U_0(s,a)-U_n(s,a)]\}$$0, and empirical evidence shows significantly improved robustness and sample-efficiency (Shar et al., 2020).

6. Applications and Empirical Performance

Envelope Q-Learning has achieved state-of-the-art results in several domains:

• Preference-agnostic multi-agent path finding (MAiF): Mean-field control with envelope Q-learning allows training a single universal value function for all preferences, providing superior flexibility and performance in formation maintenance, dynamic objective shifting, and large-scale decentralized settings (Lin et al., 2024).
• Multi-objective vehicular networks: Envelope Q-learning in multi-objective RL (MORL) for vehicular URLLC settings outperforms single-policy weighted-sum approaches and model-based baselines in terms of transportation and telecommunication rewards, handoff rate, safety, and Pareto-front coverage (Yan et al., 2024).
• Convex envelope approximation in stochastic control: A Q-learning variant yields accurate numerical approximations of convex envelopes for non-convex functions, related to PDE viscosity solutions, with results visually matching classical finite-difference solvers (Borkar et al., 2023).

A summary table of prominent instantiations is given below:

Algorithm	Envelope Type	Application Domain
LBQL	Upper/lower value bounds	Tabular RL/stochastic MDPs
MFC-EQ	Convex in $$U_{n+1}(s,a) = \Pi_{[-\rho,+\infty)}\{U_n(s,a)+\beta_n[\widehat Q^U_0(s,a)-U_n(s,a)]\}$$1	Multi-agent path finding
Envelope-DDQN	Convex in $$U_{n+1}(s,a) = \Pi_{[-\rho,+\infty)}\{U_n(s,a)+\beta_n[\widehat Q^U_0(s,a)-U_n(s,a)]\}$$2	Multi-objective vehicular RL
Convex Envelope RL	Functional envelope	Function approximation/PDE

Empirical evidence suggests envelope constraints yield faster, more robust convergence, and in multi-objective settings, enable a single trained architecture to instantly adapt to arbitrary user-specified preferences without retraining (Shar et al., 2020, Lin et al., 2024, Yan et al., 2024).

7. Connections and Outlook

Envelope Q-Learning unifies and extends a range of regularization, planning, and multi-objective learning strategies in reinforcement learning. Its bounding and convexification constructs are closely related to:

• Information relaxation duality (for value-function upper bounds).
• Preference-conditional universal function approximation (solving all scalarized tasks at once).
• Pareto-front and coverage set computation in multi-objective RL.

Theoretical and empirical results establish its value for both single-objective and multi-objective tasks, especially where robustness to hyperparameters, sample efficiency, and preference-adaptivity are critical. Active research continues on function-approximation extensions, higher-dimensional convexity enforcement, and applications to constrained, dynamic, and decentralized environments. Further directions include integration with distributional RL, policy regularization, and application-specific convex envelope constructions for continuous control and stochastic optimal stopping (Shar et al., 2020, Lin et al., 2024, Borkar et al., 2023, Yan et al., 2024).