Generalized Thresholded Lexicographic Ordering (gTLO)

• The gTLO framework extends traditional TLO by integrating threshold conditioning directly into a single deep Q-network for efficient multi-objective decision-making.
• It generalizes over the continuous threshold space, enabling simultaneous computation of a continuum of Pareto-optimal policies in complex, high-dimensional environments.
• Empirical evaluations show that gTLO achieves superior sample efficiency, faster convergence, and comprehensive non-convex Pareto front coverage compared to prior methods.

Generalized Thresholded Lexicographic Ordering (gTLO) is a deep reinforcement learning framework designed for multi-objective sequential decision-making, where solutions are required to satisfy explicit priority and threshold constraints across competing objectives. It extends classical Thresholded Lexicographic Ordering (TLO) by integrating generalization across the preference (threshold) space directly into the function approximation architecture, which enables efficient and scalable computation of non-linear multi-objective policies in complex, high-dimensional environments (Dornheim, 2022).

1. Formal Definition and Theoretical Foundations

Thresholded Lexicographic Ordering was originally proposed to address the limitations of linear scalarization methods in multi-objective reinforcement learning (MORL), particularly their inability to recover non-convex regions of the Pareto front. Let $$\mathbf Q(s,a) = \left(Q_1(s,a), Q_2(s,a), \ldots, Q_I(s,a)\right) \in \mathbb{R}^I$$ be a vector of Q-values for $I$ objectives, and $\mathbf t = (t_1, \ldots, t_{I-1}, t_I = +\infty)$ user-specified thresholds.

The classical TLO policy is defined via the thresholded Q-values:

$Q^t_i(s,a) = \min\{Q_i(s,a), t_i\}$

Action selection uses the lexicographic predicate: $$\text{sup}(a^*, a; s, i) = \begin{cases} Q^t_i(s,a^*) > Q^t_i(s,a) \ \quad \text{or}~[ Q^t_i(s,a^*) = Q^t_i(s,a) ~ \land~ (i=I~ \lor~ \text{sup}(a^*, a; s, i+1))] \end{cases}$$ An action $a^*$ is selected if it is strictly superior under this recursive comparison to all competitors in state $s$.

gTLO generalizes this setup: it learns a single, parameterized, threshold-conditioned Q-function,

$$\mathcal{Q}(s, a, \mathbf{t}; \theta) \approx \mathbf{Q}(s, a; \mathbf{t})$$

which, for any threshold setting $\mathbf{t}$, induces a TLO policy $$\pi_{\text{gTLO}}(s; \mathbf{t}) = \arg\max_{a\in A}^{\text{TLO}} \mathcal{Q}(s, a, \mathbf{t}; \theta)$$. Thus, the policy space is directly indexed by the (potentially continuous) threshold parameters (Dornheim, 2022, Tercan et al., 2024).

2. Algorithmic Design and Training Procedure

Core Bellman-Style Update

gTLO departs from conventional Q-learning by employing a threshold-conditioned Q-network and restricting Bellman backups to threshold-feasible action sets. After observing a transition tuple $(s, a, \mathbf{r}, s', \mathbf{t})$ (with $\mathbf{r}\in \mathbb{R}^I$), the update for each $i$-th objective utilizes

$$\tau_{\,i} = r_i + \gamma\, \max_{a' \in \tilde{A}_{(t, i, s')}} \mathcal{Q}_i(s', a'; \mathbf{t}; \theta^-) - \mathcal{Q}_i(s,a;\mathbf{t};\theta)$$

where $\tilde{A}_{(t, i, s')}$ encodes which actions pass the threshold tests up to index $i-1$: $$\tilde{A}_{(t, i, s')} = \begin{cases} \hat{A}_{(t, i-1, s')} & \text{if } \hat{A}_{(t, i-1, s')} \neq \varnothing \ \{ \pi_{\text{gTLO}}(s'; \mathbf{t}) \} & \text{otherwise} \end{cases}$$ The sum of per-objective Huber losses is minimized over mini-batches, and $\theta$ is periodically copied to $\theta^-$ as in standard DQN practice (Dornheim, 2022).

Deep Network Architecture

• State encoder: Three convolutional layers for image input or two for mesh data.
• Multi-headed architecture: A separate head for each objective $i$, each receiving its respective prefixes of the threshold vector $\mathbf{t}$ concatenated to the encoded state.
• Activation and loss: ReLU activations with objective-wise Huber loss.

A single network thus parameterizes policies for all possible user preferences throughout training and inference.

3. Generalization Over the Preference Space

Traditional MORL methods such as Multi-Objective Fitted Q-Iteration (MOFQ), CN, or Envelope Q-Learning (EQL) generalize over linear scalarization weights $\mathbf{w}$. In contrast, gTLO generalizes over the threshold vector $\mathbf{t}$ by embedding it directly into the Q-network’s computation graph. At each training step, $\mathbf{t}$ is sampled (often uniformly) and used for that batch, enabling sample-efficient, multi-policy learning—all policies corresponding to the full threshold continuum are represented within a single network.

This contrasts sharply with "outer-loop" TLO variants that require training and maintaining a separate network for each distinct policy preference—a procedure that incurs prohibitive data and computational costs in high-dimensional or continuous threshold spaces (Dornheim, 2022).

4. Empirical Results and Quantitative Evaluation

The gTLO framework was evaluated on both the canonical Deep Sea Treasure (DST) problem and a real-world manufacturing control task:

Approach	HV(0,–25)	Precision	Recall	F₁
gTLO (ours)	1154.6 ± 0.8	0.99 ± 0.02	0.98 ± 0.04	0.985 ± 0.022
gLinear	762.0 ± 0.0	1.00 ± 0.00	0.20 ± 0.00	0.334 ± 0.00
dTLQ	907.7 ± 190.6	0.96 ± 0.07	0.66 ± 0.05	0.78 ± 0.04
outer-loop gTLO	1150.0 ± 14.3	0.98 ± 0.04	0.98 ± 0.04	0.98 ± 0.04

• Hypervolume (HV): gTLO closely matches the true Pareto front (1,155), outperforming both linear scalarization (gLinear) and prior TLO methods.
• Precision/Recall/F₁: gTLO achieves high-coverage, while gLinear is limited to the convex hull extremes of the Pareto front.
• Convergence: Inner-loop gTLO converges roughly twice as fast as outer-loop gTLO (61k vs 140k steps).
• Deep Drawing Control: gTLO outperforms outer-loop TLO, with gLinear failing to cover the non-convex mid-region.

These results demonstrate the superior sample efficiency, generalization, and Pareto front coverage of the gTLO approach (Dornheim, 2022).

5. Extension Concepts and Research Directions

Extensions to the gTLO framework explored in recent work include (Tercan et al., 2024, Dornheim, 2022):

• State-dependent or adaptive thresholds: $\tau_i(s)$ or time-varying threshold vectors, to capture dynamically changing preferences or constraints.
• Soft lexicographic slacks: Replacing hard threshold caps with slack or buffer functions, allowing for partial satisfaction or indifference regions.
• Lex-Pareto and partial orderings: Supporting groups of objectives with equal importance or hybrid lexicographic–Pareto structures.
• Dynamic threshold tracking: Thresholds or slack budgets updated online via dual-variable or meta-gradient techniques.
• Gradient projection refinements: Projections onto intersections of parameterized cones for policy-gradient style updates, enabling satisfaction of nonstationary or context-sensitive priority constraints.

A plausible implication is that combining gTLO with Constrained MDP dual-Lagrangian approaches could enable automatic, theoretically-grounded threshold/slack adaptation for real-world requirements.

6. Advantages, Limitations, and Open Problems

The primary advantages of gTLO are:

• Non-convex Pareto coverage: gTLO recovers non-convex portions of the solution set unattainable by linear scalarization.
• Single-network, multi-policy efficiency: Unified training over all threshold settings, yielding major improvements in data efficiency compared to outer-loop approaches.
• Deep RL scalability: gTLO can be directly implemented in deep Q-network architectures and achieves state-of-the-art MORL performance on standard benchmarks (Dornheim, 2022).

Key limitations and open challenges include:

• Applicability is limited to finite-horizon MDPs where thresholded objectives are terminal; extending gTLO to cases with nonzero intermediate rewards remains an open question.
• The TLO-style off-policy update is prone to large biases if not carefully restricted as per Equation (5) in (Dornheim, 2022).
• In stochastic or partially observed environments, if important system parameters are unobserved (e.g., friction in manufacturing), full recovery of the true Pareto front is generally impossible.
• Future work outlined includes combining gTLO with Double-DQN and Dueling Network architectures, prioritized replay, extension to continuous action spaces, and dynamic threshold scheduling.

7. Relationship to Related Lexicographic and Thresholded Methods

Generalized TLO builds upon and generalizes prior multi-objective RL approaches such as Gábor et al., 1998, Vamplew et al., 2011, and subsequent works on generalized scalarization and non-linear preference parameterization (MOFQ, CN, EQL). Compared to classical methods that operate on static preference parameters or require exhaustive outer-loop training, gTLO uniquely joins non-linear, thresholded priority enforcement with direct preference generalization, and is the first inner-loop deep-MORL approach to reliably solve non-convex tasks via preference-conditioned learning (Dornheim, 2022).

Recent proposals in the literature also seek to generalize TLO beyond static thresholds, toward adaptive, state-dependent, or slack-based lexicographic constraint satisfaction, and integrate with advanced policy-gradient projection and meta-learning machinery (Tercan et al., 2024). This line of work emphasizes the potential of flexible, efficient, and principled preference-based RL for high-stakes multi-objective decision domains.