Trajectory Q-Learning: Theory and Applications

• Trajectory Q-Learning (TQL) is a reinforcement learning method that estimates value functions over full trajectories rather than individual state-action pairs.
• It employs history-based critics and non-Markovian policies to optimize risk measures and guarantee monotonic policy improvements.
• TQL is applied in diverse settings including risk-sensitive tasks, transfer learning in 5G networks, and economic pricing, demonstrating enhanced convergence and sample efficiency.

Trajectory Q-Learning (TQL) refers to a family of algorithms within reinforcement learning (RL) and transfer RL that learn value functions over entire trajectories, rather than state-action pairs. TQL has been instantiated as both classical tabular Q-learning (where “trajectory” is used to differentiate synchronous from asynchronous or single-trajectory learning) and as sequence/trajectory-centric algorithms in risk-sensitive RL, transfer RL, and applied RL domains. The term TQL is also sometimes used as an abbreviation for Tabular Q-Learning in economics and algorithmic pricing literature. The following sections delineate the principles, major algorithmic variants, theoretical properties, empirical behaviors, and applied methodologies of TQL.

1. Core Concepts and Algorithmic Definition

Trajectory Q-Learning in its modern formalization is defined by the learning of value functions or critics conditioned on histories or sequences (trajectories) rather than merely on the current state or state-action pair. The generic objective is to estimate the expected or risk-distorted return over full episodes, with updates taking into account the evolution of state, action, and reward tuples along a trajectory.

In risk-sensitive RL contexts, as in "Is Risk-Sensitive Reinforcement Learning Properly Resolved?" (Zhou et al., 2023), TQL is instantiated by a history-based non-Markovian critic:

$$Z^\pi(h_t, a_t) = \sum_{i=0}^{t-1} \gamma^i r(s_i, a_i) + \gamma^t Z^\pi(\{s_t\}, a), \quad h_t = (s_0, a_0, ..., s_t)$$

The core update targets the optimization of a risk measure (such as CVaR, Wang, POW) over complete trajectories, leveraging historical conditioning for unbiased policy improvement and value estimation. In transfer RL, as in "Transfer Reinforcement Learning for 5G-NR mm-Wave Networks" (Elsayed et al., 2020), TQL employs transfer mechanisms where value functions from source tasks (possibly learned synchronously) are mapped onto target tasks to accelerate convergence.

In the algorithmic pricing literature (e.g., "Algorithmic Collusion in Dynamic Pricing with Deep RL" (Deng et al., 4 Jun 2024)), TQL is frequently synonymous with classical tabular Q-learning, which updates:

$$Q_{t+1}(s, a) = (1-\alpha) Q_t(s,a) + \alpha\left[ R_{t+1} + \gamma \max_{a'} Q_t(s', a') \right]$$

where learning proceeds synchronously or asynchronously along sampled trajectories.

2. Theoretical Properties and Policy Guarantee

Trajectory Q-Learning distinguishes itself through several salient theoretical properties:

• Non-Markovian Value Estimation: By conditioning critics on full or partial trajectory histories, TQL overcomes the limitations of Markov state-dependent policies and achieves unbiased optimization for risk measures that are non-additive or non-decomposable, as rigorously shown in (Zhou et al., 2023). Local optimization via Bellman recursion fails for general distortion risk measures (except for the mean), but global trajectory-based approaches are theoretically sound.
• Contraction and Convergence: The history-relied operator (HR) used for critic updating in TQL is a $\gamma$-contraction in the Wasserstein metric, guaranteeing the existence and uniqueness of fixed points for trajectory-conditioned policies and critics (see Theorem 2 in (Zhou et al., 2023)).
• Policy Improvement Guarantee: TQL policy updates yield monotonic improvement or preservation of the risk-distorted return, in contrast to dynamic programming approaches that may degrade performance for certain risk objectives (Theorem 3 in (Zhou et al., 2023)).
• Transfer RL Sample Efficiency: Multi-task TQL algorithms incorporating value transfer (as in (Elsayed et al., 2020, Chai et al., 1 Feb 2025)) achieve provable regret reductions in composite or transfer settings, particularly when task dynamics can be decomposed as low-rank plus sparse.

3. Algorithmic Variants and Implementation

3.1 Risk-Sensitive Trajectory Q-Learning

• History-based Critic Design: TQL utilizes sequence encoders (e.g. GRUs, LSTMs) to produce estimates $Q(h,a;\tau)$ over discrete quantile indices $\tau$, reconstructing the value distribution over trajectories. The policy is trained to maximize the relevant risk-distorted return, $\beta[Z_\theta(h,a)]$.
• Learning Losses: Quantile regression losses (QR-Huber), coupled with policy gradient or actor-critic optimization, ensure stability and robust modeling of the full return distribution.
• Parallel Markovian Critics: For stabilization and bootstrapping, a Markovian critic is trained in parallel.

3.2 Transfer Reinforcement Learning

• Q-value Transfer with Mapping: TQL combines a transferred Q-value from a source task (expert agent) with locally learned Q-values. For target state-action pair $(s_t, a_t)$:

$Q(s_t, a_t) = Q_t(s_t, a_t) + Q_l(s_t, a_t)$

where $Q_t$ is the transferred Q-value via inter-task mapping and $Q_l$ is updated as in standard Q-learning (Elsayed et al., 2020).

• Composite MDPs and UCB-TQL: In "Transition Transfer Q-Learning for Composite Markov Decision Processes" (Chai et al., 1 Feb 2025), UCB-TQL exploits low-rank plus sparse decompositions, transferring the low-rank component from source to target while learning sparse corrections. Regret is bounded as $\tilde{O}(\sqrt{e H^5 N})$ with $e$ sparse differences.

3.3 Tabular TQL in Economics and Applied RL

• Synchronous vs. Asynchronous Single-Trajectory TQL: Finite-time convergence of asynchronous single-trajectory Q-learning (also called TQL) matches synchronous Q-learning rates under sufficient exploration, with sample complexity

$$T \gtrsim \frac{|\mathcal{S}|^2|\mathcal{A}|^2}{(1-\gamma)^5} \frac{1}{\varepsilon^2}$$

as established in (Qu et al., 2020).

• Behavior in Algorithmic Pricing: TQL robustly induces tacit collusion, supra-competitive pricing, volatility, and sensitivity in simulated duopolies absent exploration countermeasures (Deng et al., 4 Jun 2024, Deng et al., 14 Mar 2025).

4. Applications and Empirical Performance

TQL is deployed across diverse domains:

• Risk-Sensitive RL: Empirically validated in gridworld and continuous control tasks, TQL achieves superior performance in CVaR, CPW, Wang, and other risk-sensitive metrics relative to distributional RL baselines (Zhou et al., 2023). The modeling of trajectory-conditioned critics allows flexible adaptation across risk preferences.
• Transfer RL: In mobile 5G mmWave networks, joint user-cell association and beam selection using TQL provides up to 29% convergence speedup over standard Q-learning and maintains robust performance under user mobility and dynamic clustering (Elsayed et al., 2020). UCB-TQL for composite MDPs readily generalizes knowledge across high-dimensional tasks with sparse differences (Chai et al., 1 Feb 2025).
• Credit Assignment: Trajectory-space smoothing for dense guidance rewards within Q-learning, as in (Gangwani et al., 2020), yields dramatic improvements in learning speed and reliability under sparse/delayed reward environments.
• Economic Games and Pricing: Tabular TQL (tabular Q-learning) repeatedly converges to collusive or dispersed pricing equilibria, yielding highly supra-Nash prices, instability, and sensitivity to learning rate and information accessibility. Competition with DRL agents, such as PPO or DQN, attenuates collusion and restores price competition (Deng et al., 4 Jun 2024, Deng et al., 14 Mar 2025).

5. Advantages, Limitations, and Comparative Summary

Dimension	TQL (Trajectory Q-Learning)	Classical Q-Learning	DRL (PPO/DQN)
Policy Structure	History/trajectory-dependent (non-Markovian)	Markovian, state-action	Markovian, function approx
Risk Optimization	Global, unbiased for general distortion risks	Only for mean, biased for CVaR, etc.	Policy-gradient, robust
Transfer Ability	Efficient transfer via mapping, composite MDPs	None; slow convergence	Flexible, scalable
Collusion Tendency	High in tabular economic settings (supra-competitive outcomes)	High (tabular)	Low (policy gradient)
Credit Assignment	Dense, immediate via trajectory smoothing	Sparse/delayed: slow	Depends on reward struct.
Sample Efficiency	High in mobile/transfer/sparse-reward contexts	Low with large state/action spaces	High with function approx
Robustness	High in dynamic/risk-sensitive/transfer environments	Modest; sensitive to parameters	High (with proper design)

A plausible implication is that TQL’s principal merits lie in settings requiring global risk measures, transfer efficiency, rapid convergence in dynamic resource management, and dense temporal credit assignment, whereas tabular instantiations pose collusion and instability risks in economic games. TQL’s non-Markovian architecture and transfer mechanisms position it distinctly from classical Q-learning and deep policy-gradient approaches.

6. Contextualization and Future Research

Recent research on TQL highlights limitations in classical RL for risk-sensitive objectives, the efficacy of history-conditioned critics and policy representation, and the criticality of transfer mechanisms in complex, high-dimensional domains. The contrast between results in risk-sensitive RL (Zhou et al., 2023), transfer RL (Chai et al., 1 Feb 2025, Elsayed et al., 2020), and pricing games (Deng et al., 4 Jun 2024, Deng et al., 14 Mar 2025) underscores the breadth and variability of TQL’s manifestations. The expansion of TQL into deep RL architectures and function-approximation domains—for effective generalization, scalable learning, and robust policy development—remains an ongoing avenue for investigation, with theoretical guarantees and practical algorithms evolving to accommodate multi-task, risk-sensitive, and dynamic environments.