Quasimetric Reinforcement Learning (QRL)

• Quasimetric Reinforcement Learning (QRL) is an approach that uses asymmetric distance functions to capture directional costs in control and planning tasks.
• It enforces the triangle inequality in value functions, enhancing multi-step trajectory estimation and sample efficiency through methods like IQE and NbQl.
• QRL applications span robotics, transfer learning, and offline planning, demonstrating improved performance in environments with irreversible dynamics.

Quasimetric Reinforcement Learning (QRL) is an approach to reinforcement learning that incorporates quasimetrics—distance functions that are not necessarily symmetric but satisfy the triangle inequality—in the representation and learning of value functions, state costs, and planning objectives. Harnessing quasimetric structure allows QRL to model directional asymmetries fundamental to control, planning, and navigation in environments with irreversible dynamics or asymmetric costs. This article reviews key QRL principles, algorithmic formulations, geometric and representation-theoretic advances, sample efficiency analyses, practical applications, and emerging directions.

1. Quasimetric Foundations and Value Function Geometry

In goal-conditioned RL, the optimal cost-to-go (value function) $V^\ast(s, g)$ for reaching a goal $g$ from state $s$ is inherently asymmetric. Recent theoretical work has demonstrated that $-V^\ast(s, g)$ is a quasimetric: it satisfies the triangle inequality ($d(s, g) \leq d(s, s') + d(s', g)$) and the identity ($d(x, x) = 0$), but generally not symmetry (Wang et al., 2023). This geometric property justifies constraining learned value functions to the space of quasimetrics. Several neural architectures, such as Interval Quasimetric Embeddings (IQE) (Wang et al., 2022), are designed to enforce such constraints directly in latent space.

The triangle inequality regulator is particularly fundamental: it ensures compositionality of costs across multi-step trajectories, prevents the value function from underestimating long-range costs, and reflects the realities of irreversible dynamics (e.g., up/downhill asymmetry in navigation (Hossain et al., 22 Oct 2024), one-way doors in maze RL (Wang et al., 2022), or asymmetric transitions in classical control).

2. Algorithmic Formulations and Metric Space Extensions

A range of RL algorithms have extended Q-learning to handle quasimetrics and continuous (metric) state-action spaces:

• Net-based Q-learning (NbQl): Constructs an $\varepsilon$-net $N_\varepsilon$ of state-action pairs for efficient update and exploration. All state-action pairs are mapped to their nearest net point via a metric $D$, and Q-values/visit counts are maintained only for the net points. This leads to sample complexity that depends on metric covering number and dimension, not the raw size of the state-action space (Song et al., 2019).
• Multi-task RL with Planning Quasi-Metric (PQM): Separates dense, unsupervised learning of an asymmetric BeLLMan quasi-metric $f(s, s', a)$—the expected minimal steps to reach $s'$ from $s$ via $a$—from fast, task-specific selection of target states ("aimers") inside goal sets. The PQM generalizes across tasks, providing rapid policy transfer for differently defined goals (Micheli et al., 2020).
• Adversarial Intrinsic Motivation (AIM): Uses the expected time-to-goal as a quasimetric to ground the Wasserstein-1 distance between the agent's state-visitation and goal distributions, incentivizing policies that minimize expected steps to the goal (Durugkar et al., 2021). This distance is learned adversarially via the dual Kantorovich potential, yielding an intrinsic reward that smoothly reflects the agent's progress.
• Projective Quasimetric Planning (ProQ): In offline RL, ProQ learns an IQE-style quasimetric to encode directional reachability, uses it as both a repulsive energy (for uniform keypoint coverage) and a directional cost (for short-horizon graph-based planning), and couples the geometry with an out-of-distribution detector to restrict sub-goal selection to reachable regions (Kobanda et al., 23 Jun 2025).

3. Representation Learning and Embedding Structures

Rigorous quasimetric modeling in RL requires embedding methods that preserve triangle inequality and identity, while efficiently representing asymmetry:

• IQE (Interval Quasimetric Embedding): IQE reshapes latent vectors into interval matrices and computes asymmetric union lengths per dimension; aggregated via sum or max-mean reductions. IQE guarantees triangle inequality, positive homogeneity, and can universally approximate any quasimetric (Wang et al., 2022).
• Metric Residual Network (MRN): Decomposes the quasimetric value function into symmetric and asymmetric terms (e.g., $$Q(s,a,g) = -[d_{\text{sym}}(h_{sa}, h_{sg}) + d_{\text{asym}}(h_{sa}, h_{sg})]$$), suitable for both sparse and dense reward settings (Valieva et al., 13 Sep 2024).
• Quasimetric Embeddings in Navigation: In QuasiNav, the state-action cost function is embedded via an asymmetric norm, $|v|_H = w^\top \max(v, 0) + (1-w)^\top \max(-v, 0)$, allowing explicit modeling of directional terrain costs (Hossain et al., 22 Oct 2024).

4. Sample Efficiency, Dense Rewards, and Theoretical Guarantees

The efficiency of QRL arises from two main sources: (i) structural inductive bias due to the triangle inequality constraint, and (ii) improved feedback signal via metric-based generalization or dense rewards:

• Sample Complexity Bounds: NbQl demonstrates that cumulative regret scales as $O(H^2 T^{(d+1)/(d+2)})$ for an optimal $\varepsilon$-net, nearly matching bandit lower bounds for Lipschitz metrics (Song et al., 2019). QRL achieves exact or approximate recovery of optimal value under dual constrained maximization over quasimetric functions—even in complex environments (Wang et al., 2023).
• Dense Reward Setting: Contrary to earlier claims that dense rewards worsen sample complexity in goal-conditioned RL, recent results show that triangle inequality and quasimetric structure are preserved under potential-based dense reward shaping, provided the shaping function is admissible ($\phi(s,a,g) \geq Q^\ast(s, a, g)$). This allows dense rewards to potentially improve sample efficiency without sacrificing structure (Valieva et al., 13 Sep 2024).

5. Applications and Empirical Validation

Quasimetric models have found use in several domains:

• Robotic Manipulation and Navigation: Task transfer accelerates via PQM models in bit-flip and MuJoCo robotic arm tasks (Micheli et al., 2020); QuasiNav provides robust, safe navigation in challenging real and simulated terrains via quasimetric embeddings and adaptive constraint tightening in CMDPs, outperforming symmetric-cost RL approaches in success rates, energy consumption, and safety (Hossain et al., 22 Oct 2024).
• Long-horizon Offline RL: ProQ leverages quasimetric geometry to discover uniformly covered, in-distribution keypoints, facilitating sub-goal planning and outperforming baselines on challenging PointMaze benchmarks (Kobanda et al., 23 Jun 2025).
• Multitask and Transfer Learning: Dense, unsupervised learning of environmental quasimetrics enables rapid adaptation to new goal definitions through lightweight aimers (Micheli et al., 2020).
• Quantum RL Paradigms: Although quantum RL is distinct, quantum search and superposition have been used for trajectory optimization and return calculation in quantum-MDP models (Su et al., 24 Dec 2024, Behera et al., 30 Jan 2025). These works illustrate the potential interplay between geometric RL and quantum-enhanced search or learning.

6. Open Directions and Theoretical Implications

Recent QRL advances suggest future work may focus on:

• Extending quasimetric learning to more general cost structures and risk-sensitive objectives (e.g., energy, safety, risk).
• Improved scalability through adaptive net-construction or combined continuous-discrete representations.
• Integration of OOD/out-of-manifold detection to ensure reliable sub-goal generation and planning, especially in offline and transfer RL.
• Exploration of optimal transport, adversarial imitation, and distribution-matching methods grounded in quasimetrics.
• Extension to online, multi-agent, or hierarchical RL settings, where directionality and asymmetric costs are inherent.

7. Summary Table: Core QRL Algorithmic Elements

Paper/Method	Quasimetric Structure	Sample Efficiency Mechanism
NbQl (Song et al., 2019)	$\varepsilon$-net over metric space	Regret bound via covering dimension
PQM+Aim (Micheli et al., 2020)	BeLLMan quasi-metric $f(s,s',a)$	Dense unsupervised updates, transfer
QRL (Wang et al., 2023)	Triangle inequality, IQE	Dual-constrained maximization
MRN (Valieva et al., 13 Sep 2024)	Symmetric + asymmetric decomposition	Dense reward shaping, admissibility
QuasiNav (Hossain et al., 22 Oct 2024)	Asymmetric terrain cost	CMDP with adaptive constraint tightening
ProQ (Kobanda et al., 23 Jun 2025)	IQE, repulsive keypoint loss, OOD detection	Graph-based planning over keypoints

Quasimetric Reinforcement Learning synthesizes geometric, representation-theoretic, and algorithmic innovations to address the challenges of directional asymmetry, efficient exploration, and robust planning in complex environments. By enforcing and leveraging the triangle inequality, QRL enables theoretically sound and empirically validated advances in sample efficiency, transfer generalization, and safe navigation.