Transitive RL: Value Learning via Divide and Conquer (2510.22512v1)

Abstract: In this work, we present Transitive Reinforcement Learning (TRL), a new value learning algorithm based on a divide-and-conquer paradigm. TRL is designed for offline goal-conditioned reinforcement learning (GCRL) problems, where the aim is to find a policy that can reach any state from any other state in the smallest number of steps. TRL converts a triangle inequality structure present in GCRL into a practical divide-and-conquer value update rule. This has several advantages compared to alternative value learning paradigms. Compared to temporal difference (TD) methods, TRL suffers less from bias accumulation, as in principle it only requires $O(\log T)$ recursions (as opposed to $O(T)$ in TD learning) to handle a length-$T$ trajectory. Unlike Monte Carlo methods, TRL suffers less from high variance as it performs dynamic programming. Experimentally, we show that TRL achieves the best performance in highly challenging, long-horizon benchmark tasks compared to previous offline GCRL algorithms.

• The paper introduces Transitive RL, a divide-and-conquer framework that reduces bias by cutting down Bellman recursions to O(log T) in long-horizon tasks.
• It leverages the triangle inequality and expectile regression to robustly approximate values, outperforming traditional TD and Monte Carlo methods.
• Empirical results and ablation studies demonstrate TRL’s scalable performance in offline GCRL, though its reliance on deterministic dynamics limits broader application.

Transitive RL: Value Learning via Divide and Conquer

Introduction and Motivation

Transitive Reinforcement Learning (TRL) introduces a divide-and-conquer paradigm for value learning in offline goal-conditioned RL (GCRL). The central challenge addressed is the curse of horizon, where bias accumulation in temporal difference (TD) learning impedes scalability to long-horizon tasks. TRL leverages the triangle inequality inherent in shortest-path problems to construct value update rules that, in theory, require only $O(\log T)$ Bellman recursions for a trajectory of length $T$, as opposed to $O(T)$ in standard TD methods. This approach aims to mitigate both bias accumulation (prevalent in TD) and high variance (characteristic of Monte Carlo methods), providing a scalable solution for long-horizon, goal-reaching tasks.

Figure 1: Transitive RL is based on the divide-and-conquer paradigm, which can in theory reduce the number of Bellman recursions to $O(\log T)$ in the best case, unlike TD-based methods.

Theoretical Framework

TRL operates in deterministic Markov processes, exploiting the triangle inequality property of temporal distances: for any states $s, w, g$, $d^*(s, g) \leq d^*(s, w) + d^*(w, g)$. This translates to the value function as $V^*(s, g) \geq V^*(s, w) V^*(w, g)$. The transitive Bellman update is defined as:

$$V(s, g) \gets \begin{cases} \gamma^0 & \text{if } s = g \ \gamma^1 & \text{if } (s, g) \in \mathcal{E} \ \max_{w \in \mathcal{S}} V(s, w) V(w, g) & \text{otherwise} \end{cases}$$

For function approximation, direct maximization over all subgoals leads to overestimation. TRL circumvents this by restricting maximization to in-trajectory subgoals and replacing the hard max with expectile regression, which approximates the maximum in a statistically robust manner. The value loss is weighted to prioritize shorter trajectory chunks, further stabilizing learning.

Implementation Details

TRL is implemented as follows:

1. Value Function Training: For each sampled trajectory, select indices $i < j$ and a random subgoal $k$ between them. Update $Q(s_i, a_i, s_j)$ using expectile regression on $Q(s_i, a_i, s_k) Q(s_k, a_k, s_j)$, with distance-based re-weighting.
2. Policy Extraction: After value learning, extract the policy via reparameterized gradients (DDPG+BC) or rejection sampling, depending on the task's multimodality.

Pseudocode for TRL is provided in the paper, and the method is compatible with large-scale datasets and high-dimensional continuous control environments.

Empirical Evaluation

TRL is evaluated on challenging long-horizon tasks (e.g., humanoid maze navigation, robotic puzzle solving) and standard OGBench benchmarks. The results demonstrate that TRL consistently matches or outperforms individually tuned TD-$n$ and MC baselines, without requiring task-specific tuning of $n$.

Figure 2: TRL matches the best, individually tuned TD-$\bm{n}$ baseline, without needing to set $\bm{n}$.

On standard OGBench tasks, TRL achieves competitive or superior performance compared to state-of-the-art GCRL algorithms, including those based on hard and soft triangle inequality constraints.

Figure 3: TRL achieves strong performance on standard OGBench tasks. While TRL is not specifically designed for short-horizon tasks, it outperforms or matches previous GCRL methods on a standard benchmark.

Ablation studies reveal that expectile regression ($\kappa > 0.5$), behavioral subgoal selection, and distance-based re-weighting are critical for stability and performance.

Figure 4: Ablation paper on the expectile $\bm{\kappa}$. Performance is sensitive to the expectile parameter, especially in long-horizon tasks.

Practical and Theoretical Implications

TRL provides a scalable value learning framework for offline GCRL, particularly effective in long-horizon domains. The divide-and-conquer approach offers logarithmic recursion depth, reducing bias accumulation and obviating the need for careful horizon-dependent hyperparameter tuning. The method is robust to dataset suboptimality, as behavioral subgoals suffice for effective learning.

However, TRL's reliance on deterministic dynamics and the triangle inequality limits its applicability in stochastic environments. Extending divide-and-conquer value learning to general reward-based RL and stochastic settings remains an open research direction. The successor temporal distance framework and planning-based extensions may offer pathways to address these limitations.

Conclusion

Transitive RL demonstrates that divide-and-conquer value learning, grounded in the triangle inequality, is a viable and effective strategy for scalable offline goal-conditioned RL. The empirical results substantiate its advantages over TD and MC paradigms, especially in long-horizon tasks. Future work should explore generalizations to stochastic environments and broader RL settings, potentially enabling scalable, bias-resistant value learning across a wider array of domains.