Horizon Adaptive Offline Policy Learning via Value Stitching

Abstract: Learning accurate value functions plays a decisive role for reinforcement learning (RL) agents to solve long-horizon, complex tasks. Conventional temporal-difference (TD) learning objectives suffer from value-estimation bias that accumulates over the horizon, while extended-horizon modeling methods, such as n-step TD backups and Q-chunking, adopt a rigid, fixed-horizon value-modeling recipe that is often not flexible enough to capture complex value structures in long-horizon, multi-stage tasks. In this paper, we show that enabling value updates with dynamic horizon composition can yield a strong offline policy learning scheme. Our method, Horizon Adaptive Offline Policy Learning via VAlue STitching (VAST), replaces fixed-horizon backups with recursive, horizon-adaptive value composition. Its key ingredient is to couple value optimization with a future state- and horizon-length-conditioned auxiliary value function that is learned through direct data supervision, and a stitching policy that optimally selects the reward-maximizing horizon length and future sub-goal to achieve horizon-adaptive value stitching. This design enables direct estimation and compositional "stitching" of variable-length returns grounded in actionable sub-goal states, providing an accurate and greedily exploitable value-supervision signal for offline policy optimization. Across 50 tasks on OGBench, VAST outperforms fixed-step, extended-horizon methods, and generative-value offline RL baselines, achieving strong performance particularly in high-complexity, long-horizon decision-making tasks.

• The paper introduces VAST, a horizon-adaptive offline RL method that decouples value learning from fixed-step backups to enable dynamic, horizon-conditioned value composition.
• It leverages a two-level policy with diffusion-based high-level stitching and goal-conditioned behavioral cloning for improved performance on complex, long-horizon tasks.
• Empirical results on OGBench show VAST outperforms state-of-the-art baselines by up to 2.5x, confirming its efficacy in manipulation and navigation tasks.

Horizon Adaptive Offline Policy Learning via Value Stitching

Motivation and Problem Statement

The value estimation in reinforcement learning (RL) is fundamentally constrained by temporal-difference (TD) learning protocols, particularly under long-horizon, complex tasks where value-estimation bias accumulates due to recursive bootstrapping. Extended-horizon methods (e.g., n-step TD, action chunking) improve supervision but enforce rigid, fixed-horizon modeling, impeding flexibility required for adaptive temporal decisions. Hierarchical and option-based frameworks provide temporal abstraction yet remain structurally and computationally heavy, often restricted to two discrete temporal scales. Thus, the challenge persists: achieving robust, horizon-adaptive value modeling and policy extraction in offline RL, especially in regimes where compositional planning and dynamic credit assignment are critical.

Methodology: Value Stitching with Temporal Adaptivity

The core contribution is VAST (Horizon Adaptive Offline Policy Learning via Value Stitching), which systematically decouples value learning from fixed-step backups through recursive horizon-adaptive value composition. This instantiates a horizon-conditioned auxiliary value function $G(s, s^+, k)$, learned both via Monte Carlo regression and temporal compositional consistency, and a stitching policy $\pi_{\text{stitch}}$ parameterized by expressive diffusion/flow models.

The G-function formalizes the expected discounted return between any two reachable states under a specific horizon, supporting variable-length return estimation and anchoring value supervision in actionable sub-goals. Temporal compositionality is established theoretically, enabling decomposition of long-horizon returns across intermediate states—a principle critical for segmental planning and robust value propagation. The stitching Bellman operator then leverages this G-function to identify optimal horizon-lengths and sub-goal states, facilitating adaptive backup and contractive value iteration.

Policy extraction under VAST is two-level: the high-level $\pi_{\text{stitch}}$ proposes optimal horizon-target pairs $(k, s_k)$, while the low-level execution policy $\pi_{\text{exec}}$ performs goal-conditioned behavioral cloning, further refined via test-time rejection sampling for reward maximization. Both policies are parameterized as flow/diffusion networks to maximize model expressivity and facilitate scalable planning.

Empirical Evaluation

Evaluations are conducted on OGBench—a comprehensive benchmark for offline goal-conditioned RL spanning 50 tasks (30 basic, 20 hard) across manipulation and navigation domains. VAST is compared to 8 baselines (including IQL [35], FQL [52], QC [17], VALUE-FLOWS [21], FLOQ [20]), covering single/multi-step TD, chunked value, and generative-value approaches.

Key empirical findings:

• Superior performance in long-horizon, high-complexity tasks: VAST achieves best or near-best aggregate success rates across nearly all domains, with pronounced gains in manipulation (scene, cube, puzzle) and large/giant antmaze navigation.
• Pronounced gains on hard domains: VAST outperforms all baselines, showing an overall score over 2.5x higher than leading alternatives in complex planning settings.
• Sensitivity to horizon and compositionality: Ablations reveal that performance increases with horizon limit $K$ up to task-dependent thresholds, after which overly extended horizons may degrade estimation quality due to increased modeling requirements. Compositional weight $\lambda$ enhances G-function generalization, but excessive regularization is suboptimal.
• Benefit of two-level reward maximization: Test-time rejection sampling at the execution level consistently increases success rates, validating the hierarchical reward maximization design.

Theoretical Foundation and Implications

VAST instantiates horizon-adaptive value learning grounded in temporal compositionality, rigorously proved via contraction mapping properties. This enables bootstrapping on optimal horizon-target pairs, mitigating error propagation inherent to fixed-step TD and unlocking robust, greedy value supervision. By decoupling value estimation from fixed backups and employing temporally compositional objectives, VAST effectively addresses local optima in planning, preserves critical short-range signals, and scales to complex, multi-stage tasks.

Practical implications include significantly improved scalability and flexibility for offline RL in robotics, sequential decision-making, and real-world settings where heterogeneous temporal demands arise. The horizon-adaptive paradigm also provides a theoretical bridge between classical hierarchical/option frameworks and lightweight, expressive generative policy classes.

Future Directions

The horizon-adaptive architecture of VAST is readily extensible to advanced generative policy optimization (e.g., more aggressive diffusion methods [46, 76]), model-based RL, and deep representation learning architectures (Transformers/MoEs). Further research should explore the integration of VAST with these directions to enhance robustness, controllability, and generalization in offline RL. Computational efficiency remains a consideration, but the trade-off for increased flexibility and performance is justified.

Conclusion

VAST delivers a principled and practical advancement in offline RL by enabling dynamic horizon composition for value estimation and policy extraction. Strong empirical results, especially on complex long-horizon benchmarks, establish its efficacy and scalability. The theoretical underpinnings guarantee stable convergence and compositional robustness. Future research should further harness horizon adaptivity for advanced policy optimization and scalable real-world deployment.

---

Reference:

"Horizon Adaptive Offline Policy Learning via Value Stitching" (2606.21136)