Dynamic Execution Horizon Prediction for Chunk-based Robot Policies

Abstract: Action chunking has become a standard design in modern robot policies, from diffusion/flow policies to vision-language-action models, where the policy predicts a sequence of actions and executes a fixed number of them instead of acting one step at a time. However, this paradigm relies on a key assumption: a fixed execution horizon. During chunk execution, the policy operates open-loop, which is particularly problematic for fine-grained manipulation tasks that require frequent replanning. In practice, the execution horizon is typically chosen through empirical tuning and is highly task-dependent. To this end, we propose Dynamic Execution Horizon Prediction (DEHP), an effective method that trains a lightweight execution-horizon prediction branch using online reinforcement learning while keeping the pretrained chunk policy completely frozen. This makes the method compatible with black-box chunk policies and isolates the effect of adapting the execution horizon from changes to the underlying action generator. Across our evaluations, DEHP improves the success rate of different high-precision and long-horizon manipulation tasks by a large margin. Our qualitative analysis further shows that DEHP predicts shorter execution horizons during fine-grained stages of the task and longer horizons during free-space motion. In this way, DEHP balances the efficiency of open-loop chunk execution with the reactivity of closed-loop single-step control. Project page: https://dehp-chunking.github.io/

• The paper introduces Dynamic Execution Horizon Prediction, a categorical policy head that selects how many actions to execute from each predicted chunk while keeping the base robot policy frozen.
• The method models variable-length execution as a semi-MDP and trains horizon selection with PPO, using step-consistent discounting and a state-only critic to preserve standard return estimates.
• DEHP improves success by 23.03% on average in noisy insertion tasks and raises FurnitureBench one-leg success from 70.30% to 95.18%, with short horizons during precision phases and longer horizons during transport.

Motivation and problem statement

Chunk-based policies—spanning ACT, Diffusion Policy, and most modern vision-language-action models—predict a sequence of future actions and execute a fixed number of them before replanning. This design improves temporal coherence and control stability, but it embeds a strong assumption: the execution horizon is fixed throughout the task and must be tuned empirically per task. The authors identify this as a structural weakness. A long execution horizon yields smooth, efficient open-loop behavior but poor reactivity; a short horizon increases responsiveness but sacrifices stability and efficiency. Because manipulation tasks alternate between free-space motion (where open-loop commitment is safe) and contact-rich fine-grained phases (where frequent replanning is essential), no single fixed horizon is appropriate across task phases.

The paper proposes Dynamic Execution Horizon Prediction (DEHP): a lightweight categorical head that, at each decision point, selects how many actions from the predicted chunk to execute, conditioned on the current observation and the full action chunk. Crucially, the pretrained base policy remains completely frozen, so DEHP is compatible with black-box chunk policies and cleanly isolates the effect of horizon adaptation from changes to the action generator.

Method

The joint policy factorizes as $$\pi(\mathbf{a}_{1:H}, h \mid s) = \pi_{\mathrm{act}}(\mathbf{a}_{1:H} \mid s)\,\pi_{\mathrm{len}}(h \mid s, \mathbf{a}_{1:H})$$, where $\pi_{\mathrm{act}}$ is any frozen chunk policy and $\pi_{\mathrm{len}}$ is trained online with RL using sparse binary rewards. The induced process over variable-length chunks is formulated as a semi-MDP.

Three technical components support the training procedure:

• Return invariance theorem: the paper proves that the chunk-level discounted objective $\mathbb{E}_\pi[\sum_k \gamma^{t_k}\bar{R}_k]$ exactly equals the base-MDP discounted return, because chunks are contiguous and cover every timestep once. This justifies evaluating the chunk-level policy with ordinary step-level returns.
• State-only critic: since chunks and horizons are sampled on-policy from the current policy, an action-conditioned critic over chunks (as required by off-policy Q-chunking methods) is unnecessary; a state-value critic suffices.
• Step-discounted GAE at chunk boundaries: the TD residual uses $\gamma^{h_k}$ to discount across a chunk of length $h_k$, and the GAE decay weights later residuals by elapsed environment time rather than by the number of chunk decisions, keeping credit assignment consistent with the base MDP.

PPO is applied directly to $\pi_{\mathrm{len}}$. Because the head is categorical, the importance ratio reduces to a difference of two log-softmax outputs, avoiding the likelihood-evaluation difficulties that motivate specialized objectives for diffusion or flow base policies in DPPO-style fine-tuning. The critic is distributional (201 atoms on $[0,1]$, two-hot projection), following recent evidence that classification losses improve value learning. The horizon head is initialized uniformly over $\{1,\dots,H\}$ rather than warm-started from a tuned fixed horizon, so the learned schedule is not biased toward any single execution length.

Experimental results

All experiments use a frozen MLP-based Diffusion Policy as the base, trained on state-machine-collected demonstrations with waypoint randomization and injected action noise. Evaluations cover four tasks: multi-stage peg insertion and bimanual needle–syringe insertion in IsaacLab, and one-leg and round-table assembly in FurnitureBench.

Multi-stage insertion under noise. The optimal fixed horizon depends on noise level: executing 6 actions per chunk is best under low noise, while 2 actions wins at noise level 0.15—direct empirical confirmation that a single fixed horizon cannot serve all conditions. DEHP achieves the highest overall success rate at every noise level tested, improving overall success by 23.03% on average over the best fixed-horizon BC baseline. At zero noise, DEHP reaches 93.17% overall success versus 71.50% for fixed-horizon BC; even at noise 0.15 it attains 61.43% versus 37.73%. Gains concentrate in Stage 1, which has the tightest clearance (2 mm), indicating that adaptive replanning matters most in precision-critical phases. Notably, DEHP outperforms every fixed-horizon baseline even when the base policy is trained on the largest demonstration set (1000 trajectories), which implies the performance gap stems from the fixed-horizon assumption itself rather than limited data.

Long-horizon assembly. On FurnitureBench one-leg, DEHP reaches 95.18% success versus 70.30% for the best fixed-horizon baseline. On round-table, DEHP lifts success from 29.90% to over 93.80% under low randomization and from 18.50% to 53.03% under medium randomization. An important practical finding: one-leg and round-table favor different fixed horizons despite identical controllers and similar demonstrations, underscoring that per-task manual tuning is unavoidable under fixed-horizon execution—tuning that DEHP eliminates by adapting online.

Bimanual needle–syringe insertion. On this high-precision bimanual task (3 mm needle opening, 2 mm syringe tip), the best fixed-horizon policy achieves only 10.20% success; DEHP raises the same frozen policy to 29.00%. This is the weakest absolute result in the paper and the authors concede that such precision-critical tasks may require combining horizon adaptation with low-level policy fine-tuning.

Learned horizon schedules

Qualitative analysis shows interpretable, phase-dependent schedules: DEHP predicts short horizons (often near 1 during the tightest insertion stage) during grasping, alignment, and insertion, and longer horizons during approach and transport. The same pattern appears in the bimanual task, with shortening during final mid-air alignment. These schedules reconcile the efficiency of open-loop chunk execution with the reactivity of closed-loop control, and they explain why gains concentrate in tight-tolerance stages.

Limitations and open questions

The paper is explicit about its boundaries. First, because the base policy is frozen, DEHP's ceiling is bounded by the quality of the pretrained chunk generator; combining dynamic horizons with low-level fine-tuning is left unexplored. Second, all experiments use state-based observations for controlled evaluation; extension to image-based policies is claimed to be straightforward given DEHP's black-box treatment of the base policy, but is not demonstrated. Third, the method requires online RL rollouts in the target environment, so its applicability to settings without fast simulators or safe exploration is not addressed. Finally, whether the learned horizon schedules transfer across tasks or must be relearned per environment remains an open question.

Conclusion

This paper reframes the execution horizon in chunk-based robot policies as a state-conditioned decision variable rather than a fixed hyperparameter. By formulating dynamic horizon selection as a semi-MDP and training only a small categorical head with chunk-level PPO against a frozen base policy, DEHP delivers large success-rate improvements—up to roughly 25 percentage points on assembly tasks and a 23% average gain under noise—without modifying the underlying action generator. The learned schedules align with task-phase reactivity requirements, providing both performance gains and interpretability. The main open direction is integrating horizon prediction with base-policy fine-tuning for tasks whose difficulty exceeds what replanning alone can recover.