Mixture-of-Horizon Training

• Mixture-of-Horizon Training is a method that supervises models over varying temporal segments, balancing precise local control with long-term predictive stability.
• It is applied in contexts such as robotic action chunking and turbulence forecasting to improve robustness and overall performance.
• The approach employs techniques like horizon-specific fusion and stride-conditioned operators, yielding superior metrics in complex sequential tasks.

Searching arXiv for the cited papers to ground the article in current preprints. Mixture-of-horizon training denotes a class of temporal learning strategies in which a model is trained over multiple horizons rather than a single fixed prediction length. In the literature, the idea appears explicitly as “Mixture of Horizons (MoH)” for action-chunked vision-language-action policies and, in a domain-specific form, as multi-stepsize training for turbulence neural operators, where the model learns a family of stride-conditioned advancement operators instead of one fixed-step map. The common motivation is that horizon choice is structurally consequential: short horizons favor local accuracy and stabilizability, whereas long horizons favor foresight or shallower rollout depth, but neither regime is uniformly optimal across tasks or phases of execution (Jing et al., 24 Nov 2025, Pan et al., 14 Apr 2026).

1. Core definition and motivating trade-offs

In action-chunked robotic policies, the horizon is the action chunk length $H$. At timestep $t$, the policy predicts a chunk

$$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$

The reported trade-off is explicit: longer horizons provide stronger global foresight but degrade fine-grained accuracy, while shorter horizons sharpen local control yet struggle on long-term tasks. Mixture-of-horizon training addresses this by exposing the policy to multiple chunk lengths simultaneously instead of forcing a single global choice of $H$ (Jing et al., 24 Nov 2025).

In turbulence forecasting, the analogous control variable is the relative stride $s$, with one-step and multi-step advancement written as

$u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$

Here the trade-off is non-monotonic. If the sampling interval is too large, the mapping becomes harder because snapshots are weakly correlated; if the interval is too small, rollouts become very deep and repeatedly compose nearly redundant transitions, which can worsen stability. The multi-stepsize formulation therefore treats stable long-horizon prediction at fine temporal resolution as requiring learning more than one step size (Pan et al., 14 Apr 2026).

A concise unifying interpretation is that mixture-of-horizon training replaces a single temporal discretization choice with a supervised distribution over temporal scales. In one case the scale is the number of future control actions predicted jointly; in the other it is the temporal separation between fluid states.

2. Formal problem formulations

For action-chunking, the multi-horizon construction begins from a maximum horizon $H$ and a candidate set

$$\mathcal{H} = \{h_1, \dots, h_N\}, \quad h_1 < \dots < h_N = H.$$

Given a ground-truth chunk $A_t = (a_{t,1}, \dots, a_{t,H})$, the method forms truncated variants

$$A_t^{(h)} = (a_{t,1}, \dots, a_{t,h}) \in \mathbb{R}^{h \times d_a}.$$

These horizon-specific views are then processed in parallel and later fused. The important point is that supervision spans several valid prefixes of the same future action segment, so the model is jointly exposed to short- and long-range structure (Jing et al., 24 Nov 2025).

For turbulence, the formulation is not expressed as chunk truncation but as a family of stride-parameterized operators. The exact operator family is

$t$0

with

$t$1

The learned approximation is written

$t$2

and the training objective averages data-fidelity over sampled strides: $t$3 There is no explicit multistep unrolling in the training loss; rather, the model is directly supervised on multiple temporal separations. This makes the method very close to mixture-of-horizon training conceptually, even though the paper frames the horizon variable as requested relative stride $t$4 (Pan et al., 14 Apr 2026).

These two formulations differ in representation but coincide in objective structure: both replace single-horizon supervision with a distribution over admissible temporal spans.

3. Architectural realizations

In the VLA setting, MoH is a plug-and-play strategy for full-attention action transformers. Each truncated chunk is padded back to length $t$5, and a horizon-specific attention mask disables positions $t$6. This permits all horizons to be processed in one batched forward pass, while the VLM prefix or contextual encoding is computed only once. For each $t$7, the shared action transformer produces hidden states $t$8 and horizon-specific action predictions $t$9. A single linear gate then computes per-step horizon weights

$$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$0

and fuses the valid horizon outputs as

$$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$1

The paper explicitly interprets this as a horizon-wise mixture-of-experts in which horizons, rather than separate subnetworks, act as experts for each action position (Jing et al., 24 Nov 2025).

In turbulence, the multi-horizon mechanism is realized through a mixture-of-experts neural operator built on IFactFormer-m. The lifting and projection modules remain outside the mixture; specialization occurs in latent space. The architecture contains a shared expert $$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$2, routed experts $$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$3, and a stride-indexed corrector $$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$4, combined as

$$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$5

$$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$6

$$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$7

$$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$8

Routing is conditioned explicitly on stride through a log-scale parameterization,

$$A_t = (a_{t,1}, \dots, a_{t,H}) = (a_t, \dots, a_{t+H-1}) \in \mathbb{R}^{H \times d_a}.$$9

with Gaussian routing scores

$H$0

followed by top-$H$1 sparse activation. The reported setting is $H$2, yielding smooth overlap between nearby dyadic scales while keeping computation sparse (Pan et al., 14 Apr 2026).

The contrast is instructive. MoH in robotics keeps the backbone shared and mixes horizon-specific predictions at the output level. Ms-MoE in turbulence embeds the horizon variable inside conditional computation and learns a scale-specialized operator family. Both are valid realizations of multi-horizon supervision, but they occupy different points in the design space of parameter sharing and specialization.

4. Objectives and deployment regimes

For VLA policies, MoH optimizes a composite loss

$H$3

Here $H$4 is the loss on the fused prediction, $H$5 is the sum of losses on individual horizon branches, and $H$6 is a balance term intended to prevent collapse to a preferred horizon. The reported coefficients are $H$7 and $H$8. The balance regularizer uses horizon-usage statistics and penalizes uneven usage via the coefficient of variation,

$H$9

At inference time, the same framework supports a dynamic mode based on cross-horizon consensus: disagreement between horizon-specific actions and the fused action is monitored stepwise, a threshold is estimated from the first $s$0 steps, and only the stable executable prefix is committed before replanning. The paper describes this as a self-truncating executable chunk (Jing et al., 24 Nov 2025).

For turbulence, the training procedure samples a stride $s$1 uniformly from $s$2, samples a valid time index $s$3, and predicts $s$4 from the input window. At inference, the same network can be queried at any admissible stride,

$s$5

and long trajectories are then generated by repeated autoregressive composition at the chosen stride: $s$6 This deployment model differs from the robotic setting in that there is no output fusion across multiple strides during a single prediction. Instead, multi-horizon training yields a single stride-conditioned operator family that can be queried flexibly at rollout time (Pan et al., 14 Apr 2026).

A plausible implication is that mixture-of-horizon training has at least two operational modes. One mode performs explicit multi-horizon fusion during prediction; the other uses multi-horizon supervision to learn a conditional family of predictors, with horizon selection deferred to deployment.

5. Empirical record

Setting	Representative outcomes	Source
VLA robotic action chunking	$s$7: 93.8 $s$8 95.1; $s$9: 97.7 $u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$0 99.0; $u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$1: 95.2 $u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$2 96.4 on LIBERO; dynamic inference yields 2.5$u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$3 higher throughput	(Jing et al., 24 Nov 2025)
Turbulence neural operators	More stable long-horizon rollouts and improved agreement with long-time-averaged statistics on turbulent channel flow and HIT; training data at up to 20 times finer temporal resolution than previous studies	(Pan et al., 14 Apr 2026)

In VLA manipulation, the reported benchmarks include LIBERO, RoboTwin 2.0, and real-robot tasks. LIBERO comprises four suites—Spatial, Object, Goal, and Long—with 10 tasks and 500 demonstrations per suite. Under the mixed-task setting, $u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$4 with MoH reaches 99\% average success on LIBERO after only 30k training iterations, and the paper reports especially strong gains on the Long suite. Horizon-density ablations further show that a single horizon $u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$5 gives 97.7 average success, $u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$6 gives 98.3, and the denser set $u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$7 gives 99.0. On real-robot tasks, the qualitative observations reported are faster, more decisive grasps, less dithering, better long-horizon planning, and better low-level precision (Jing et al., 24 Nov 2025).

In turbulence, the experiments use filtered DNS data for turbulent channel flow and forced homogeneous isotropic turbulence. The channel-flow study considers $u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$8 and $u^{n+1}=G_1(u^n), \qquad u^{n+s}=G_s(u^n).$9, where these are reported as $H$0 and $H$1 finer than the earlier $H$2, respectively. Across channel flow and HIT, the recurring pattern is that FNO becomes unstable or produces NaNs under deep rollout, IFactFormer remains usable but accumulates structural or statistical bias, and Ms-MoE-IFactFormer retains better structural fidelity and better agreement with long-time-averaged statistics such as $H$3, $H$4, RMS fluctuation profiles, kinetic energy spectra, and PDFs of normalized longitudinal velocity increments and normalized vorticity magnitude. The reported parameter counts are 53.1M for FNO, 0.9M for IFactFormer, and 1.4M at $H$5 or 2.2M at $H$6 for Ms-MoE-IFactFormer (Pan et al., 14 Apr 2026).

Taken together, these results support a domain-general interpretation: mixture-of-horizon training is motivated not merely by forecasting farther ahead, but by improving the robustness of the learned predictor under free-running deployment.

6. Scope, boundaries, and adjacent formulations

The term should not be applied indiscriminately to every long-horizon or horizon-aware model. A useful boundary case is FaST, a heterogeneity-aware Mixture-of-Experts framework for long-horizon spatial-temporal graph forecasting. FaST uses a core HA-MoE module, a heterogeneity-aware router with spatial and temporal expert biases, and adaptive graph agent attention to support horizons $H$7. However, it is explicitly not a “Mixture-of-Horizon Training” method in the sense of mixing multiple forecast horizons in the objective; the mixture mechanism addresses heterogeneity across nodes and temporal contexts, and the model is trained end-to-end on the chosen horizon using Huber loss rather than a dedicated horizon-mixture objective (Zhao et al., 8 Jan 2026).

A second nearby but distinct formulation is Horizon Imagination for diffusion world models in reinforcement learning. Its central contribution is an on-policy imagination process that denoises multiple future observations in parallel, together with a stable discrete action-sampling mechanism and a Horizon schedule that decouples generation horizon $H$8, denoising budget $H$9, and decay horizon $$\mathcal{H} = \{h_1, \dots, h_N\}, \quad h_1 < \dots < h_N = H.$$0, including sub-frame budgets where $$\mathcal{H} = \{h_1, \dots, h_N\}, \quad h_1 < \dots < h_N = H.$$1. This can reasonably be viewed as a kind of multi-budget, horizon-scheduled imagination framework, but the paper also states that it is not a classical mixture-of-horizons objective because it does not mix multiple environment rollout lengths in the loss (Cohen et al., 8 Feb 2026).

The resulting taxonomy is narrow but clear. In the strict sense, mixture-of-horizon training refers to training on multiple temporal horizons rather than only one. Explicit realizations may fuse horizon-specific predictions, as in robotic action chunking, or learn a single conditional family of horizon- or stride-parameterized operators, as in turbulence forecasting. By contrast, methods that are merely long-horizon, horizon-aware, or schedule-aware do not automatically qualify unless multi-horizon supervision is part of the training formulation itself.