Multi-Objective Temporal Masking (MOTM)

• Multi-Objective Temporal Masking (MOTM) is a masked reconstruction strategy that uses distinct masking patterns to instill varied temporal reasoning modes in time series models.
• It employs multiple masking techniques—predictive, point, multi-block, single-block, and mixed—to mimic extrapolation, interpolation, and global abstraction in a unified pretraining framework.
• By integrating point-wise tokenization with a U-shaped multi-scale Transformer in Zeus, MOTM efficiently supports forecasting, imputation, anomaly detection, and classification without task-specific tuning.

Multi-Objective Temporal Masking (MOTM) is a masked reconstruction training strategy introduced in Zeus, a unified tuning-free Time Series Foundation Model (TSFM), to support heterogeneous time series analysis tasks within a single pretraining framework. MOTM was designed to expose the model to multiple corruption patterns, each corresponding to a distinct temporal reasoning mode—extrapolation, interpolation, and global abstraction—while training all of them through a shared quantile reconstruction loss on masked positions only. In Zeus, MOTM functions as the pretraining core that enables a single model to address forecasting, imputation, anomaly detection, and classification without task-specific fine-tuning (Fu et al., 2 Jul 2026).

1. Problem setting and motivation

Zeus is framed around a tuning-free objective: it is intended to work out-of-the-box across multiple downstream tasks without task-specific adaptation. The paper identifies two major obstacles to that goal. The first is a granularity-versus-scalability dilemma. Many prior TSFMs use patch tokenization to reduce sequence length and improve efficiency, but this sacrifices point-level temporal detail, which is harmful for imputation, especially point-missing imputation, and for anomaly detection, where precise local deviations matter. Zeus therefore adopts point-wise tokenization to preserve fidelity, but combines it with a multi-scale hierarchy so that long sequences remain tractable (Fu et al., 2 Jul 2026).

The second, and more central, obstacle is that downstream tasks require heterogeneous inductive biases. Forecasting needs extrapolation; imputation needs interpolation; anomaly detection needs sensitivity to both local variation and global consistency; classification requires global abstraction, often with shapelet-like local patterns too. The paper explicitly argues that a single monolithic objective such as BERT-style masked reconstruction or GPT-style autoregressive generation is insufficient to induce all of these behaviors simultaneously. MOTM was introduced to address this training dilemma by using multiple masking patterns within one unified masked reconstruction framework (Fu et al., 2 Jul 2026).

A common misconception is to treat MOTM as a generic masking heuristic. In Zeus, it is more specific than that: the masking patterns are chosen because they correspond to distinct operational behaviors that the model must learn during pretraining. This suggests that MOTM is best understood as a mechanism for inducing task-relevant temporal reasoning modes, rather than merely as stochastic corruption.

2. Formal definition and objective

MOTM is a masked reconstruction strategy in which Zeus reconstructs masked values using a quantile loss computed only on masked positions. For each training instance, the overall corruption ratio is sampled from

$p \sim \mathcal{U}(0, 0.5),$

and the masked budget is then defined as

$L_p = \lfloor Tp \rfloor,$

where $T$ is the sequence length. The paper states that the temporal scope is chosen to cover short-, mid-, and long-term dependencies, after which one masking strategy is sampled from a set of strategies (Fu et al., 2 Jul 2026).

For a binary mask $\mathcal{M} \in \{0,1\}^T$, the reconstruction loss is

$$\mathcal{L}=\frac{1}{|Q||\mathcal{M}|}\sum_{t:\mathcal M_t=1} \sum_{q\in Q} \begin{cases} q(y_t-\hat{y}_t^q), & \hat{y}_t^q \leq y_t, \ (1-q)(\hat{y}_t^q-y_t), & \hat{y}_t^q > y_t. \end{cases}$$

Here, $Q$ is the set of quantile levels, $\hat{y}_t^q$ is the prediction at quantile $q$, and the loss is evaluated only on masked positions. Zeus uses nine quantiles,

$Q = \{0.1,0.2,\dots,0.9\}.$

This formulation makes MOTM more than a standard representation-learning objective. It combines masking with probabilistic reconstruction. A plausible implication is that the quantile-based target encourages the pretrained model to encode uncertainty structure as part of its generic time series representation, although the paper presents this primarily as a practical design choice rather than a formal uncertainty theory (Fu et al., 2 Jul 2026).

3. Masking strategies and induced temporal behaviors

MOTM contains five masking strategies. Each one instantiates a different corruption pattern and is intended to train a different inductive bias.

The predictive mask removes a suffix of the sequence. Given probability $p$, the last

$L_p = \lfloor Tp \rfloor,$0

time steps are masked. This is the forecasting-style pattern and is intended to teach extrapolation and long-range causal prediction. Operationally, the model observes an unmasked prefix and must predict the masked suffix (Fu et al., 2 Jul 2026).

The point mask randomly masks individual time steps. This targets point-wise interpolation and local continuity. Because missing positions are isolated, reconstruction depends heavily on nearby context and point-level fidelity (Fu et al., 2 Jul 2026).

The multi-block mask removes several contiguous segments. The paper specifies the following procedure: given budget $L_p = \lfloor Tp \rfloor,$1, sample block lengths $L_p = \lfloor Tp \rfloor,$2, continue sampling until

$L_p = \lfloor Tp \rfloor,$3

and then randomly distribute the blocks along the sequence. This simulates structured missingness and forces interpolation under nontrivial gaps (Fu et al., 2 Jul 2026).

The single-block mask removes one long contiguous segment from an arbitrary position. This is intended to encourage global consistency. Because a large region is absent, local micro-pattern matching alone is insufficient, so the model must infer the broader regime and reconstruct a segment consistent with overall context. The paper identifies this as especially helpful for contextual anomaly detection and classification (Fu et al., 2 Jul 2026).

The mixed mask combines multiple masking strategies, pairing simpler masks such as multi-block and point mask with harder masks such as predictive and single-block. This yields a composite corruption regime with varying difficulty (Fu et al., 2 Jul 2026).

The mapping from mask type to behavior is explicit in Zeus. Predictive masking corresponds to future extrapolation; point and multi-block masking correspond to interpolation under local or structured missingness; single-block masking corresponds to holistic or global reconstruction. MOTM’s central claim is therefore not merely that diversity of masking helps, but that different mask families teach different temporal competencies within one pretraining objective.

4. Architectural coupling in Zeus

MOTM is tightly coupled to Zeus’s architecture rather than being an architecture-agnostic add-on. Zeus first applies point-wise tokenization, preserving full temporal detail. This is particularly important because MOTM includes point masking, which would be poorly matched to patch tokenization. The architectural design thus directly supports the granularity required by the masking objective (Fu et al., 2 Jul 2026).

Zeus then uses a U-shaped multi-scale Transformer with a fine-to-coarse-to-fine hierarchy. At fine scale, local detail is preserved; at coarse scale, global dependencies are captured efficiently; and the upsampling path refines fine-grained structure. The paper states that MOTM and this hierarchy were designed together. Point-mask and multi-block mask benefit from fine-scale representations, while single-block and predictive mask benefit from coarse-scale abstractions and long-range context (Fu et al., 2 Jul 2026).

In this co-design, MOTM supplies the training signals that require both local and global reasoning, and the U-shaped hierarchy supplies the representational capacity to realize those behaviors efficiently. Zeus presents this as solving two dilemmas jointly: the architecture dilemma of granularity versus scalability, and the training dilemma of incompatible inductive biases across tasks. MOTM addresses the latter, while the multi-scale hierarchy addresses the former (Fu et al., 2 Jul 2026).

This suggests that MOTM should not be interpreted independently of Zeus’s tokenization and hierarchy. In Zeus, the masking scheme and the representational scaffold are mutually reinforcing components of a single pretraining design.

5. Training pipeline and implementation

Although the paper does not provide a standalone pseudocode block specifically for MOTM, it describes a clear training pipeline. For each training sequence, Zeus first samples sequence length or crop from a piecewise uniform distribution: $L_p = \lfloor Tp \rfloor,$4 from $L_p = \lfloor Tp \rfloor,$5, $L_p = \lfloor Tp \rfloor,$6 from $L_p = \lfloor Tp \rfloor,$7, and $L_p = \lfloor Tp \rfloor,$8 from $L_p = \lfloor Tp \rfloor,$9. It then samples the corruption ratio $T$0, computes the mask budget $T$1, selects one masking strategy from predictive, point, multi-block, single-block, or mixed, replaces the selected positions with $T$2, feeds the masked sequence into Zeus, and computes the quantile reconstruction loss only at masked positions over nine quantiles (Fu et al., 2 Jul 2026).

Implementation details that are explicitly relevant to MOTM include the use of AdamW, $T$3 training steps, batch size $T$4, maximum context length $T$5, FlashAttention, and exclusion of padded tokens from attention. The paper also notes that Zeus uses a variable masking ratio because prior work often uses a fixed ratio, and Zeus intentionally avoids overfitting to a single missingness level (Fu et al., 2 Jul 2026).

These details matter because MOTM is intended to cover a broad range of dependency scales and missingness regimes. The variable sequence lengths and variable masking ratios mean that the pretraining signal is not tied to one temporal scale or one corruption severity. A plausible implication is that this broadens the support of the learned representation across downstream task conditions, although the paper’s direct claim is limited to avoiding overfitting to a single missingness level.

6. Empirical contribution, scope, and related objectives

The paper reports an ablation study in which one masking strategy is removed at a time while the expected masking ratio is held fixed by reallocating probability mass to the remaining strategies. Removing the predictive mask causes a clear drop on GIFT-Eval forecasting, which the paper interprets as confirmation that predictive masking is crucial for extrapolation and forecasting precision. Removing the multi-block mask degrades imputation, indicating its importance for interpolation and local pattern reconstruction. Removing the single-block mask consistently degrades anomaly detection and classification, supporting the claim that single-block reconstruction teaches global consistency. The empirical conclusion is that each masking type contributes a different capability, and the combined objective is better than any single masking regime alone (Fu et al., 2 Jul 2026).

Additional evidence is provided through comparison with patch-based TSFMs. The paper states that patch-based models such as MOMENT degrade sharply when moving from patch-missing pretraining assumptions to point-missing imputation. In a separate comparison, MOMENT’s imputation performance drops by about $T$6 MSE and $T$7 MAE on average when evaluated under point-level missing instead of patch-level missing. This motivates Zeus’s combination of point-wise tokenization and MOTM (Fu et al., 2 Jul 2026).

The paper also identifies several limitations. Zeus uses a channel-independent strategy for multivariate time series, so MOTM operates mostly on univariate channels independently; multivariate dependencies are not explicitly modeled, and future work may integrate methods like CoRA. Zeus is evaluated on five downstream task families, but tasks such as time series segmentation and causal discovery are not yet supported or explored. Classification remains challenging: fully non-parametric classification via 1-NN is strong but not always state of the art against fully supervised baselines, partly because of very short and high-dimensional UEA sequences and highly heterogeneous label semantics. Finally, MOTM is presented as a practical unified masking strategy rather than as a theoretically derived optimal objective (Fu et al., 2 Jul 2026).

In relation to adjacent work, MOTM differs from temporal multi-objective formulations that combine reconstruction and future prediction without explicit masking. The paper “Temporal Shift -- Multi-Objective Loss Function for Improved Anomaly Fall Detection” defines a windowed objective comprising reconstruction loss on overlapping frames and prediction loss on future frames, with $T$8, and explicitly characterizes the method as a combination of shifting, forecasting, and reconstruction rather than masking in the strict transformer sense (Denkovski et al., 2023). That comparison clarifies MOTM’s specificity: MOTM is a masked modeling strategy for time series pretraining, whereas Temporal Shift is a multi-objective windowed loss for video anomaly detection. The resemblance lies in their shared attempt to train multiple temporal behaviors within one objective; the difference lies in the mechanism by which those behaviors are induced.

Taken together, the available evidence presents MOTM as the pretraining engine of Zeus: a family of masking schemes unified by quantile-based masked reconstruction, designed to induce extrapolation, interpolation, and global abstraction in one model, and empirically validated through task-specific ablations and broad tuning-free evaluation (Fu et al., 2 Jul 2026).