URTF Benchmark Overview

• URTF Benchmark is a comprehensive evaluation framework that standardizes long-term forecasting by integrating synthetic, simulated, and real-world datasets.
• It employs a diverse set of baseline models, including LSTM, Transformer, and linear methods, with unified evaluation protocols and metrics.
• Empirical findings demonstrate that simpler linear approaches can outperform complex models under certain conditions, challenging conventional forecasting assumptions.

URTF (“Unified Long-Term Time-Series Forecasting”) is a comprehensive benchmark established to systematically assess the performance and generalizability of time-series forecasting methodologies over very long horizons and across heterogeneous domains. The benchmark consists of carefully curated and standardized datasets covering synthetic, simulated, and real-world temporal dynamics, alongside a unified evaluation protocol, diverse model baselines, and detailed empirical findings. URTF serves as a foundational testbed for the development and comparative analysis of forecasting models capable of handling intricate, multi-scale, and high-dimensional time-series data (Cyranka et al., 2023).

1. Dataset Suite and Standardization

URTF aggregates eight principal datasets from three broad domains:

• Robotic Control (MuJoCo-based simulations): Half-Cheetah (17D), Walker2D (17D), and Hopper (11D) comprise continuous-control benchmark tasks.
• Synthetic Chaotic Dynamics: Kuramoto–Sivashinsky PDE (100 spatial modes) and Mackey–Glass (scalar, chaotic) provide challenging nonlinear dynamics.
• Real-World Sensing:
  • Electricity: Load from 321 feeders.
  • Traffic: Flow from 862 sensors.
  • ETT: Hourly energy/time dataset (7 variables).
  • Weather: 21 meteorological variables from station logs.

Each component is partitioned into standardized trajectories of up to 2,000 points. Training (M₁=8,000) and test (M₂=2,000) splits are constructed for all but the real-world sensor datasets, which are chronologically segmented (first 70% for training, last 30% for testing). All data are min-max rescaled to [0,1] and further standardized (zero mean, unit variance) by variable. Sequences are extracted via sliding windows:

• At time $T$:
  • Lookback $L$: $X_T = s_{T-L+1:T}$
  • Forecast horizon $H$: $Y_T = s_{T+1:T+H}$
  • with $L+H = N$ (window length). Training and evaluation are exclusively conducted on disjoint subsets.

2. Model Baselines and Custom Extensions

URTF includes a spectrum of methods:

• LSTM: Standard two-layer, 256-unit recurrent models, direct multi-step ($\ell_2$) loss.
• DeepAR: Autoregressive probabilistic LSTM with Gaussian outputs and curriculum training for increasing forecast horizons.
• NLinear: Single linear projection from input windows $\mathbb{R}^{L \times D}$ to $\mathbb{R}^{H \times D}$.
• N-HiTS: Hierarchical time-series module using residual stacks (5 stacks, 2 layers each, ReLU activation).
• PatchTST: Transformer model with patch embedding (patch size=16, stride=8), 4 heads, 512-dim embedding, 6 layers.
• LatentODE: Continuous-time model with GRU-ODE encoder, Gaussian VAE latent state, and NODE (Dormand–Prince) decoder.
• Latent NLinear (Editor's term): Custom architecture that encodes $X$ into a latent initial state $L$0 and produces forecasts via a learned linear mapping: $L$1.
• Curriculum DeepAR: Horizon grows over epochs: at epoch $L$2, $L$3. This facilitates stable training on long sequences.

Batch size, number of epochs, and lookback/horizon settings are standardized: batch size 128, 50 epochs, $L$4, $L$5.

3. Evaluation Protocols and Metrics

Performance is quantified using:

• Mean Squared Error (MSE):

$L$6

• Mean Absolute Error (MAE):

$L$7

• Relative $L$8 Error (for KS PDE):

$L$9

All results represent the mean and standard deviation over 5 independent random seeds (both data shuffling and parameter initialization).

4. Empirical Findings and Analysis

Several major trends emerged from systematic URTF evaluation:

• Linear Models (NLinear, N-HiTS): These outperform RNN and Transformer models for long-horizon forecasting ($X_T = s_{T-L+1:T}$0) on deterministic or weakly stochastic systems (MuJoCo, KS PDE), sometimes achieving half the MSE of deep alternatives.
• Real-World Series: DeepAR and PatchTST are most effective for high-variance, noisy temporal data and short lookback windows ($X_T = s_{T-L+1:T}$1); linear models are competitive as $X_T = s_{T-L+1:T}$2 grows.
• Continuous-Time Models: LatentODE achieves superior error on irregularly sampled series, especially in the presence of missing data, notably with the KS PDE and high drop-out.
• Latent NLinear Extension: Demonstrates further 5–10% MSE improvements for MuJoCo and KS PDE, showing the benefit of combining linear projection with learned latent embeddings.
• Dataset-Dependent Outcomes: Linear or mildly nonlinear dynamics favor simple models; attention and nonlinearity become essential for complex, heteroskedastic observations. Transformers risk overfitting if $X_T = s_{T-L+1:T}$3 is not carefully set.

5. Benchmark Implications and Recommendations

URTF reveals that:

• Simple Multi-Step Linear Models provide unexpectedly strong baselines for long-horizon tasks, especially under stable or low-dimension dynamics.
• Continuous-Time ODE Solvers are necessary for irregular-sampling or dynamically flexible prediction.
• Transformer Models (PatchTST) excel with moderate horizon and noise, but over-parameterization for extreme horizons is discouraged.
• No Universal Best Model: Users should validate several families due to the strong interaction between task statistics, lookback size, and model design.

Recommended directions include expansion to episodic and multi-modal series, Bayesian or generative uncertainty quantification, non-autonomous/delayed dynamics modeling, and hybrid architectures that adaptively determine complexity (Cyranka et al., 2023).

6. Significance and Role in the Literature

URTF is positioned as the first unified benchmark for evaluating long-term time-series forecasting over a rich set of domains and temporal scales. It enables reproducible, apples-to-apples performance comparisons. The demonstration of linear methods’ efficacy challenges prevailing assumptions regarding model complexity, anchoring further inquiry on the interplay between data properties, model architecture, and forecast horizon. URTF informs both application-specific benchmarking (e.g., in energy or traffic analytics) and the theoretical development of time-series models.

7. Summary Table of Datasets and Models

Dataset Domain	Example(s)	Model Type
Robotic Control	Half-Cheetah, Walker2D, Hopper	LSTM, NLinear, PatchTST, LatentODE, N-HiTS
Synthetic Dynamics	KS PDE, Mackey–Glass	LSTM, LatentODE, Linear variants
Real-World Sensing	Electricity, Traffic, Weather, ETT	DeepAR, PatchTST, N-HiTS, NLinear

This structure facilitates rigorous, multi-faceted evaluation and substantiates future development of robust, scalable forecasting algorithms across science and engineering domains (Cyranka et al., 2023).