OccamVTS: Distilled Vision for Forecasting

• The paper introduces OccamVTS, a knowledge distillation framework that transfers useful low-level textural features from large vision models to lightweight student networks.
• It employs a dual-pathway architecture combining temporal encoding and visual augmentation with pyramid-style alignment, correlation, and feature distillation losses.
• Empirical results show that aggressive parameter compression improves forecasting performance across long-term, few-shot, and zero-shot scenarios by filtering out irrelevant high-level semantics.

OccamVTS is a knowledge distillation framework for time series forecasting that transfers predictive information from large vision models (LVMs) into lightweight student networks with approximately $1\%$ of the original parameters. The method is motivated by the observation that visual encoders can improve forecasting when time series are rendered as images, yet most of the transferred visual capacity is unnecessary or detrimental because time series align with low-level textural features rather than high-level semantics. OccamVTS therefore uses pre-trained LVMs as frozen privileged teachers and distills only the temporal and textural components that remain useful for forecasting, combining pyramid-style feature alignment with correlation and feature distillation losses (Lyu et al., 3 Aug 2025).

1. Research context and problem setting

Time series forecasting is fundamental to diverse applications, and a recent line of work applies LVMs to forecasting by converting numerical sequences into visual representations. The rationale is partly perceptual: humans often interpret time-series plots visually, identifying edges, textures, trends, and periodic structure from the curve morphology itself. In the same vein, modern vision backbones, including ViTs and CNNs pre-trained on ImageNet, are effective at extracting low-level textural features such as edges, frequency patterns, and gradients; these properties are also present in visualized time series, including recurrence plots and spectrograms. Prior works named in this context include TimesNet, VisionTS, and Time VLM, all of which convert numerical sequences into 2D “images” and fine-tune or adapt LVMs for improved forecasting accuracy (Lyu et al., 3 Aug 2025).

OccamVTS is formulated as a response to a specific inefficiency in that paradigm. The central claim is not that vision transfer is ineffective, but that its benefits are concentrated in a very small subspace of the teacher model. The framework reports that while vision models enhance forecasting performance, $99\%$ of their parameters are unnecessary for time series tasks, and that aggressive compression can improve accuracy by removing overfitting to irrelevant visual features. In this formulation, the teacher is treated as a source of privileged supervision rather than as the deployment model itself.

2. Cross-modal premise: low-level texture versus high-level semantics

A defining premise of OccamVTS is that time series are visually closer to repetitive textures than to semantically rich natural images. The supporting analysis uses t-SNE on MAE features extracted from ImageNet images and from benchmark time-series datasets including ECL, Weather, Electricity, and ETT. In that feature space, time-series plots cluster near images of repetitive textures such as meshes and curtains, while remaining far from semantically rich scenes such as waterbirds and parachutes (Lyu et al., 3 Aug 2025).

The reported conclusion is that LVM early layers, which act as edge and frequency detectors, align with time series, whereas deeper semantic features do not and can impair forecasting by overfitting to irrelevant concepts. This observation is structurally important because it changes the purpose of cross-modal transfer. Instead of adapting a full semantic vision model to a temporal task, OccamVTS isolates the low-level inductive bias that appears transferable and suppresses the higher-level representations that act as semantic noise.

A common misunderstanding in visually augmented forecasting is that stronger vision pretraining should monotonically improve downstream forecasting. OccamVTS explicitly reports the opposite: larger transferred semantic capacity can be counterproductive when the target signal is purely numerical and forecasting-relevant structure resides in edges, textures, and frequency components. The framework’s compression strategy is therefore not merely an efficiency measure; it is part of the predictive hypothesis.

3. Architecture and teacher–student pipeline

OccamVTS is a three-stage teacher–student distillation pipeline that transforms time series into dual representations, uses a frozen large vision backbone as a privileged teacher, and trains a tiny student with specialized distillation objectives (Lyu et al., 3 Aug 2025).

The cross-modal representation module has two pathways. The temporal pathway uses a 1D patch-embedding followed by a Transformer encoder to capture sequence patterns. The visual pathway performs “plot→image” augmentation through FFT, periodic encodings, 1D→2D convolutions, and bilinear interpolation. This dual design preserves direct temporal modeling while also exposing the series to visual feature extraction.

The teacher model is a large pre-trained vision network $V^T(\cdot)$, with examples including MAE-Base, CLIP, and ResNet-101. Teacher visual features are fused with temporal embeddings through cross-attention, after which the teacher generates forecasts $Y^T$ and attention maps $P^T$. The teacher remains frozen during training.

The student model uses a lightweight backbone $V^S(\cdot)$, with examples including Tiny-ViT, EfficientNet-B0, and MobileNet-V3. It shares the same fusion mechanism and forecasting head as the teacher but operates at reduced dimensionality. In the high-level training loop, the model computes temporal embeddings $H_t$, applies visual augmentation to obtain $I_{\mathrm{vis}}$, forms fused teacher and student representations, predicts $Y^T_{\mathrm{pred}}$ and $Y^S_{\mathrm{pred}}$, and optimizes a weighted sum of task and distillation losses. Student parameters and distillation parameters are updated, whereas the teacher remains fixed.

4. Multi-scale alignment and distillation objectives

The technical core of OccamVTS lies in three complementary losses: pyramid-style feature alignment, correlation distillation, and feature distillation. These are combined with the forecasting task loss into a single objective (Lyu et al., 3 Aug 2025).

For pyramid-style feature alignment, the teacher and student provide multi-scale fused representations $99\%$0 and $99\%$1, where $99\%$2 and $99\%$3. Two learned projections,

$99\%$4

map teacher and student features into a shared latent space. The alignment loss is

$99\%$5

By matching multiple scales, the student is encouraged to preserve both fine- and coarse-grained structure.

Correlation distillation operates on self-attention maps $99\%$6. With temperature $99\%$7, the loss is

$99\%$8

This transfers temporal dependency patterns rather than only matching pointwise outputs.

Feature distillation compares the student’s aligned feature $99\%$9 with the teacher’s fused feature $V^T(\cdot)$0: $V^T(\cdot)$1 The summary assigns distinct roles to the three terms: the MSE term preserves absolute magnitudes of predictive cues, the cosine term emphasizes directional alignment and filters scale noise, and the KL term aligns output distributions and further filters spurious semantic activations. Together, $V^T(\cdot)$2 is described as transferring temporal and textural knowledge while suppressing high-level semantic noise.

The total objective is

$V^T(\cdot)$3

with

$V^T(\cdot)$4

The weights $V^T(\cdot)$5 are learnable and are initialized, for example, to $V^T(\cdot)$6.

5. Parameter compression, optimization, and data protocol

OccamVTS is explicitly designed around extreme compression. Teachers such as MAE-Huge are described as having $V^T(\cdot)$7 and approximately $V^T(\cdot)$8M parameters, while students based on Tiny-ViT, EfficientNet-B0, or MobileNet-V3 use $V^T(\cdot)$9 and approximately $Y^T$0–$Y^T$1M parameters. By selecting $Y^T$2 and pruning most attention heads and MLP channels, the student retains roughly $Y^T$3 of the teacher’s weights (Lyu et al., 3 Aug 2025).

The summary also gives a mathematical intuition for this design: if the predictive subspace of dimension $Y^T$4 captures at least $Y^T$5 of the variance of the teacher’s fused features, then projection into that low-dimensional subspace preserves almost all forecasting information while discarding bulky semantic components. This is presented as intuition rather than as a proved theorem.

Training uses AdamW with initial learning rate $Y^T$6 and weight decay $Y^T$7, batch size $Y^T$8, a frozen teacher, and updates to the student and distillation parameters. The temperature $Y^T$9 is learned in $P^T$0 via a sigmoid transform, and early stopping is applied after $P^T$1 epochs of no validation improvement. The data protocol specifies $P^T$2 train, $P^T$3 validation, and $P^T$4 test for long-term forecasting; $P^T$5 train for few-shot experiments; and cross-dataset transfer for zero-shot evaluation.

These details matter because OccamVTS does not frame compression as post hoc model pruning. Instead, compression is embedded in the student architecture and in the distillation objective from the outset, with the teacher serving as a privileged but non-deployed source of supervision.

6. Empirical results, ablations, and implications

The reported empirical evaluation covers long-term, few-shot, zero-shot, and short-term forecasting. Long-term experiments use horizons $P^T$6 on ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, and Traffic, with MSE and MAE as metrics. Few-shot evaluation uses the same datasets with $P^T$7 training data. Zero-shot evaluation uses cross-transfer among ETT variants. Short-term evaluation uses the M4 competition with horizons $P^T$8–$P^T$9 and metrics SMAPE, MASE, and OWA (Lyu et al., 3 Aug 2025).

The main quantitative claims are specific. In long-term forecasting, OccamVTS achieves the best MSE on all seven datasets; on ETTh2, the reported result is $V^S(\cdot)$0 versus PatchTST at $V^S(\cdot)$1, corresponding to $V^S(\cdot)$2. In few-shot evaluation, with only $V^S(\cdot)$3 of the training data, the framework still outperforms strong baselines on $V^S(\cdot)$4 datasets by $V^S(\cdot)$5–$V^S(\cdot)$6. In zero-shot transfer between ETT variants, the reported performance drop is less than $V^S(\cdot)$7, compared with up to $V^S(\cdot)$8 for other methods. In short-term forecasting on M4, the reported values are $V^S(\cdot)$9 versus ETSformer at $H_t$0, and $H_t$1 versus $H_t$2.

Ablation studies are consistent with the dual-pathway and multi-loss design. Removing the visual pathway makes MSE $H_t$3 worse on Weather. Removing the temporal encoder yields $H_t$4 worse MSE. Dropping either $H_t$5 or $H_t$6 causes a $H_t$7–$H_t$8 loss in accuracy. Teacher-only models without a student and student-only models without knowledge distillation both underperform the full system by $H_t$9–$I_{\mathrm{vis}}$0.

Efficiency results report teacher sizes up to $I_{\mathrm{vis}}$1M parameters and student sizes between $I_{\mathrm{vis}}$2M and $I_{\mathrm{vis}}$3M, corresponding to approximately $I_{\mathrm{vis}}$4–$I_{\mathrm{vis}}$5 of teacher scale. Inference is $I_{\mathrm{vis}}$6–$I_{\mathrm{vis}}$7 faster on ETTh1, with less than $I_{\mathrm{vis}}$8 memory reduction. The framework also reports the counterintuitive outcome that pruning $I_{\mathrm{vis}}$9 of vision parameters improves generalization by removing overfit semantic features.

The broader implication is that the useful transferable content of vision pretraining for forecasting may be concentrated in low-level textural and frequency-sensitive representations rather than in category-level semantics. The summary also identifies limitations. OccamVTS still depends on a visual augmentation pipeline, which may need adaptation for irregularly sampled or multivariate series. It may incur information loss when rare high-level patterns become predictive in domain-specific tasks, with ECG wave shapes given as an example. Proposed future directions include distillation from multi-modal foundation models such as CLIP with text prompts, mixture-of-experts teachers specialized for different seasonalities or domains, and incorporation of additional modalities such as audio or tabular metadata through joint distillation.