Time2Vec: Neural Temporal Embedding

• Time2Vec is a model-agnostic vector representation that converts scalar time into a learnable k+1 dimensional embedding combining linear trends with periodic patterns.
• It integrates seamlessly with architectures like RNNs, Transformers, and CNNs, enhancing performance in tasks such as forecasting, recommendation, and biosignal analysis.
• Empirical studies demonstrate up to 15% accuracy improvement and robust performance by optimizing the fusion of linear and periodic components.

Time2Vec is a model-agnostic vector representation for encoding scalar time indices in neural networks. Unlike raw timestamps or hand-engineered temporal features, Time2Vec provides a learnable, k+1-dimensional embedding that jointly captures non-periodic trends and periodic patterns such as seasonality or cycles. It is compatible with a wide range of architectures—including RNNs, Transformers, and CNNs—and is applicable across tasks ranging from financial forecasting and recommendation systems to biosignal analysis and gesture recognition (Kazemi et al., 2019, Ma et al., 3 Feb 2025, Bui et al., 18 Apr 2025, Hristov et al., 2 Feb 2026).

1. Formal Definition and Mathematical Structure

Time2Vec maps a scalar time $τ∈ℝ$ to an embedding vector $\phi(τ)∈ℝ^{k+1}$. The first component is a linear function capturing non-periodic drift, while the remaining $k$ components are periodic functions (generally sine) with learnable frequencies and phase shifts:

$\phi(\tau)[0] = \omega_0\,\tau + \varphi_0$

$$\phi(\tau)[i] = \sin(\omega_i\,\tau + \varphi_i), \quad i=1,\dots,k$$

where $\omega_0,\dots,\omega_k \in ℝ$ are learnable frequencies and $\varphi_0,\dots,\varphi_k \in ℝ$ are learnable phase offsets. All parameters are optimized end-to-end with the downstream model's loss.

Key properties include:

• Adaptive periodicity: Each learned frequency $\omega_i$ allows the model to recover any periodic cycle (e.g., weekly, yearly, arbitrary) directly from the data.
• Non-periodic trend: The linear term ($\omega_0 \tau$) enables modeling of smooth drifts and extrapolative behavior.
• Rescaling invariance: Shifting the time unit (e.g., days $\rightarrow$ seconds) can be compensated by scaling $\omega_i$.
• Plug-and-play: The embedding can be concatenated or added to arbitrary features, requiring no change to model architecture (Kazemi et al., 2019).

2. Integration in Neural Architectures

Time2Vec serves as a universal temporal embedding, directly substitutable for scalar time inputs in neural architectures:

• RNNs/LSTMs: Concatenate $\phi(\tau)$ to the input at each time step (Kazemi et al., 2019).
• CNNs: Append $\phi(\tau)$ along the channel dimension for time-indexed data (Kazemi et al., 2019).
• Transformers: Replace fixed sinusoidal positional encodings with learnable Time2Vec embeddings, either by concatenation or addition to token embeddings (Bui et al., 18 Apr 2025, Hristov et al., 2 Feb 2026).

Example integration in a PyTorch-style module:

class Time2Vec(nn.Module): def __init__(self, k): super().__init__() self.k = k self.omega = nn.Parameter(torch.randn(k+1)) self.phase = nn.Parameter(torch.randn(k+1)) def forward(self, t): lin = self.omega[0] * t + self.phase[0] t_rep = t.expand(*t.shape[:-1], self.k) w_rep = self.omega[1:].view(*([1]*(t.dim()-1) + [self.k])) p_rep = self.phase[1:].view(*([1]*(t.dim()-1) + [self.k])) per = torch.sin(w_rep * t_rep + p_rep) return torch.cat([lin.unsqueeze(-1), per], dim=-1)

(Kazemi et al., 2019, Hristov et al., 2 Feb 2026)

Several downstream pipelines refine this basic integration:

• Attention-enhanced variants: Post-Time2Vec, apply multi-head self-attention to the temporal embeddings, as in deep sequence models for forecasting (Ma et al., 3 Feb 2025).
• Normalized fusion: In low-density sensor applications, normalize both the spatial and temporal latent outputs before additive fusion to prevent destructive interference (Hristov et al., 2 Feb 2026).

3. Empirical Performance and Comparative Studies

Time2Vec has been benchmarked across synthetic, recommender, financial, and biosignal datasets.

Original study findings (Kazemi et al., 2019):

• Replacing raw time with Time2Vec embeddings always improved or matched reference performance, with no deterioration observed.
• Gains of ~10–15% accuracy on challenging classification tasks (e.g., event-based MNIST, audio spike classification).
• Up to ~5% recall@10 improvement on real-world recommendation data (e.g., Last.FM, StackOverflow).
• Periodic activations (sine, triangle, modulo) consistently outperform non-periodic alternatives.
• Inclusion of the linear term is particularly beneficial for long-horizon, asynchronous, or non-stationary sequences.

Financial forecasting (Ma et al., 3 Feb 2025, Bui et al., 18 Apr 2025):

• In Bitcoin transaction fee forecasting, Time2Vec and Time2Vec+Attention provided moderate improvements, but underperformed relative to traditional models (e.g., SARIMAX, Prophet) in limited data settings. This suggests neural time embeddings may require larger training sets for strong generalization (Ma et al., 3 Feb 2025).
• In equity index prediction, a Transformer Encoder with Time2Vec outperformed fixed sinusoidal positional encodings, LSTM, and RNN baselines by 9.3% RMSE and required only one third as many parameters. Multi-feature selection based on cross-correlation further improved accuracy, yielding statistically significant gains in prediction metrics when using Time2Vec (Bui et al., 18 Apr 2025).

Biosignals and sEMG-based gesture recognition (Hristov et al., 2 Feb 2026):

• Integration of Time2Vec into transformer-based models for two-channel sEMG achieved an F1-score of 95.7% ± 0.20%, surpassing both conventional transformers (fixed encoding) and recurrent CNN-LSTM models.
• Architectural optimization indicated that a balanced split between spatial and temporal capacity, with normalized additive fusion, yields the most stable and accurate results.

4. Design Choices and Implementation Guidelines

Best practices for deploying Time2Vec in neural pipelines, as established across studies:

• Embedding dimension $(k+1)$: Recommended to select $k \in \{16,32,64\}$; larger $k$ (e.g., $k=64$) offers more expressivity but may encourage overfitting or instability on short sequences (Kazemi et al., 2019). In recent works, $k=64$ is standard (Ma et al., 3 Feb 2025, Bui et al., 18 Apr 2025, Hristov et al., 2 Feb 2026).
• Parameter initialization:
  • Frequencies: $\omega_i \sim U(0,1)$ or $\mathcal{N}(0,0.1)$, to avoid premature high-frequency oscillations.
  • Phases: $\varphi_i \sim U(0,2\pi)$ or $\mathcal{N}(0,0.1)$.
  • Linear term: $\omega_0=1.0$, $\varphi_0=0$ (Kazemi et al., 2019).
• Optimizer and regularization: Use standard optimizer (Adam) and learning rate schedule from the base network; moderate weight decay ($10^{-5}$) can curtail overfitting in small-data settings (Kazemi et al., 2019, Hristov et al., 2 Feb 2026).
• Activation function: Default to $\sin$; alternative periodic activations (triangle, mod) may be used if justified by domain prior, but sine generally provides the broadest empirical utility (Kazemi et al., 2019).
• Fusion strategy: For architectures combining spatial and temporal embeddings, independently normalize each before additive fusion to avoid magnitude mismatch (Hristov et al., 2 Feb 2026).
• Curriculum training: For noisy or limited data, a two-stage regime—aggressive data augmentation followed by fine-tuning—enhances generalization (Hristov et al., 2 Feb 2026).
• Rescaling check: Confirm rescaling invariance by verifying that changing the time unit (e.g., days $\rightarrow$ seconds) does not affect model output, owing to adaptive $\omega_i$ (Kazemi et al., 2019).

5. Theoretical and Practical Rationale

Time2Vec is motivated by the limitations of scalar- or hand-engineered time features, which do not expose periodic structure and require domain-specific tuning. By learning both the linear drift and an adaptive bank of periodic components with trainable frequencies and phases, Time2Vec allows the model to:

• Discover latent cycles or seasonality directly from data, without manual feature design.
• Support non-periodic and extrapolative trends via the linear term.
• Adjust to non-stationary or warped temporal grids, as demonstrated in gesture recognition under speed/acceleration variation (Hristov et al., 2 Feb 2026).
• Maintain model performance under unit or scale changes, enabling plug-and-play replacement in diverse application domains (Kazemi et al., 2019).

In Transformer-based architectures, replacing rigid positional encoding with Time2Vec significantly improves the capacity to model both short- and long-range dependencies, especially when combined with multi-feature selection and attention (Bui et al., 18 Apr 2025).

6. Notable Variants and Applications

Several studies have developed and evaluated variants of the Time2Vec paradigm:

• Time2Vec with self-attention: Applying attention mechanisms over sequences of Time2Vec embeddings to refine temporal context representations for forecasting (Ma et al., 3 Feb 2025).
• Multi-feature Time2Vec: Aggregating highly correlated related features via normalized cross-correlation and geometric mean before Time2Vec embedding for multi-asset forecasting (Bui et al., 18 Apr 2025).
• Normalized additive fusion: Independent layer normalization of spatial and temporal streams prior to combination, improving robustness in sensor modalities where spatial resolution is limited (Hristov et al., 2 Feb 2026).
• Task diversity: Time2Vec has been applied to discrete event modeling (synthetic periodicity, event-based MNIST), sequential recommendation systems (StackOverflow, Last.FM), financial asset prediction (indices, cryptocurrencies), and biosignals (sEMG gesture classification) (Kazemi et al., 2019, Ma et al., 3 Feb 2025, Bui et al., 18 Apr 2025, Hristov et al., 2 Feb 2026).

Typical results indicate that learned, flexible temporal representations obtained by Time2Vec outperform both fixed basis encodings (sinusoidal) and architectures with only raw time inputs in sequence modeling tasks.

7. Limitations and Future Directions

Limitations observed in current literature include:

• Data regime sensitivity: In time series with limited historical data, high-parameter Time2Vec-based models (especially when paired with attention and deep MLP heads) can suffer from overfitting and high estimation variance, occasionally underperforming statistical baselines (e.g., SARIMAX) (Ma et al., 3 Feb 2025).
• Domain shift sensitivity: In biosignal contexts, direct transfer between subjects produces accuracy degradation, but rapid calibration protocols can quickly restore performance (Hristov et al., 2 Feb 2026).
• Parameter tuning: The selection of embedding dimension, activation function, and initialization is moderately task-dependent and benefits from targeted empirical tuning and ablation (Kazemi et al., 2019, Bui et al., 18 Apr 2025, Hristov et al., 2 Feb 2026).

Potential directions include adaptive sparsification of the periodic basis, hybridization with domain-specific temporal kernels, and further integration with advanced feature selection pipelines.

---

Time2Vec provides a trainable, model-agnostic basis for learning smooth, periodic, and non-periodic temporal dependencies, offering consistent improvements over fixed or raw time features in neural sequence modeling when deployed with appropriately scaled data and capacity (Kazemi et al., 2019, Ma et al., 3 Feb 2025, Bui et al., 18 Apr 2025, Hristov et al., 2 Feb 2026).