Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings

Abstract: Transformer-based architectures have advanced sequence modeling in language and vision, yet general-purpose representation learning for heterogeneous multivariate time series remains underexplored. We introduce CHARM (Channel-Aware Representation Model), which incorporates channel-level textual descriptions into a Transformer encoder equivariant to channel order. CHARM is trained with a Joint Embedding Predictive Architecture (JEPA) and a novel loss promoting informative, temporally stable embeddings; latent-space prediction encourages robustness to sensor noise while description-aware gating provides interpretability through learned inter-channel relationships. Across anomaly detection, classification, and short- and long-term forecasting, the learned embeddings achieve strong performance using only a linear probe. Performance is driven primarily by the JEPA objective and conditioning architecture, with text descriptions serving as channel identifiers for cross-dataset generalization.

• The paper introduces CHARM, a 7.1-million-parameter channel-aware encoder that combines description-conditioned temporal convolutions and attention with JEPA latent prediction for heterogeneous multivariate time series.
• The model matches or exceeds much larger forecasting models while achieving 79.6% average UEA classification accuracy, 99.8% accuracy on UCI Hydraulic Systems, and a 0.754 adjusted F1 on 46 UCR anomaly datasets.
• The results show that JEPA provides the largest performance gain, while sensor descriptions mainly identify and separate channels rather than contribute substantial domain knowledge, with quadratic attention and metadata quality limiting scalability.

Overview

CHARM (Channel-Aware Representation Model) is a self-supervised foundation embedding model for heterogeneous multivariate time series that integrates textual channel descriptions directly into a Transformer encoder while remaining equivariant to channel ordering. The model is trained with a Joint-Embedding Predictive Architecture (JEPA), predicting latent representations of masked target segments rather than raw signal values, and is evaluated via lightweight downstream heads on classification, anomaly detection, and forecasting benchmarks (2605.31580). The central empirical claim is that a 7.1M-parameter encoder with frozen embeddings matches or exceeds substantially larger forecasting-specific foundation models such as Moirai-Large and Toto on several benchmarks.

Motivation and positioning

The authors argue that existing time series foundation models fall into two camps with complementary weaknesses. Forecasting-centric models (TimesFM, Moirai, Chronos, iTransformer) couple pretraining to a point-prediction loss, producing representations that are specialized and brittle for non-forecasting tasks. Reconstruction-based embedding models (MOMENT, UniTS) require imputing raw sensor values, which entangles semantic structure with sensor noise and domain-specific artifacts. JEPA-style latent prediction offers an alternative: by operating entirely in embedding space, it filters low-level noise and avoids the negative-sampling sensitivities of contrastive learning. A second gap the paper targets is channel-awareness: most multivariate models treat channels as anonymous streams, discarding metadata about sensor identity and semantics. CHARM addresses both gaps simultaneously, using channel descriptions not to align modalities (as in TENT or JoLT, which rely on instance-level text labels) but as per-sensor conditioning signals available at scale without instance-level annotation.

Architecture

CHARM processes an input tuple $(\mathbf{T}, \mathbf{D}, \mathbf{pos})$ of measurements, textual descriptions, and positional indices through two stages.

Contextual temporal convolutional network. A dilated TCN featurizes $\mathbf{T} \in \mathbb{R}^{T\times C}$ into per-time, per-channel embeddings $\mathbf{T}_c \in \mathbb{R}^{T\times C\times H}$. Description embeddings from a frozen text encoder condition the TCN in two ways: soft gates $$\mathbf{G}_c = \text{sigmoid}(\mathbf{E}_d \mathbf{W}_g)$$ modulate each convolutional layer per channel, controlling effective receptive field; and convolutional kernels themselves are generated from description embeddings ($\mathbf{G}_k = \mathbf{E}_d \mathbf{W}_k$). The authors position this as a generalized, channel-aware extension of patch-based featurization that avoids manual patch-size tuning across domains.

Contextual attention layers. The core Transformer block extends scaled dot-product self-attention over all $CT$ channel–time tokens with two description-conditioned additions: (i) an inter-channel gating matrix $$\mathbf{G}_d = \text{ReLU}(\mathbf{Z} - \mathbf{S})$$, where $\mathbf{S}$ is pairwise description similarity and $\mathbf{Z}$ a learned similarity threshold; with large $\lambda_G$, this acts as a learned mask suppressing cross-attention between dissimilar channels; and (ii) a symmetric inter-channel time-offset tensor $\mathbf{T} \in \mathbb{R}^{T\times C}$0 encoding dependency strength between channels at lag $\mathbf{T} \in \mathbb{R}^{T\times C}$1, constructed from concatenated description pairs so that $\mathbf{T} \in \mathbb{R}^{T\times C}$2 by construction. Rotary position embeddings are applied to queries and keys, broadcast across channels. Channel-order equivariance is verified empirically (max output difference $\mathbf{T} \in \mathbb{R}^{T\times C}$3 under random permutations across 8 datasets).

The full pretrained context encoder has 7.1M parameters; the additional TCN and text-attention modules contribute roughly 2M parameters (~25% of model size).

Self-supervised training

Training follows the I-JEPA recipe: a context encoder optimized by backpropagation, a target encoder updated by exponential moving average, and a narrow predictor composed only of contextual attention layers with learnable mask placeholder tokens. Two SSL tasks are used — causal prediction (context precedes target) and smoothing (overlapping context/target) — with uniform segment masking and channel-selective masking perturbations applied only to the context view, simulating sensor outages and network dropouts.

The loss combines three terms: an $\mathbf{T} \in \mathbb{R}^{T\times C}$4 discrepancy between predicted and target embeddings at fine granularity, plus progressively aggregated versions over channels and over channels-and-time, encouraging consistency at multiple resolutions. Two regularizers promote sparsity of inter-channel structure: $\mathbf{T} \in \mathbb{R}^{T\times C}$5 pushes the gating threshold matrix toward 1 (suppressing unnecessary cross-channel attention), and $\mathbf{T} \in \mathbb{R}^{T\times C}$6 penalizes the magnitude of time-offset dependencies. Pretraining used ~30 datasets spanning industrial machinery, wearables, ETT/Weather/Illness forecasting corpora, and UEA subsets, on 8×A100 GPUs for roughly 18 hours.

Downstream performance

Forecasting. On the LSF benchmark suite, frozen CHARM embeddings with per-dataset linear probes outperform billion-parameter models on ETTm2 (0.209 vs. 0.266 MSE for Toto) and ETTh2 (0.324 vs. 0.349), and remain competitive elsewhere. With end-to-end fine-tuning (CHARM+NLH FT), CHARM attains the lowest MSE on 3 of 5 datasets, including Weather at 0.222. The authors note MAE leaders skew toward models trained with MAE-aligned objectives, explaining their relative MAE gap. On Exchange Rate — excluded from pretraining — frozen CHARM with a linear head is competitive with supervised iTransformer, PatchTST, and DLinear, evidencing zero-shot transfer.

Classification. On a 21-dataset UEA subset, frozen CHARM + SVM achieves 79.6% average accuracy versus 78.5% for T-Rep and 77.6% for MiniROCKET; finetuning raises this to 80.9% with the most dataset wins (5). On the held-out UCI Hydraulic Systems benchmark, frozen embeddings with a linear SVM reach 99.8% valve-condition accuracy, exceeding the best published supervised baseline (96.0%, 1D-CNN).

Anomaly detection. On SKAB, CHARM achieves F1 0.86 with the lowest missed-alarm rate (12.69%) among representation-learning and classical baselines, though its false-alarm rate (19.35%) is higher than TS2Vec or T-Rep. On 46 UCR univariate anomaly datasets, CHARM averages adjusted F1 0.754 versus 0.684 for MOMENT and 0.627 for TimesNet, with 24 per-dataset wins.

Ablations and attribution

The ablation program isolates component contributions with a consistent finding: the JEPA objective is the single largest driver of performance. Replacing latent prediction with a quantile reconstruction loss under identical architecture and hyperparameters costs 4.4 percentage points of classification accuracy — the largest single-factor effect measured — while also degrading forecasting MSE. Architectural contributions rank next: text-conditioned convolutions improve accuracy from 68.0% to 71.4% relative to a text-free TCN, and combining time-offset attention with gating yields the best configuration; patch-based featurization underperforms the TCN substantially (63.0% vs. 68.0%). Naive additive injection of text embeddings into all layers actively hurts (60.5%), indicating principled integration matters.

Notably, the role of text semantics itself is modest. Shuffling channel–text correspondence costs 2.6pp, whereas replacing descriptions with random embeddings costs only 0.7pp, and even wrong-domain descriptions degrade accuracy by fewer than 8 of 2,002 test samples. The authors conclude that text functions primarily as a channel-addressing mechanism decoupling the encoder from fixed sensor topology — enabling cross-dataset generalization — rather than as a source of domain knowledge. This is a candid qualification of the multimodal framing in the title.

Interpretability emerges as a byproduct: on ETT, the learned gating matrix evolves during training into an asymmetric causal pattern in which Oil Temperature attends to all input channels but is not attended to by them, matching the known data-generating semantics.

Limitations and open questions

The paper concedes two principal constraints. First, computational scaling: because attention is computed over all $\mathbf{T} \in \mathbb{R}^{T\times C}$7 channel–time tokens without patching or downsampling, cost grows as $\mathbf{T} \in \mathbb{R}^{T\times C}$8, restricting evaluation to UEA subsets with fewer than 50 channels and 1,500 timestamps and capping usable context length. Proposed mitigations (gating-based channel pre-filtering, downsampling before the TCN) are sketched but not validated. Second, dependence on description quality: although the model is robust to noisy or wrong-domain text, high-quality sensor descriptions are required, and manual curation for UEA was labor-intensive and does not scale; automated LLM-generated descriptions are suggested but not systematically evaluated. Additional caveats include the absence of hyperparameter sweeps due to compute limits, the fact that MOMENT was not pretrained on SKAB (leaving its headroom unknown), and reliance on published rather than reproduced numbers for many baselines. Open questions include whether efficient attention variants preserve the equivariance and gating behavior, and how far multi-task post-training can push the observed fine-tuning gains.

Conclusion

CHARM demonstrates that a compact (~7M-parameter) encoder trained with a JEPA objective and description-conditioned featurization and attention produces time-series embeddings competitive with much larger foundation models across classification, anomaly detection, and forecasting, frequently using only linear probes. The evidence supports a specific division of labor: latent-space prediction supplies robustness and transferability, architectural text conditioning supplies channel identity and interpretability, and the semantic content of descriptions contributes comparatively little beyond disambiguation. The quadratic attention bottleneck and the metadata requirement remain the binding constraints on scaling the approach.