---
title: 'UniCA: Unified Covariate Adaptation'
url: https://www.emergentmind.com/topics/unica
type: topic
---

# UniCA: Unified Covariate Adaptation

UniCA (Unified Covariate Adaptation) is a lightweight, modular framework designed for adapting large, pretrained Time Series Foundation Models (TSFMs) to general covariate-aware forecasting tasks involving diverse, heterogeneous covariates, such as categorical metadata, image series, and text features. UniCA enables TSFMs—whose architectures and pretraining routines are typically restricted to real-valued inputs—to leverage arbitrary covariate information without altering the foundation model’s parameters, significantly broadening their applicability to realistic scenarios and multimodal forecasting pipelines [2506.22039].

## 1. Covariate-Aware Forecasting: Problem Formulation

Covariate-aware forecasting extends classic univariate or multivariate time series forecasting by integrating external variables that can be static or time-varying, and possibly multimodal. The model's objective is to predict a future target segment $y_{T+1:T+H} \in \mathbb{R}^{H \times 1}$, given historical target $y_{1:T} \in \mathbb{R}^{T \times 1}$, static covariates $s \in \mathbb{R}^N$ (e.g., item/location IDs), and dynamic covariates $C_{1:T+H} \in \mathbb{R}^{T+H \times M}$. These covariates may be:

- Homogeneous real-valued series (e.g., temperature)
- Discrete categorical sequences (e.g., product IDs)
- Multimodal data (e.g., satellite images, diagnostic text)

Formally, the forecasting function is:
$$
\hat{y}_{T+1:T+H} = f(y_{1:T}, C_{1:T+H}, s)
$$

Traditional TSFMs, such as Chronos-Bolt and TimesFM, accept only real-valued covariates and assume channel independence, rendering them inadequate for ingesting discrete or multimodal signals without disrupting pretrained temporal representations [2506.22039].

## 2. Covariate Homogenization

UniCA addresses this limitation through covariate homogenization: a process that transforms all heterogeneous covariates into dense, real-valued series representations aligned with the forecasting horizon. The approach consists of the following mapping pipelines:

- **Categorical Sequences:** Each token $c_k \in \{1, ... V\}$ is mapped via an embedding matrix $E_c \in \mathbb{R}^{V \times d}$,
  $$
  h^{(c)}_k = E_c[c_k] \in \mathbb{R}^d.
  $$
- **Image Sequences:** Each image $i_k$ is encoded by a convolutional neural network (CNN) or pretrained vision encoder,
  $$
  h^{(i)}_k = f_\text{img}(i_k) \in \mathbb{R}^d.
  $$
- **Text Sequences:** Segment $t_k$ is transformed by an encoder (e.g., GPT2) and linear projection,
  $$
  h^{(t)}_k = W_\text{txt} \cdot \text{gpt2}(t_k) + b_\text{txt} \in \mathbb{R}^d.
  $$

Each covariate’s embedding $H^{(\text{het})} \in \mathbb{R}^{(T+H) \times d}$ is input to a Covariate Homogenizer (typically a single linear projection),
$$
\widetilde{C}^{(\text{het})} = \text{CH}(H^{(\text{het})}) = H^{(\text{het})} W_\text{ch} + b_\text{ch} \in \mathbb{R}^{(T+H) \times d^{\text{het}}}
$$
yielding unified covariate tensors with dimensionality $d^{\text{het}}$ (typically $d^{\text{het}}=4$).

All homogenized covariates and native real-valued covariates are concatenated channel-wise to form the final input $C_{1:T+H} \in \mathbb{R}^{(T+H) \times M}$, where $M$ is the total effective number of covariate channels.

## 3. Unified Attention-Based Covariate Fusion

UniCA introduces two-stage, attention-based fusion mechanisms for conditioned covariate injection, situated before and after the frozen temporal encoder of the TSFM:

- **Pre-Fusion (Past Covariates):** Past target embeddings $Y_\text{tok}$ and covariate tokens $C_\text{tok}$ are fused for each time patch via a conditional attention pooling. For static features $S = \rho(s)$, attention weights
  $$
  \alpha_{p,m} = \text{softmax}_m \left( \frac{Q_p \cdot K_{p,m}^T}{\sqrt{d}} \right)
  $$
  are computed with $Q_p = Y_\text{tok}[p,:]$, $K_{p,m} = \text{Linearity}([Y_\text{tok}[p,:]; S])$, and values $V_{p,m} = \text{GRN}(C_\text{tok}[p,m,:])$.

  The attended covariate pool $C_\text{pool}[p] = \sum_{m=1}^M \alpha_{p,m} V_{p,m}$ is fused into the patch token using a Gated Linear Unit (GLU):
  $$
  X_\text{pre}[p] = Y_\text{tok}[p] + \text{GLU}(C_\text{pool}[p]).
  $$

- **Post-Fusion (Future Covariates):** Future covariates, tokenized as $C_\text{tok}^\text{fut}$, are fused to the temporal encoder’s output $X^1$ via the same conditional pooling. The sequence $Z = [X^1; C_\text{fut}]$ undergoes standard Transformer attention to yield updated hidden states, with forecast tokens $\widehat{X}$ passed to the predictor.

Both fusion points are modular; empirical results show negligible sensitivity to the precise fusion locations.

## 4. Adaptation Protocol and Training

UniCA treats the TSFM (tokenizer $\mathcal{T}$, encoder $\mathcal{E}$, predictor $\mathcal{P}$) as an immutable backbone:
$$
\widehat{y} = \mathcal{P} \circ \text{PostFusion} \circ \mathcal{E} \circ \text{PreFusion} \circ \mathcal{T}(y_{1:T}),
$$
training only the adaptation modules: embedding layers ($E_c, E_m$), the Covariate Homogenizer, attention pooling, and GLUs—which collectively comprise 1–5% of the total parameters. The loss is the quantile loss (CRPS) across multiple quantiles $q \in [0.1, \ldots, 0.9]$:
$$
L_{QL} = \sum_{q} \sum_{\tau=1}^H \left[ q \cdot (y_{T+\tau} - \widehat{y}_{\tau,q})_+ + (1 - q) \cdot (\widehat{y}_{\tau,q} - y_{T+\tau})_+ \right] / H.
$$
Targets are normalized by Reversible Instance Normalization before training and inverse normalized at inference. Training proceeds with the Adam optimizer, learning rates in $\{10^{-3}, 10^{-4}, 10^{-5}, 10^{-6}\}$, early stopping, and batch sizes $\{8, 16, 32, 64\}$ [2506.22039].

## 5. Empirical Evaluation

Experiments on 12 unimodal and multiple multimodal covariate-aware forecasting benchmarks establish the following:

- **Unimodal Covariate Tasks:** Electricity price forecasting (EPF subsets), retail sales (M5, PDB, Spain), BDG-2 load, and GEFCom datasets.
- **Multimodal Benchmarks:** MMSP (64x64x4 satellite images + load/NWP features for solar power), Time-MMD (text reports + series in multiple domains).

Key metrics include MAE, MAPE, MSE (all normalized to naive baselines), and CRPS for probabilistic performance.

| Method                       | MSE↓   | MAE↓  | MAPE↓ | CRPS↓  |
|------------------------------|--------|-------|-------|--------|
| Chronos-Bolt (0-shot)        | 0.418  | 0.526 | 0.514 | 0.460  |
| Chronos-Bolt (UniCA)         | 0.383  | 0.509 | 0.506 | 0.429  |
| TimesFM (SFT)                | 0.100  | 0.258 |  —    |   —    |
| TimesFM (UniCA, MMSP)        | 0.098  | 0.229 |  —    |   —    |
| TimesFM (UniCA, Time-MMD)    |   —    | 0.652 |   —   | 0.645  |
| Time-LLM/TTM (Time-MMD)      |   —    | 0.682 |   —   | 0.681  |

Ablations demonstrate:
- Plug-in Covariate Homogenizer to TFT/TiDE on MMSP: MAE reduced by 5–55%, MAPE by 38–60%
- Homogenization dim. ($d^{\text{het}}$): performance saturates at $d^{\text{het}}=4$
- Homogenizer: single linear layer is optimal for most use cases

All empirical comparisons use >95% frozen parameters, isolating gains to the UniCA adaptation.

## 6. Implementation and Design Principles

- **Backbones:** Chronos-Bolt (T5 encoder-decoder on LOTSA), TimesFM (decoder-only pretraining)
- **Covariate Homogenizer:** Linear projection, $d \rightarrow d^{\text{het}}=4$
- **Image Encoder:** 4-layer CNN for satellite data
- **Text Encoder:** GIST embeddings
- **Optimization:** Adam, ReduceLROnPlateau, 4 × RTX 3090 GPUs
- **Parameter Efficiency:** Only 1–5% of the architecture is trainable.

Performance robustness is observed with respect to homogenizer architecture and fusion position. Linear projection for homogenization suffices; more complex MLPs confer negligible additional benefits.

## 7. Significance and Scope

UniCA establishes a practical, plug-and-play methodology for leveraging the generalization strengths of large pretrained TSFMs in heterogeneous, covariate-rich environments without sacrificing scalability or requiring end-to-end fine-tuning. By decoupling covariate processing from time series backbone dynamics, UniCA enables flexible, modular adaptation to production pipelines, supports any combination of covariate modalities, and achieves state-of-the-art results on a spectrum of real-world forecasting tasks [2506.22039].

Source: https://www.emergentmind.com/topics/unica