---
title: Weighted Contrastive Adaptation (WECA)
url: https://www.emergentmind.com/topics/weighted-contrastive-adaptation-weca
type: topic
---

# Weighted Contrastive Adaptation (WECA)

Weighted Contrastive Adaptation (WECA) is a training algorithm for multivariate time-series forecasting that explicitly accounts for anomaly-awareness when learning representations, targeting improved performance under distribution shift scenarios without degrading accuracy on normal data. WECA introduces a weighted contrastive objective, interpolating between invariance to benign augmentations and the preservation of anomaly-specific information. Its principal application, as demonstrated in the context of nationwide ATM cash logistics, is to enhance forecaster robustness when encountering realistic, domain-informed anomaly events [2512.07569].

## 1. Objective Formulation and Loss Derivation

WECA extends instance-wise contrastive learning (InfoNCE) for representations $\{z_{i,t}\}$ of normal windows and $\{\tilde z_{i,t}\}$ of anomaly-augmented windows produced by an encoder $g_\phi$. For each window $i$ and time $t$, a similarity weight $w^{(i,t)} \in [0,1]$ scales the strength of alignment in the loss. The core weighted InfoNCE term is
\[
\ell_{\mathrm{WECA}}^{(i,t)} = -w^{(i,t)}\;\log \frac{\exp\!\bigl(\langle z_{i,t},\tilde z_{i,t}\rangle/\tau\bigr)}
    {\sum_{j=1}^B\left[\exp\!\bigl(\langle z_{i,t},\tilde z_{j,t}\rangle/\tau\bigr) + \mathbb{1}_{j\neq i} \exp\!\bigl(\langle z_{i,t},z_{j,t}\rangle/\tau\bigr) \right]},
\]
where $\langle\cdot,\cdot\rangle$ denotes the dot-product. The full batch loss combines the contrastive term and a mean absolute error (MAE) forecast loss with balance coefficient $\lambda$:
\[
\mathcal{L} = \frac{1}{B}\sum_{i=1}^B \mathcal{L}_{\mathrm{forecast}}^{(i)} + 
\lambda \frac{1}{BT'} \sum_{i=1}^B \sum_{t=1}^{T'} \ell_{\mathrm{WECA}}^{(i,t)},
\]
where
\[
\mathcal{L}_{\mathrm{forecast}}^{(i)} = \frac{1}{H} \sum_{h=1}^H \lVert y_{i,h}-\hat y_{i,h} \rVert_1,
\]
with decoder $h_\psi$, forecast targets $y_i$, and per-timestep outputs $\hat y_{i,h}=h_\psi(g_\phi(\mathbf{x}_i))_h$.

## 2. Contrastive Pairing and Anomaly Augmentation

The contrastive setup defines, for each sample $i$ at position $t$, a positive pair $(z_{i,t},\tilde z_{i,t})$ and negative pairs both as $(z_{i,t},\tilde z_{j,t})$ for all $j\neq i$ (augmented negatives), and $(z_{i,t},z_{j,t})$ (other normal negatives). Anomaly-augmented samples are synthesized at the input level: for each $\mathbf{x}_i$, the last window positions (tail) are perturbed using an anomaly function $a(n) = A\cdot n\cdot e^{-B n^C}$, where $(A,B,C)$ are stochastically sampled to mimic historically observed anomaly statistics (e.g. $A\sim\mathcal{N}(74120, 20000^2)$, $B=0.39$, $C\sim\mathcal{N}(0.806,0.3^2)$). The generated anomaly $a(n)$ is injected into the tail of $\mathbf{x}_i$ and propagated into the forecast horizon, yielding $\tilde{\mathbf{x}}_i$.

## 3. Weight Function Design and Principle

The weight $w^{(i,t)}$ is a continuous function of the input-level perturbation magnitude, typically of the form $w^{(i,t)} = \exp(-\alpha \lVert \mathbf{x}_i-\tilde{\mathbf{x}}_i \rVert_2)$, with $\alpha$ chosen so domain-plausible (benign) variations yield $w\approx 0.9\text{--}1.0$ (strong invariance imposed), while major anomaly-like deviations yield $w\approx 0.1\text{--}0.3$ (weak or no invariance). This soft weighting causes the encoder to enforce invariance only for small (non-anomalous) perturbations, while explicitly retaining anomaly-specific features otherwise. Consequently, WECA can interpolate between pure contrastive learning (full invariance) and no contrastive adaptation, preserving anomaly detectability while improving normal operational robustness.

## 4. Training Protocol and Network Architecture

The WECA training loop proceeds as follows:

```python
# WECA Training Algorithm (Pseudocode)
for each training step:
    1. Sample batch {(x_i, y_i)}_{i=1}^B
    2. For each i:
         - Sample anomaly parameters (A, B, C)
         - Generate anomaly a_i(n) = A*n*exp(−B n^C)
         - Form augmented input x~_i = x_i + tail(a_i)
    3. Compute Z = g_phi({x_i}), Z~ = g_phi({x~_i})
    4. Compute ŷ_i = h_psi(z_i)  # normal branch
    5. Compute forecast loss L_forecast = (1/B) Σ_i MAE(y_i, ŷ_i)
    6. Compute weights w^(i,t) = exp(−α * ||x_i − x~_i||_2)
    7. Compute L_WEC = (1/(B·T')) Σ_{i=1}^B Σ_{t=1}^{T'} ℓ_WECA^{(i,t)}
    8. Total loss: L = L_forecast + λ·L_WEC
    9. Backpropagate/update parameters via Adam (lr=1e−3)
```

The backbone architecture is TimesNet, which achieves optimal symmetric mean absolute percentage error (SMAPE) on normal data. The contrastive head projects encoded features to a $D$-dimensional space for dot-product similarity.

## 5. Hyperparameter Selection and Weighting Strategy

Selected hyperparameters:

- Learning rate: $10^{-3}$ (Adam optimizer)
- Batch size: $128$
- Contrastive weight $\lambda$: $1$ (tuned via validation)
- InfoNCE temperature $\tau$: usually $0.1$ or absorbed into dot product scaling
- Weighting function parameter $\alpha$: tuned so that small (domain-plausible) perturbations produce $w\approx 0.9\text{--}1.0$ and large-scale anomalies $w\approx 0.1\text{--}0.3$
- Early stopping: monitored on validation MAE

*Editor's term*: "Dynamic similarity weighting" refers to the weighting mechanism for anchoring contrastive invariance according to anomaly severity.

## 6. Empirical Evaluation and Performance

The primary empirical benchmark uses approximately 1,300 ATMs' daily withdrawal data over two years, with a rolling-origin 70/10/20 training/validation/test split. Anomaly-augmented data is generated using the described injection procedure.

Evaluation metric: SMAPE (%) on a 14-day forecast horizon.

Performance summary:

| Method | Normal Data (ND) SMAPE ± std | Δ vs NT | Anomaly Data (AD) SMAPE ± std | Δ vs NT |
|--------|-----------------------------|---------|-------------------------------|---------|
| NT     | 28.73                       | —       | 37.91                         | —       |
| FT     | 31.50±0.87                  | +2.77   | 30.69±2.03                    | −7.22   |
| CL-IL  | 28.62±0.97                  | −0.11   | 33.09±0.96                    | −4.82   |
| WECA   | 28.70±1.00                  | −0.03   | 31.78±1.93                    | −6.13   |

WECA achieves a 6.13 percentage point reduction in SMAPE on anomaly-affected test data compared to the normally trained baseline (NT), while only marginally affecting normal-data performance (0.03 pp difference). Instance-level contrastive learning (CL-IL) offers some anomaly robustness but with weaker margins; fine-tuning on anomaly-only data (FT) yields the largest anomaly gain but with a significant loss on normal data, consistent with catastrophic forgetting [2512.07569].

## 7. Implementation and Applicability Considerations

- The anomaly injection process is fully reproducible with fixed random seeds and parameter distributions.
- WECA approximately doubles encoder passes per batch (normal + augmented), but the total computational overhead remains under 20% of standard GPU training time.
- The similarity weight $w^{(i,t)}$ can be adapted to rely on alternative distance metrics or learned heuristics.
- WECA is compatible with any deep forecaster architecture that exposes intermediate latent representations; no property is exclusive to TimesNet.
- Training is most stable when the forecasting backbone is pre-trained on normal data and then fine-tuned with the WECA joint objective.
- The method requires access to domain knowledge or distributions for realistic anomaly magnitude modeling, especially for applications like cash logistics where external events influence future behavior.

By calibrating the trade-off between invariance and anomaly awareness, WECA provides reliable forecasting resilience across severe real-world distribution shifts without compromising peacetime operational accuracy [2512.07569].

Source: https://www.emergentmind.com/topics/weighted-contrastive-adaptation-weca