Consistency-Regularized Information Bottleneck

• The paper introduces CRIB, a framework that bypasses imputation by directly predicting from partial observations using an information bottleneck to learn compact and predictive latent representations.
• CRIB employs a unified-variate self-attention mechanism to simultaneously model intra- and inter-variate dependencies, improving forecasting under varying missing data patterns.
• The model integrates a consistency regularization term to stabilize the latent representations, thereby enhancing robustness against high rates of missingness.

Searching arXiv for the CRIB paper and the baseline/related methods mentioned, so the article can cite them precisely. Consistency-Regularized Information Bottleneck (CRIB) is a framework for multivariate time series forecasting with missing values that was introduced by Yang et al. as part of a broader re-examination of the dominant imputation-then-prediction paradigm in MTSF-M. Rather than first reconstructing missing entries and then forecasting on the filled series, CRIB performs direct prediction from partially observed time series, using the Information Bottleneck principle to learn latent representations that are compact yet predictive, a unified-variate attention mechanism to model intra- and inter-variate dependencies, and a consistency regularization term to improve robustness under severe missingness (Yang et al., 27 Sep 2025).

1. Problem setting and conceptual motivation

CRIB is formulated for real-world multivariate time series $X \in \mathbb{R}^{N \times T}$ with a mask $M \in \{0,1\}^{N \times T}$, which induces observed entries $X^{o}$ and missing entries $X^{m}$. The forecasting target is the next $S$ steps, denoted $Y \in \mathbb{R}^{N \times S}$ (Yang et al., 27 Sep 2025).

The motivating claim is that missing values are common in deployed settings, including sensor dropouts, transmission glitches, and irregular sampling. Within that setting, prior work is characterized as relying predominantly on an imputation module that fills missing values before a forecasting module processes the completed series. Yang et al. argue that this workflow contains a structural weakness: there is no ground truth for the missing entries during inference, so the imputation step may introduce systematic distortions rather than recover the latent data-generating process.

This leads to a conceptual shift. CRIB treats forecasting under missingness not as a reconstruction-first problem but as a representation-learning problem over partial observations. The central premise is that the model should extract the predictive content of $X^{o}$ while suppressing noise or uncertainty associated with unobserved positions. This suggests a departure from explicit recovery of $X^{m}$ toward selective retention of forecast-relevant information.

2. Reassessment of imputation-based forecasting

A central contribution associated with CRIB is the empirical claim that imputation without direct supervision can corrupt the underlying data distribution and actively degrade prediction accuracy (Yang et al., 27 Sep 2025). The paper reports two specific failure modes.

First, imputed values may drift from the original distribution. The study identifies this through t-SNE and correlation plots, which are used to illustrate that imputation can distort inter-variate structure rather than preserve it. Second, the paper reports error propagation: a sophisticated imputer combined with a forecaster often underperforms a naive model applied directly to $X^{o}$.

These observations function as a criticism of a common assumption in the literature: that better reconstruction of missing values necessarily improves downstream forecasting. The reported evidence instead indicates that, in the absence of supervised targets for $X^{m}$, the imputation stage can become a source of inductive bias mismatch. A plausible implication is that forecasting performance under missingness may depend more on robust partial-observation encoding than on explicit gap filling.

The paper positions CRIB as a direct-prediction alternative to two-stage and end-to-end imputation-then-forecast approaches. In the reported experiments, this framing is evaluated against 12 strong baselines, including BRITS, SAITS, SPIN, GRIN, BiTGraph, and modern forecasters such as PatchTST, iTransformer, and DLinear, both with and without imputation.

3. Information Bottleneck formulation

CRIB is built on the Information Bottleneck (IB) principle. From the observed series $M \in \{0,1\}^{N \times T}$0, it learns a latent variable $M \in \{0,1\}^{N \times T}$1 that is simultaneously compressed and informative about the future target $M \in \{0,1\}^{N \times T}$2. The objective is written as

$M \in \{0,1\}^{N \times T}$3

where

$M \in \{0,1\}^{N \times T}$4

This formulation encodes the trade-off between retaining predictive signal and discarding superfluous information associated with partially observed inputs (Yang et al., 27 Sep 2025).

Under standard variational bounds, the paper derives two tractable surrogate losses. The compactness term upper-bounds $M \in \{0,1\}^{N \times T}$5: $M \in \{0,1\}^{N \times T}$6 with prior $M \in \{0,1\}^{N \times T}$7 and variational posterior

$M \in \{0,1\}^{N \times T}$8

The informativeness term lower-bounds $M \in \{0,1\}^{N \times T}$9. Assuming a Gaussian decoder $X^{o}$0, the lower bound becomes proportional to the negative prediction error: $X^{o}$1

Within CRIB, IB is therefore not an abstract regularizer but the organizing principle of representation learning. It operationalizes the idea that the encoder should compress away nuisance variation introduced by missingness while preserving the statistics required for accurate multistep prediction.

4. Encoder architecture and unified-variate attention

The encoder-predictor pipeline begins with patching and embedding. Each variate’s length-$X^{o}$2 series is partitioned into $X^{o}$3 non-overlapping patches of size $X^{o}$4. Sinusoidal temporal embeddings are added, and a small Temporal Convolutional Network (TCN) produces local patch features

$X^{o}$5

This stage converts the partially observed input into a patch-level representation suitable for global interaction modeling (Yang et al., 27 Sep 2025).

CRIB then applies what the paper terms unified-variate self-attention. The tensor $X^{o}$6 is flattened into $X^{o}$7 tokens, and standard multi-head self-attention is used: $X^{o}$8 Because every token can attend to every other token, temporal dependencies within a variate and cross-series dependencies across variates are handled by a single mechanism.

The significance of this design lies in its refusal to separate temporal modeling from inter-variate modeling. The paper explicitly attributes to unified attention the ability to capture both intra-variate and inter-variate dependencies, avoiding a more fragmented architecture in which different modules handle time and cross-series structure independently. A plausible implication is that such flattening is especially useful when missing values make local neighborhood assumptions unreliable.

Prediction is performed by a simple two-layer MLP mapping

$X^{o}$9

to

$X^{m}$0

The resulting architecture therefore comprises three main stages: local patch encoding, globally shared attention across flattened tokens, and direct horizon prediction.

5. Consistency regularization and full objective

The paper argues that, at high missing rates, an IB-guided encoder may overfit to the limited observed positions. CRIB addresses this by enforcing representation invariance across a more severely corrupted view of the same partially observed input (Yang et al., 27 Sep 2025).

The augmentation procedure starts from $X^{m}$1. An additional 10% of observed entries are randomly masked, and small Gaussian noise is added to the remaining values, yielding $X^{m}$2. Both $X^{m}$3 and $X^{m}$4 are encoded to latent representations $X^{m}$5 and $X^{m}$6. The consistency penalty is then

$X^{m}$7

This term is intended to stabilize training by making the latent representation less sensitive to further corruption of an already incomplete signal. In effect, the regularizer treats robustness to additional masking and small perturbations as a desirable property of the learned representation.

The full optimization target is

$X^{m}$8

where $X^{m}$9 balance compression, prediction accuracy, and representation stability. The paper reports Adam with learning rate $S$0, attention with 2 layers and 4 heads, patch length $S$1, historical window $S$2, forecast horizon $S$3, and embedding dimension $S$4 tuned from 32 to 128 with default 64. The IB weights $S$5 and consistency weight $S$6 are selected via validation, typically with $S$7 and $S$8. The reported sensitivity analysis indicates that moderate $S$9 improves robustness, whereas over-regularization under extreme missingness can underfit correlation structure.

6. Empirical behavior, reported performance, and interpretation

CRIB was evaluated on four real-world datasets: PEMS-BAY, Metr-LA, ETTh1, and Electricity. The experimental protocol injects synthetic missing rates of 20%, 40%, 60%, and 70% under point, block, or column masks (Yang et al., 27 Sep 2025).

Against 12 baselines, the paper reports that CRIB achieves the lowest MAE and MSE in every scenario. It further reports up to 18% average MAE reduction on ETTh1 and 13% on PEMS-BAY compared with the best competitor. The results are described as remaining stable even at 70% missingness, in contrast to imputation-based two-stage pipelines, which are said to degrade sharply.

These findings are used to support the broader claim that bypassing imputation can be preferable when forecasting is the ultimate objective. The paper’s interpretation is that the combination of IB-based compression, unified attention, and consistency regularization allows the model to preserve essential predictive signals while filtering noise introduced by missingness. This suggests that the principal advantage of CRIB is not reconstruction fidelity but representation selectivity under incomplete observation.

The reported implementation is accompanied by publicly available code at the repository linked by the authors. Within the scope of the paper, CRIB is therefore presented both as a methodological proposal and as evidence for a more general reframing of MTSF-M: direct prediction from partial observations may be a more reliable design principle than unsupervised imputation followed by forecasting.