Bounded Context Management for Tabular Foundation Models on Stream Learning

Abstract: Tabular stream learning requires predictions on sequentially arriving examples under distribution shift. While standard methods adapt by updating model states, tabular foundation models (TFMs) make predictions conditioned on a labeled context in an in-context manner, making them a natural alternative for stream learning. This shifts the challenge from how to update the model to how to manage the context. We propose a future information view that yields three practical requirements for context management: preserve recent examples, retain uncertain examples, and remove redundant examples. We instantiate these requirements as CURE (Context management via Uncertainty-aware admission and Redundancy aware Eviction), a context-managing policy with entropy-gated admission and redundancy-aware eviction. Across seven streams, CURE shows up to 27.0% relative improvement over classical stream learners, remains robust across multiple TFM backbones, and ranks first among other policy variants. Code and datasets are available at https://github.com/morcellinus/CURE-ICML-FMSD.

• The paper introduces Cure, a dual-bank context management strategy that exploits recency, uncertainty, and redundancy for streaming tabular foundation models.
• It develops a future-information framework, translating in-context learning metrics into practical entropy and redundancy filters for effective adaptation.
• Experimental results demonstrate robust performance gains and memory efficiency across multiple TFM architectures and diverse streaming datasets.

Bounded Context Management for Streaming Tabular Foundation Models

Problem Motivation and Paradigm Shift

Tabular stream learning demands robust prediction algorithms capable of handling sequential, non-stationary input with stringent memory and real-time constraints. Traditionally, this has been addressed by incrementally adapting model states (e.g., updating tree statistics or ensemble weights) to accommodate distributional drift and concept changes.

Tabular Foundation Models (TFMs) introduce a salient shift in this paradigm. Instead of persistent weight updates, TFMs adapt via in-context learning—conditioning predictions on a fixed-size, recent context of labeled examples. Consequently, the locus of adaptation migrates from model state updates to the strategic selection and maintenance of context under bounded memory.

The core challenge thus becomes designing context management policies that select which historical examples to retain such that, for future queries under evolving distributions, information content and adaptability are maximized.

Theoretical Foundation: Future-Information View

The paper formulates a future-information criterion for context utility, specific to in-context adaptation in online streams. The context $D_t$ at step $t$ is scored by:

$$\mathcal J_t(D_t) = \mathbb{E}_{x'\sim \mathcal P_{t,X}^+}[I(D_t ; Y_{x'} \mid x')]$$

Here, $x'$ is sampled from the near-future feature distribution $\mathcal P_{t,X}^+$, and $I$ denotes the (conditional) pointwise mutual information with respect to label prediction. This metric encapsulates the value of contextual samples for informing predictions over the immediate future.

Because this objective is algorithmically inaccessible (future queries and labels are unobserved), the authors decompose it into three tractable, theoretically motivated signals:

• Recency: Recent samples approximate the forthcoming distributional regime due to local stationarity. Retaining recent data ensures rapid adaptation to concept drift.
• Uncertainty: High predictive entropy at the time of observation signals examples likely to provide significant information gain upon label revelation, as formalized by a lower-bound connection between entropy and local mutual information.
• Redundancy: Locally similar, same-class examples offer diminishing returns in terms of incremental information. Drop candidates are identified via nearest neighbors in representation space under the same class label.

This decomposition is formally justified through a series of assumptions and lemmas in the appendix, establishing the connection between theoretical future-label utility and accessible context-policy signals.

Cure: A Dual-Bank Context Management Strategy

Building on the theoretical signals, the authors propose Cure—a context management policy for streaming with TFMs that operationalizes recency, uncertainty, and redundancy.

Figure 1: Cure's dual-bank structure manages context by combining short-term FIFO recency with entropy-filtered long-term retention, and redundancy-aware eviction.

• Short Bank ($S_t$): Implements a sliding window (FIFO) mechanism, guaranteeing that all recent examples are retained for a fixed window, harnessing the recency signal.
• Long Bank ($L_t$): Upon overflow from the short bank, candidates are admitted to the long bank if their normalized predictive entropy (captured at prediction time) exceeds a threshold $\tau$. This admits only samples likely to be informative according to the uncertainty signal.
• Eviction: When the long bank exceeds its budget, redundancy-aware removal is performed. The overrepresented class is identified, and within it, the closest pair in representation space is located. The instance further from the recent centroid is evicted, minimizing potential information loss.

This explicit instantiation is performant, theoretically grounded, and computationally efficient.

Experimental Results

Cure is benchmarked on seven data streams, spanning both real-world and synthetic abrupt-drift scenarios. Backbone robustness is demonstrated across four prominent TFM architectures: TabICL-v2, TabPFN-v2.5, TabDPT-v1, and LimiX-v1. Performance is evaluated against strong MOA baselines such as VFDT, EFDT, ARF, SRP, LevBag, and BOLE, with prequential accuracy under the standard test-then-train protocol.

Prequential accuracy trajectories are consistent with significant and persistent performance gains.

Figure 2: Prequential accuracy of Cure and MOA baselines on NOAA, exhibiting early separation and stable leadership by Cure.

Figure 3: Prequential accuracy gap on METER, demonstrating Cure's sustained top performance.

Figure 4: Cure maintains clear superiority on RIALTO throughout the stream.

Figure 5: Cure outperforms baselines on POSTURE-No8, including highly imbalanced regimes.

Figure 6: Cure's advantage on POKER, a large-scale, extremely imbalanced stream.

Figure 7: Strong and stable Cure trajectory on NOMAO.

Figure 8: Robust performance on AGR(A), synthetic abrupt drift.

Strong numerical results include:

• Best prequential accuracy on all evaluated streams with TabICL-v2.
• Maximum relative improvement up to 27.0% (e.g., from 71.49% to 90.80% on METER).
• Consistent benefits across multiple TFM backbones: Cure outperforms DualFIFO in 17/18 backbone-dataset pairs.
• Ablations and controlled variants confirm the complementarity of recency, uncertainty, and redundancy: Cure achieves the best average rank among all tested alternatives.

Additionally, results indicate Cure's low sensitivity to the entropy threshold and negligible computational overhead relative to TFM inference cost.

Implications and Prospective Directions

Practical Implications

This work positions context-policy design as the primary adaptation lever for TFMs in streaming scenarios. Cure not only demonstrates that context-limited TFMs can compete with, and outperform, state-mutating baselines but also highlights the role of context signals—especially uncertainty and redundancy—in maintaining performance under concept drift without model retraining.

The dual-bank abstraction and entropy-gated admission could be further leveraged in memory-constrained or real-time deployment contexts, such as embedded systems or online A/B testing infrastructure, where retraining is prohibitive.

Theoretical Ramifications and Future Work

This research advances the formalism for in-context adaptation under memory constraints, adapting information-theoretic concepts to streaming learning settings. Extensions may include:

• Adaptive tuning of short/long bank allocation, entropy thresholds, or redundancy metrics in response to detected drift.
• Generalization to regression or multi-modal settings, potentially involving richer uncertainty quantification.
• Integration of more elaborate base TFM architectures or semantic- and representation-aware redundancy criteria, possibly leveraging contrastive objectives or meta-learning signals.
• Exploration of transfer learning scenarios, devising context policies suitable for continual or federated learning under privacy constraints.

Conclusion

The paper establishes that effective bounded context management, driven by jointly exploiting recency, uncertainty, and redundancy, enables tabular foundation models to achieve robust, state-of-the-art performance on dynamic streaming tasks. Cure, a dual-bank context management policy, operationalizes this theoretical perspective and demonstrates consistent advantages—both in terms of predictive accuracy and memory efficiency—across a variety of streams and TFM backbones. These results redirect attention from model-state adaptation toward strategic context construction, thereby opening new avenues in online, in-context learning theory and practice (2606.18677).