Training-Conditional Cumulative Regret

• The paper demonstrates that training-conditional cumulative regret quantifies test-phase loss via KL divergence and minimax strategies, guiding optimal predictor designs.
• It extends classic regret analysis to Rényi divergences, sequential decision-making, and federated learning with tight tail behavior and risk guarantees.
• This framework informs adaptive exploration and robust risk calibration in reinforcement learning, bandit settings, and nonstationary online learning scenarios.

Training-conditional cumulative regret quantifies the predictive or decision-theoretic loss incurred during a phase where algorithms are permitted to adaptively interact with data (“training”), and then characterizes performance, risk, or error in the subsequent application (“test” or “evaluation”) phase conditional on that specific training trajectory. This notion sharpens classic performance metrics by tracking how learning during training—potentially under individual noise, nonstationarity, or strategic constraints—determines future regret, risk, or coverage guarantees. The training-conditional perspective bridges universal prediction, adaptive online learning, and sequential decision making, providing exact information-theoretic, minimax, and tail-behavior characterizations across prediction, bandits, reinforcement learning, and federated optimization.

1. Formal Definitions and Universal Prediction Framework

In batch universal prediction, training-conditional cumulative regret (also called “minimal batch regret”) is precisely formulated in terms of KL divergence or its generalizations. Given a parametric family of distributions $\mathcal{P} = \{p_\theta : \theta \in \Theta\}$ over a finite alphabet $\mathcal{X}$, one observes a training sequence $X^m \sim p_\theta^{\otimes m}$ and predicts an evaluation sequence $Y^n \sim p_\theta^{\otimes n}$. Predictors are conditional distributions $\hat p(y^n|x^m)$.

Under logarithmic loss, the expected training-conditional regret is

$$R_{n|m}(\hat p, \theta) = \mathbb{E}_{X^m, Y^n \sim p_\theta}\left[ \log \frac{p_\theta(Y^n)}{\hat p(Y^n | X^m)} \right]$$

which can be written equivalently as the conditional KL divergence: $$R_{n|m}(\hat p, \theta) = D\left(p_\theta(Y^n) \;||\; \hat p(Y^n|X^m) \mid X^m\right)$$ The minimax training-conditional regret is

$$R_{n|m}(\mathcal{P}) = \inf_{\hat p}\sup_{\theta \in \Theta} R_{n|m}(\hat p, \theta)$$

This exact value is given by the “conditional regret-capacity theorem,” stating that $R_{n|m}(\mathcal{P})$ equals the supremum, over priors $w$ on $\mathcal{X}$0, of the conditional mutual information between $\mathcal{X}$1 and $\mathcal{X}$2 given $\mathcal{X}$3,

$\mathcal{X}$4

The optimal predictor is the Bayesian mixture using the posterior $\mathcal{X}$5 computed from the maximizing prior $\mathcal{X}$6 (Bondaschi et al., 14 Aug 2025).

For the binary memoryless source class, (e.g., $\mathcal{X}$7 Bernoulli), the precise minimax regret is

$\mathcal{X}$8

expressing that the regret for predicting $\mathcal{X}$9 samples after seeing $X^m \sim p_\theta^{\otimes m}$0 is controlled by the effective information gain from training.

2. Extensions to General Divergence and Information Measures

Training-conditional regret admits generalization from logarithmic (Shannon) loss to Rényi-$X^m \sim p_\theta^{\otimes m}$1 divergences. This leads to conditional Sibson’s mutual information as the relevant information-theoretic quantity: $X^m \sim p_\theta^{\otimes m}$2 Via analogous minimax duality arguments, the minimax regret equals $X^m \sim p_\theta^{\otimes m}$3, where $X^m \sim p_\theta^{\otimes m}$4 denotes conditional Sibson mutual information of order $X^m \sim p_\theta^{\otimes m}$5. The minimax-optimal predictor is the “conditional $X^m \sim p_\theta^{\otimes m}$6-NML” form, a normalized Bayesian mixture over the parameter space using the maximizing prior $X^m \sim p_\theta^{\otimes m}$7. In the binary memoryless case, these metrics admit closed forms: $X^m \sim p_\theta^{\otimes m}$8 establishing a bridge between universal prediction regret lower bounds and channel-/Sibson-capacities (Bondaschi et al., 14 Aug 2025).

3. Sequential and Federated Online Learning Perspectives

In online stochastic optimization and federated learning, training-conditional cumulative regret appears as the performance metric after grouping regret by training epochs or client synchronization points. For $X^m \sim p_\theta^{\otimes m}$9 clients over $Y^n \sim p_\theta^{\otimes n}$0 rounds,

$Y^n \sim p_\theta^{\otimes n}$1

Grouping by epochs where points $Y^n \sim p_\theta^{\otimes n}$2 are synchronized, and conditioning all statements on the realized sequence of stochastic gradients (training data), one obtains “training-conditional” high-probability regret bounds such as: $Y^n \sim p_\theta^{\otimes n}$3 when using appropriate adaptive quantization and synchronization protocols (e.g., CEAL algorithm). The conditional probability is with respect to the event in which quantization and sampling noise bounds both hold (Salgia et al., 2023).

The training-conditional framework here provides explicit trade-off analyses: tuning quantization precision, sampling depth, and step-sizes to balance regret versus total communication cost—an aspect not addressed by classic “simple regret”-based analysis.

4. Training-Conditional Regret in Reinforcement and Bandit Settings

In sequential contextual bandits or episodic reinforcement learning, training-conditional cumulative regret formalizes the downstream impact of exploration during the training phase. Specifically, after a learning (training) episode of horizon $Y^n \sim p_\theta^{\otimes n}$4, the learner outputs a warm-start policy for deployment in test phase $Y^n \sim p_\theta^{\otimes n}$5, leading to total regret

$Y^n \sim p_\theta^{\otimes n}$6

Here, $Y^n \sim p_\theta^{\otimes n}$7 is the cumulative regret in training, and $Y^n \sim p_\theta^{\otimes n}$8 is the simple regret in evaluation, both of which are inextricably linked by the training-conditional principle: improved test-phase optimality demands excess exploration—and thus higher regret—in training (Xu et al., 2024).

Fundamental lower bounds show, for nonadaptive policies,

$Y^n \sim p_\theta^{\otimes n}$9

which translates, for $\hat p(y^n|x^m)$0, to an unavoidable $\hat p(y^n|x^m)$1 test-phase regret unless additional exploration ($\hat p(y^n|x^m)$2-mixed policies) is injected. Tuning the exploration rate achieves a Pareto frontier between minimizing training-phase and evaluation-phase regret.

5. Instance-Dependent and Tail Characterizations

Recent analyses in episodic MDPs with unknown transition dynamics extend training-conditional regret to the full tail distribution: $\hat p(y^n|x^m)$3 where $\hat p(y^n|x^m)$4 is an instance-dependent baseline governed by the global optimality gap, and $\hat p(y^n|x^m)$5 is a transition threshold depending on the exploration bonus parameter $\hat p(y^n|x^m)$6. The results yield high-resolution, training-conditional guarantees on risk at every regret level, crucial for safety-critical or distributionally-robust applications (Khodadadian et al., 23 Nov 2025).

The tuning parameter $\hat p(y^n|x^m)$7 determines the optimal trade-off: smaller $\hat p(y^n|x^m)$8 approaches minimax optimal expected regret, while larger $\hat p(y^n|x^m)$9 controls extreme outlier probability.

6. Adaptation to Nonstationarity: Online Conformal Prediction

In online conformal prediction for nonstationary data streams, training-conditional cumulative regret arises as a coverage calibration metric: $$R_{n|m}(\hat p, \theta) = \mathbb{E}_{X^m, Y^n \sim p_\theta}\left[ \log \frac{p_\theta(Y^n)}{\hat p(Y^n | X^m)} \right]$$0 Algorithms employ stage/round decompositions with drift detection for both change-point and smooth-drift models. Provable minimax optimal upper and matching lower bounds are established: $$R_{n|m}(\hat p, \theta) = \mathbb{E}_{X^m, Y^n \sim p_\theta}\left[ \log \frac{p_\theta(Y^n)}{\hat p(Y^n | X^m)} \right]$$1

$$R_{n|m}(\hat p, \theta) = \mathbb{E}_{X^m, Y^n \sim p_\theta}\left[ \log \frac{p_\theta(Y^n)}{\hat p(Y^n | X^m)} \right]$$2

These rates demonstrably hold under both split-conformal (pretrained scores) and full-conformal (online-trained, stable predictors) regimes. Sublinear training-conditional regret ensures valid coverage at each time and robust adaptation to unknown forms of nonstationarity (Liang et al., 18 Feb 2026).

7. Broader Implications, Trade-Offs, and Algorithmic Design

Training-conditional cumulative regret sharpens the classical minimax paradigm by accounting for the dependence structure induced by training, conditioning all learning-theoretic guarantees on the realized stochasticity, adaptation path, and exploration schedule. Its information-theoretic, tail, and minimax lower bound characterizations enable practitioners to:

• Quantify the unavoidable trade-offs between present (training) and future (deployment) regret.
• Achieve robust, fine-grained risk control vital for contexts with downstream objectives or distribution shift (health, education, federated analytics).
• Guide adaptive exploration schedules to balance global regret and risk as a function of environment nonstationarity, instance difficulty, and communication constraints.
• Connect universal prediction, statistical learning, online optimization, and reinforcement learning via common conditional mutual information principles, and generalize those to Rényi and Sibson information.
• Calibrate exploration and communication efficiency in distributed and federated contexts by leveraging conditional law-of-iterated-logarithm-type results and epoch-dependent synchronization schemes.

Training-conditional regret thus serves both as a sharp quantifier of learning limits and as a practical design principle for adaptive, robust, and efficient algorithms in nonstationary, high-stakes, and resource-constrained applications (Bondaschi et al., 14 Aug 2025, Khodadadian et al., 23 Nov 2025, Liang et al., 18 Feb 2026, Salgia et al., 2023, Xu et al., 2024).