Deterministic Sequencing of Exploration & Exploitation

• DSEE is a sequential decision-making paradigm that deterministically alternates between exploration and exploitation phases to optimize learning in multi-armed bandits and reinforcement learning.
• It achieves provable near-optimal regret bounds by tailoring exploration schedules to problem parameters, providing strong performance guarantees across diverse settings.
• Its design simplifies multi-agent coordination and resource-constrained applications, making it valuable for practical tasks like reliable wireless routing and decentralized learning.

Deterministic Sequencing of Exploration and Exploitation (DSEE) is a sequential decision-making paradigm that deterministically schedules intervals of exploratory and exploitative actions to optimize learning and performance in multi-armed bandits (MAB), reinforcement learning (RL), and combinatorial online optimization problems. In contrast to randomized or purely adaptive exploration strategies, DSEE strictly separates exploration and exploitation phases according to a deterministic, pre-established schedule, yielding provable performance guarantees under general reward and transition distributions. DSEE frameworks have demonstrated near-optimal regret scaling in classical MABs, decentralized and combinatorial extensions, non-stationary environments, and practical networking tasks such as reliable routing in wireless mesh networks.

1. Core DSEE Framework and Algorithm

DSEE alternates between block-structured exploration—where each action or “arm” in a bandit or RL setting is sampled in a round-robin manner—and subsequent exploitation phases in which decisions are made according to empirical estimates obtained only from prior exploration samples. The critical design aspect is the deterministic, parameterized definition of the number and timing of exploration periods, typically growing at a controlled rate with time horizon $T$, number of arms, or other structural parameters.

Pseudocode Outline for MAB (Light-Tailed Rewards)

Consider a $K$-armed bandit with reward means $\mu_i$ ($i=1,\dots,K$):

$w$5

With $w$ large enough, DSEE achieves $O(\log T)$ regret for light-tailed rewards (Vakili et al., 2011). The exploration schedule $m(T)$ is the sole tunable parameter, adjusting the trade-off between exploration and exploitation.

2. Theoretical Guarantees and Regret Analysis

DSEE achieves optimal or near-optimal regret rates, dependent on distributional assumptions:

• Light-tailed rewards: With $m(T) = K \lceil w \log T \rceil$, for sufficient $w$, DSEE achieves

$$R_\pi(T) = O \left( \sum_{i\ne i^*} \frac{\log T}{\Delta_i} \right)$$

where $K$0 is the mean-gap to the optimal arm (Vakili et al., 2011).

• Heavy-tailed rewards (moment $K$1 exists): With $K$2 for $K$3, DSEE achieves $K$4 regret.
• Combinatorial bandits: When the arms depend on unknown edge weights or costs, DSEE samples basis elements (e.g., links in a path) in exploration, and solves the combinatorial optimization (e.g., shortest path) in exploitation, yielding regret scaling as $K$5 rather than exponential in the number of arms (Vakili et al., 2011).
• Near-logarithmic regret in networking: In anypath routing over wireless mesh networks, the DSEE-augmented algorithm achieves regret

$K$6

where $K$7 is the number of nodes and $K$8 is the maximal neighbor set size. This is near-logarithmic in $K$9 and quadratic in $\mu_i$0 (Nourzad et al., 2024).

3. DSEE in Non-Stationary and Resource-Constrained Environments

Variants such as Limited-Memory DSEE (LM-DSEE) adapt the exploration-exploitation schedule for non-stationary bandit problems:

• Abruptly-changing MAB: If the environment permits at most $\mu_i$1 abrupt changes ($\mu_i$2), and block lengths grow polynomially, LM-DSEE achieves regret

$\mu_i$3

(Wei et al., 2018).

• Slowly-varying MAB: If the arm mean can drift by $\mu_i$4, LM-DSEE, with appropriate phase length scaling, satisfies

$\mu_i$5

with $\mu_i$6, for design-capped $\mu_i$7 (Wei et al., 2018).

Memory resets at each block ensure the algorithm is not misled by stale samples, and phase lengths can be tuned for change sensitivity.

4. DSEE for Combinatorial and Decentralized Bandit Extensions

DSEE generalizes seamlessly to:

• Combinatorial bandit settings (e.g., shortest-path, minimum spanning tree): By sampling structural primitives (edges, links) during exploration, and solving for the optimal structure in exploitation based on empirical means, the regret is polynomial in the number of components (Vakili et al., 2011).
• Decentralized multi-player bandits (with collisions): When $\mu_i$8 players each interact with shared arms and collisions, exploration phases are offset across players to avoid collisions, maintaining independent estimates. Exploitation proceeds using local empirical best arms, and overall system regret matches the single-player DSEE scaling (Vakili et al., 2011).
• Markovian or restless bandits: DSEE applies by sampling each arm for a block of steps to estimate steady-state rewards and transitions, ensuring $\mu_i$9 regret under light-tailed conditions.

5. DSEE Integration into Reinforcement Learning

DSEE has been extended to model-based RL for Markov Decision Processes (MDPs) (Gupta et al., 2022):

• Algorithmic structure: Alternating epochs of exploration (uniformly random action selection) and exploitation (policy derived from robust MDP using empirical reward and transition estimates), with epoch lengths growing as a function of iteration $i=1,\dots,K$0.
• Robust policy computation: After exploration, the agent computes empirical estimates $i=1,\dots,K$1, $i=1,\dots,K$2, constructs uncertainty sets, and derives a robust policy via Bellman equation minimax optimization.
• Regret bound: For finite $i=1,\dots,K$3 and ergodic sampling,

$i=1,\dots,K$4

for cumulative discounted-value-function regret (Gupta et al., 2022).

• Trade-offs: DSEE avoids the random interruptions of exploitation typical in confidence-bound or optimism-based algorithms, and is suitable where deterministic, predictable decision phases are preferred.

6. DSEE in Reliable Wireless Networking

In multi-hop wireless mesh networks, DSEE forms the basis for fully-online, reliable routing under link uncertainty (Nourzad et al., 2024):

• Problem mapping: Each directed link $i=1,\dots,K$5 acts as a Bernoulli arm with mean delivery probability $i=1,\dots,K$6. The routing objective is to learn all $i=1,\dots,K$7 to minimize cumulative routing cost via Shortest Anypath Routing.
• Exploration phase: Each node broadcasts dummy packets, updating empirical $i=1,\dots,K$8.
• Exploitation phase: Fixed empirical means are used in the Shortest Anypath First (SAF) algorithm to select forwarding sets $i=1,\dots,K$9:

$w$0

Real packet transmissions update statistics incrementally, allowing continued learning.

• Regret guarantees: The approach ensures that

$w$1

under general assumptions. This outperforms stochastic Thompson-Sampling-based schemes (TSOR) in network- and neighbor-size scaling.

• Operational impact:
  • Rapid estimation error decay: $w$2 shrinks at $w$3 rate.
  • Adaptivity to link dynamics via periodic re-exploration.
  • Provable reliability and resilience in practical routing deployments.

7. Comparison to Alternative Exploration Strategies

DSEE contrasts with continuously randomized or confidence-bound exploration strategies:

• UCB and optimistic algorithms: Continuous, often random, exploration with confidence-adjusted action selection at every time step; adaptivity but irregular phase transitions and increased computational overhead.
• Sliding-window and reset-on-change: Adaptive to non-stationarity, but require randomization, online confidence interval computation, and potentially higher storage/computational resources (Wei et al., 2018).
• DSEE strengths: Deterministic, predictable schedules; minimal sample-storage (often $w$4); explicitly tunable exploration density; extensibility to non-stationary, combinatorial, and decentralized settings (Vakili et al., 2011, Wei et al., 2018, Nourzad et al., 2024, Gupta et al., 2022).

The leading constant in DSEE regret bounds can be larger than in optimally-tuned adaptive schemes, but its deterministic phase structure is advantageous for energy-efficient scheduling, multi-agent coordination, and applications with strict operational constraints.

---

References:

• (Nourzad et al., 2024): Smart Routing with Precise Link Estimation: DSEE-Based Anypath Routing for Reliable Wireless Networking
• (Wei et al., 2018): On Abruptly-Changing and Slowly-Varying Multiarmed Bandit Problems
• (Vakili et al., 2011): Deterministic Sequencing of Exploration and Exploitation for Multi-Armed Bandit Problems
• (Gupta et al., 2022): Deterministic Sequencing of Exploration and Exploitation for Reinforcement Learning