--- title: Explore-Then-Commit (ETC) Methods url: https://www.emergentmind.com/topics/explore-then-commit-etc type: topic --- # Explore-Then-Commit (ETC) Methods Explore-Then-Commit (ETC) denotes a family of policies that allocate an initial segment of the horizon to pure exploration and then irrevocably commit to a decision rule for the remainder. In the literature considered here, the committed object ranges from a single arm in stochastic bandits to a controller in partially observable linear systems, an open-loop action sequence in nonstationary linear bandits with latent dynamics, a stable matching in decentralized two-sided markets, and an arm selected under a finite-time risk-return criterion [1605.08988][2002.09174][2002.00082][2510.16208][2408.08690][1904.13387]. The defining feature is the same across domains: ETC separates information acquisition from exploitation, so its performance is governed by the tension between the cost of exploration and the consequences of committing with imperfect information. ## 1. Canonical formulation In the classical stochastic bandit setting, ETC is a two-stage strategy. For a $K$-armed bandit with unknown means $\mu_1,\dots,\mu_K$, one picks an exploration budget $n$, pulls each arm $n$ times, computes empirical means $\hat\mu_i$, and then commits to $$ i^*=\arg\max_i \hat\mu_i $$ for the remaining $T-Kn$ rounds [2002.09174]. In the two-armed Gaussian formulation, ETC can be stated more generally through an exploration stopping time $\tau\le T$, an $\mathcal F_\tau$-measurable decision $\hat a\in\{1,2\}$, alternating uniform sampling during $t\le \tau$, and commitment to $\hat a$ for $t>\tau$ [1605.08988]. The standard regret expressions already expose the ETC structure. In the $K$-armed case, $$ R(T)=T\mu_1-\mathbb E\Bigl[\sum_{t=1}^T r_t\Bigr] =\sum_{i:\Delta_i>0}\Delta_i\,\mathbb E[N_i(T)], \qquad \Delta_i=\mu_1-\mu_i, $$ whereas in the two-armed Gaussian case with $\mu_1>\mu_2$ and $\Delta=|\mu_1-\mu_2|$, $$ R_\mu^\pi(T)=\Delta\,\mathbb E[N_2(T)], $$ with $$ N_2(T)=\frac{\tau\wedge T}{2}+(T-\tau)_+\,\mathbf 1\{\hat a=2\}. $$ This yields the basic decomposition $$ \Delta\,\mathbb E[(\tau\wedge T)/2] \le R_\mu^\pi(T) \le \frac{\Delta}{2}\,\mathbb E[\tau\wedge T]+\Delta T\,\mathbb P_\mu\{\tau\le T,\hat a\ne a^*\}, $$ so ETC regret is the sum of an exploration term and a post-commitment misidentification term [1605.08988]. A related finite-horizon formulation appears in risk-averse bandits. There, an ETC policy is a pair $(\pi_{\text{explore}},\pi_{\text{commit}})$: the learner performs pure experimentation for $\tau$ rounds and then commits to one fixed arm for $H=T-\tau$ exploitations [1904.13387]. This preserves the stagewise architecture of ETC while changing the criterion used to select the committed arm. ## 2. Regret structure and the classical suboptimality of single-stage ETC The main negative result attached to ETC is not that it fails to achieve sublinear regret, but that a single exploration phase followed by a single commitment phase is generally asymptotically suboptimal relative to fully sequential policies. In the two-armed Gaussian setting, fixed-design ETC with known gap $\Delta$ and deterministic $\tau=2n$ has $$ R_\mu^n(T)=\Delta\bigl[n+T\,\Phi(-\Delta\sqrt{n/2})\bigr], $$ and the optimal fixed budget satisfies $$ \inf_n R_\mu^n(T)\sim 4\,\frac{\log T}{\Delta}\qquad (T\to\infty). $$ A matching lower bound gives $\inf_n R_\mu^n(T)\ge (4-o(1))\log T/\Delta$ [1605.08988]. Allowing a data-dependent stopping time improves the constant but not the basic limitation. The SPRT-based ETC rule, which alternates arms until $$ \bigl|\hat\mu_{1,n}-\hat\mu_{2,n}\bigr| \ge \frac{\log(T\Delta^2)}{n\Delta}, \qquad \tau=2n, $$ achieves $$ R_\mu^{\rm SPRT}(T)\le \frac{\log(eT\Delta^2)}{\Delta} +\frac{4\sqrt{\log(T\Delta^2)}+4}{\Delta}+\Delta, $$ hence $R(T)=O(\log T/\Delta)$, but no ETC can achieve an asymptotic constant smaller than $1/\Delta$ in the known-gap case [1605.08988]. When $\Delta$ is unknown, the gap is larger. A fixed-confidence best-arm identification subroutine with risk $\delta=1/T$ yields BAI-ETC with asymptotic behavior $R\sim 4\log T/\Delta$, and any ETC with unknown $\Delta$ has asymptotic cost at least $4\log T/\Delta$ [1605.08988]. By contrast, fully sequential algorithms in the same paper attain smaller constants: $\Delta$-UCB satisfies $\limsup R/\log T\le 1/(2\Delta)$ in the known-gap case, and UCB$^*$ has asymptotic regret $2\log T/\Delta$ in the unknown-gap case. The standard conclusion is therefore precise: ETC is not uniformly optimal because its forced separation of exploration and exploitation creates a factor-two asymptotic penalty in the two-armed Gaussian model [1605.08988]. This corrects a common misconception. ETC is not merely a crude fixed-design heuristic; stopping-time variants such as SPRT-ETC are statistically refined. The limitation is structural: once commitment occurs, no further information can correct an early error. ## 3. Double ETC and batched asymptotic optimality "Double Explore-then-Commit: Asymptotic Optimality and Beyond" shows that the preceding negative conclusion is specific to single-stage ETC rather than to stagewise learning itself [2002.09174]. The proposed Double ETC (DETC) introduces two exploration phases and two commitment phases. In the known-gap two-armed version, DETC uses $$ T_0=\Bigl\lceil\frac{2\ln(T\Delta^2)}{\Delta^2}\Bigr\rceil, \qquad T_1=\Bigl\lceil\frac{4\ln(T\Delta^2)}{\Delta^2}\Bigr\rceil. $$ Stage I samples both arms until each has been pulled $T_0$ times. Stage II commits temporarily to the empirical best arm $i'$ until it has been pulled $T_1$ times. Stage III re-opens exploration of the other arm $j'$ and continues until $$ 2(1-\varepsilon_T)\,t_2^2\,\bigl|\mu' - \hat\mu_{j'}(t_2)\bigr|<\ln(T\Delta^2) $$ fails, or $t_2=T$. Stage IV commits to the arm with larger empirical mean [2002.09174]. The unknown-gap version replaces $T_0,T_1$ with data-dependent confidence tests and uses $T_0=\lfloor \ln^2 T\rfloor$. The paper’s interpretation is that the second exploration phase compares the temporarily chosen arm against fresh pulls of the unchosen arm, so the decisive sampling noise after Stage II comes from only one arm. This yields asymptotic optimality: $$ \lim_{T\to\infty}\frac{R(T)}{\ln T}=\frac{2}{\Delta} $$ in the unknown-gap two-arm case, and for $K$ arms $$ \lim_{T\to\infty}\frac{R(T)}{\ln T} =\sum_{i:\Delta_i>0}\frac{2}{\Delta_i}. $$ The same paper extends DETC to batched bandits and proves that a batched version attains the same asymptotic regret with only $O(1)$ rounds [2002.09174]. A plausible implication is that ETC should be understood less as a single algorithm than as a design principle. Once the phase structure is allowed to include re-entry into exploration, ETC-type methods can recover asymptotic optimality while retaining low round complexity. ## 4. ETC in partially observable linear quadratic control In control, ETC appears in a more structured form. "Regret Minimization in Partially Observable Linear Quadratic Control" studies an unknown discrete-time partially observable linear system $$ x_{t+1}=A x_t + B u_t + w_t,\qquad y_t=C x_t + z_t, $$ with Gaussian process and measurement noise, per-step cost $$ c_t=y_t^\top Q y_t + u_t^\top R u_t, $$ and comparison against the optimal average cost under the information pattern $u_t(\{y_1,\dots,y_t\})$ [2002.00082]. The proposed algorithm, ExpCommit, dedicates the first $T_{\exp}$ steps to pure exploration by drawing $$ u_t\sim \mathcal N(0,\sigma_u^2 I). $$ From the resulting input-output data, it forms overlapping $H$-length regressors and estimates the truncated Markov-parameter matrix $$ G=[0,CB,CAB,\dots,CA^{H-2}B] $$ through least squares. Under $\rho(A)<1$ and sufficiently large $T_{\exp}$, the paper proves $$ \|\hat G-G\|_2 \le \frac{R_w+R_e+R_z}{\sigma_u\sqrt{T_{\exp}-H+1}} $$ with high probability. Ho-Kalman is then applied to recover an order-$n$ realization $(\hat A,\hat B,\hat C)$ up to a similarity transform, and high-probability confidence sets for $A,B,C$ are constructed [2002.00082]. The commit phase is not a static arm choice. ExpCommit selects an optimistic model $\tilde\Theta$ inside the confidence set, computes the corresponding steady-state gains $(P,K,L)$, and applies the certainty-equivalent controller $$ u_t=-K\,\hat x_{t|t}, $$ with the state estimate updated by the Kalman filter associated with $\tilde\Theta$ [2002.00082]. The regret analysis relies on a Bellman-optimality identity and a decomposition of the commit-phase regret into $$ R_1+R_2+\cdots+R_{11}+O(T^{2/3}), $$ where different terms capture mismatches in $A,B,C,P,L$ and in noise cross-moments. Choosing $T_{\exp}=T^{2/3}$ yields $$ R(T)=\tilde O(T^{2/3}). $$ A central technical point is stability. Under the assumptions $\rho(A)<1$, controllability, observability, and uniform contractibility, the paper shows that the optimistic covariance and filter gain satisfy $$ \|\tilde P-P\|=\tilde O(\max\{\beta_A,\beta_C\}),\qquad \|\tilde L-L\|=\tilde O(\max\{\beta_A,\beta_C\}), $$ and that latent-state estimates and observations remain uniformly bounded for all $t\le T$, preventing blow-up during commitment [2002.00082]. ## 5. ETC for nonstationary linear bandits with latent dynamics "Explore-then-Commit for Nonstationary Linear Bandits with Latent Dynamics" specializes ETC to a bandit problem in which rewards depend on both actions and latent states, and the latent dynamics also depend on actions [2510.16208]. The model is $$ x_0=0,\qquad x_{t+1}=A x_t + B u_t + w_t, $$ with unknown stable $A$, action set $U=[-1,1]^p$ or $U=\{\pm 1\}^p$, and bilinear reward $$ r_t=x_t^\top C u_t + z_t. $$ The benchmark is the best open-loop sequence $$ u_{0:T}^\star=\arg\max_{u_{0:T}\in\{\pm 1\}^{p(T+1)}} \frac12\,u_{0:T}^\top S_T u_{0:T}, $$ where $S_T=M_T+M_T^\top$ and $M_T$ is a block-Toeplitz matrix with blocks $(M_T)_{ij}=C A^{j-i-1}B$ for $i