IDEA: Information Filter Directed Exploration

• IDEA is an information-directed decision-making framework that selects actions by balancing immediate rewards with targeted uncertainty reduction using metrics such as mutual information and KL divergence.
• It integrates control-theoretic techniques, including Kalman filtering and observability analysis, to guide action selection in various domains like bandits, active sensing, and reinforcement learning.
• IDEA adapts to different settings through computationally efficient policies that outperform traditional methods by reducing sample complexity and enhancing empirical performance.

Searching arXiv for the core IDEA/IDS papers to ground the article in the cited literature. arXiv search query: (Russo et al., 2014) Information-Directed Sampling Russo Van Roy arXiv search query: (Qin et al., 2023) Dual-Directed Algorithm Design for Efficient Pure Exploration Information Filter Directed Exploration Action-selection (IDEA) denotes an information-directed approach to sequential decision-making in which actions are chosen not only for immediate utility, but also for how efficiently they reduce task-relevant uncertainty. The name appears explicitly for a control-theoretic linear bandit driven by a Linear Gaussian Dynamical System (LGDS), where IDEA combines predicted reward with a covariance-reduction term grounded in Kalman filtering and observability (Gornet et al., 1 Oct 2025). Closely related formulations appear under other names, most notably Information-Directed Sampling (IDS), Information-Directed Selection, Bayesian value-of-information filtering, and information-gain-driven active sensing, each defining the “information” term with respect to a different latent object such as the optimal action, a worst-case alternative, a posterior pose distribution, or a residual offline-to-online uncertainty state (Russo et al., 2014).

1. Genealogy and conceptual scope

The conceptual lineage of IDEA is rooted in several adjacent literatures. In online optimization with partial feedback, IDS selects actions by minimizing a ratio between squared expected single-period regret and mutual information about the optimal action (Russo et al., 2014). In Bayesian information filtering, a system forwards an item not only for its immediate expected relevance but also for the value of information its feedback provides for future items in the same category, yielding a directed-exploration rule with a value-of-information index (Zhao et al., 2014). In pure exploration, the dual-directed framework replaces regret minimization by a maximin allocation objective and derives Information-Directed Selection as a hyperparameter-free sampling distribution from KKT stationarity (Qin et al., 2023). In active tactile perception, the action score is the expected information gain between the current belief and a hypothetical posterior under a one-step lookahead Bayesian filter (Murali et al., 2021). In offline-to-online reinforcement learning, the relevant uncertainty is the conditional mutual information left after conditioning on the offline dataset, and IDS is applied to that residual uncertainty (Chen, 28 May 2026).

A consistent theme across these formulations is that IDEA is not tied to a single universal score. Instead, the common structure is an action-selection rule in which the exploration term is explicitly targeted at uncertainty that matters for the downstream decision problem. Depending on the model class, that uncertainty may be represented by Shannon entropy over the optimal arm, KL divergence to alternative instances, Gaussian posterior covariance, an information-form precision matrix, or a state-space filtering error.

2. IDS as the canonical regret–information template

In the original IDS formulation, let $A$ be a finite action set, $A^*$ the unknown optimal action, $Y_{t,a}$ the observation if action $a$ is taken at time $t$, and $\mathcal{F}_t$ the history up to time $t$. The one-step IDS rule is

$$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$

where

$$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$

is the expected single-period regret, and

$$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$

is the information gain about the optimal action (Russo et al., 2014).

The same framework admits a distributional variant that minimizes

$A^*$0

over action distributions $A^*$1, with $A^*$2 and $A^*$3. A key structural property is that the minimizing distribution has support on at most two actions. This yields a computational reduction from an optimization over the simplex to a search over arm pairs and a one-dimensional convex program.

The information term has several equivalent forms. Besides the entropy-reduction expression, IDS uses

$A^*$4

which interprets the information gain as an expected KL divergence between conditional predictive distributions. This is the point at which “information-directed” exploration becomes task-specific: the action is evaluated by how much it is expected to reduce uncertainty about $A^*$5, not about the full parameter $A^*$6.

The regret analysis is organized around the information ratio. Under bounded rewards or sub-Gaussian assumptions, the cumulative information gain satisfies

$A^*$7

and if $A^*$8 uniformly, then the Bayesian regret obeys

$A^*$9

Model-specific values given for $Y_{t,a}$0 are $Y_{t,a}$1 in the general worst case, $Y_{t,a}$2 under full-information feedback, $Y_{t,a}$3 for linear bandits with $Y_{t,a}$4-dimensional features, and $Y_{t,a}$5 for semi-bandit feedback with $Y_{t,a}$6 selected components among $Y_{t,a}$7.

For Bernoulli, Gaussian, and linear bandits, the same paper also develops computable approximations. In particular, the variance-based surrogate

$Y_{t,a}$8

satisfies $Y_{t,a}$9, which leads to $a$0-IDS. In linear-Gaussian models, this becomes

$a$1

with $a$2 constructed from posterior means conditioned on the identity of $a$3.

3. Dual-directed pure-exploration formulations

In pure-exploration problems, the object of interest is no longer cumulative reward but the sample complexity required to answer a query $a$4 with high confidence. The dual-directed framework characterizes the instance complexity through

$a$5

where $a$6 is a sampling allocation over $a$7 arms, $a$8 indexes the elementary sub-queries or “culprits,” and

$a$9

is the information collected against culprit $t$0 (Qin et al., 2023).

The algorithmic contribution is a primal–dual reformulation of this maximin problem. Define

$t$1

Under strictly positive optimal proportions, there exist dual weights $t$2 such that

$t$3

and

$t$4

The Euler-type identity

$t$5

implies that $t$6 is a probability distribution over arms. Information-Directed Selection is then the per-culprit sampling rule $t$7: identify the leading culprit

$t$8

and sample an arm according to $t$9.

This yields a direct IDEA-style policy for pure exploration. In Gaussian best-arm identification, the culprit set is the set of challengers to the current best arm, and the induced sampling rule reduces to a top-two allocation with data-driven probabilities rather than a fixed tuning parameter. In thresholding bandits, $\mathcal{F}_t$0 and the rule degenerates to $\mathcal{F}_t$1, so the active culprit arm is sampled deterministically. In Gaussian $\mathcal{F}_t$2-best-arm identification, the same machinery applies after replacing the gap by $\mathcal{F}_t$3.

The main asymptotic statements are task-specific. For Gaussian best-arm identification with known variance, top-two Thompson sampling combined with Information-Directed Selection satisfies $\mathcal{F}_t$4 strongly and

$\mathcal{F}_t$5

matching the lower bound. The same optimal exponent $\mathcal{F}_t$6 is established for thresholding bandits and $\mathcal{F}_t$7-best-arm identification with the corresponding IDS specializations.

4. Information-form and linear-Gaussian realizations

A distinct but related use of the term “information filter” appears when the posterior is maintained in precision form. In linear-Gaussian bandits framed through IDS, the posterior over $\mathcal{F}_t$8 is Gaussian, the precision matrix is $\mathcal{F}_t$9, and the update is

$t$0

In this setting, IDEA is implemented by approximating

$t$1

through posterior sampling and using

$t$2

as a tractable surrogate for information about $t$3 (Russo et al., 2014). The resulting action rule minimizes $t$4, or its two-action distributional relaxation, while the information-form posterior representation keeps the computation tractable.

The explicit acronym IDEA is used in a more specialized LGDS setting in which the latent state evolves as

$t$5

and the reward is the scalar measurement

$t$6

with $t$7 itself acting as the measurement vector (Gornet et al., 1 Oct 2025). The Kalman predictor covariance update is

$t$8

IDEA then scores each action by

$t$9

The first term is the predicted reward, and the second term is the square root of the trace of the rank-1 covariance reduction induced by measuring in direction $$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$0.

This formulation makes the “information filter” interpretation literal. The exploration bonus is not arm-local optimism of the form $$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$1; it is a state-estimation term tied to how the measurement improves observability of the latent dynamics. The associated observability Gramian is

$$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$2

and the paper’s interpretation is that actions with high observability value reduce uncertainty for all future reward predictions, not only for the chosen action.

5. Domain-specific instantiations

In Bayesian information filtering, IDEA appears as a forwarding rule for a stream of categorized items. Each category $$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$3 has a latent relevance probability $$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$4 with Beta posterior parameters $$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$5, and forwarding an item yields reward $$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$6 with $$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$7 while discarding yields $$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$8 (Zhao et al., 2014). The category-wise Bellman recursion is

$$a_t \in \arg\min_a \Psi_t(a), \qquad \Psi_t(a) := \frac{(\Delta_t(a))^2}{g_t(a)},$$9

where $$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$0. The directed-exploration index is

$$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$1

Forwarding is optimal when $$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$2, equivalently when $$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$3. The structural results $$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$4, monotonic increase in $$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$5, and $$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$6 show that the policy is explicitly more exploratory than pure exploitation in low-evidence cold-start states.

In active tactile object pose estimation, the state is an object pose $$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$7, with rotation estimated by the Translation-Invariant Quaternion Filter (TIQF) using pairwise point-cloud differences that eliminate translation from the rotation update (Murali et al., 2021). An action is a ray tuple $$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$8 fired from faces of a bounding box around the current estimate. The action-selection step is one-step lookahead: for each candidate action, generate a hypothetical tactile measurement via ray–mesh intersection, compute the hypothetical posterior under TIQF, and rank actions by a divergence between the current prior and the hypothetical posterior. The paper evaluates KL divergence, Rényi divergence with $$\Delta_t(a) := \mathbb{E}[\mu_{A^*} - \mu_a \mid \mathcal{F}_t]$$9, a Fisher information metric proxy, Bhattacharyya distance, and squared $$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$0-Wasserstein distance, all in closed form for Gaussian priors and posteriors. A Gaussian mutual-information proxy,

$$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$1

is also a natural information-form interpretation of the same Bayesian filtering setup.

In offline-to-online reinforcement learning, the target of exploration is the residual uncertainty after conditioning on an offline dataset $$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$2. The central quantity is

$$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$3

the conditional mutual information between the learning target $$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$4 and the online trajectories (Chen, 28 May 2026). The regularized IDS family is defined by

$$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$5

with vanilla IDS given by $$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$6. In a known-dynamics Bayesian linear-reward model, the per-stage information gain is

$$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$7

and the total residual information has the log-determinant form

$$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$8

The same work isolates a warm-start regime with a dominated but informative probe: vanilla IDS selects the probe, while Thompson sampling never does, producing a constant-factor Bayesian regret separation.

6. Empirical behavior, comparisons, and limitations

Across regret-minimization bandits, IDS and its variance-based surrogate display strong empirical performance. For Bernoulli bandits with Beta priors, IDS outperforms Thompson sampling, Bayes-UCB, and several UCB variants and achieves the lowest average regret in the reported trials; for independent Gaussian bandits, $$g_t(a) := I(A^*; Y_{t,a} \mid \mathcal{F}_t) = H(A^* \mid \mathcal{F}_t) - \mathbb{E}_{Y_{t,a}}\!\left[H(A^* \mid \mathcal{F}_t, Y_{t,a})\right]$$9-IDS is state-of-the-art and competitive with tuned GPUCB; for linear bandits, $A^*$00-IDS substantially outperforms Thompson sampling and GPUCB and is competitive with tuned heuristics (Russo et al., 2014).

In pure exploration, the dual-directed IDS-based algorithms are reported to produce substantial reductions in expected sample complexity relative to classical top-two methods with fixed $A^*$01, elimination methods such as UGapE and KL-LUCB, and game-based or lazy mirror-ascent approaches, often by $A^*$02–$A^*$03 or more in the Gaussian and Bernoulli experiments described (Qin et al., 2023). In linear bandits, the corresponding information-form method significantly outperforms m-LinGapE, MisLid, and Frank–Wolfe Samplers, with complexity close to the lower bound $A^*$04.

The LGDS-specific IDEA algorithm is evaluated on randomly generated environments with $A^*$05, horizon $A^*$06, and several parameter distributions. It achieves the best median normalized regret across distributions and the lowest interquartile ranges except in heavy-tailed Cauchy settings, while Kalman-UCB generally performs worse, particularly under Bernoulli and Exponential parameter draws (Gornet et al., 1 Oct 2025). The same paper also proposes a Wasserstein-distance-based metric $A^*$07 for predicting whether Kalman-UCB or IDEA will perform better from the steady-state covariance structure.

Application results follow the same pattern of task-targeted information use. In the TIQF pose-estimation experiments on the Stanford Bunny, pose error decreases monotonically with additional actively selected tactile measurements; $A^*$08 is achieved with approximately $A^*$09 touches, and the evaluated criteria show similar accuracy, with Rényi and Wasserstein exhibiting comparatively lower variance (Murali et al., 2021). In offline-to-online RL, controlled bandit experiments and D4RL experiments are reported to validate the mechanism that IDS is most useful when offline data is informative but leaves biased or low-probability residual uncertainty; on six D4RL-v2 locomotion tasks, ROID exceeds BOORL’s summed online score by $A^*$10 points (Chen, 28 May 2026). In the arXiv information-filtering study, the optimal Bayesian policy substantially outperforms pure exploitation near the prior mean cost threshold and is especially useful in cold-start regimes (Zhao et al., 2014).

Several caveats recur across these literatures. First, the theoretical guarantees of classic IDS are stated in terms of mutual information about $A^*$11, not directly about $A^*$12, so surrogate criteria require separate justification (Russo et al., 2014). Second, Information-Directed Selection in pure exploration is not the same object as the original regret-to-information ratio: it is a KKT-derived allocation rule for the active culprit in a maximin complexity problem (Qin et al., 2023). Third, the active tactile work is explicitly myopic, uses deterministic hypothetical measurements from ray–mesh intersection, and does not estimate mutual information by sampling (Murali et al., 2021). Fourth, the offline-to-online theory is proved in a known-dynamics Bayesian linear-reward model; the deep ensemble implementation uses disagreement as a posterior surrogate, and a full theory for such surrogates remains open (Chen, 28 May 2026). Finally, in the LGDS setting the available theory establishes linear lower bounds and a unified upper bound for optimism-based methods rather than sublinear regret guarantees, so IDEA there should be understood as a model-based exploration rule tailored to partially observed dynamical rewards rather than a standard stochastic-bandit algorithm (Gornet et al., 1 Oct 2025).