Optimal Target Selection Algorithm

• Optimal target selection algorithms are adaptive, non-myopic methods that choose actions to reduce estimation errors and enhance information gain.
• They employ a context-tree weighting approach with KT estimators to predict future observations and update cost-to-go functions online.
• These algorithms outperform traditional CRLB and RL methods by dynamically modeling variable-length Markov dependencies for robust tracking.

Optimal target selection algorithms are designed to choose actions, measurements, or data acquisition strategies that optimize a defined objective, often under uncertainty. In the context of sequential waveform selection for adaptive target tracking, the objective is typically to minimize estimation error or to maximize information gain in the presence of unknown and potentially complex temporal dependencies in the underlying radar scene. Universal learning approaches rooted in source coding have recently enabled performance guarantees that are unattainable by traditional estimation-theoretic or myopic reinforcement learning strategies.

1. Universal Sequential Waveform Selection: Problem Definition

The universal sequential waveform selection algorithm is constructed for active sensors, notably radars, performing target tracking in environments where the mapping between past actions (waveforms) and observations admits a $U$th order Markov process with finite but unknown memory depth $U$. At each tracking step, the system must select a probing waveform from a finite alphabet in order to optimize a long-term cost function—often combining tracking uncertainty, innovation, or general utility criteria.

The chosen strategy is non-myopic: the waveform selection at each step is based not only on current state but also on the inferred (and possibly long) temporal dependencies of the environment. The underlying challenge is managing these dependencies without prior knowledge of the true memory depth or transition probabilities.

2. Methodology: Context-Tree Weighting and Online Optimization

The core methodological innovation is recasting the waveform-agile tracking problem as an online stochastic control process with variable memory. The environment’s behavior is modeled as a stationary source parsed into variable-length phrases, each defining a distinct context. The algorithm maintains a context tree in which every node is associated with:

• An estimate of transition probabilities for the next observation, given a particular sequence of past actions and measurements (the context).
• A running estimate of the cost-to-go, obtained recursively via BeLLMan-like updates.

The context tree is updated using a multi-alphabet extension of the Context-Tree Weighting (CTW) algorithm. Each node’s probability for a future event is estimated by a Krischevsky–Trofimov (KT) estimator:

$$P_{\mathrm{KT}}(y|s)=\frac{N(s,y)+\frac{1}{2}}{\sum_{y'\in\mathcal{Y}} (N(s,y') + \frac{|\mathcal{Y}|}{2})}$$

where $N(s,y)$ is the number of times outcome $y$ has followed context $s$. For every new observation-action pair, the context tree is recursively updated—combining local estimate via KT with weighted probabilities from child nodes.

Cost-to-go at each node is updated as:

$$\widehat{J}(\text{context}) \leftarrow \min_{w} \sum_{\text{next observation}} \hat{P}(\text{next} | \text{context}) \left[g(\text{current, next}) + \gamma \widehat{J}(\text{updated context})\right]$$

Here, $\gamma$ is the discount factor and $g$ is the instantaneous cost. This recursion closely mirrors dynamic programming, but crucially, all terms (transition probabilities and costs) are learned online from data.

3. Comparison to Estimation-Theoretic and Markov Decision Process Approaches

Conventional solutions to the waveform selection problem—such as minimizing the Cramér-Rao lower bound (CRLB) on measurement error—are only valid under high SNR and restrictive modeling assumptions (e.g., Gaussianity, linearity, short memory). Such methods are typically forced to operate myopically due to their computational requirements and model fragility.

Recent reinforcement learning (RL) approaches have attempted to address long-term planning by formulating the problem as a Markov Decision Process (MDP), with policies found by value iteration or Q-learning. However, RL requires explicit knowledge or accurate estimation of the Markov process order, and in radar tracking scenarios the environment’s temporal memory length is unknown and possibly variable. Standard RL algorithms become suboptimal when their assumed model order is mismatched to the true order.

The universal CTW-based strategy does not presuppose the length of the environment’s memory. It is theoretically proven to be asymptotically BeLLMan optimal over the class of all processes that can be modeled as an unknown-order Markov process with finite $U$—even as $U$ is unknown and dynamically inferred online.

4. Mathematical Formulation and Algorithm Structure

The approach can be summarized in the following mathematical structure:

Context Tree Update:

At time $t$, context $s$ is defined by the last $k$ steps of the history, for each $0 \leq k \leq D$, where $D$ is the maximum context depth (chosen much larger than expected true memory order). Each node $s$ stores:

• $P_{\mathrm{KT}}(y | s)$ via event counts,
• a weighted mixture:

$$P^w_{s} = \begin{cases} \frac{1}{2}P_{\mathrm{KT}}^{s} + \frac{1}{2}\prod_{a\in\Sigma}P^w_{as} & \text{if } |s| < D \ P_{\mathrm{KT}}^{s} & \text{if } |s| = D \end{cases}$$

where $as$ denotes the extended context by one symbol.

BeLLMan Recursion:

For context $(s^{t-U+1}, u^{t-U}, \ldots, s^t, u^{t-1})$, compute

$$J(s^{t-U+1} \ldots s^t, u^{t-U} \ldots u^{t-1}) = \min_{w_t} \sum_{s_{t+1}} P(s_{t+1} | s^{t-U+1}, u^{t-U} \ldots, s^t, u^{t-1}, w_t) \left[ g(s^t, u^{t-1}, w_t, s_{t+1}) + \gamma J(\text{updated context}) \right]$$

This is dynamically estimated online, with transition probabilities given by the context tree.

Action (Waveform) Selection Rule:

At every step, randomly select a waveform (with small probability for exploration) or else select greedily according to current minimizer of the BeLLMan recursion at the active context node.

As the sequence grows, the context tree is updated online. When the window of recent history covers the true memory length $U$ and sufficient statistics have been collected, the estimates of transition probabilities and cost-to-go converge to the true optimal values, guaranteeing asymptotic optimality (in the sense of long-term cost minimization).

5. Applications and Theoretical Guarantees

The algorithm’s flexibility allows it to adapt across various practical tracking environments:

• Waveform-Agile Radar: Sequential selection of transmit waveforms in radar tracking where clutter, interference, or target-scene behavior exhibit unknown memory beyond typical one-step Markovianity.
• Adversarial/Evasive Scenarios: Where the target or channel adapts in response to the radar’s actions, such as in electronic warfare or spectrum-sharing.
• General Sensor or Control Systems: Any online control or measurement problem with discrete actions and finite-memory stochastically generated observations.

The explicit proof in the source establishes that, for any process with finite Markov order $U$, the universal CTW-based algorithm attains the BeLLMan-optimal cost (as the discount factor $\gamma \to 1$ and exploration is annealed).

CTW is known to be computationally efficient, with update/runtime complexity scaling linearly in the depth of the context tree and logarithmically in the data length. Redundancy bounds (difference between the code length and the best possible for any fixed Markov order) are $O(\log N)$ for data of length $N$. Additionally, the approach is robust to model mismatch and is not sensitive to rare events dominating gradients, which is a limitation of one-step estimation approaches.

6. Implications and Future Directions

The adoption of universal learning and context-tree weighting into adaptive waveform selection opens several research avenues:

• Robust, model-free radar and sensor systems that automatically tune decision strategies to the true underlying structure of the environment.
• Meta-learning extensions, where information acquired in one operational regime (e.g., with a specific target or channel class) can be transferred and used to bootstrap learning in related contexts.
• Further investigation of worst-case or risk-sensitive objectives in addition to average-case BeLLMan-optimality.
• Extension to continuous state/action spaces via quantization or tree-based representations with variable-length coding.

This universal selection strategy represents a rigorously grounded, computationally viable solution to non-myopic, optimal target (waveform) selection under minimal statistical assumptions in adaptive tracking and control.