FRONT: Foresighted Policy with Interference

• The framework FRONT extends classical contextual bandit models by integrating interference, where actions impact future rewards through an ISO term.
• It employs online least squares estimation with an ε-greedy strategy and force-pull mechanism to ensure robust parameter estimation despite spillover effects.
• Its foresighted policy minimizes both immediate and consequential regret by carefully mapping past and future interference into scalar decision metrics.

Foresighted Online Policy with Interference (FRONT) is a principled framework for sequential decision making that generalizes the classical contextual bandit paradigm by explicitly modeling and counteracting interference—an agent’s action affecting not only its immediate outcome but also future rewards through spillover effects. Unlike myopic approaches that maximize individual instantaneous utility, FRONT is constructed to optimize cumulative rewards by considering how present decisions modulate the interference structure across subsequent phases.

1. Formal Model of Interference and Outcome

The FRONT paradigm augments the standard contextual bandit or online decision-making setup with an additive outcome model that incorporates historical and future interference. The conditional mean outcome for individual $t$ is parameterized as: $$μ(x_t, κ_t, a_t) = (1 - a_t) φ(x_t)^\top β_0 + a_t φ(x_t)^\top β_1 + κ_t γ$$ where:

• %%%%1%%%% is the observed context,
• $a_t \in \{0,1\}$ denotes action (e.g., treatment assignment),
• $φ(x_t)$ is a feature transformation,
• $β_0, β_1$ are action-specific coefficients,
• $γ$ quantifies interference strength,
• $κ_t = \sum_{s=1}^{t-1} w_{ts} a_s$ is an exposure mapping aggregating past actions with design weights $w_{ts}$.

Optimal policy prescriptions are "foresighted" in that the action rule at time $t$ is derived by: $$a_t^* = \mathbb{I}\left\{ φ(x_t)^\top(β_1 - β_0) + ζ_t γ \geq 0 \right\}$$ where $ζ_t = \sum_{s=t+1}^\infty w_{st}$ encodes the predicted total future impact of taking action $a_t$ on downstream interference—a term termed "Interference on Subsequent Outcome (ISO)" in the FRONT literature (Xiang et al., 17 Oct 2025).

2. Handling Interference in Online Learning

A key challenge for FRONT is exposure mapping: constructing summary statistics $\kappa_t$ and $ζ_t$ to reduce the potentially high-dimensional or networked interference structure into scalar forms that are statistically and computationally manageable. The design weights $w_{ts}$ must satisfy decay or normalization properties to keep the interference terms stable as $t$ grows.

To maintain estimator identifiability and avoid degeneracy in the covariate matrix—where interference can cause singularities—the method adopts an $\epsilon$-greedy exploration strategy:

• With probability $1-\epsilon_t$, the agent executes the foresighted optimal action based on estimated parameters.
• With probability $\epsilon_t$, the agent randomly explores.

A "force-pull" mechanism is triggered in degenerate design regimes, introducing artificial variation into $\kappa_t$ to ensure sufficient exploration for robust parameter estimation.

3. Statistical Theory: Estimator Properties

FRONT supports online least squares estimation. For parameter vector $\theta$ containing $(β_0, β_1, γ)$, the online estimator is: $$\hat{\theta}_t = \left(\frac{1}{t} \sum_{s=1}^t z_s z_s^\top\right)^{-1} \left(\frac{1}{t}\sum_{s=1}^t z_s y_s\right)$$ with $$z_s = ((1-a_s)φ(x_s)^\top,\, a_sφ(x_s)^\top,\, κ_s)^\top$$.

The estimator admits nontrivial tail bounds for the $\ell_1$ error: $$\Pr\left\{ \|\hat{\theta}_t - \theta\|_1 \leq h \right\} \geq 1 - 4d_1 \exp\left(- \frac{t\epsilon_t^2 C^2 h^2}{2d^2 \sigma^2 L_z^2}\right) - 4\exp\left(- \frac{t\epsilon_t^2 C^2 h^2}{8d^2 \sigma^2 L_z^2} \right)$$ assuming $t \epsilon_t^2 \to \infty$, bounded design and noise conditions, and well-chosen exploration schedules.

Moreover, the estimator is asymptotically normal: $$\sqrt{t}(\hat{\theta}_t - \theta) \xrightarrow{d} \mathcal{N}(0, S)$$ where $S$ is a block matrix incorporating signal and interference parameters.

4. Foresighted Versus Myopic Regret Analysis

FRONT introduces two regret quantities:

• $R_1(T)$, which measures cumulative loss relative to the optimal foresighted policy based strictly on observed rewards.
• $R_2(T)$, which incorporates "consequential regret"—the total latent loss including future interference effects propagated by current decisions.

The optimal foresighted policy, by construction, minimizes both regret forms sublinearly: $$R_1(T) = \mathcal{O}_p \left(\sum_t \epsilon_t + T^{3/4} + |\mathcal{F}_T| \right)$$ and similarly for $R_2(T)$, where $|\mathcal{F}_T|$ is the size of the force-pull set.

This property sharply discriminates FRONT from myopic or naive methods; short-sighted policies can incur linear regret due to cumulative interference amplification.

5. Implementation and Practical Impact

FRONT is operationalized via online least squares and policy evaluation in a sequential loop:

• At each time $t$, the agent observes $(x_t, \kappa_t)$, computes the decision score incorporating ISO, and updates parameter estimates via the most recent sample.
• Exploration is scheduled adaptively: $\epsilon_t$ is set to maintain statistical efficiency given interference structure and data degeneracy.
• The architecture is agnostic to underlying domain, provided interference can be mapped to scalar forms through careful weighting.

Applied to urban hotel profits, the iso-aware decision rule consistently outperforms myopic and naive benchmarks in cumulative profit, validating the efficacy of the interference-corrected strategy in practical networked environments.

6. Mathematical Formulary

Table: Principal Model Components in FRONT (Xiang et al., 17 Oct 2025)

Component	Formula/Definition	Role
Outcome model	$\mu(x_t, \kappa_t, a_t)$ as above	Encodes reward, interference
Exposure mapping	$\kappa_t = \sum_{s=1}^{t-1} w_{ts} a_s$	Scalar summary of past actions
ISO term	$\zeta_t = \sum_{s=t+1}^{\infty} w_{st}$	Future impact quantification
Online estimator	as described above	Parameter learning
Decision rule	$a_t^* = \mathbb{I}\{... \geq 0\}$	Foresighted policy
Regret bounds	$R_1(T), R_2(T)$ as above	Performance characterization

This formalism clarifies that interference is not a nuisance but a modeling primitive which, if incorporated into both estimation and action selection, can be leveraged to maximize long-term utility in online systems.

7. Significance and Broader Context

FRONT closes a methodological gap by making the mutual influence of actions explicit—forecasting how present interventions shape the conditions for future choice. The encompassing theoretical analysis covers estimator behavior, regret properties, and practical reach. The key insight is that foresight, expressed mathematically via ISO and exposure mapping, is essential for robust sequential optimization in any domain permeated by interference, and that tailored exploration (including force-pull mechanisms) is integral to sustaining statistical identifiability.

By grounding sequential decision making in interference-aware models, and rigorously quantifying tail risks and asymptotic inference properties, FRONT establishes a new standard for online policy optimization in interconnected environments. This approach constitutes a template for future development across online experimentation, networked recommender systems, and policy learning in social settings.