Logged Counterfactual Exposures

• LCE is a framework for offline counterfactual inference that uses logged exposures—including contexts, outputs, and propensities—to evaluate or learn new policies.
• It leverages estimators such as IPS, DPM, and doubly robust methods to handle issues like deterministic logging, variance explosion, and support mismatch.
• Extensions of LCE address sequential recommendation, continuous actions, and causal effect estimation, enhancing its robustness against reward sparsity and estimation bias.

Logged Counterfactual Exposures (LCE) denotes a family of offline learning settings in which a target policy is learned, evaluated, or analyzed from exposures produced by a logging policy, using recorded contexts, exposed outputs or items, propensity information or probability ratios, and observed or inferred outcomes. In machine translation, LCE is the setting where a new policy is learned and evaluated purely from a log of past interactions without active exploration (Lawrence et al., 2017). In semi-supervised batch learning from logged data, exposures may have known feedback or missing feedback while retaining logged propensity scores (Aminian et al., 2022). In sequential recommendation, LCE is instantiated as the joint use of logged system exposure trajectories and counterfactually augmented exposure trajectories (Zhao et al., 18 Apr 2025). In causal estimation from stochastic algorithms, the logged record additionally contains an unexposed alternative and the ratio of its probability to the exposed item’s probability (Athey et al., 7 Jul 2026).

1. Core representations of logged exposures

Across the cited literature, LCE is organized around the system’s realized exposure mechanism rather than around fully supervised labels. The exposed object may be a translation, a semantic parse, a discrete action, a sequence of recommendations, a continuous action, or a realized item paired with a logged alternative. What remains invariant is that the log preserves enough of the historical decision process to support counterfactual inference.

Setting	LCE unit	Logged fields
Machine translation	$(x_i, y_i, \delta_i, p_i)$	source sentence, displayed translation, reward, propensity
Neural semantic parsing	$(x_i, y_i, r_i, p_i)$	question, logged parse, human bandit feedback, propensity
Semi-supervised batch learning	$S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$	known-feedback and missing-feedback exposures
Sequential recommendation	$S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$	exposure sequence and feedback alphabet $\mathcal{B}$
Pairwise causal logging	$(E_i,U_i,\rho_i,Y_i)$ plus features	exposed item, unexposed item, logged ratio, outcome

In machine translation, the ingredients align directly with structured prediction: input $x$ is a source sentence, logged exposure $y$ is the translation displayed by the logging policy $h_0$, user feedback $\delta$ is an observed reward signal, and logging propensity is $(x_i, y_i, r_i, p_i)$0. In deployed web MT, logging is often exploration-free and deterministic: for each $(x_i, y_i, r_i, p_i)$1, the logger displays a single top-1 translation $(x_i, y_i, r_i, p_i)$2, with $(x_i, y_i, r_i, p_i)$3 for the displayed $(x_i, y_i, r_i, p_i)$4 and $(x_i, y_i, r_i, p_i)$5 for all other translations for that $(x_i, y_i, r_i, p_i)$6 (Lawrence et al., 2017).

In semantic parsing, the logging policy outputs exactly one machine-readable parse per question, and the target sequence-to-sequence model is trained from human bandit feedback on that logged output. In semi-supervised CRM, the exposure is any logged action shown by the logging policy together with its propensity, regardless of whether feedback is observed. In sequential recommendation, the basic object is not a single exposure but a trajectory of exposures and feedback events. In the pairwise causal formulation, the log is enriched with one specific unexposed item and a logged relative-probability ratio, enabling within-pair causal contrasts (Lawrence et al., 2018).

2. Estimands and counterfactual estimators

Two closely related objective conventions appear in the literature. Reward-based formulations seek to maximize future expected reward, while cost-based CRM formulations minimize expected cost. In MT, the target expected reward is

$(x_i, y_i, r_i, p_i)$7

In semi-supervised batch learning, the target policy $(x_i, y_i, r_i, p_i)$8 minimizes the true risk

$(x_i, y_i, r_i, p_i)$9

with $S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$0 (Lawrence et al., 2017, Aminian et al., 2022).

The canonical off-policy estimator is inverse propensity scoring:

$S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$1

Under correct propensities, positivity, and ignorability, IPS is unbiased. To control variance, self-normalized IPS defines $S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$2 and uses

$S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$3

which reduces variance but introduces a small ratios-of-means bias (Lawrence et al., 2017).

When logging is deterministic, MT and semantic parsing adopt deterministic propensity matching. In MT,

$S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$4

Its self-normalized form is

$S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$5

A practical doubly controlled estimator augments importance weighting with a learned reward model $S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$6:

$S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$7

and a variance-minimizing scalar $S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$8 yields the $S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}$9-DC variant (Lawrence et al., 2017).

Semi-supervised CRM introduces truncation at level $S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$0:

$S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$1

The same work also uses SNIPS in deep-model experiments:

$S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$2

Because the regularizer is feedback-independent, it can be evaluated on both observed-feedback and missing-feedback exposures (Aminian et al., 2022).

Structured MT experiments parameterize the target policy as a Gibbs policy over a featureized structured space,

$S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$3

with

$S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$4

This links counterfactual objectives directly to stochastic-gradient policy updates in structured prediction (Lawrence et al., 2017).

3. Deterministic logging, support failure, and estimator degeneracy

A recurring theme in LCE is that deterministic exposure logging breaks positivity. In web MT, the logger displays only one translation per source sentence, so support outside the logger’s choices is effectively zero in the enormous structured output space $S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$5. The MT analysis proves that IPS and DPM can be maximized by setting $S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$6 for all logged tuples with positive reward, regardless of their actual quality ordering. Under such spiky assignments,

$S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$7

are maximal, even though low-reward items are indiscriminately boosted (Lawrence et al., 2017).

Self-normalization does not remove the pathology completely. If $S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$8 is the set of logged tuples with maximum reward $S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)$9, then assigning positive probability to just one tuple in $\mathcal{B}$0 and zero probability to all other tuples maximizes the reweighted objective. The resulting failure mode is “spiky-at-the-top”: the model collapses onto a single best-logged exposure while ignoring other good options. The same analysis highlights importance-weight explosion when $\mathcal{B}$1 is small or when the target policy concentrates on a few logged items, reducing effective sample size and destabilizing learning (Lawrence et al., 2017).

Semantic parsing exhibits a related deterministic-log regime. The historical system outputs exactly one parse per question, so classical IPS correction is not available. The paper therefore uses DPM and reweighting control variates. Reweighted DPM,

$\mathcal{B}$2

reduces variance and avoids degeneracy with bias $\mathcal{B}$3. For stochastic optimization, the denominator is evaluated under previous parameters $\mathcal{B}$4, producing the one-step-late objective DPM+OSL. At token granularity, the objective becomes

$\mathcal{B}$5

and DPM+T+OSL combines token-level rewards with log-wide normalization (Lawrence et al., 2018).

The semantic parsing results show why granularity matters. Sequence-level binary rewards assign zero to any partially incorrect parse; token-level feedback provides positive signal for partially correct queries, yielding finer blame assignment. The paper states that minibatch-only normalization loses the control variate’s favorable properties, so normalization must be computed over the entire log, with periodic refresh of the denominator under previous parameters (Lawrence et al., 2018).

4. Semi-supervised, Bayesian, and regularized CRM

A second major line of LCE research treats the exposure log itself as a source of regularization, even when reward feedback is sparse or partially missing. Semi-supervised batch learning from logged data derives an upper bound on the variance of importance-weighted costs in terms of

$\mathcal{B}$6

and a high-probability bound for truncated IPS exposes the same KL and reverse-KL terms as complexity controls. This motivates the objective

$\mathcal{B}$7

where the IPS term uses only known-feedback exposures and the regularizer is computed on all exposures, including missing-feedback LCE (Aminian et al., 2022).

Two feedback-independent regularizers are proposed:

$\mathcal{B}$8

The reverse-KL estimator reduces to weighted cross entropy,

$\mathcal{B}$9

The paper reports that reverse KL, instantiated as WCE-S2BL, is particularly robust when feedback is sparse and logging is weak. As the ratio of missing-feedback exposures to known-feedback exposures grows, accuracy improves, consistent with the stated estimation-error analysis (Aminian et al., 2022).

Bayesian CRM arrives at a different but closely related regularization principle. For a posterior $(E_i,U_i,\rho_i,Y_i)$0 over hypotheses and prior $(E_i,U_i,\rho_i,Y_i)$1, the PAC-Bayesian bound for truncated IPS involves $(E_i,U_i,\rho_i,Y_i)$2. In the mixed-logit model class, this yields an upper bound

$(E_i,U_i,\rho_i,Y_i)$3

which motivates logging-policy regularization,

$(E_i,U_i,\rho_i,Y_i)$4

Here $(E_i,U_i,\rho_i,Y_i)$5 can encode the logging policy’s softmax parameters when they are known. The convex WNLL-LPR objective is a weighted negative log-likelihood with a logging-policy-centered penalty, and the paper reports that LPR methods are simpler and more computationally efficient than POEM, with POEM about $(E_i,U_i,\rho_i,Y_i)$6 slower per epoch than IPS-LPR (London et al., 2018).

Regularized control of variance also appears in MT. The cited remedies include self-normalization, clipping $(E_i,U_i,\rho_i,Y_i)$7, entropy regularization

$(E_i,U_i,\rho_i,Y_i)$8

and doubly controlled estimators that shrink learning toward a reward model over the output space. The MT paper states that DC variants “smooth out” deterministic components and avoid degenerate solutions while improving MT performance (Lawrence et al., 2017).

5. Sequential recommendation and counterfactual exposure augmentation

In sequential recommendation, LCE is defined over exposure trajectories rather than isolated actions. For user $(E_i,U_i,\rho_i,Y_i)$9, the logged exposure sequence is

$x$0

where $x$1 is the $x$2-th exposed item and $x$3 is the feedback, typically binary with $x$4. The problem is formulated as an offline RL MDP $x$5, with state

$x$6

action $x$7, and reward

$x$8

Non-clicked exposures receive a nonzero reward, while clicks receive a higher reward (Zhao et al., 18 Apr 2025).

CaseRec uses a Decision Transformer formulation. The trajectory is

$x$9

with return-to-go

$y$0

The state encoder adds item and behavior embeddings, $y$1, passes them through a GRU, and feeds the resulting trajectory tokens into a unidirectional transformer with MHA and FFN. Unlike vanilla DT, the recommender predicts the next high-reward action, defined as the next interacted item after time $y$2, through a cross-entropy objective over the item vocabulary (Zhao et al., 18 Apr 2025).

The distinctive LCE contribution is counterfactual augmentation of system exposure. A logged sequence prefix is modified by replacing an exposed item $y$3 with a counterfactual $y$4, predicting counterfactual feedback $y$5 with a transformer-based user simulator $y$6, and rolling out further counterfactual exposures for $y$7 steps. Two strategies are used. Random augmentation samples counterfactual items uniformly from $y$8, simulating exposures under a random policy. Self-Improving augmentation perturbs the item embedding by Gaussian noise, selects the cosine-similarity nearest neighbor, and autoregressively rolls out later items using the current recommender. The former broadens exploration; the latter keeps counterfactuals near the logged trajectory manifold (Zhao et al., 18 Apr 2025).

The training loop pre-trains the recommender on the original logged dataset $y$9, iteratively builds an augmented set $h_0$0 until an augmentation ratio $h_0$1 is reached, and then trains on $h_0$2. The simulator is trained with BCE on logged trajectories:

$h_0$3

The recommender uses $h_0$4 on both logged and augmented trajectories (Zhao et al., 18 Apr 2025).

Empirically, CaseRec-R and CaseRec-S outperform baselines on ZhihuRec, Tenrec, and KuaiRand-15 policies. On KuaiRand-Random, where models are trained on KuaiRand-15policies but evaluated under unbiased random exposure, only CaseRec variants achieve substantial gains: Recall@20 rises from the best baseline $h_0$5 to $h_0$6 for CaseRec-R and $h_0$7 for CaseRec-S, while NDCG@20 rises from $h_0$8 to $h_0$9 and $\delta$0, respectively. Coverage@K is consistently highest for CaseRec-R, and moderate augmentation ratios improve performance before very large $\delta$1 values degrade it due to noise (Zhao et al., 18 Apr 2025).

6. Extensions beyond discrete single-action logs

One extension treats the action as continuous and the propensity as a density. In this formulation, each logged exposure is

$\delta$2

with overlap and positivity requiring $\delta$3 on the support of the target policy. The continuous-action IPS estimator is

$\delta$4

and SNIPS uses the same density ratio inside a self-normalized average. Because direct-method integration over $\delta$5 is non-trivial offline for continuous stochastic policies, the cited work concentrates on IPS variants rather than a fully DR estimator (Zenati et al., 2020).

The paper’s modeling contribution is a joint kernel embedding on $\delta$6,

$\delta$7

with Nyström approximation over action anchors and a contextual mean action defined by a soft-argmin:

$\delta$8

Optimization uses logarithmic soft-clipping

$\delta$9

and a proximal point algorithm. Offline evaluation relies on SNIPS and the effective sample size diagnostic

$(x_i, y_i, r_i, p_i)$00

with the stated rule of thumb to reject estimates when $(x_i, y_i, r_i, p_i)$01. The released CoCoA dataset contains about $(x_i, y_i, r_i, p_i)$02k logged exposures from a real-world continuous-action system (Zenati et al., 2020).

A different extension uses LCE for causal feature-effect estimation rather than policy value estimation. Here the log contains the exposed item $(x_i, y_i, r_i, p_i)$03, an unexposed alternative $(x_i, y_i, r_i, p_i)$04, outcome $(x_i, y_i, r_i, p_i)$05, and the logged ratio

$(x_i, y_i, r_i, p_i)$06

The pairwise probabilities are recovered by

$(x_i, y_i, r_i, p_i)$07

Given an antisymmetric treatment-characterizing function $(x_i, y_i, r_i, p_i)$08, the paper defines a pseudo treatment $(x_i, y_i, r_i, p_i)$09 and identifies conditional treatment-averaged causal effects through transformed outcomes of the form $(x_i, y_i, r_i, p_i)$10. In the unequal-probability case,

$(x_i, y_i, r_i, p_i)$11

The resulting estimator uses only the realized outcome, the specific unexposed alternative, and the pairwise probability ratio, and the paper states that identification holds even with unobserved confounders that affect both user preferences and the identity of the considered pair, provided the within-pair exposure algorithm is stochastic (Athey et al., 7 Jul 2026).

7. Assumptions, diagnostics, and reported empirical behavior

LCE methods rely on a common set of assumptions, but different papers emphasize different failure modes. Positivity or support requires that actions later chosen by the target policy have nonzero probability under the logger; deterministic logging violates this for all but the displayed output in MT and semantic parsing. Unconfoundedness or ignorability is assumed in standard off-policy formulations, and SUTVA rules out interference between units. Propensity correctness is essential in semi-supervised CRM, Bayesian CRM, and continuous-action IPS, while the pairwise causal formulation additionally requires correct logging of the exposed–unexposed pair and of the ratio $(x_i, y_i, r_i, p_i)$12 (Lawrence et al., 2017, Aminian et al., 2022, Athey et al., 7 Jul 2026).

Variance diagnostics are central. The MT guidance recommends tracking effective sample size,

$(x_i, y_i, r_i, p_i)$13

inspecting the distribution of importance weights, clipping extreme weights, and using bootstrap confidence intervals for SNIPS; the same section also notes Delta-method approximations for ratio-of-means confidence intervals. Continuous-action evaluation likewise relies on ESS and bootstrap CIs, and the sequential recommendation paper recommends monitoring simulator drift on held-out data (Lawrence et al., 2017, Zenati et al., 2020, Zhao et al., 18 Apr 2025).

The required logging granularity depends on the domain. MT and semantic parsing require the exact output $(x_i, y_i, r_i, p_i)$14, the context $(x_i, y_i, r_i, p_i)$15, and the logged or implied propensity. Sequential recommendation requires impression-level histories with sufficiently long exposure sequences; the cited implementation keeps the last $(x_i, y_i, r_i, p_i)$16 exposed items and last $(x_i, y_i, r_i, p_i)$17 clicked items, pads to fixed length, and trains a transformer with $(x_i, y_i, r_i, p_i)$18 layers, $(x_i, y_i, r_i, p_i)$19 heads, embedding dimension $(x_i, y_i, r_i, p_i)$20, maximum trajectory interaction steps $(x_i, y_i, r_i, p_i)$21, maximum state interaction steps $(x_i, y_i, r_i, p_i)$22, augmentation length $(x_i, y_i, r_i, p_i)$23, and augmentation ratio $(x_i, y_i, r_i, p_i)$24 varied from $(x_i, y_i, r_i, p_i)$25 to $(x_i, y_i, r_i, p_i)$26 (Zhao et al., 18 Apr 2025).

Reported empirical behavior is consistently aligned with the underlying variance and support analysis. In semantic parsing with human feedback on $(x_i, y_i, r_i, p_i)$27 valid logged question–query pairs, DPM+T+OSL achieves $(x_i, y_i, r_i, p_i)$28 F1 against a supervised baseline of $(x_i, y_i, r_i, p_i)$29; with simulated feedback on $(x_i, y_i, r_i, p_i)$30 logged exposures, DPM+T+OSL reaches $(x_i, y_i, r_i, p_i)$31, and updating the OSL denominator once per validation, four times per epoch, is markedly better than computing it only once at the beginning (Lawrence et al., 2018). In semi-supervised batch learning, with sparse feedback $(x_i, y_i, r_i, p_i)$32 and weak logging $(x_i, y_i, r_i, p_i)$33, WCE-S2BL achieves $(x_i, y_i, r_i, p_i)$34 accuracy on CIFAR-10 versus BanditNet $(x_i, y_i, r_i, p_i)$35 and logging policy $(x_i, y_i, r_i, p_i)$36, and the same work reports that increasing the number of missing-feedback exposures improves accuracy (Aminian et al., 2022). In MT, the stated practical conclusion is that DC or $(x_i, y_i, r_i, p_i)$37-DC should be preferred for learning under deterministic logging, with SNIPS and clipping as evaluation defaults and entropy regularization used to prevent probability collapse (Lawrence et al., 2017).

Taken together, these results suggest that LCE is not a single estimator or a single data schema, but a design space for offline inference from historical exposures. Its recurring technical problems are support mismatch, variance blow-up, deterministic-logging degeneracy, and reward sparsity. Its recurring technical responses are reweighting, truncation or clipping, entropy or KL control, reward modeling, log-wide normalization, exposure augmentation, and richer logging of alternatives or missing-feedback exposures.