---
title: Logged Counterfactual Exposures
url: https://www.emergentmind.com/topics/logged-counterfactual-exposures-lce
type: topic
---

# Logged Counterfactual Exposures

Logged Counterfactual Exposures (LCE) denotes a family of offline learning settings in which a target policy is learned, evaluated, or analyzed from exposures produced by a logging policy, using recorded contexts, exposed outputs or items, propensity information or probability ratios, and observed or inferred outcomes. In machine translation, LCE is the setting where a new policy is learned and evaluated purely from a log of past interactions without active exploration [1711.08621]. In semi-supervised batch learning from logged data, exposures may have known feedback or missing feedback while retaining logged propensity scores [2209.07148]. In sequential recommendation, LCE is instantiated as the joint use of logged system exposure trajectories and counterfactually augmented exposure trajectories [2504.13482]. In causal estimation from stochastic algorithms, the logged record additionally contains an unexposed alternative and the ratio of its probability to the exposed item’s probability [2607.05792].

## 1. Core representations of logged exposures

Across the cited literature, LCE is organized around the system’s realized exposure mechanism rather than around fully supervised labels. The exposed object may be a translation, a semantic parse, a discrete action, a sequence of recommendations, a continuous action, or a realized item paired with a logged alternative. What remains invariant is that the log preserves enough of the historical decision process to support counterfactual inference.

| Setting | LCE unit | Logged fields |
|---|---|---|
| Machine translation | \((x_i, y_i, \delta_i, p_i)\) | source sentence, displayed translation, reward, propensity |
| Neural semantic parsing | \((x_i, y_i, r_i, p_i)\) | question, logged parse, human bandit feedback, propensity |
| Semi-supervised batch learning | \(S=\{(x_i,a_i,p_i,c_i)\},\; S_u=\{(x_j,a_j,p_j)\}\) | known-feedback and missing-feedback exposures |
| Sequential recommendation | \(S^u=(v_1^u,b_1^u,\dots,v_n^u,b_n^u)\) | exposure sequence and feedback alphabet \(\mathcal{B}\) |
| Pairwise causal logging | \((E_i,U_i,\rho_i,Y_i)\) plus features | exposed item, unexposed item, logged ratio, outcome |

In machine translation, the ingredients align directly with structured prediction: input \(x\) is a source sentence, logged exposure \(y\) is the translation displayed by the logging policy \(h_0\), user feedback \(\delta\) is an observed reward signal, and logging propensity is \(p=h_0(y\mid x)\). In deployed web MT, logging is often exploration-free and deterministic: for each \(x\), the logger displays a single top-1 translation \(y^*\), with \(p=1\) for the displayed \(y^*\) and \(p=0\) for all other translations for that \(x\) [1711.08621].

In semantic parsing, the logging policy outputs exactly one machine-readable parse per question, and the target sequence-to-sequence model is trained from human bandit feedback on that logged output. In semi-supervised CRM, the exposure is any logged action shown by the logging policy together with its propensity, regardless of whether feedback is observed. In sequential recommendation, the basic object is not a single exposure but a trajectory of exposures and feedback events. In the pairwise causal formulation, the log is enriched with one specific unexposed item and a logged relative-probability ratio, enabling within-pair causal contrasts [1805.01252].

## 2. Estimands and counterfactual estimators

Two closely related objective conventions appear in the literature. Reward-based formulations seek to maximize future expected reward, while cost-based CRM formulations minimize expected cost. In MT, the target expected reward is
$$
R(h)=\mathbb{E}_{x\sim p(x)}\mathbb{E}_{y\sim h(\cdot\mid x)}[\Delta(x,y)].
$$
In semi-supervised batch learning, the target policy \(\pi_\theta\) minimizes the true risk
$$
R(\pi_\theta)\triangleq \mathbb{E}_{P_X}\big[\mathbb{E}_{\pi_\theta(A\mid X)}[c(A,X)]\big],
$$
with \(c\in[-1,0]\) [1711.08621; 2209.07148].

The canonical off-policy estimator is inverse propensity scoring:
$$
\hat{R}_{IPS}(h)=\frac{1}{n}\sum_{i=1}^{n}\frac{h(y_i\mid x_i)}{p_i}\,\delta_i.
$$
Under correct propensities, positivity, and ignorability, IPS is unbiased. To control variance, self-normalized IPS defines \(w_i=\frac{h(y_i\mid x_i)}{p_i}\) and uses
$$
\hat{R}_{SNIPS}(h)=\frac{\sum_{i=1}^{n} w_i\,\delta_i}{\sum_{i=1}^{n} w_i},
$$
which reduces variance but introduces a small ratios-of-means bias [1711.08621].

When logging is deterministic, MT and semantic parsing adopt deterministic propensity matching. In MT,
$$
\hat{R}_{DPM}(h)=\frac{1}{n}\sum_{i=1}^{n} h(y_i\mid x_i)\,\delta_i.
$$
Its self-normalized form is
$$
\hat{R}_{DPM+R}(h)=\frac{\sum_i h(y_i\mid x_i)\,\delta_i}{\sum_i h(y_i\mid x_i)}.
$$
A practical doubly controlled estimator augments importance weighting with a learned reward model \(\hat{\Delta}(x,y)\):
$$
\hat{R}_{DC}(h)=\frac{1}{n}\sum_{i=1}^{n}\Big(\hat{\Delta}(x_i;h)+\frac{h(y_i\mid x_i)}{p_i}\big[\delta_i-\hat{\Delta}(x_i,y_i)\big]\Big),
$$
and a variance-minimizing scalar \(c\) yields the \(c\)-DC variant [1711.08621].

Semi-supervised CRM introduces truncation at level \(\nu\in(0,1]\):
$$
\hat{R}_\nu(\pi_\theta;S)=\frac{1}{n}\sum_{i=1}^{n} c_i\,w_\nu(a_i,x_i),\qquad
w_\nu(a,x)\triangleq \frac{\pi_\theta(a\mid x)}{\max(\nu,\pi_0(a\mid x))}.
$$
The same work also uses SNIPS in deep-model experiments:
$$
\mathrm{SNIPS}(\pi_\theta;S)=
\frac{\sum_{i=1}^{n} c_i\,\frac{\pi_\theta(a_i\mid x_i)}{\pi_0(a_i\mid x_i)}}
{\sum_{i=1}^{n} \frac{\pi_\theta(a_i\mid x_i)}{\pi_0(a_i\mid x_i)}}.
$$
Because the regularizer is feedback-independent, it can be evaluated on both observed-feedback and missing-feedback exposures [2209.07148].

Structured MT experiments parameterize the target policy as a Gibbs policy over a featureized structured space,
$$
h_w(y\mid x)=
\frac{\exp\big(\alpha\, w^\top \phi(x,y)\big)}
{\sum_{y'\in\mathcal{Y}(x)} \exp\big(\alpha\, w^\top \phi(x,y')\big)},
$$
with
$$
\nabla \log h_w(y\mid x)=
\alpha\!\left(\phi(x,y)-\sum_{y'}\phi(x,y')h_w(y'|x)\right).
$$
This links counterfactual objectives directly to stochastic-gradient policy updates in structured prediction [1711.08621].

## 3. Deterministic logging, support failure, and estimator degeneracy

A recurring theme in LCE is that deterministic exposure logging breaks positivity. In web MT, the logger displays only one translation per source sentence, so support outside the logger’s choices is effectively zero in the enormous structured output space \(\mathcal{Y}(x)\). The MT analysis proves that IPS and DPM can be maximized by setting \(h(y_i\mid x_i)=1\) for all logged tuples with positive reward, regardless of their actual quality ordering. Under such spiky assignments,
$$
\hat{R}_{IPS}(h)=\frac{1}{n}\sum_i \frac{1}{p_i}\delta_i,\qquad
\hat{R}_{DPM}(h)=\frac{1}{n}\sum_i \delta_i
$$
are maximal, even though low-reward items are indiscriminately boosted [1711.08621].

Self-normalization does not remove the pathology completely. If \(D^{max}\) is the set of logged tuples with maximum reward \(\delta_{max}>0\), then assigning positive probability to just one tuple in \(D^{max}\) and zero probability to all other tuples maximizes the reweighted objective. The resulting failure mode is “spiky-at-the-top”: the model collapses onto a single best-logged exposure while ignoring other good options. The same analysis highlights importance-weight explosion when \(p_i\) is small or when the target policy concentrates on a few logged items, reducing effective sample size and destabilizing learning [1711.08621].

Semantic parsing exhibits a related deterministic-log regime. The historical system outputs exactly one parse per question, so classical IPS correction is not available. The paper therefore uses DPM and reweighting control variates. Reweighted DPM,
$$
\hat{R}_{DPM+R}(\pi_w)=
\frac{\frac{1}{n}\sum_{t=1}^{n}\delta_t\,\pi_w(y_t\mid x_t)}
{\frac{1}{n}\sum_{t=1}^{n}\pi_w(y_t\mid x_t)},
$$
reduces variance and avoids degeneracy with bias \(O(1/n)\). For stochastic optimization, the denominator is evaluated under previous parameters \(w'\), producing the one-step-late objective DPM+OSL. At token granularity, the objective becomes
$$
\hat{R}_{DPM+T}(\pi_w)=
\frac{1}{n}\sum_{t=1}^{n}\left(\sum_{j=1}^{|y_t|}\delta_{t,j}\log \pi_w(y_{t,j}\mid y_{t,<j},x_t)\right),
$$
and DPM+T+OSL combines token-level rewards with log-wide normalization [1805.01252].

The semantic parsing results show why granularity matters. Sequence-level binary rewards assign zero to any partially incorrect parse; token-level feedback provides positive signal for partially correct queries, yielding finer blame assignment. The paper states that minibatch-only normalization loses the control variate’s favorable properties, so normalization must be computed over the entire log, with periodic refresh of the denominator under previous parameters [1805.01252].

## 4. Semi-supervised, Bayesian, and regularized CRM

A second major line of LCE research treats the exposure log itself as a source of regularization, even when reward feedback is sparse or partially missing. Semi-supervised batch learning from logged data derives an upper bound on the variance of importance-weighted costs in terms of
$$
\min\big(KL(\pi_\theta\|\pi_0),\,KL(\pi_0\|\pi_\theta)\big),
$$
and a high-probability bound for truncated IPS exposes the same KL and reverse-KL terms as complexity controls. This motivates the objective
$$
\min_\theta\;\hat{R}_\nu(\pi_\theta;\mathcal{I}_{\mathrm{obs}})
+\lambda\,\Omega(\pi_\theta;\mathcal{I}_{\mathrm{obs}}\cup\mathcal{I}_{\mathrm{miss}}),
$$
where the IPS term uses only known-feedback exposures and the regularizer is computed on all exposures, including missing-feedback LCE [2209.07148].

Two feedback-independent regularizers are proposed:
$$
\Omega_{\mathrm{KL}}(\pi_\theta)=KL(\pi_\theta(A\mid X)\,\|\,\pi_0(A\mid X)),
\qquad
\Omega_{\mathrm{RKL}}(\pi_\theta)=KL(\pi_0(A\mid X)\,\|\,\pi_\theta(A\mid X)).
$$
The reverse-KL estimator reduces to weighted cross entropy,
$$
\hat{L}_{\mathrm{WCE}}(\pi_\theta)=
\sum_{a\in\mathcal{A}}\frac{1}{m_a}
\sum_{(x,a,p)\in S\cup S_u}\big[-p\log(\pi_\theta(a\mid x))\big].
$$
The paper reports that reverse KL, instantiated as WCE-S2BL, is particularly robust when feedback is sparse and logging is weak. As the ratio of missing-feedback exposures to known-feedback exposures grows, accuracy improves, consistent with the stated estimation-error analysis [2209.07148].

Bayesian CRM arrives at a different but closely related regularization principle. For a posterior \(\rho\) over hypotheses and prior \(\rho_0\), the PAC-Bayesian bound for truncated IPS involves \(\mathrm{KL}(\rho\|\rho_0)\). In the mixed-logit model class, this yields an upper bound
$$
\mathrm{KL}(\rho\|\rho_0)\le
\frac{\|\mu-\theta_0\|^2}{2\sigma_0}
+\frac{d}{2}\ln\frac{\sigma_0}{\sigma},
$$
which motivates logging-policy regularization,
$$
\lambda\|\mu-\theta_0\|^2.
$$
Here \(\theta_0\) can encode the logging policy’s softmax parameters when they are known. The convex WNLL-LPR objective is a weighted negative log-likelihood with a logging-policy-centered penalty, and the paper reports that LPR methods are simpler and more computationally efficient than POEM, with POEM about \(25\%\) slower per epoch than IPS-LPR [1806.11500].

Regularized control of variance also appears in MT. The cited remedies include self-normalization, clipping \(w_i^{(c)}=\min\{w_i,c\}\), entropy regularization
$$
\hat{R}(h)+\lambda\sum_{i=1}^{n}H(h(\cdot\mid x_i)),
$$
and doubly controlled estimators that shrink learning toward a reward model over the output space. The MT paper states that DC variants “smooth out” deterministic components and avoid degenerate solutions while improving MT performance [1711.08621].

## 5. Sequential recommendation and counterfactual exposure augmentation

In sequential recommendation, LCE is defined over exposure trajectories rather than isolated actions. For user \(u\), the logged exposure sequence is
$$
S^u=(v_1^u,b_1^u,v_2^u,b_2^u,\dots,v_n^u,b_n^u),
$$
where \(v_j^u\) is the \(j\)-th exposed item and \(b_j^u\in\mathcal{B}\) is the feedback, typically binary with \(\mathcal{B}=\{0,1\}\). The problem is formulated as an offline RL MDP \(\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)\), with state
$$
s_t=(v_1,b_1,\dots,v_t,b_t),
$$
action \(a_t=v_{t+1}\), and reward
$$
r_t=
\begin{cases}
r_{uni}, & b_{t+1}=0,\\
r_{int}, & b_{t+1}=1.
\end{cases}
$$
Non-clicked exposures receive a nonzero reward, while clicks receive a higher reward [2504.13482].

CaseRec uses a Decision Transformer formulation. The trajectory is
$$
\tau=(\hat{R}_1,s_1,a_1,\dots,\hat{R}_{T-1},s_{T-1},a_{T-1},\hat{R}_T,s_T),
$$
with return-to-go
$$
\hat{R}_t=\sum_{t'=t}^{T}\gamma^{t'}r_{t'}.
$$
The state encoder adds item and behavior embeddings, \(\mathbf{x}_i=\mathbf{v}_i+\mathbf{b}_i\), passes them through a GRU, and feeds the resulting trajectory tokens into a unidirectional transformer with MHA and FFN. Unlike vanilla DT, the recommender predicts the next high-reward action, defined as the next interacted item after time \(t\), through a cross-entropy objective over the item vocabulary [2504.13482].

The distinctive LCE contribution is counterfactual augmentation of system exposure. A logged sequence prefix is modified by replacing an exposed item \(v_k\) with a counterfactual \(v_k^*\), predicting counterfactual feedback \(b_k^*\) with a transformer-based user simulator \(g_\phi(s_t,a_t)\), and rolling out further counterfactual exposures for \(h\) steps. Two strategies are used. Random augmentation samples counterfactual items uniformly from \(\mathcal{V}\), simulating exposures under a random policy. Self-Improving augmentation perturbs the item embedding by Gaussian noise, selects the cosine-similarity nearest neighbor, and autoregressively rolls out later items using the current recommender. The former broadens exploration; the latter keeps counterfactuals near the logged trajectory manifold [2504.13482].

The training loop pre-trains the recommender on the original logged dataset \(\mathcal{D}\), iteratively builds an augmented set \(\mathcal{D}_{aug}\) until an augmentation ratio \(\delta\) is reached, and then trains on \(\mathcal{D}_{aug}\). The simulator is trained with BCE on logged trajectories:
$$
\mathcal{L}_{sim}=
-\sum_t \big[b_t\log \sigma(z_t)+(1-b_t)\log(1-\sigma(z_t))\big].
$$
The recommender uses \(\mathcal{L}_{rec}\) on both logged and augmented trajectories [2504.13482].

Empirically, CaseRec-R and CaseRec-S outperform baselines on ZhihuRec, Tenrec, and KuaiRand-15 policies. On KuaiRand-Random, where models are trained on KuaiRand-15policies but evaluated under unbiased random exposure, only CaseRec variants achieve substantial gains: Recall@20 rises from the best baseline \(0.0055\) to \(0.0229\) for CaseRec-R and \(0.0224\) for CaseRec-S, while NDCG@20 rises from \(0.0018\) to \(0.0099\) and \(0.0086\), respectively. Coverage@K is consistently highest for CaseRec-R, and moderate augmentation ratios improve performance before very large \(\delta\) values degrade it due to noise [2504.13482].

## 6. Extensions beyond discrete single-action logs

One extension treats the action as continuous and the propensity as a density. In this formulation, each logged exposure is
$$
e_i=(x_i,a_i,r_i,\pi_0(a_i\mid x_i)),
$$
with overlap and positivity requiring \(\pi_0(a\mid x)>0\) on the support of the target policy. The continuous-action IPS estimator is
$$
\widehat{R}_{IPS}(T)=
\frac{1}{n}\sum_{i=1}^{n}\frac{T(a_i\mid x_i)}{\pi_0(a_i\mid x_i)}\,r_i,
$$
and SNIPS uses the same density ratio inside a self-normalized average. Because direct-method integration over \(T(\cdot\mid x)\) is non-trivial offline for continuous stochastic policies, the cited work concentrates on IPS variants rather than a fully DR estimator [2004.11722].

The paper’s modeling contribution is a joint kernel embedding on \(X\times A\),
$$
K\big((x,a),(x',a')\big)=
\langle \phi_x(x),\phi_x(x')\rangle
\cdot
\exp\!\left(-\frac{\|a-a'\|^2}{\sigma^2}\right),
$$
with Nyström approximation over action anchors and a contextual mean action defined by a soft-argmin:
$$
\mu_\beta^{CCP}(x)=
\frac{\sum_{a\in Z} a\,\exp(-\gamma\,\eta_\beta(x,a))}
{\sum_{a'\in Z}\exp(-\gamma\,\eta_\beta(x,a'))}.
$$
Optimization uses logarithmic soft-clipping
$$
c(w,M)=
\begin{cases}
w, & w\le M,\\
a(M)\log(w+a(M)-M), & w>M,
\end{cases}
$$
and a proximal point algorithm. Offline evaluation relies on SNIPS and the effective sample size diagnostic
$$
n_{\mathrm{eff}}(\pi_\theta)=
\frac{\left(\sum_i w_i\right)^2}{\sum_i w_i^2},
$$
with the stated rule of thumb to reject estimates when \(n_{\mathrm{eff}}/n<0.01\). The released CoCoA dataset contains about \(120\)k logged exposures from a real-world continuous-action system [2004.11722].

A different extension uses LCE for causal feature-effect estimation rather than policy value estimation. Here the log contains the exposed item \(E_i\), an unexposed alternative \(U_i\), outcome \(Y_i\), and the logged ratio
$$
\rho_i=\frac{\pi_i(U_i)}{\pi_i(E_i)}.
$$
The pairwise probabilities are recovered by
$$
\pi_i(E_i)=\frac{1}{1+\rho_i},\qquad
\pi_i(U_i)=\frac{\rho_i}{1+\rho_i}.
$$
Given an antisymmetric treatment-characterizing function \(h(v,v')=-h(v',v)\), the paper defines a pseudo treatment \(D_i^h\) and identifies conditional treatment-averaged causal effects through transformed outcomes of the form \(2D_i^hY_i\). In the unequal-probability case,
$$
D_i^h=
\begin{cases}
\frac{1+\rho_i}{2}, & h(v_i,v_i^N)>0,\\[4pt]
-\frac{1+\rho_i}{2\rho_i}, & h(v_i,v_i^N)<0,\\
0, & \text{otherwise}.
\end{cases}
$$
The resulting estimator uses only the realized outcome, the specific unexposed alternative, and the pairwise probability ratio, and the paper states that identification holds even with unobserved confounders that affect both user preferences and the identity of the considered pair, provided the within-pair exposure algorithm is stochastic [2607.05792].

## 7. Assumptions, diagnostics, and reported empirical behavior

LCE methods rely on a common set of assumptions, but different papers emphasize different failure modes. Positivity or support requires that actions later chosen by the target policy have nonzero probability under the logger; deterministic logging violates this for all but the displayed output in MT and semantic parsing. Unconfoundedness or ignorability is assumed in standard off-policy formulations, and SUTVA rules out interference between units. Propensity correctness is essential in semi-supervised CRM, Bayesian CRM, and continuous-action IPS, while the pairwise causal formulation additionally requires correct logging of the exposed–unexposed pair and of the ratio \(\rho_i\) [1711.08621; 2209.07148; 2607.05792].

Variance diagnostics are central. The MT guidance recommends tracking effective sample size,
$$
ESS=\frac{\left(\sum_i w_i\right)^2}{\sum_i w_i^2},
$$
inspecting the distribution of importance weights, clipping extreme weights, and using bootstrap confidence intervals for SNIPS; the same section also notes Delta-method approximations for ratio-of-means confidence intervals. Continuous-action evaluation likewise relies on ESS and bootstrap CIs, and the sequential recommendation paper recommends monitoring simulator drift on held-out data [1711.08621; 2004.11722; 2504.13482].

The required logging granularity depends on the domain. MT and semantic parsing require the exact output \(y\), the context \(x\), and the logged or implied propensity. Sequential recommendation requires impression-level histories with sufficiently long exposure sequences; the cited implementation keeps the last \(200\) exposed items and last \(50\) clicked items, pads to fixed length, and trains a transformer with \(2\) layers, \(8\) heads, embedding dimension \(d=64\), maximum trajectory interaction steps \(T=20\), maximum state interaction steps \(L=10\), augmentation length \(h=10\), and augmentation ratio \(\delta\) varied from \(0.2\) to \(10\) [2504.13482].

Reported empirical behavior is consistently aligned with the underlying variance and support analysis. In semantic parsing with human feedback on \(995\) valid logged question–query pairs, DPM+T+OSL achieves \(58.44\pm0.09\) F1 against a supervised baseline of \(57.45\); with simulated feedback on \(22{,}765\) logged exposures, DPM+T+OSL reaches \(64.41\pm0.05\), and updating the OSL denominator once per validation, four times per epoch, is markedly better than computing it only once at the beginning [1805.01252]. In semi-supervised batch learning, with sparse feedback \((\rho=0.2)\) and weak logging \((\tau=10)\), WCE-S2BL achieves \(69.40\pm0.47\) accuracy on CIFAR-10 versus BanditNet \(50.38\pm0.55\) and logging policy \(43.45\), and the same work reports that increasing the number of missing-feedback exposures improves accuracy [2209.07148]. In MT, the stated practical conclusion is that DC or \(c\)-DC should be preferred for learning under deterministic logging, with SNIPS and clipping as evaluation defaults and entropy regularization used to prevent probability collapse [1711.08621].

Taken together, these results suggest that LCE is not a single estimator or a single data schema, but a design space for offline inference from historical exposures. Its recurring technical problems are support mismatch, variance blow-up, deterministic-logging degeneracy, and reward sparsity. Its recurring technical responses are reweighting, truncation or clipping, entropy or KL control, reward modeling, log-wide normalization, exposure augmentation, and richer logging of alternatives or missing-feedback exposures.

Source: https://www.emergentmind.com/topics/logged-counterfactual-exposures-lce