---
title: Single-Policy Concentrability in RL
url: https://www.emergentmind.com/topics/single-policy-concentrability
type: topic
---

# Single-Policy Concentrability in RL

Single-policy concentrability is a coverage condition that measures whether an offline data distribution, behavior policy, or sampling distribution adequately covers the state–action occupancy of one fixed target policy rather than all policies. In episodic finite-horizon form, the coefficient compares $d_h^{\pi}(s,a)$ against a reference marginal $\mu_h(s,a)$ at each stage; in discounted form, it compares discounted occupancies $d^\pi$ against a sampling distribution $\mu$ or $\nu$. The central motivation is statistical: if the target policy places mass on state–action pairs that the data almost never visit, off-policy estimation and optimization become unstable or impossible. Across offline RL, policy-search dynamic programming, contextual bandits, goal-conditioned RL, preference-based RL, and constrained MDPs, single-policy concentrability serves as a weaker alternative to all-policy or uniform concentrability, often yielding sharper guarantees because it only requires coverage of the policy that matters for the task [2106.04895].

## 1. Formal definitions and core intuition

In the episodic finite-horizon setting of policy finetuning, for any policy $\pi$ and time step $h\in\{1,\dots,H\}$, the state–action visitation probability is
\[
d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).
\]
Fixing a deterministic optimal policy $\pi_\star$ and a reference policy $\mu$, the single-policy concentrability coefficient is
\[
C^\star=\max_{h\in[H],\;(s,a)\in\mathcal S\times\mathcal A}
\frac{d_h^{\pi_\star}(s,a)}{d_h^\mu(s,a)},
\qquad\text{with the convention }0/0=0.
\]
This is the worst-case ratio between how often the optimal policy visits a state–action pair and how often the reference policy visits it [2106.04895].

A closely related finite-horizon definition fixes an arbitrary target policy $\pi$ and offline data marginals $\mu_h$:
\[
C_\pi:=\max_{h=1,\dots,H}\max_{(s,a)}\frac{d_h^\pi(s,a)}{\mu_h(s,a)}<\infty.
\]
This form appears in offline RL with function approximation, where the assumption is sometimes stated for one policy $\pi$ even when the main body of a paper uses a uniform constant $C$ for all admissible marginals $\nu_h$ [2510.03494].

In discounted infinite-horizon MDPs, the analogous object is defined using discounted occupancies. One standard form is
\[
d^\pi(s,a)=(1-\gamma)\sum_{t=0}^\infty \gamma^t\Pr(s_t=s,a_t=a\mid \pi),
\qquad
C_\pi=\max_{(s,a)}\frac{d^\pi(s,a)}{\mu(s,a)},
\]
or, in policy-search analyses, via the discounted occupancy measure
\[
d_{\pi,\mu}=(1-\gamma)\mu(I-\gamma P_\pi)^{-1},
\qquad
C_{\pi^*}\text{ such that }d_{\pi^*,\mu}\le C_{\pi^*}\nu.
\]
These formulations differ in notation but preserve the same basic meaning: the data-generating distribution must dominate the target policy’s occupancy [1306.0539].

The intuition is uniform across these settings. The numerator measures where the target or optimal policy needs accurate information; the denominator measures how much evidence the offline dataset or reference policy provides there. If $d_h^\mu(s,a)$ is tiny while $d_h^{\pi_\star}(s,a)$ is large, then the data have poor coverage exactly where optimal control depends on them, and statistical instability follows [2106.04895]. A closely related interpretation in offline RL with trajectories is that whenever $\pi$ puts weight on $(s,a)$, the data distribution $\mu$ should put at least a $1/C_\pi$ fraction of that mass there, so empirical averages under $\mu$ still “see” the regions relevant to $\pi$ [2510.03494].

## 2. Relation to stronger and alternative concentrability notions

The defining contrast is with all-policy, uniform, or sup-concentrability. In finite-horizon notation,
\[
C_{\sup}=\sup_\pi\max_{h,s,a}\frac{d_h^\pi(s,a)}{d_h^\mu(s,a)},
\]
which requires the data distribution to cover every policy’s occupancy, not merely the target policy or optimal policy. This is explicitly described as often much stronger, and it can be infinite even when the single-policy coefficient is finite [2106.04895]. The same distinction appears in contextual bandits, where
\[
C^\Pi=\sup_{\pi\in\Pi,s,a}\frac{\pi(a\mid s)}{\mu(a\mid s)}
\]
is stronger than $C^{\pi^*}$, and in function-approximation settings where an all-policy $D^2$-type coverage constant dominates the single-policy quantity $D^2_{\pi^*}$ [2605.09214; 2502.06051].

Several papers emphasize that this weakening is not merely cosmetic. In offline goal-conditioned RL, the dataset assumption is only that the regularized optimum $\pi_\alpha^*$ satisfies
\[
C^{\pi_\alpha^*}\le C_\alpha^*<\infty,
\]
rather than requiring coverage for every candidate policy [2302.03770]. In preference-based RL, the relevant target-dependent coefficient $C_r(\mathcal G_r,\pi,\rho)$ is constructed so that only the policy to be learned must be covered, and it can be upper bounded by the per-trajectory concentrability coefficient rather than by a uniform condition over all policies [2305.14816]. In direct preference optimization, PEPO is analyzed with a single-policy coefficient $C^\star$ defined for a comparator policy $\pi^\star$, in contrast to an all-policy quantity $C^{\rm all}$ that is generally much larger [2602.06239].

The literature also places single-policy concentrability alongside other, nonuniform notions. One alternative is average-case or integral concentrability, which bounds $\sum_h\|\frac{d_h^\pi}{d_h^\mu}\|_{L^p}$-type quantities rather than a supremum [2106.04895]. Another is aggregated concentrability, which is induced by a function class and state aggregation rather than the original MDP states. For offline policy evaluation with value-function realizability but without Bellman completeness, the key hardness parameter becomes an aggregated coefficient $\bar C(M,\Phi,\mu,\pi)$, not the original single-policy $C(M,\mu,\pi)$ [2403.17091]. A further generalization is sequence-level concentrability
\[
C_\tau=\sup_{\tau\in\mathbb T_\mu}\frac{d^\pi(\tau)}{\mu(\tau)},
\]
which measures coverage over whole trajectories or length-$l$ decision sequences rather than one-step marginals [2506.11172].

A recurring point is that single-policy coverage is usually presented as the minimal coverage assumption for the task at hand. In policy evaluation or optimization for one target policy, it is weaker than uniform coverage and better aligned with the actual inferential objective [2510.03494]. This suggests that single-policy concentrability is best viewed not as a universal replacement for all other coverage conditions, but as the target-policy-specific baseline against which stronger assumptions should be justified.

## 3. Sample-complexity role in tabular offline RL and policy finetuning

The concept became especially prominent in the theoretical study of policy finetuning, where the learner interacts online but also has access to a reference policy $\mu$ that is close to the optimal policy in the single-policy-concentrability sense. In episodic tabular MDPs with $S$ states, $A$ actions, and horizon $H$, the offline reduction algorithm that simply executes $\mu$ and then runs offline policy optimization finds an $\varepsilon$-near-optimal policy within
\[
\widetilde O(H^3SC^\star/\varepsilon^2)
\]
episodes [2106.04895]. The same work proves an information-theoretic lower bound
\[
\Omega(H^3S\min\{C^\star,A\}/\varepsilon^2)
\]
for any policy-finetuning algorithm, including adaptive ones. Together, these results imply that, up to logarithmic factors, offline reduction is unimprovable when $C^\star\le A$, while purely online RL with cost $\widetilde O(H^3SA/\varepsilon^2)$ is unimprovable when $C^\star\ge A$ [2106.04895].

That paper also studies partial coverage up to an intermediate step $h_\star\le H$:
\[
\max_{h\le h_\star}\max_{s,a}\frac{d_h^{\pi_\star}(s,a)}{d_h^\mu(s,a)}\le C^{\rm partial},
\]
with no assumption for later steps. The resulting hybrid algorithm HOOVI uses optimistic exploration for steps $h_\star+1$ to $H$ and offline pessimistic updates for steps $1$ to $h_\star$, with sample complexity roughly
\[
\widetilde O\!\Bigl(
\underbrace{H^2h_\star S C^{\rm partial}}_{\text{offline part}}
+
\underbrace{(H-h_\star)^3SA(C^{\rm partial})^2}_{\text{online part}}
\Bigr)\Big/\varepsilon^2.
\]
This interpolates between pure offline and pure online regimes [2106.04895].

In discounted policy-search dynamic programming, single-policy concentrability predates this tabular offline RL line. For Conservative Policy Iteration, with $C_{\pi^*}$ defined through $d_{\pi^*,\mu}\le C_{\pi^*}\nu$, the returned policy satisfies
\[
\mu(v^*-v_{\pi_{k^*}})\le \frac{C_{\pi^*}}{(1-\gamma)^2}(\epsilon_{k^*}+\rho).
\]
For Non-Stationary DPI,
\[
\mu(v^*-v_{\sigma_k\cdots})\le \frac{C_{\pi^*}}{1-\gamma}\max_{1\le i\le k}\epsilon_i+2\gamma^kV_{\max}.
\]
These analyses explicitly compare $C_{\pi^*}$ with larger multi-step constants $C^{(1)}$ and $C^{(2)}$, concluding that single-policy constants can be arbitrarily smaller while preserving meaningful global guarantees [1306.0539].

A closely related result for local policy search uses
\[
C_{\mu,\nu}=\|d_{\mu,\pi_*}/\nu\|_\infty
\]
and shows that any $\varepsilon$-approximate local optimum $\hat\pi$ satisfies
\[
E_{s\sim\mu}[v_*(s)-v_{\hat\pi}(s)]
\le
\frac{1}{(1-\gamma)^2}C_{\mu,\nu}\bigl(E_\nu(\Pi)+(1-\gamma)\varepsilon\bigr).
\]
That comparison is notable because the paper contrasts this single-policy quantity with a much larger DPI-style constant involving a supremum over all policies and all time steps [1306.1520].

## 4. Function approximation, trajectory structure, and realizability

Under function approximation, single-policy concentrability remains central but its sufficiency depends on additional structure. In offline RL with realizability and density-ratio modeling, PRO-RL assumes that the discounted occupancy of the $\alpha$-regularized optimal policy satisfies
\[
\forall s,a:\quad \frac{\tilde d^\alpha(s,a)}{d^D(s,a)}\le B_w,
\]
and then obtains polynomial sample complexity under realizability and boundedness of the value and density-ratio classes [2202.04634]. The same target-policy-only assumption appears in CORAL, where $C^\pi=\inf\{C:d^\pi(s,a)/\mu(s,a)\le C\ \forall(s,a)\}$ and $C^\star=C^{\pi^\star}$ enter the MIS-based augmented-Lagrangian analysis for bandits, contextual bandits, and RL [2211.00716].

A more recent line studies what trajectory data can and cannot change. One positive result shows that in finite-horizon offline RL with linear $q^\pi$-realizability and concentrability, trajectory data suffice for statistically efficient policy evaluation and policy optimization. Specialized to a single policy $\pi$, the concentrability assumption is
\[
\max_{h\in[H]}\max_{(s,a)}\frac{\nu_h(s,a)}{\mu_h(s,a)}\le C_\pi,
\]
and the main rates are
\[
n=\widetilde\Theta(C^5H^7d^3/\varepsilon^2+L^2)
\]
for policy evaluation and
\[
n=\widetilde\Theta(C^3H^7d^3/\varepsilon^2+L^2)
\]
for policy optimization [2510.03494]. The key technical tool is a change-of-measure lemma,
\[
E_{(s,a)\sim \nu_h}f(s,a)\le C\,E_{(s,a)\sim\mu_h}f(s,a),
\]
which converts target-policy expectations into data-distribution expectations [2510.03494].

At the same time, another result gives a negative answer for offline policy evaluation with value-function realizability but without Bellman completeness. It shows that the sample complexity is governed not by the original single-policy coefficient
\[
C(M,\mu,\pi)=\max_{h\in[H-1]}\sup_{s\in\mathcal S_h,a\in\mathcal A}\frac{d_h^\pi(s,a)}{\mu_h(s,a)},
\]
but by an aggregated concentrability coefficient $\bar C(M,\Phi,\mu,\pi)$ defined on a state aggregation induced by the function class [2403.17091]. The paper further shows that $\bar C$ may grow exponentially with horizon even when the original $C$ is small and the data are admissible, and that a generic reduction converts hard admissible-data instances into hard trajectory-data instances. The stated consequence is that trajectory data offer no extra benefits over admissible data for this OPE setting [2403.17091].

Taken together, these results delimit the scope of single-policy concentrability under function approximation. It can be sufficient in settings with additional trajectory structure and $q^\pi$-realizability [2510.03494], yet insufficient for general OPE under value-function realizability alone, where the relevant hardness is induced by aggregation [2403.17091]. A plausible implication is that “single-policy concentrability” is not one theorem but a family of task-dependent coverage assumptions whose adequacy depends strongly on the representational assumptions attached to them.

## 5. Variants across adjacent decision-making settings

Single-policy concentrability has been adapted well beyond standard offline RL. In offline goal-conditioned RL, the coverage condition is defined over state, action, and goal:
\[
C^\pi=\min\Bigl\{C\ge0:\forall s,a,g,\ \frac{d^\pi(s,a,g)}{\mu(s,a,g)}\le C\Bigr\},
\]
and the analysis assumes $C^{\pi_\alpha^*}\le C_\alpha^*<\infty$ for the $\chi^2$-regularized optimum [2302.03770]. Under realizability and a lower bound $\pi(a\mid s,g)\ge \tau>0$, the deterministic-case theorem gives
\[
J(\pi^*)-J(\hat\pi)\lesssim
\Bigl[
\frac{V_{\max}^3(C_\alpha^*)^3\log(1/\tau)\log(|\mathcal V||\Pi|/\delta)}
{\tau^2N^{1/4}}
\Bigr]^{1/3},
\]
which implies
\[
N=\widetilde O((C_\alpha^*)^{12}/\varepsilon^{12})
\]
to achieve $\varepsilon$ suboptimality [2302.03770].

In offline preference-based RL, where feedback is available as preferences between trajectory pairs, the single-policy coefficient is defined by
\[
C_r(\mathcal G_r,\pi,\rho):=
\max\!\Bigl\{0,\sup_{r\in\mathcal G_r}
\frac{
\mathbb E_{\tau^0\sim\pi,\tau^1\sim\rho}
[(r^\star(\tau^1)-r^\star(\tau^0))-(r(\tau^1)-r(\tau^0))]
}{
\sqrt{
\mathbb E_{\tau^0\sim\mu_0,\tau^1\sim\mu_1}
\bigl|(r^\star(\tau^1)-r^\star(\tau^0))-(r(\tau^1)-r(\tau^0))\bigr|^2
}}
\Bigr\}.
\]
It is upper bounded by the square root of the per-trajectory concentrability coefficient when $\rho=\mu_1$ [2305.14816]. The paper’s upper bound is
\[
J(\hat\pi;r^\star)-J(\pi;r^\star)
\le
\sqrt{
\frac{c\,C_r(\mathcal G_r,\pi,\rho)^2\kappa^2
\log(\mathcal N_{\mathcal G_r}(1/N)/\delta)}
{N}
},
\]
while the lower bounds show that per-step concentrability is insufficient and per-trajectory coverage is necessary up to constants [2305.14816].

In offline contextual bandits with forward-KL regularization, the tabular single-policy constant is
\[
C^{\pi^*}:=\sup_{s\in\mathcal S,a\in\mathcal A}\frac{\pi^*(a\mid s)}{\mu(a\mid s)},
\]
and the function-approximation analysis also introduces
\[
D_{\pi^*}^2:=\mathbb E_{(s,a)\sim \rho\times\pi^*}[D_\mu^2(s,a)].
\]
The paper proves the first $\widetilde O(\varepsilon^{-1})$ rates for forward-KL offline CBs under such single-policy conditions:
\[
n=\widetilde \Theta\!\bigl(\eta(C^{\pi^*})^2|\mathcal S||\mathcal A|/\varepsilon\bigr)
\]
in the tabular case and
\[
n=\widetilde \Theta\!\bigl(\eta C^{\pi^*}D_{\pi^*}^2\log N_{\mathcal G}(\varepsilon)/\varepsilon\bigr)
\]
under function approximation [2605.09214]. Reverse-KL-regularized offline contextual bandits admit a related sharp rate
\[
n\ge c\,\eta\,D_{\pi^*}^2\,\varepsilon^{-1}\,\mathrm{polylog}(\cdots)
\]
under single-policy concentrability, with a matching lower bound establishing the necessity of multiplicative $D_{\pi^*}^2$ dependence [2502.06051].

Additional variants include offline CMDPs, where the optimal safe policy’s occupancy must belong to a deviation-controlled set $D(\phi)$ and
\[
C^*=\inf\{\phi\ge1:\exists\,\pi^*\in\Pi^*\text{ such that }\nu^{\pi^*}\in D(\phi)\},
\]
leading to an offline CMDP lower bound
\[
\Omega\!\Bigl(\frac{\min\{|\mathcal S||\mathcal A|,|\mathcal S|+I\}C^*}{(1-\gamma)^3\varepsilon^2}\Bigr)
\]
and a near-optimal primal-dual upper bound up to logarithmic and $(1-\gamma)^{-1}$ factors [2207.06147]. In sparse offline RL with corruption robustness, the relevant quantity becomes a sparse covariance-ratio condition
\[
\kappa=\max_{h\in[H]}\max_{z:\|z\|_0\le 2s}\frac{z^\top\Sigma_{\star,h}z}{z^\top\Sigma_h z},
\]
which controls coverage only along sparse directions relevant to the optimal policy [2512.24768].

## 6. Limitations, failure modes, and broader significance

The most basic limitation is support mismatch. If there exists $(h,s,a)$ such that $d_h^{\pi_\star}(s,a)>0$ but $d_h^\mu(s,a)=0$, then $C^\star=\infty$ and offline reduction completely fails [2106.04895]. The same principle recurs in discounted settings: guarantees typically assume that whenever the target or comparator policy assigns positive mass, the behavior distribution does as well [2602.06239]. This is why many papers treat finiteness of the single-policy constant as the minimum requirement for nonvacuous guarantees.

A second limitation is that the coefficient can hide severe multi-step phenomena. Sequence-level analysis defines
\[
C_\tau=\sup_{\tau\in\mathbb T_\mu}\frac{d^\pi(\tau)}{\mu(\tau)}
\]
for length-$l$ trajectories and shows
\[
C_\tau\le C^l,
\]
because the sequence ratio factorizes as $\prod_{t=0}^{l-1}\pi(a_t\mid s_t)/\mu(a_t\mid s_t)$ [2506.11172]. The resulting upper bound on cumulative $Q$-error scales as
\[
\mathbb E_{\tau\sim d^\pi}\Bigl[\sum_{t=0}^{l-1}\gamma^t(Q^*(s_t,a_t)-Q^\pi(s_t,a_t))\Bigr]
\le
\frac{2R_{\max}C_\tau}{1-\gamma}\varepsilon
\le
\frac{2R_{\max}C^l}{1-\gamma}\varepsilon,
\]
exhibiting potential exponential amplification along rare multi-step patterns [2506.11172]. The same paper argues that poisoning only rare patterns can collapse effective coverage and degrade agent performance severely, which casts single-step coverage diagnostics as potentially incomplete for security analysis [2506.11172].

A third limitation concerns representational mismatch. In large or continuous spaces, the notion must be generalized through density-ratio estimation, covariance conditions, or function-class-dependent constructions, and controlling the resulting analogue of $C^\star$ is more challenging [2106.04895]. This is visible in aggregated concentrability for OPE [2403.17091], sparse covariance-ratio conditions [2512.24768], and D$^2$-type function-class coverage in contextual bandits [2605.09214; 2502.06051]. The practical interpretation is consistent across these works: what matters is not merely whether the raw dataset visits the target policy’s support, but whether it does so in a form that is learnable under the chosen representation.

A final theme is that hybrid RL can remove the need for single-policy concentrability altogether in some non-tabular settings. In linear MDPs, hybrid algorithms RAPPEL and HYRULE achieve PAC and regret guarantees without assuming $C^*<\infty$, instead decomposing the feature space into offline-covered and online-explored subspaces and measuring coverage through subspace-specific quantities such as $c_{\rm off}$ and $d_{\rm on}$ [2408.04526]. This does not invalidate single-policy concentrability; rather, it identifies a regime where limited online exploration can replace a restrictive offline coverage assumption.

Single-policy concentrability therefore occupies a precise place in modern learning theory for sequential decision making. It is weaker than all-policy coverage, often sufficient for sharp upper and lower bounds, and naturally tailored to target-policy evaluation or optimization. At the same time, it is not universally decisive: under weak realizability assumptions, sequence-level effects, aggregation effects, sparse high-dimensional structure, adversarial corruption, or hybrid exploration can shift the relevant hardness parameter away from the classical ratio of occupancies. This suggests that single-policy concentrability is best regarded as a canonical but not exhaustive language for coverage.

Source: https://www.emergentmind.com/topics/single-policy-concentrability