Single-Policy Concentrability in RL

• Single-policy concentrability is defined as the worst-case ratio of the target policy's state–action occupancy to that of a given data or behavior distribution.
• It relaxes stronger uniform coverage conditions by focusing solely on the target or optimal policy, enabling sharper guarantees in offline RL.
• It plays a crucial role in offline RL, influencing sample complexity and stability in both tabular and function approximation settings.

Single-policy concentrability is a coverage condition that measures whether an offline data distribution, behavior policy, or sampling distribution adequately covers the state–action occupancy of one fixed target policy rather than all policies. In episodic finite-horizon form, the coefficient compares $d_h^{\pi}(s,a)$ against a reference marginal $\mu_h(s,a)$ at each stage; in discounted form, it compares discounted occupancies $d^\pi$ against a sampling distribution $\mu$ or $\nu$. The central motivation is statistical: if the target policy places mass on state–action pairs that the data almost never visit, off-policy estimation and optimization become unstable or impossible. Across offline RL, policy-search dynamic programming, contextual bandits, goal-conditioned RL, preference-based RL, and constrained MDPs, single-policy concentrability serves as a weaker alternative to all-policy or uniform concentrability, often yielding sharper guarantees because it only requires coverage of the policy that matters for the task (Xie et al., 2021).

1. Formal definitions and core intuition

In the episodic finite-horizon setting of policy finetuning, for any policy $\pi$ and time step $h\in\{1,\dots,H\}$, the state–action visitation probability is

$d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$

Fixing a deterministic optimal policy $\pi_\star$ and a reference policy $\mu$, the single-policy concentrability coefficient is

$\mu_h(s,a)$0

This is the worst-case ratio between how often the optimal policy visits a state–action pair and how often the reference policy visits it (Xie et al., 2021).

A closely related finite-horizon definition fixes an arbitrary target policy $\mu_h(s,a)$1 and offline data marginals $\mu_h(s,a)$2: $\mu_h(s,a)$3 This form appears in offline RL with function approximation, where the assumption is sometimes stated for one policy $\mu_h(s,a)$4 even when the main body of a paper uses a uniform constant $\mu_h(s,a)$5 for all admissible marginals $\mu_h(s,a)$6 (Tkachuk et al., 3 Oct 2025).

In discounted infinite-horizon MDPs, the analogous object is defined using discounted occupancies. One standard form is

$\mu_h(s,a)$7

or, in policy-search analyses, via the discounted occupancy measure

$\mu_h(s,a)$8

These formulations differ in notation but preserve the same basic meaning: the data-generating distribution must dominate the target policy’s occupancy (Scherrer, 2013).

The intuition is uniform across these settings. The numerator measures where the target or optimal policy needs accurate information; the denominator measures how much evidence the offline dataset or reference policy provides there. If $\mu_h(s,a)$9 is tiny while $d^\pi$0 is large, then the data have poor coverage exactly where optimal control depends on them, and statistical instability follows (Xie et al., 2021). A closely related interpretation in offline RL with trajectories is that whenever $d^\pi$1 puts weight on $d^\pi$2, the data distribution $d^\pi$3 should put at least a $d^\pi$4 fraction of that mass there, so empirical averages under $d^\pi$5 still “see” the regions relevant to $d^\pi$6 (Tkachuk et al., 3 Oct 2025).

2. Relation to stronger and alternative concentrability notions

The defining contrast is with all-policy, uniform, or sup-concentrability. In finite-horizon notation,

$d^\pi$7

which requires the data distribution to cover every policy’s occupancy, not merely the target policy or optimal policy. This is explicitly described as often much stronger, and it can be infinite even when the single-policy coefficient is finite (Xie et al., 2021). The same distinction appears in contextual bandits, where

$d^\pi$8

is stronger than $d^\pi$9, and in function-approximation settings where an all-policy $\mu$0-type coverage constant dominates the single-policy quantity $\mu$1 (Zhao et al., 9 May 2026, Zhao et al., 9 Feb 2025).

Several papers emphasize that this weakening is not merely cosmetic. In offline goal-conditioned RL, the dataset assumption is only that the regularized optimum $\mu$2 satisfies

$\mu$3

rather than requiring coverage for every candidate policy (Zhu et al., 2023). In preference-based RL, the relevant target-dependent coefficient $\mu$4 is constructed so that only the policy to be learned must be covered, and it can be upper bounded by the per-trajectory concentrability coefficient rather than by a uniform condition over all policies (Zhan et al., 2023). In direct preference optimization, PEPO is analyzed with a single-policy coefficient $\mu$5 defined for a comparator policy $\mu$6, in contrast to an all-policy quantity $\mu$7 that is generally much larger (Barla et al., 5 Feb 2026).

The literature also places single-policy concentrability alongside other, nonuniform notions. One alternative is average-case or integral concentrability, which bounds $\mu$8-type quantities rather than a supremum (Xie et al., 2021). Another is aggregated concentrability, which is induced by a function class and state aggregation rather than the original MDP states. For offline policy evaluation with value-function realizability but without Bellman completeness, the key hardness parameter becomes an aggregated coefficient $\mu$9, not the original single-policy $\nu$0 (Jia et al., 2024). A further generalization is sequence-level concentrability

$\nu$1

which measures coverage over whole trajectories or length-$\nu$2 decision sequences rather than one-step marginals (Zhou et al., 12 Jun 2025).

A recurring point is that single-policy coverage is usually presented as the minimal coverage assumption for the task at hand. In policy evaluation or optimization for one target policy, it is weaker than uniform coverage and better aligned with the actual inferential objective (Tkachuk et al., 3 Oct 2025). This suggests that single-policy concentrability is best viewed not as a universal replacement for all other coverage conditions, but as the target-policy-specific baseline against which stronger assumptions should be justified.

3. Sample-complexity role in tabular offline RL and policy finetuning

The concept became especially prominent in the theoretical study of policy finetuning, where the learner interacts online but also has access to a reference policy $\nu$3 that is close to the optimal policy in the single-policy-concentrability sense. In episodic tabular MDPs with $\nu$4 states, $\nu$5 actions, and horizon $\nu$6, the offline reduction algorithm that simply executes $\nu$7 and then runs offline policy optimization finds an $\nu$8-near-optimal policy within

$\nu$9

episodes (Xie et al., 2021). The same work proves an information-theoretic lower bound

$\pi$0

for any policy-finetuning algorithm, including adaptive ones. Together, these results imply that, up to logarithmic factors, offline reduction is unimprovable when $\pi$1, while purely online RL with cost $\pi$2 is unimprovable when $\pi$3 (Xie et al., 2021).

That paper also studies partial coverage up to an intermediate step $\pi$4: $\pi$5 with no assumption for later steps. The resulting hybrid algorithm HOOVI uses optimistic exploration for steps $\pi$6 to $\pi$7 and offline pessimistic updates for steps $\pi$8 to $\pi$9, with sample complexity roughly

$h\in\{1,\dots,H\}$0

This interpolates between pure offline and pure online regimes (Xie et al., 2021).

In discounted policy-search dynamic programming, single-policy concentrability predates this tabular offline RL line. For Conservative Policy Iteration, with $h\in\{1,\dots,H\}$1 defined through $h\in\{1,\dots,H\}$2, the returned policy satisfies

$h\in\{1,\dots,H\}$3

For Non-Stationary DPI,

$h\in\{1,\dots,H\}$4

These analyses explicitly compare $h\in\{1,\dots,H\}$5 with larger multi-step constants $h\in\{1,\dots,H\}$6 and $h\in\{1,\dots,H\}$7, concluding that single-policy constants can be arbitrarily smaller while preserving meaningful global guarantees (Scherrer, 2013).

A closely related result for local policy search uses

$h\in\{1,\dots,H\}$8

and shows that any $h\in\{1,\dots,H\}$9-approximate local optimum $d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$0 satisfies

$d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$1

That comparison is notable because the paper contrasts this single-policy quantity with a much larger DPI-style constant involving a supremum over all policies and all time steps (Scherrer et al., 2013).

4. Function approximation, trajectory structure, and realizability

Under function approximation, single-policy concentrability remains central but its sufficiency depends on additional structure. In offline RL with realizability and density-ratio modeling, PRO-RL assumes that the discounted occupancy of the $d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$2-regularized optimal policy satisfies

$d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$3

and then obtains polynomial sample complexity under realizability and boundedness of the value and density-ratio classes (Zhan et al., 2022). The same target-policy-only assumption appears in CORAL, where $d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$4 and $d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$5 enter the MIS-based augmented-Lagrangian analysis for bandits, contextual bandits, and RL (Rashidinejad et al., 2022).

A more recent line studies what trajectory data can and cannot change. One positive result shows that in finite-horizon offline RL with linear $d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$6-realizability and concentrability, trajectory data suffice for statistically efficient policy evaluation and policy optimization. Specialized to a single policy $d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$7, the concentrability assumption is

$d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$8

and the main rates are

$d_h^\pi(s,a)=\Pr_\pi(s_h=s,\;a_h=a).$9

for policy evaluation and

$\pi_\star$0

for policy optimization (Tkachuk et al., 3 Oct 2025). The key technical tool is a change-of-measure lemma,

$\pi_\star$1

which converts target-policy expectations into data-distribution expectations (Tkachuk et al., 3 Oct 2025).

At the same time, another result gives a negative answer for offline policy evaluation with value-function realizability but without Bellman completeness. It shows that the sample complexity is governed not by the original single-policy coefficient

$\pi_\star$2

but by an aggregated concentrability coefficient $\pi_\star$3 defined on a state aggregation induced by the function class (Jia et al., 2024). The paper further shows that $\pi_\star$4 may grow exponentially with horizon even when the original $\pi_\star$5 is small and the data are admissible, and that a generic reduction converts hard admissible-data instances into hard trajectory-data instances. The stated consequence is that trajectory data offer no extra benefits over admissible data for this OPE setting (Jia et al., 2024).

Taken together, these results delimit the scope of single-policy concentrability under function approximation. It can be sufficient in settings with additional trajectory structure and $\pi_\star$6-realizability (Tkachuk et al., 3 Oct 2025), yet insufficient for general OPE under value-function realizability alone, where the relevant hardness is induced by aggregation (Jia et al., 2024). A plausible implication is that “single-policy concentrability” is not one theorem but a family of task-dependent coverage assumptions whose adequacy depends strongly on the representational assumptions attached to them.

5. Variants across adjacent decision-making settings

Single-policy concentrability has been adapted well beyond standard offline RL. In offline goal-conditioned RL, the coverage condition is defined over state, action, and goal: $\pi_\star$7 and the analysis assumes $\pi_\star$8 for the $\pi_\star$9-regularized optimum (Zhu et al., 2023). Under realizability and a lower bound $\mu$0, the deterministic-case theorem gives

$\mu$1

which implies

$\mu$2

to achieve $\mu$3 suboptimality (Zhu et al., 2023).

In offline preference-based RL, where feedback is available as preferences between trajectory pairs, the single-policy coefficient is defined by

$\mu$4

It is upper bounded by the square root of the per-trajectory concentrability coefficient when $\mu$5 (Zhan et al., 2023). The paper’s upper bound is

$\mu$6

while the lower bounds show that per-step concentrability is insufficient and per-trajectory coverage is necessary up to constants (Zhan et al., 2023).

In offline contextual bandits with forward-KL regularization, the tabular single-policy constant is

$\mu$7

and the function-approximation analysis also introduces

$\mu$8

The paper proves the first $\mu$9 rates for forward-KL offline CBs under such single-policy conditions: $\mu_h(s,a)$00 in the tabular case and

$\mu_h(s,a)$01

under function approximation (Zhao et al., 9 May 2026). Reverse-KL-regularized offline contextual bandits admit a related sharp rate

$\mu_h(s,a)$02

under single-policy concentrability, with a matching lower bound establishing the necessity of multiplicative $\mu_h(s,a)$03 dependence (Zhao et al., 9 Feb 2025).

Additional variants include offline CMDPs, where the optimal safe policy’s occupancy must belong to a deviation-controlled set $\mu_h(s,a)$04 and

$\mu_h(s,a)$05

leading to an offline CMDP lower bound

$\mu_h(s,a)$06

and a near-optimal primal-dual upper bound up to logarithmic and $\mu_h(s,a)$07 factors (Chen et al., 2022). In sparse offline RL with corruption robustness, the relevant quantity becomes a sparse covariance-ratio condition

$\mu_h(s,a)$08

which controls coverage only along sparse directions relevant to the optimal policy (Tran et al., 31 Dec 2025).

6. Limitations, failure modes, and broader significance

The most basic limitation is support mismatch. If there exists $\mu_h(s,a)$09 such that $\mu_h(s,a)$10 but $\mu_h(s,a)$11, then $\mu_h(s,a)$12 and offline reduction completely fails (Xie et al., 2021). The same principle recurs in discounted settings: guarantees typically assume that whenever the target or comparator policy assigns positive mass, the behavior distribution does as well (Barla et al., 5 Feb 2026). This is why many papers treat finiteness of the single-policy constant as the minimum requirement for nonvacuous guarantees.

A second limitation is that the coefficient can hide severe multi-step phenomena. Sequence-level analysis defines

$\mu_h(s,a)$13

for length-$\mu_h(s,a)$14 trajectories and shows

$\mu_h(s,a)$15

because the sequence ratio factorizes as $\mu_h(s,a)$16 (Zhou et al., 12 Jun 2025). The resulting upper bound on cumulative $\mu_h(s,a)$17-error scales as

$\mu_h(s,a)$18

exhibiting potential exponential amplification along rare multi-step patterns (Zhou et al., 12 Jun 2025). The same paper argues that poisoning only rare patterns can collapse effective coverage and degrade agent performance severely, which casts single-step coverage diagnostics as potentially incomplete for security analysis (Zhou et al., 12 Jun 2025).

A third limitation concerns representational mismatch. In large or continuous spaces, the notion must be generalized through density-ratio estimation, covariance conditions, or function-class-dependent constructions, and controlling the resulting analogue of $\mu_h(s,a)$19 is more challenging (Xie et al., 2021). This is visible in aggregated concentrability for OPE (Jia et al., 2024), sparse covariance-ratio conditions (Tran et al., 31 Dec 2025), and D$\mu_h(s,a)$20-type function-class coverage in contextual bandits (Zhao et al., 9 May 2026, Zhao et al., 9 Feb 2025). The practical interpretation is consistent across these works: what matters is not merely whether the raw dataset visits the target policy’s support, but whether it does so in a form that is learnable under the chosen representation.

A final theme is that hybrid RL can remove the need for single-policy concentrability altogether in some non-tabular settings. In linear MDPs, hybrid algorithms RAPPEL and HYRULE achieve PAC and regret guarantees without assuming $\mu_h(s,a)$21, instead decomposing the feature space into offline-covered and online-explored subspaces and measuring coverage through subspace-specific quantities such as $\mu_h(s,a)$22 and $\mu_h(s,a)$23 (Tan et al., 2024). This does not invalidate single-policy concentrability; rather, it identifies a regime where limited online exploration can replace a restrictive offline coverage assumption.

Single-policy concentrability therefore occupies a precise place in modern learning theory for sequential decision making. It is weaker than all-policy coverage, often sufficient for sharp upper and lower bounds, and naturally tailored to target-policy evaluation or optimization. At the same time, it is not universally decisive: under weak realizability assumptions, sequence-level effects, aggregation effects, sparse high-dimensional structure, adversarial corruption, or hybrid exploration can shift the relevant hardness parameter away from the classical ratio of occupancies. This suggests that single-policy concentrability is best regarded as a canonical but not exhaustive language for coverage.