Variational Gradient Dominance (VGD)

• Variational Gradient Dominance (VGD) is a condition that links global suboptimality to the best feasible descent direction in policy space.
• It differs from stronger assumptions like completeness and coverage by accommodating misspecification through an error floor.
• VGD underpins non-Euclidean first-order methods (SDPO, CPI, PMD) that achieve explicit convergence rates and sample complexity bounds in agnostic reinforcement learning.

Searching arXiv for the target paper and closely related reinforcement-learning optimization papers. Variational gradient dominance (VGD) is a structural condition that relates global suboptimality to a feasible first-order improvement gap. In agnostic policy learning, where the policy class $\Pi$ is not assumed to contain an optimal policy, VGD serves as the assumption under which policy optimization can be analyzed as constrained first-order optimization in a non-Euclidean policy space. In "Convergence and Sample Complexity of First-Order Methods for Agnostic Reinforcement Learning" (Sherman et al., 6 Jul 2025), VGD is presented as strictly weaker than completeness and coverage assumptions and as the key condition enabling convergence and sample-complexity guarantees for Steepest Descent Policy Optimization (SDPO), Conservative Policy Iteration (CPI) interpreted through Frank-Wolfe, and on-policy Policy Mirror Descent (PMD).

1. Formal definition

For a policy class $\Pi \subset \Delta(\mathcal A)^{\mathcal S}$, VGD with respect to the value function $V$ is defined by the inequality

$$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$

Equivalently, in the abstract constrained optimization formulation used later in the analysis,

$$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$

The term

$$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$

is the variational gradient gap: it quantifies the best feasible descent direction available inside $\Pi$. The additive term $\varepsilon_{\mathrm{vgd}}$ is an error floor that allows approximate rather than exact dominance (Sherman et al., 6 Jul 2025).

This formulation is significant because it is neither an unconstrained Polyak-Łojasiewicz-type condition nor a stationarity criterion tied to a particular parametrization. The gap is defined directly over the feasible class, so the condition is intrinsic to optimization over $\Pi$. In the reinforcement-learning setting of agnostic policy learning, this makes VGD compatible with misspecification: the best attainable comparator is the best policy in the class, not an externally optimal policy.

2. Relation to completeness, coverage, and misspecification

The framework distinguishes VGD from two stronger assumptions that are standard in reinforcement-learning theory: completeness and coverage. Completeness means that for every $\pi\in\Pi$, the class contains a policy greedy with respect to $\Pi \subset \Delta(\mathcal A)^{\mathcal S}$0; explicitly, there exists $\Pi \subset \Delta(\mathcal A)^{\mathcal S}$1 such that

$\Pi \subset \Delta(\mathcal A)^{\mathcal S}$2

Coverage is the bounded distribution mismatch condition

$\Pi \subset \Delta(\mathcal A)^{\mathcal S}$3

where $\Pi \subset \Delta(\mathcal A)^{\mathcal S}$4 and $\Pi \subset \Delta(\mathcal A)^{\mathcal S}$5 is the optimal in-class policy over the relevant class (Sherman et al., 6 Jul 2025).

The paper also defines an approximate completeness error,

$\Pi \subset \Delta(\mathcal A)^{\mathcal S}$6

A key lemma shows that coverage plus approximate completeness implies VGD: $\Pi \subset \Delta(\mathcal A)^{\mathcal S}$7 In particular, if $\Pi \subset \Delta(\mathcal A)^{\mathcal S}$8 is complete then $\Pi \subset \Delta(\mathcal A)^{\mathcal S}$9-VGD holds.

The importance of this implication is conceptual as well as technical. VGD is weaker in two distinct ways: it does not require the greedy policy itself to belong to $V$0, and it permits misspecification through the error floor $V$1. A common misunderstanding is to identify VGD with completeness or bounded distribution mismatch; in this framework that identification is incorrect. VGD is the relaxed condition that survives when the policy class is only competitive in-class rather than globally realizable.

3. Non-Euclidean first-order optimization viewpoint

The central reduction is from agnostic policy learning to first-order optimization in a non-Euclidean space. Using the policy gradient theorem,

$V$2

and a local smoothness property,

$V$3

policy updates can be treated as first-order steps over a constrained policy set (Sherman et al., 6 Jul 2025).

Within this reduction, VGD is the mechanism that converts local first-order progress into global suboptimality control. Smoothness alone yields stationarity-type guarantees. VGD upgrades those guarantees to actual suboptimality bounds. The paper explicitly characterizes this as a gradient-dominance phenomenon generalized to a variational, constrained, and non-Euclidean setting.

Convexity of $V$4 is structurally important in this formulation. For SDPO and PMD, the update takes the proximal form

$V$5

with $V$6 either a squared norm or a Bregman divergence. For Frank-Wolfe/CPI, convexity is required for the update

$V$7

to remain in the feasible class. More broadly, convexity makes the quantity

$V$8

a meaningful feasible-direction gap and ensures that the algorithm optimizes over a stable decision set.

4. Consequences for policy optimization algorithms

Under convexity of $V$9 and $$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$0-VGD, the paper derives guarantees for three classical policy optimization methods (Sherman et al., 6 Jul 2025).

Steepest Descent Policy Optimization is analyzed as constrained steepest descent in policy space, especially with the $$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$1 action norm: $$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$2 The stated theorem gives a $$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$3-rate with additional dependence on $$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$4 and $$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$5. The proof uses the dependence of the local smoothness constant on $$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$6, together with the choices $$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$7 and $$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$8. The result is explicitly a non-Euclidean steepest descent guarantee in policy space rather than parameter space. The paper further emphasizes improved error dependence from $$V(\pi) - \min_{\pi^\star\in \Pi} V(\pi^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde \pi \in \Pi}\langle \nabla V(\pi), \pi - \tilde \pi\rangle + \varepsilon_{\mathrm{vgd}} \qquad \forall \pi\in \Pi.$$9 to $$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$0 and improved dependence on $$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$1 when using $$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$2 geometry.

Conservative Policy Iteration, viewed through the lens of Frank-Wolfe, uses the update

$$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$3

If $$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$4 satisfies $$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$5-VGD and $$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$6, then

$$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$7

This is an $$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$8 rate, sharper than the $$f(x) - \min_{x^\star\in \mathcal X} f(x^\star) \le \mathcal C_{\mathrm{vgd}} \max_{\tilde x\in\mathcal X}\langle \nabla f(x), x-\tilde x\rangle + \varepsilon_{\mathrm{vgd}}.$$9-type guarantees cited for the original CPI analysis. The argument uses smoothness of $$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$0 in the $$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$1 norm: $$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$2 A notable qualification is that this CPI result does not require convexity of $$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$3 in the same way as SDPO and PMD. However, the original CPI algorithm is not actor-oracle efficient because it stores all previous policies.

DA-CPI is introduced to address that efficiency issue by approximating the convex combination step with another oracle call, making it actor-efficient. Under convexity and VGD, the resulting bound has a $$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$4-type tradeoff with additional approximation terms and the same $$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$5 floor.

Policy Mirror Descent is treated as a Bregman proximal method in policy space: $$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$6 For the Euclidean regularizer $$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$7, PMD and $$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$8-SDPO coincide. The main theorem gives a $$\max_{\tilde \pi\in \Pi}\langle \nabla V(\pi), \pi-\tilde \pi\rangle$$9-type guarantee with dependence on $\Pi$0 and $\Pi$1. In the more general Bregman-proximal analysis, if the regularizer is $\Pi$2-strongly convex and $\Pi$3-smooth, then approximate optimality in the subproblem yields

$\Pi$4

Taken together, these results show that VGD is not algorithm-specific. It is the shared assumption under which several classical policy updates admit suboptimality guarantees in misspecified, agnostic settings.

5. Sample complexity and statistical translation

The iteration bounds are converted into environment-interaction guarantees through concentration of empirical policy-update objectives around their expectations. The analysis uses an on-policy sampler and a covering-number argument over $\Pi$5. The resulting bounds are polynomial in the covering number

$\Pi$6

and avoid dependence on $\Pi$7 (Sherman et al., 6 Jul 2025).

For total sample budget $\Pi$8, the strongest headline sample-complexity result stated for SDPO is of order $\Pi$9, with dependence on $\varepsilon_{\mathrm{vgd}}$0, $\varepsilon_{\mathrm{vgd}}$1, $\varepsilon_{\mathrm{vgd}}$2, $\varepsilon_{\mathrm{vgd}}$3, $\varepsilon_{\mathrm{vgd}}$4, $\varepsilon_{\mathrm{vgd}}$5, and $\varepsilon_{\mathrm{vgd}}$6. For CPI, the stated bound has the cleaner $\varepsilon_{\mathrm{vgd}}$7-type rate: $\varepsilon_{\mathrm{vgd}}$8 For DA-CPI and PMD with $\varepsilon_{\mathrm{vgd}}$9 regularization, the final rates are again of $\Pi$0-type, reflecting the same local-smoothness and exploration tradeoff.

The paper explicitly treats the exact exponents as secondary to the structural message. The main point is that all three algorithms admit agnostic guarantees under VGD, with bounds depending on the VGD constant, horizon, action cardinality, and covering number, but not on the full state-space size. This suggests that VGD supports a notion of statistical complexity governed by class geometry rather than by tabular state enumeration.

6. Empirical assessment and interpretation

The empirical analysis evaluates whether VGD is plausible in standard benchmark environments by estimating the ratio

$\Pi$1

in Cartpole-v1, Acrobot-v1, SpaceInvaders-MinAtar, and Breakout-MinAtar, using an $\Pi$2-SDPO / $\Pi$3-PMD-style procedure (Sherman et al., 6 Jul 2025).

The reported behavior is that the empirical VGD constant remains moderate throughout training and typically decreases to around $\Pi$4 or below as the algorithm converges. In Cartpole and Acrobot, the method appears to converge to the global minimum. In the harder MinAtar tasks, local optima matter, and the observed floor is interpreted as an effective $\Pi$5.

The significance of these observations is limited but concrete. They do not establish VGD as a universal property of policy classes, but they do indicate that the condition is not merely a formal device in the tested environments. A plausible implication is that VGD may be easier to satisfy or estimate in practice than completeness or coverage, especially in agnostic settings where exact greedy closure is unrealistic.

7. Conceptual role within agnostic reinforcement learning

Within the framework of agnostic reinforcement learning, VGD functions as the assumption that makes policy-space first-order methods analyzable without requiring realizability of an optimal policy. The paper’s central message is that agnostic policy learning can be treated through a non-Euclidean first-order optimization lens once the policy class satisfies VGD, replacing the stronger completeness and coverage assumptions that dominate much of the literature (Sherman et al., 6 Jul 2025).

This reframing has two consequences. First, it separates optimization geometry from parametrization: the updates are posed directly in policy space through local norms or Bregman divergences, rather than through a particular actor parameterization. Second, it separates misspecification from convergence analysis: the residual $\Pi$6 encodes the irreducible floor associated with an imperfect class, while $\Pi$7 controls how efficiently variational first-order progress converts into objective improvement.

In that sense, VGD is best understood not as a single algorithmic trick but as a unifying condition for constrained non-Euclidean policy optimization. It explains why smooth first-order policy updates can yield global in-class guarantees in agnostic settings, and it clarifies the precise sense in which those guarantees are weaker in assumptions yet still quantitatively explicit.