Privileged Value Functions (PVF) in Reinforcement Learning

• Privileged Value Functions are training-time critics that use task-relevant information unavailable to the policy—such as reference answers, solved states, or sibling rollouts—to improve token-level value estimates without changing the expected policy-gradient objective.
• In classical reinforcement learning, the related term proto-value function describes spectral features derived from state-transition graphs, while reward-dependent proto-value functions incorporate immediate rewards to represent task-specific value structure more effectively.
• In combinatorics on words, the analogous concept is privileged complexity, which counts privileged factors by length; for the Thue–Morse word, this complexity is unbounded but also contains arbitrarily long intervals of zero values.

Privileged Value Functions (PVF) denotes distinct concepts in reinforcement learning and combinatorics on words. In large-language-model reinforcement learning, a Privileged Value Function is a critic conditioned on task-relevant information unavailable to the policy, producing token-level baselines intended to reduce gradient variance without changing the expected policy-gradient objective. In classical reinforcement learning, closely related terminology refers to reward-dependent proto-value functions, which modify topology-based spectral representations using observed rewards. In combinatorics on words, “PVF” is not an abbreviation used by the relevant paper; its natural counterpart is the privileged complexity function, which counts privileged factors at each length.

1. Terminological scope and disambiguation

The term Privileged Value Function is used explicitly in “Le Critique: Privileged Value Functions for LLM Reinforcement Learning” (Venkatraman et al., 17 Aug 2026). There, PVFs are critic-side representations for RL with LLM policies. The policy acts only on its ordinary history, whereas the value model receives additional training-time information. The resulting prediction is used as a token-level baseline in the policy-gradient estimator.

A separate line of work studies proto-value functions (PVFs), spectral basis functions derived from the topology of a sampled state space. “Shaping Proto-Value Functions via Rewards” (Narayanan et al., 2015) introduces reward-dependent proto-value functions (RPVFs) by incorporating immediate rewards into the diffusion or similarity operator used to construct the spectral basis. In this usage, the abbreviation PVF refers to proto-value function, not Privileged Value Function.

In combinatorics on words, privileged factors are defined recursively through complete first returns. The paper “Privileged Factors in the Thue-Morse Word” (Peltomäki, 2013) uses the privileged complexity function

$A_w(n)=|\operatorname{Pri}_w(n)|,$

where $\operatorname{Pri}_w(n)$ is the set of privileged factors of length $n$ occurring in an infinite word $w$. The abbreviation “PVF” is absent from that paper. If PVF is used to denote the sequence of counts of privileged factors by length, it corresponds to $A_w$, and in the Thue–Morse case to $A_t$.

These meanings should not be conflated:

Usage	Object	Principal information source
LLM reinforcement learning	Privileged Value Function	Critic history plus admissible privileged context
Classical reinforcement learning	Proto-value function or RPVF	State-space graph, optionally modified by rewards
Combinatorics on words	Privileged complexity function $A_w(n)$	Privileged factors occurring at length $n$

2. Privileged Value Functions in LLM reinforcement learning

Consider a batch of responses $\tau_i=(y_{i,1},\ldots,y_{i,T_i})$ to prompts $x_i$, with policy history

$\operatorname{Pri}_w(n)$0

A standard token-normalized policy-gradient estimator assigns an advantage $\operatorname{Pri}_w(n)$1 to each token. An ordinary value function predicts the expected return from the policy history:

$\operatorname{Pri}_w(n)$2

A Privileged Value Function augments the critic input with $\operatorname{Pri}_w(n)$3:

$\operatorname{Pri}_w(n)$4

The associated Monte Carlo advantage is

$\operatorname{Pri}_w(n)$5

The policy continues to sample according to

$\operatorname{Pri}_w(n)$6

and does not condition its generation on $\operatorname{Pri}_w(n)$7. Thus, PVF is a training-time critic mechanism rather than an inference-time policy input.

The central distinction is between policy information and critic information. The policy is restricted to information available in the actual task history, while the critic may use reference solutions, verifier rubrics, latent environment state, solved states, or independently sampled sibling trajectories. This additional information can make return prediction easier and can produce more informative token-level credit assignment than a sequence-level baseline.

Examples of privileged inputs include an oracle answer to a mathematical problem, a proof sketch, a gold code patch, a solved Sudoku grid, or the other responses and rewards in a rollout group. For trajectory $\operatorname{Pri}_w(n)$8, sibling information can be represented as

$\operatorname{Pri}_w(n)$9

The leave-one-out baseline is a simple PVF:

$n$0

3. Variance reduction and policy-gradient validity

The usefulness of PVF follows from conditional prediction. The optimal conditional-mean predictor with additional information satisfies

$n$1

Consequently, an accurately learned PVF can reduce the residual variance of the advantage. The paper measures this through explained variance:

$n$2

With $n$3, the residual is the policy advantage, so explained variance directly measures reduction in return and advantage variance relative to a constant baseline.

The additional information must satisfy a baseline-independence condition. A sufficient condition is

$n$4

Equivalently, the baseline must satisfy

$n$5

Under this condition, PVF changes the variance of the policy-gradient estimator without changing its expectation. A fixed reference answer and independently sampled sibling trajectories are admissible. By contrast, the current trajectory’s future tokens, realized return, later verifier feedback, or future environment states caused by the current action are not admissible because they may depend on the token whose gradient is being estimated.

This restriction distinguishes PVF from retrospective self-distillation. A self-distillation teacher may use completed-trajectory feedback to construct a distribution-matching objective, thereby changing the optimization objective and potentially the policy optimum. PVF uses privileged information only inside an auxiliary baseline and preserves the original RL objective under the stated independence condition.

PVF is particularly relevant to group-relative methods such as GRPO. GRPO uses the group mean

$n$6

and assigns the sequence-level advantage $n$7 to every token. This avoids a separately trained critic but provides no token-level differentiation. A learned value function can operate with $n$8, while PVF can additionally use task-relevant information to improve prediction. The tradeoff is that PVF requires suitable privileged data and additional value-model infrastructure.

4. Training architecture and TETHER

A typical PVF pipeline initializes an LLM policy and a value model consisting of a copy of the base policy with a randomly initialized value head. The critic is warmed up, rollouts are collected, responses are evaluated by a binary outcome verifier or environment reward, and privileged context is attached to each trajectory and token prefix. The evaluator computes

$n$9

after which the advantages are frozen for policy optimization:

$w$0

The value trainer operates asynchronously using a FIFO replay buffer. The evaluator and trainer may run on separate replicas or GPUs, so value optimization need not block policy rollout and training. The reported infrastructure uses a FIFO replay buffer of 256 trajectories, with each trajectory reused at most twice. At inference time, the policy does not require the privileged information.

The experiments use Qwen3-4B-Instruct-2507, Qwen3.5-4B for MiniF2F, and 8-way H200 GPU nodes. Evaluated settings include Reasoning Gym with ground-truth answers, CodeIO with sibling responses and rewards, Sudoku with a complete solved grid, and MiniF2F for formal mathematics in Lean. The main value experiments use binary cross-entropy value loss, a value learning rate of $w$1, 20 value warmup updates, and $w$2.

The paper introduces TETHER, an adaptive interpolation between a leave-one-out group baseline and a learned value baseline. With

$w$3

and $w$4, TETHER uses

$w$5

The corresponding advantage is

$w$6

The endpoint $w$7 gives the leave-one-out baseline, while $w$8 gives the ordinary token-level value baseline. The coefficient is fitted from return-prediction error and smoothed with $w$9. The coefficient estimated from one batch is applied only to the next batch, preventing the current rewards from determining their own baseline.

TETHER can itself be interpreted as a simple PVF because the leave-one-out baseline summarizes privileged sibling-return information. The richer CodeIO PVF conditions on full sibling responses and rewards, whereas TETHER compresses sibling information into a scalar.

5. Empirical findings and limitations

PVF is reported to outperform the ordinary value-function baseline in all displayed PVF experiments. In Reasoning Gym with $A_w$0, ordinary value-function learning plateaus earlier while PVF continues improving. With $A_w$1, the gain is smaller because group information already reduces variance. In CodeIO, PVF using sibling responses and rewards surpasses both the ordinary value function and the group mean. In Sudoku, PVF produces the largest improvement, attributed to the difficulty of determining whether a partial grid is compatible with a globally valid solution without access to the solved grid.

PVF has higher explained variance than the ordinary value function in every reported environment. The improvement is smallest in Reasoning Gym with $A_w$2 and largest in Sudoku. This supports the interpretation that PVF improves policy learning primarily through advantage-variance reduction rather than through a new policy-training objective.

TETHER improves over the ordinary value baseline in all four tested settings. Relative to the group mean, it outperforms Mean on Reasoning Gym and MiniF2F, matches Mean on CodeIO, and narrows but does not eliminate the gap with Mean on Sudoku. The adaptive coefficient generally begins at zero and moves away from zero as the critic becomes useful.

Several limitations qualify these results. Value training and inference require additional accelerator capacity, and value-backed runs are not exactly compute-matched with group-mean runs. The experiments use approximately 4B-parameter models and selected reasoning environments, so scaling behavior for larger models and production-scale RL systems is not established. PVF also depends on task-specific privileged information, which may be unavailable in open-ended RLHF settings. Additional context can be difficult for a finite critic to exploit, and excessive or poorly formatted context can hinder value learning.

Critic lag and off-policy effects remain relevant. Replay and asynchronous evaluation improve utilization but do not eliminate the consequences of policy drift. The main experiments use $A_w$3, preserving unbiased Monte Carlo returns but potentially producing substantial variance on long sequences. Lower $A_w$4 may reduce variance while introducing dependence on critic accuracy and potential bias. PVF also provides no direct inference-time benefit: the deployed policy ordinarily lacks the privileged answer, solved state, or sibling trajectories.

6. Proto-value functions and reward-dependent representations

In classical reinforcement learning, proto-value functions are topology-based spectral features. Given a graph $A_w$5 of sampled states with adjacency matrix $A_w$6, define the degree matrix $A_w$7, combinatorial Laplacian $A_w$8, normalized Laplacian

$A_w$9

and random-walk diffusion matrix

$A_t$0

Ordinary PVFs are eigenvectors of a graph diffusion or Laplacian operator. In the diffusion formulation,

$A_t$1

and the first $A_t$2 eigenvectors form the feature matrix

$A_t$3

These features provide a compact basis for linear value-function approximation. Their motivation is that slowly varying modes of state connectivity can approximate value functions, particularly when the state graph captures the transition geometry relevant to a policy.

Ordinary PVFs can fail when the state graph is symmetric but rewards are asymmetric. Topologically equivalent states receive equivalent treatment from $A_t$4, even when one state is near a goal and another is associated with a mine or negative reward. With a truncated basis, this produces representation error: the learning algorithm may estimate the weights accurately while remaining unable to express the task-induced asymmetry.

“Shaping Proto-Value Functions via Rewards” (Narayanan et al., 2015) introduces reward-dependent proto-value functions by modifying the local diffusion weights. For neighboring states $A_t$5 and $A_t$6, the reward-based transition weight is a softmax over immediate rewards:

$A_t$7

The resulting matrix $A_t$8 is reward-weighted, row-normalized, and remains graph-based because only connected states receive nonzero weight. RPVFs are eigenvectors satisfying

$A_t$9

with the top $A_w(n)$0 eigenvectors forming

$A_w(n)$1

The representation is then used in representational policy iteration with LSTDQ. Reward shaping and RPVF construction are distinct: reward shaping changes the scalar learning signal, whereas RPVFs change the feature basis itself. In the reported $A_w(n)$2 grid experiment, ordinary PVFs with $A_w(n)$3 obtained aggregate value $A_w(n)$4, compared with $A_w(n)$5 for the optimal policy. RPVFs with $A_w(n)$6 and $A_w(n)$7 obtained $A_w(n)$8. The first two eigenfunctions were identical for $A_w(n)$9 and $n$0, while the third and fourth differed.

The RPVF formulation is task dependent and assumes that immediate rewards provide useful local preference information. It can be inadequate for strongly delayed, deceptive, or strategically nonlocal rewards. Since $n$1 may be directed or nonsymmetric, its eigenvectors can also require care concerning normalization, orthogonality, and numerical stability. RPVFs remain truncated representations and can fail when reward-relevant structure lies beyond the selected eigenvectors.

7. Privileged complexity functions in combinatorics on words

A complete first return to a word $n$2 begins and ends with $n$3 and contains exactly two occurrences of $n$4, with overlapping occurrences permitted. Privileged words are defined inductively: the empty word and every letter are privileged, and a word is privileged if and only if it is a complete first return to a shorter privileged word.

For an infinite word $n$5, let $n$6 be its set of factors of length $n$7, and let $n$8 be its privileged factors of that length. The privileged complexity function is

$n$9

The Thue–Morse word is the fixed point

$\tau_i=(y_{i,1},\ldots,y_{i,T_i})$0

of the morphism

$\tau_i=(y_{i,1},\ldots,y_{i,T_i})$1

Its square is

$\tau_i=(y_{i,1},\ldots,y_{i,T_i})$2

Using overlap-freeness, unique interpretations under $\tau_i=(y_{i,1},\ldots,y_{i,T_i})$3 and $\tau_i=(y_{i,1},\ldots,y_{i,T_i})$4, and complete-return classifications, the paper derives recursive formulas for $\tau_i=(y_{i,1},\ldots,y_{i,T_i})$5. In particular,

$\tau_i=(y_{i,1},\ldots,y_{i,T_i})$6

so only lengths $\tau_i=(y_{i,1},\ldots,y_{i,T_i})$7 and $\tau_i=(y_{i,1},\ldots,y_{i,T_i})$8 carry privileged factors among sufficiently small odd lengths. The function is nevertheless unbounded. For $\tau_i=(y_{i,1},\ldots,y_{i,T_i})$9,

$x_i$0

Hence

$x_i$1

At the same time, $x_i$2 has arbitrarily long finite intervals of zeros. If

$x_i$3

then

$x_i$4

The values immediately outside the specified boundaries are nonzero, so these are genuine finite gaps rather than an eventually zero tail.

The contrast with palindromic complexity is substantial. The palindromic complexity $x_i$5 of the Thue–Morse word is bounded, whereas $x_i$6 is unbounded. Privileged and palindromic factors also need not contain one another: $x_i$7 is a privileged non-palindrome, while $x_i$8 is a palindrome that is not privileged.

The paper establishes the general equivalence

$x_i$9

where richness means that the word contains the maximum possible number of distinct palindromic factors. At the level of complexity functions, equality of privileged and palindromic complexities at every relevant length characterizes richness. Thus privileged complexity supplies a structural comparison with palindromic complexity, while the term PVF, when used in this context, should be understood as an informal label for $\operatorname{Pri}_w(n)$00 rather than as terminology introduced by the authors.

For automatic sequences, privilegedness can be characterized without recursive reference to shorter privileged words. A word has property P if, for every prefix length $\operatorname{Pri}_w(n)$01, it has a nonempty border of length at most $\operatorname{Pri}_w(n)$02 that occurs exactly once in both the corresponding prefix and suffix. A word is privileged if and only if it has property P. This characterization is expressible in first-order arithmetic and yields automata-theoretic results: for a $\operatorname{Pri}_w(n)$03-automatic sequence, the characteristic sequence of lengths admitting privileged factors is $\operatorname{Pri}_w(n)$04-automatic, while the number of distinct privileged factors of each length is $\operatorname{Pri}_w(n)$05-regular. In the Thue–Morse case, a $\operatorname{Pri}_w(n)$06-state base-$\operatorname{Pri}_w(n)$07 automaton recognizes the privileged-factor lengths. These Boolean characteristic functions and counting sequences are distinct objects and should not be identified with one another.