Papers
Topics
Authors
Recent
Search
2000 character limit reached

Agent-Centric General Value Function

Updated 16 September 2026
  • Agent-Centric General Value Functions (ACGVFs) are an interpretive framework for General Value Functions (GVFs) grounded in an agent’s actions, goals, and capabilities, focusing on what the agent experiences, achieves, or learns.
  • ACGVFs extend conventional GVFs by predicting agent-relative quantities, such as future safety, competence, and learning progress, grounded in the agent’s own policies and observation contexts
  • ACGVF has applications in autonomous driving, robotics, and multi-agent systems, where it provides a novel approach to predicting the consequences of an agent’s decisions and behaviors

Agent-Centric General Value Function (ACGVF) is an interpretive framework for General Value Functions (GVFs) in which the predicted quantities are grounded in an agent’s own observations, actions, policies, capabilities, learning processes, goals, and interaction context. A conventional GVF predicts the expected temporally extended accumulation of a cumulant under a policy and continuation function. An ACGVF extends the subject matter of prediction to agent-relative quantities such as future safety, competence, visitation, prediction reliability, learning progress, goal attainability, or the consequences of an agent’s own policy. The term is not established as a separate formalism in the cited literature; rather, it synthesizes several lines of work on introspective prediction, predictive state construction, affordances, multi-agent value decomposition, world-oriented value representations, and agent-generated communication.

1. Formal basis and defining characteristics

A GVF is specified by a policy π\pi, a cumulant cc, and a continuation or discount function γ\gamma. For a history-dependent formulation, the return is

Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},

and the corresponding prediction is

Vc,γπ(ht)=Eπ[Gt∣Ht=ht].V^{\pi}_{c,\gamma}(h_t) = \mathbb{E}_{\pi}[G_t\mid H_t=h_t].

The cumulant need not be a task reward. It may be an observation, event indicator, sensor measurement, resource level, prediction error, or another learned signal. The continuation function may be constant, state-dependent, transition-dependent, or event-conditioned. A value of γ=0\gamma=0 terminates the prediction, whereas γ\gamma near $1$ yields a long temporal horizon.

An ACGVF is therefore best characterized by the following conditions:

  1. Agent-relative subject matter: the prediction concerns what the agent will experience, achieve, infer, or learn, rather than only an externally defined environmental variable.
  2. Policy grounding: the prediction is conditional on an executable or anticipated agent policy.
  3. Embodiment and capability grounding: the prediction depends on the agent’s action space, dynamics, sensors, morphology, or actuator limitations.
  4. Internal or agent-generated cumulants: the cumulant may be computed by the agent, including confidence, error, visitation, learning signals, or internally selected goals.
  5. Actionability: the prediction can participate in control, exploration, policy selection, option initiation, communication, or representation construction.

The distinction is between what is predicted and whose properties the prediction characterizes. A conventional GVF may ask what wall hue or battery level will be observed. An ACGVF may ask how accurate the wall-hue prediction will be, whether a braking policy will preserve safety, how likely a goal is to be reached, or how much a parameterized predictor will change.

The ACGVF interpretation is explicit only in a limited sense. “Introspective Agents: Confidence Measures for General Value Functions” proposes predictions of internally generated signals, but does not use the ACGVF terminology or define a complete formal architecture (Sherstan et al., 2016). Similarly, work on affordances, autonomous-driving safety, object-centric GVFs, and world value functions supplies concrete instances or extensions rather than a unified ACGVF theory (Graves et al., 2020, Graves et al., 2020, Nath et al., 2023, Tasse et al., 2022).

2. Internal prediction, confidence, and introspection

The most direct precursor to ACGVF is the proposal to predict the agent’s own learning processes. The introspective-agent framework extends GVFs beyond external sensorimotor signals such as battery level, joint position, wall hue, and other environmental measurements. Candidate internal cumulants include prediction errors, weight changes, meta-parameters, feature-space visitation, input statistics, confidence, and uncertainty (Sherstan et al., 2016).

If a primary GVF has prediction V^ti\hat V_t^i and temporal-difference error

δti=Ct+1i+γt+1V^ti(St+1)−V^ti(St),\delta_t^i = C_{t+1}^i+ \gamma_{t+1}\hat V_t^i(S_{t+1}) - \hat V_t^i(S_t),

then cc0 can itself become the cumulant of a secondary GVF. A secondary predictor can consequently estimate future or accumulated prediction error rather than merely record current error. The paper specifically discusses expected squared TD error, with cumulant

cc1

Low predicted squared error suggests that a primary prediction is usually accurate in a context; high predicted squared error suggests unreliability. This is distinct from raw visitation. A state may be frequently visited while its outcomes remain variable or its predictor remains systematically wrong.

Three confidence-related quantities are distinguished:

Quantity Interpretation Limitation
Visitation Experience coverage or familiarity High visitation does not guarantee correctness
Prediction error Disagreement between predictions and outcomes Error may reflect environmental change
Variance Outcome variability or uncertainty Estimation is difficult for nonstationary targets

Variance can be represented through first- and second-moment predictions,

cc2

with variance cc3. However, the underlying predictor changes during learning, so the target distribution is nonstationary.

The proposed architecture is recursive in a functional sense: an environmental cumulant produces a primary GVF; its TD error or learning signal becomes the cumulant of another GVF; further GVFs can predict visitation, variance, or additional properties of the learner. The work does not define unlimited recursion, a formal hierarchy, or a general compositional algebra. Its principal contribution is the proposal that internal learning signals should be treated as ordinary predictive cumulants (Sherstan et al., 2016).

This framework supports several interpretations of introspective state:

  • Confidence: predicted reliability of another prediction.
  • Competence: predicted ability to achieve a target outcome.
  • Learning progress: predicted reduction in error or change in parameters.
  • Epistemic coverage: predicted familiarity with a state or feature region.
  • Self-monitoring: prediction of conditions under which the agent’s own knowledge becomes unreliable.

These terms are broader interpretations rather than fully formal variables in the original work.

3. Predictive state construction and agent-centered representations

A GVF can serve as a component of recurrent state. “General Value Function Networks” constrains recurrent state components to represent predictions about the future rather than unconstrained latent features (Schlegel et al., 2018). A GVFN has recurrent state

cc4

where each component approximates a GVF,

cc5

The recurrence is RNN-like,

cc6

but the state components are constrained by specified predictive questions. Their one-step temporal-difference errors are

cc7

This produces a predictive state representation grounded in the agent’s own sensorimotor history. It need not reconstruct an objective latent physical state. Instead, it can preserve information sufficient for selected predictions and decisions. Possible components include the probability of a future observation under a policy, the expected occurrence of an event, a future resource level, or a prediction of another prediction.

GVFNs support multiple policies, cumulants, and continuation functions. Their compositional extension allows one GVF’s prediction to serve as another’s cumulant. The cited work assumes an acyclic dependency graph, Lipschitz compositional cumulants, and contraction of discounted transition operators. Under these conditions, repeated Bellman-network iteration converges to a unique fixed point. Cyclic composition can diverge (Schlegel et al., 2018).

The architecture is relevant to ACGVF because it treats internal state as a collection of agent-relevant predictions. The experiments report improved robustness to short truncated backpropagation windows in several settings, including Compass World, where the agent must infer long-term consequences from sequences of actions and observations. The results also show that prediction specification matters: useful GVFs must be learnable online, numerically well-scaled, temporally meaningful, and relevant to downstream behavior.

The representation is therefore not automatically agent-centric merely because it is recurrent or predictive. Agent-centricity depends on the selected questions. A recurrent state composed of predictions about the agent’s own future sensorimotor consequences is an ACGVF-like representation; an unconstrained latent vector is not necessarily so.

4. Affordances, safety, and action-conditioned prediction

Affordance-oriented work gives ACGVF a direct action-perception interpretation. “Affordance as general value function: A computational model” formalizes an affordance as an option together with a policy-relative GVF prediction (Graves et al., 2020). An option is

cc8

where cc9 is the initiation set, γ\gamma0 is the termination function, and γ\gamma1 is the option policy. The option’s valence is represented by a GVF predicting a cumulant γ\gamma2 under γ\gamma3 and a continuation function incorporating termination:

γ\gamma4

The proposed mapping is

γ\gamma5

A state affordance can be represented as

γ\gamma6

and an action affordance as

γ\gamma7

The cumulant specifies what counts as good or bad: safety, comfort, traversability, grasp success, stopping time, lane offset, centeredness, collision proximity, or weld quality. The same physical state can therefore have different values under different policies or concerns.

“Perception as prediction using general value functions in autonomous driving applications” applies this principle to ego-centered safety (Graves et al., 2020). It learns action-conditioned predictions of front safety, rear safety, and ego speed. The central question is whether the vehicle will remain safe after taking an action and continuing under a specified policy. Other vehicles are represented through their effects on the ego vehicle’s future cumulants rather than through explicit models of their policies or intentions.

The front-safety cumulant is binary: it is γ\gamma8 when the ego vehicle’s front safety zone is unoccupied and γ\gamma9 when it is occupied. The safety zone depends on the ego vehicle’s speed through

Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},0

The learned function is an action-conditioned Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},1 prediction. The target policy models smooth continuation around the previous action,

Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},2

The system uses TD learning with replay and collision termination through Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},3. In reported autonomous-driving experiments, GVF-based controllers produced behavior broadly comparable to an LQR-based controller, and rear-safety predictions could decline before the instantaneous rear-distance signal became unsafe. The work did not provide a formal safety guarantee, and large discounts could cause instability in long-horizon nonlinear TD learning (Graves et al., 2020).

These results illustrate a central ACGVF property: the predicted quantity is not merely an environmental feature. It is the future consequence of an ego action under an anticipated ego policy. This differs from conventional perception, full model-based predictive control, and explicit intention modeling of other agents.

5. Multi-agent, distributed, and communicative ACGVFs

ACGVFs can be defined over population context, agent-specific observations, local rewards, or communication interfaces. Several cited studies provide complementary mechanisms.

“Policy Gradient With Value Function Approximation For Collective Multiagent Planning” studies exchangeable agents whose collective state is represented by population counts (Nguyen et al., 2018). Its critic decomposes an approximate joint action-value function as

Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},4

or, for homogeneous agents,

Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},5

The local factor is agent-centered operationally, but it is not necessarily an independent agent’s true return. It may depend on shared population counts and is trained to support approximation of the global collective value. The method uses centralized training, decentralized execution, finite-horizon undiscounted returns, and return-based critic targets rather than general GVF TD learning. Its reusable ACGVF ideas are local observation-conditioned prediction, population aggregation, and local credit assignment (Nguyen et al., 2018).

“Multi-Agent Fully Decentralized Value Function Learning with Linear Convergence Rates” develops Fast Diffusion for Policy Evaluation, in which agents exchange compact corrected iterates over a communication graph and converge to a shared value-function parameter (Cassano et al., 2018). The method supports off-policy learning, eligibility traces, linear function approximation, and collective state coverage. Its original object is one shared discounted reward value, not multiple agent-specific GVFs. An ACGVF extension would require agent-indexed cumulants, continuation functions, policies, and potentially private or clustered parameters.

“Pavlovian Signalling with General Value Functions in Agent-Agent Temporal Decision Making” demonstrates a sender-centered predictive interface (Kalwar et al., 2022). A prediction-learning co-agent learns a GVF for hazard accumulation or countdown, thresholds the prediction, and transmits a binary token to a reinforcement-learning control agent. The sender-side semantics are determined by

Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},6

where the threshold Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},7 is part of the signal’s grounding. The receiver need not share the sender’s representation or understand the GVF. The architecture is therefore:

Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},8

This provides an intermediate regime between fixed signalling and fully learned communication. The prediction is learned online, but the token interface and threshold mapping are fixed. The work also demonstrates that temporal representation affects signal timing, aliasing, and receiver performance.

Off-policy and multi-prediction extensions are addressed by “A Unified Off-Policy Evaluation Approach for General Value Function” (Xu et al., 2021). GenTD evaluates multiple multidimensional and causally interrelated GVFs using stationary density-ratio correction rather than long trajectory importance ratios. Its causal-filtering class has lower-triangular dependencies:

Gt=∑k=0∞(∏j=1kγt+j)ct+k+1,G_t=\sum_{k=0}^{\infty} \left(\prod_{j=1}^{k}\gamma_{t+j}\right)c_{t+k+1},9

For ACGVF, this structure could model dependencies among an agent’s return, variance, interaction cost, uncertainty, or policy-gradient predictions. The convergence guarantees require bounded quantities, sufficient coverage, suitable density-ratio estimation, linear approximation, and an acyclic dependency structure. They do not directly cover nonstationary agents, partial observability, cyclic strategic dependencies, or nonlinear representations (Xu et al., 2021).

6. Goals, world knowledge, and object-centered predictive representations

World Value Functions (WVFs) provide a goal-conditioned ACGVF-like representation. A WVF uses an internally constructed goal space

Vc,γπ(ht)=Eπ[Gt∣Ht=ht].V^{\pi}_{c,\gamma}(h_t) = \mathbb{E}_{\pi}[G_t\mid H_t=h_t].0

and learns a goal-conditioned action-value function Vc,γπ(ht)=Eπ[Gt∣Ht=ht].V^{\pi}_{c,\gamma}(h_t) = \mathbb{E}_{\pi}[G_t\mid H_t=h_t].1 (Tasse et al., 2022, Tasse et al., 2022).

The agent modifies the reward so that terminating at an unintended goal receives a sufficiently large negative penalty. It then learns a goal-conditioned world policy and value function. In the maximal case Vc,γπ(ht)=Eπ[Gt∣Ht=ht].V^{\pi}_{c,\gamma}(h_t) = \mathbb{E}_{\pi}[G_t\mid H_t=h_t].2, the WVF represents how to reach every state regarded as an internal goal. The reported theoretical properties include:

  • mastery of reachable internal goals under the stated assumptions;
  • recovery of the original task value by maximizing over goals;
  • zero-shot transfer to new tasks that share dynamics but differ in terminal rewards;
  • possible transition-dynamics inference from the complete goal-indexed value representation;
  • Boolean composition of learned tasks under the stated reward and approximation assumptions.

WVFs are a specialized subclass of goal-oriented GVFs rather than a general ACGVF theory. Their goals are terminal-state-based, their reward transformation is prescribed, and their formal results focus on deterministic or otherwise restricted settings. They do not directly address arbitrary nonterminal cumulants, changing embodiments, or general partial observability (Tasse et al., 2022, Tasse et al., 2022).

“Discovering Object-Centric Generalized Value Functions From Pixels” introduces OC-GVFs, which discover object-like slots from images, generate cumulants from those slots, learn GVF predictions, and provide the predictions to a downstream Double DQN controller (Nath et al., 2023). The architecture is

Vc,γπ(ht)=Eπ[Gt∣Ht=ht].V^{\pi}_{c,\gamma}(h_t) = \mathbb{E}_{\pi}[G_t\mid H_t=h_t].3

For slot Vc,γπ(ht)=Eπ[Gt∣Ht=ht].V^{\pi}_{c,\gamma}(h_t) = \mathbb{E}_{\pi}[G_t\mid H_t=h_t].4, a question network produces cumulants such as

Vc,γπ(ht)=Eπ[Gt∣Ht=ht].V^{\pi}_{c,\gamma}(h_t) = \mathbb{E}_{\pi}[G_t\mid H_t=h_t].5

The GVF predictions are projected and concatenated with the main visual representation before action-value computation. The experiments report object-specific value patterns and improved adaptation in stationary and nonstationary environments.

OC-GVFs are explicitly object-centric but only indirectly agent-centric. The architecture does not explicitly encode agent-object geometry, relative pose, reachability, or affordances. Its predictions are evaluated from the agent’s current state and used by its policy, so they provide an important implementation pathway toward ACGVF, but they do not constitute a complete relational agent-centric formalism (Nath et al., 2023).

7. Learning, applications, limitations, and research directions

ACGVF systems can be trained using several regimes represented in the cited literature:

  • Online TD learning: suitable for continual predictions, internal signals, and sensorimotor interaction.
  • Recurrent TD and recurrent GTD: designed for predictive recurrent state under partial observability and, in some cases, off-policy learning (Schlegel et al., 2018).
  • Return-based actor-critic learning: useful for collective multi-agent planning but not equivalent to general GVF learning (Nguyen et al., 2018).
  • Off-policy density-ratio methods: support multiple interrelated GVFs under pre-collected data (Xu et al., 2021).
  • Meta-gradient cumulant discovery: learns predictive questions from their downstream utility for control (Nath et al., 2023).
  • Goal-conditioned Q-learning: supports internally generated goals and reusable world-oriented value representations (Tasse et al., 2022, Tasse et al., 2022).
  • Thresholded signalling: exposes a low-bandwidth predictive interface to another agent or human [(Aglibar et al., 2022)?]

The final citation above must refer to the supplied paper on signalling, which is (Kalwar et al., 2022).

Principal applications include:

  • Introspection and self-monitoring: predicting confidence, error, visitation, variance, and learning signals (Sherstan et al., 2016).
  • Adaptive exploration and caution: using uncertainty or reliability to select information-seeking or risk-sensitive behavior (Sherstan et al., 2016).
  • Autonomous driving and robotics: predicting ego safety, speed, perception, and interaction consequences (Graves et al., 2020).
  • Affordance perception: representing policy-relative action possibilities and their valence (Graves et al., 2020).
  • Collective control: decomposing population-level values into local factors and using aggregate context (Nguyen et al., 2018).
  • Distributed multi-agent prediction: learning shared values from heterogeneous datasets and local communication (Cassano et al., 2018).
  • Predictive communication: transmitting temporally extended predictions through compact signals (Kalwar et al., 2022).
  • Goal-general control and planning: learning reusable goal-conditioned world knowledge (Tasse et al., 2022, Tasse et al., 2022).
  • Object-centered visual control: discovering temporally coherent object-related predictions from pixels (Nath et al., 2023).

Important limitations remain. Confidence is not identical to prediction error, visitation, or variance. Internal cumulants are often nonstationary because the learning system itself changes. Shared-parameter and recurrent GVFs can exhibit coupled stability problems. Policy dependence complicates transfer across behavior policies. Partial observability may require histories, recurrent state, belief estimates, or predictive state representations. Off-policy learning can suffer from distribution mismatch and high-variance corrections. Multi-agent settings introduce joint coverage, nonstationary other-agent policies, correlated actions, credit assignment, and potentially cyclic causal dependencies. Nonlinear approximation generally lacks the guarantees established for restricted linear or tabular settings.

A general ACGVF research program would therefore need to address:

  1. Question discovery: selecting cumulants, policies, and continuation functions that are predictive and actionable.
  2. Agent-specific conditioning: representing identity, embodiment, local observation, belief state, and capability.
  3. Multiple predictions: supporting vector-valued, hierarchical, and compositional GVFs with shared representations.
  4. Uncertainty calibration: distinguishing epistemic coverage, aleatoric variability, systematic bias, and distribution shift.
  5. Off-policy and nonstationary learning: handling changing behavior policies, other agents, and environments.
  6. Local-global semantics: distinguishing an agent’s own return, marginal contribution to a collective objective, and conditional global value.
  7. Communication grounding: jointly relating sender predictions, signal interfaces, receiver utility, and temporal representation.
  8. Goal and affordance composition: connecting predicted consequences to option initiation, policy selection, and behavior orchestration.
  9. Evaluation: measuring prediction accuracy, calibration, transfer, safety, exploration efficiency, control quality, and actionability separately.

ACGVF is consequently best understood as a unifying research perspective rather than a single established algorithm. Its mathematical foundation is the GVF tuple Vc,γπ(ht)=Eπ[Gt∣Ht=ht].V^{\pi}_{c,\gamma}(h_t) = \mathbb{E}_{\pi}[G_t\mid H_t=h_t].6; its agent-centric character arises from grounding those elements in the agent’s own learning, embodiment, goals, observations, and actions. Introspective GVFs contribute predictions about knowledge and reliability; GVFNs contribute predictive recurrent state; affordance and autonomous-driving systems contribute action-conditioned consequences; distributed multi-agent methods contribute local-global decomposition and consensus; communication systems contribute predictive signalling; WVFs contribute internally generated goal spaces and reusable world knowledge; and object-centric GVFs contribute learned predictive structure from pixels. Together, these directions define an emerging architecture in which an agent predicts not only what will happen in the world, but also what will happen to itself, how well it knows that, what it can achieve, and how its predictions should guide subsequent behavior.

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Agent-Centric General Value Function (ACGVF).