Incremental TD Networks

• The incremental TD network framework extends traditional TD learning by predicting interdependent future events using eligibility traces and action-conditional features.
• It employs a directed acyclic question network paired with an answer network to support online learning in both discrete and continuous domains.
• Empirical studies show rapid convergence and low prediction error in modeling noisy, partially observable dynamical systems with non-Markovian characteristics.

An incremental TD (Temporal-Difference) network is a generalization of temporal-difference learning extended to a network of interdependent scalar predictions. Rather than relying on a single prediction updated from itself at a future time, incremental TD networks (TDNs) represent sets of predictions, each of which may depend on future values of other predictions as well as observations, enabling a broad class of inter-predictive relationships. They are formulated for both discrete and continuous domains, including those with continuous actions and observations, and support fully incremental online updating with eligibility traces and action-conditional representations. The incremental TD network algorithm enables learning structured predictive models—including predictive state representations—of partially observable dynamical systems, which extend beyond the representational capacity of traditional value methods and allow for predictive modeling of non-Markovian, partially observed, or continuous systems (Vigorito, 2012, Sutton et al., 2015).

1. Formal Framework and Definitions

An incremental TD network operates on a controlled dynamical system with observations $o_t \in \mathbb{R}^o$ and actions $a_t \in \mathbb{R}^a$, governed by an unknown stochastic kernel. The agent maintains a history $h_t = (o_0, a_0, o_1, a_1, ..., o_t)$, updating after each action is applied and observation received. Rather than a single value function, the model employs a set of $n$ scalar predictions $\mathbf{y}_t \in \mathbb{R}^n$ representing answers to a user-defined set of "questions" (predictions about future observations or events).

The network is defined by two main structures:

• Question Network ($Q$): A directed acyclic graph of $n$ nodes (predictions $y^i$), each with its parent as either an observation feature function $\phi_j$ or another prediction node $y^k$. Dependencies may be conditioned on the similarity between the previous action and a center, measured via a gating function $a_t \in \mathbb{R}^a$0.
• Answer Network: Parameterized by a weight matrix $a_t \in \mathbb{R}^a$1 and a feature mapping $a_t \in \mathbb{R}^a$2 that integrates previous predictions, last action, and current observation into a feature input $a_t \in \mathbb{R}^a$3, generating $a_t \in \mathbb{R}^a$4 (optionally passing through a nonlinear activation such as identity or sigmoid) (Vigorito, 2012, Sutton et al., 2015).

2. Network Architecture and Action Conditioning

Architecturally, each root node corresponds to an observation feature function, such as a Gaussian RBF $a_t \in \mathbb{R}^a$5 centered at $a_t \in \mathbb{R}^a$6. Chains (or more general DAGs) of prediction nodes are grown from these roots, with each node possibly conditioned on an action-gating function $a_t \in \mathbb{R}^a$7—also typically a Gaussian RBF over the action space. Predictions at node $a_t \in \mathbb{R}^a$8 at time $a_t \in \mathbb{R}^a$9 may thus reflect, for example, the expected value of $h_t = (o_0, a_0, o_1, a_1, ..., o_t)$0 one (or multiple) steps ahead if an action $h_t = (o_0, a_0, o_1, a_1, ..., o_t)$1 similar to $h_t = (o_0, a_0, o_1, a_1, ..., o_t)$2 is taken (Vigorito, 2012).

The feature vector at each time step is constructed as

$h_t = (o_0, a_0, o_1, a_1, ..., o_t)$3

where $h_t = (o_0, a_0, o_1, a_1, ..., o_t)$4 denotes the number of prediction nodes, $h_t = (o_0, a_0, o_1, a_1, ..., o_t)$5 and $h_t = (o_0, a_0, o_1, a_1, ..., o_t)$6 are the number of observation and action features used, and $h_t = (o_0, a_0, o_1, a_1, ..., o_t)$7 is the input dimension.

The action gating function $h_t = (o_0, a_0, o_1, a_1, ..., o_t)$8 provides a continuous responsibility signal $h_t = (o_0, a_0, o_1, a_1, ..., o_t)$9 for whether the TD error on $n$0 is incurred at time $n$1. This allows for smooth (rather than binary) conditioning and trace decay (Vigorito, 2012).

3. Incremental Update Procedure and Learning Rules

For each prediction node $n$2 at time $n$3, the incremental update is defined via:

• Target Definition ($n$4):
  • If the parent is an observation feature $n$5: $n$6
  • If the parent is another prediction $n$7: $n$8
• TD Error ($n$9):

$\mathbf{y}_t \in \mathbb{R}^n$0

• Eligibility Traces ($\mathbf{y}_t \in \mathbb{R}^n$1):

$\mathbf{y}_t \in \mathbb{R}^n$2

Each trace is maintained per output row; $\mathbf{y}_t \in \mathbb{R}^n$3 is the trace decay parameter.

• Weight Update:

$\mathbf{y}_t \in \mathbb{R}^n$4

with step size $\mathbf{y}_t \in \mathbb{R}^n$5.

The general step-wise procedure—applicable for both the real-valued continuous and the finite case—involves building $\mathbf{y}_t \in \mathbb{R}^n$6, computing current predictions $\mathbf{y}_t \in \mathbb{R}^n$7, performing the action and observing $\mathbf{y}_t \in \mathbb{R}^n$8, computing targets and TD errors per node, updating traces and weights incrementally, and shifting state. This routine supports fully incremental, online learning (Vigorito, 2012, Sutton et al., 2015).

4. Extensions from Discrete to Continuous Domains

Incremental TD networks extend naturally to continuous spaces by:

• Replacing discrete observation and action symbols with (typically Gaussian RBF) features $\mathbf{y}_t \in \mathbb{R}^n$9 and $Q$0 over continuous vector spaces $Q$1 and $Q$2;
• Using real-valued feature vectors and gating signals, permitting partial eligibility and smooth conditioning, critical for high-dimensional or noisy environments;
• Concatenating previous predictions into the feature vector, enabling representation of history and information flow, thereby facilitating learning in non-Markovian or partially observable systems (Vigorito, 2012).

These extensions ensure that eligibility traces no longer die “instantly” upon action mismatch, and ties the generalization ability of the network closely to the selection, width, and tiling of the chosen $Q$3 and $Q$4 basis functions.

5. Algorithmic Pseudocode and Workflow

A concise algorithmic outline for incremental TD network learning, capturing all the above elements, is as follows (for $Q$5 nodes, feature functions $Q$6 and $Q$7, weight matrix $Q$8, eligibility traces $Q$9, and step sizes $n$0 and $n$1):

$y^i$7 (Vigorito, 2012, Sutton et al., 2015)

6. Convergence, Computational Complexity, and Empirical Performance

Under standard linear TD learning assumptions (bounded features, small enough $n$2, and proper gating functions), incremental TD networks converge in mean to the fixed-point of the projected Bellman operator defined by the question network (Vigorito, 2012). The soft gates $n$3 do not affect convergence provided they are bounded away from zero and one as necessary.

Per-step computational cost is $n$4, driven by feature construction, prediction calculation, and $n$5 weight/traces updates. The principal memory requirement is for $n$6 weight storage and $n$7 eligibility traces (Vigorito, 2012).

Empirical studies on five domains highlighted rapid convergence and robustness under partial observability and noise:

• On controlled and uncontrolled 1D square and sine waves (continuous domains), root mean squared prediction errors converged to levels comparable to observation noise (≈0.05 in $n$8 steps with $n$9 φ functions and depth $y^i$0).
• In partially observable mountain-car, velocity was recovered via multi-step prediction chains, achieving RMSE ≈0.04.
• Depth $y^i$1 generally provided a strong trade-off between model expressiveness and stability; accuracy and convergence rates were robust even with a small number of φ and ψ basis functions (e.g., $y^i$2 per dimension).

These results demonstrate that fully incremental TD networks can reliably learn accurate, robust predictive models in continuous, noisy, partially observable dynamical systems (Vigorito, 2012).

7. Relation to Conventional TD Learning and Broader Implications

Incremental TD networks strictly generalize classical TD(0) learning. With a single node, identity feature encoding, and appropriate definition of $y^i$3 and $y^i$4, the framework collapses to standard value-function TD learning. General TD networks, however, allow each prediction $y^i$5 to target arbitrary functions of both future observations and other nodes' predictions, facilitating "learn a guess from arbitrary other guesses" (Sutton et al., 2015).

The conditioning vector $y^i$6 allows for selective or action-conditional credit assignment, supporting structured, multi-step, and action-dependent predictions in a unified update mechanism. This enables simultaneous learning of many interrelated, action-conditional predictions using local TD error signals, including predictive state representations and fixed-interval predictions that are not possible with conventional TD methods.

This broadens the applicability of the TD learning paradigm to a rich class of inter-predictive world models—enabling application to non-Markov problems, highly structured dynamical systems, and the construction of end-to-end predictive representations of environment dynamics (Sutton et al., 2015, Vigorito, 2012).