Convergence of the generalized learning dynamics

Prove convergence of the generalized TD-style learning dynamics derived via semi-gradient descent on the cost function C_{F,I}[Z_t] in Appendix “Generalized derivation of learning dynamics,” showing that the iterates of the policy π_t(s'|s) ∝ p(s'|s) exp(F[Z_t](s')) and the endogenous cue update approach the optimal state value V* and its mapped concentration Z* under appropriate conditions (e.g., via a Lyapunov argument).

Background

The paper introduces a generalized framework for deriving learning dynamics by combining a mapping F from endogenous cue concentration Z to state value and a further bijection I, yielding a cost function C_{F,I}[Z_t] whose semi-gradient descent produces TD-style updates. This generalization encompasses biologically relevant nonlinear responses (e.g., Hill-type mappings), extending beyond the specific log-exp and lin-lin couplings analyzed in the main text.

While the authors provide intuition and examples for these generalized dynamics, they explicitly state that they do not supply a formal proof of convergence. Establishing convergence would validate the generalized learning procedure and its alignment with optimal control solutions characterized by the Bellman optimality equation.