Neural Discount Models

• Neural discount models are formal frameworks that describe how the subjective value of outcomes decays with time delay, risk, or evidence reliability.
• They integrate concepts from neuroscience, psychology, economics, and machine learning using methods like exponential, hyperbolic, and quantized discount functions.
• These models are applied to explain intertemporal decision making, stabilize deep RL, and shed light on disorders such as addiction through techniques like IRL and multi-horizon training.

Neural discount models are a class of formal and computational frameworks that describe how organisms, artificial agents, and decision systems value outcomes as a function of time delay, risk, or evidence reliability, with a particular focus on mechanisms grounded in neural or neurally inspired computation. These models unify concepts from neuroscience, psychology, economics, and machine learning, with representations ranging from classical continuous discount functions to quantized, scale-invariant, and state-dependent forms, often implemented via deep neural networks. Neural discount models are central to understanding intertemporal decision making, evidence integration, reinforcement learning, and disorders characterized by altered valuation dynamics such as addiction.

1. Mathematical Foundations of Discounting

Discount functions model the decrease in subjective value of an outcome as its occurrence moves further into the future or becomes less certain. The two primary analytic forms are:

• Exponential discounting: $V_e(t) = e^{-\lambda t}$, where $\lambda > 0$ defines the exponential discount rate.
• Hyperbolic discounting: $V_h(t) = \frac{1}{1 + k t}$, where $k > 0$ is the hyperbolic discount rate.

Exponential discounting yields stationary preferences and underpins standard Markov Decision Process (MDP) theory, while hyperbolic discounting captures empirically observed preference reversals and nonstationarities in human and animal choice. Hyperbolic discounting can arise from uncertainty in the hazard (termination) rate, leading to the discount function as a Laplace transform of an exponential prior over hazard rates (Tee et al., 2020, Schultheis et al., 2022, Fedus et al., 2019).

Quantized representations posit that subjective value is not continuously variable but discretized into a finite set of steps, operationalized by a quantizer $Q_N[x] = \frac{\lfloor x 2^N\rfloor}{2^N}$ for $x \in [0,1]$, where $N$ is the bit precision. The resulting quantized discount functions are $V_{e,q}(t) = Q_N(e^{-\lambda t})$ and $V_{h,q}(t) = Q_N(1/(1 + k t))$ (Tee et al., 2020).

2. Neural and Neural-Network Implementations

Neural discount models are realized both at the level of biological neural circuits and in artificial neural network architectures:

• Biophysical models: In biological agents, evidence accumulation and discounting can be modeled by leaky integrator neurons with time-varying leak rates, where the discount parameter is tuned by hazard rate and stimulus reliability (Piet et al., 2017). Intertemporal value signals in the brain (e.g., vmPFC) are consistent with the quantized coding stipulated by finite bit precision (Tee et al., 2020).
• Deep learning and RL: Neural networks can approximate discount functions and value trajectories under arbitrary (including non-exponential) discounting using architectures such as:
  • Multi-head Q-networks, where each head predicts value at a specific exponential discount, and hyperbolic (or other) discounting is achieved by integrating over these predictions (Fedus et al., 2019).
  • Neural collocation methods, where a deep network represents the value function and is trained to satisfy the generalized HJB equation corresponding to arbitrary discount functions (Schultheis et al., 2022).
  • State-dependent discount networks, in which a neural module predicts $\gamma(s)$ as a function of the current state, with empirically validated stability gains when regulated by return-consistency objectives (Wang et al., 7 May 2026).

Table: Summary of Neural Discount Model Implementations

Model Type	Neural/Algorithmic Mechanism	Key Reference
Quantized value discounting	Value quantization operator $\lambda > 0$0	(Tee et al., 2020)
Leaky evidence accumulators	Leak rate $\lambda > 0$1 tuned by hazard	(Piet et al., 2017)
Multi-head Q-learning	Parallel discount heads in network	(Fedus et al., 2019)
Neural HJB solvers	Deep neural collocation for value	(Schultheis et al., 2022)
State-dependent RL discount	MLP-predicted $\lambda > 0$2	(Wang et al., 7 May 2026)

3. Methodologies for Fitting and Learning Discounts

Parameter estimation and model selection in neural discount models leverage both behavioral and algorithmic approaches:

• Behavioral model fitting: Parameters such as discount rate, bit precision, and inverse temperature are fitted by minimizing negative log-likelihoods of binary choices (e.g., delayed reward tasks), with information criteria (AIC, BIC) used for model selection. Cross-validation and bootstrap procedures assess robustness against overfitting and confounds (Tee et al., 2020).
• Inverse reinforcement learning (IRL): Deep IRL frameworks infer the underlying discount function (or its parameters) from agent behavior by optimizing losses that penalize deviations between observed choices and those predicted by an optimal policy under a parametric discount family. Gradients are computed via differentiation through neural HJB solvers (Schultheis et al., 2022).
• Auxiliary and multi-horizon training: Training value estimators over a grid of discount factors provides auxiliary supervision that empirically improves representational quality and policy performance, an effect observed even when the acting policy uses a fixed discount (Fedus et al., 2019).
• Return-consistency regularization: Adapting the discount factor per state introduces hazards of trivial collapse; this is countered by loss terms that enforce consistency between one-step and n-step returns under a fixed reference discount, preventing degenerate solutions (Wang et al., 7 May 2026).

4. Scale-Invariance, Memory Constraints, and Subjective Time

Some neural discount models derive temporal discounting and time perception directly from information-theoretic and memory constraints:

• Scale-invariant forecasting: Logarithmically-compressed, parallel leaky-integrator architectures yield predictive timelines that are scale-free over future intervals. Discounting with $\lambda > 0$3 yields power-law (hyperbolic) functions without predefined characteristic timescales (Tiganj et al., 2018).
• Information-theoretic subjective time: Constraints on an agent's predictive memory capacity induce renormalized time perceptions, producing discount functions (exponential or hyperbolic) as direct consequences of how predictive information grows with time horizon. This framework accounts for known empirical distortions of duration and intertemporal choice (Ortega et al., 2016).

5. Applications and Empirical Findings

Neural discount models have explanatory and predictive value across decision neuroscience, artificial intelligence, and behavioral economics:

• Human intertemporal choice: Fitting quantized discount models to behavioral data reveals that a 5-bit precision (32 levels) matches observed indifference regions in human delay discounting, supporting discrete neural coding phenomenology (Tee et al., 2020).
• Deep RL stability and efficiency: Progressive or state-/parameter-dependent discount schedules in DQN, SAC, and PPO improve convergence speed, stability, and performance across diverse tasks, including high-dimensional continuous control and real-world recommender systems (François-Lavet et al., 2015, Wang et al., 7 May 2026).
• Addiction and hierarchical RL: Differential discounting in multi-level neural architectures explains the upward propagation of drug-seeking tendencies and the amplification of impulsivity as a function of steeper temporal discount rates, aligning with empirical findings on addiction severity and neural timescale gradients in the striatum (Palod et al., 5 Jun 2025).
• Evidence integration: Rats flexibly adjust the timescale of evidence accumulation in dynamic environments, in accordance with theoretically optimal discounting based on hazard rates and sensory noise, supporting neurally-plausible accumulator models (Piet et al., 2017).

6. Open Challenges and Future Directions

Several limitations and open research directions emerge for neural discount modeling:

• The functional-neurobiological mapping between quantized/scale-invariant model parameters and specific neural populations or circuits requires direct neural measurement (e.g., BOLD signal quantization, single-unit codes) (Tee et al., 2020).
• Fully extending operator-theoretic convergence proofs for state-dependent, deep function-approximate discount learning remains an open problem (Wang et al., 7 May 2026).
• Integration with temporal abstraction frameworks (options, subgoals, termination critics) and exploration strategies in large-scale RL contexts offers a promising but underexplored direction (François-Lavet et al., 2015).
• Automated meta-learning of discount schedules or architectures, and further empirical characterization of discount-pathology links in clinical populations, are active research areas (Ortega et al., 2016, Palod et al., 5 Jun 2025).

7. Synthesis and Conceptual Significance

Neural discount models provide a rigorous, neurally-grounded mathematical and computational account of how time, evidence, and memory constraints shape value-based decision making. By linking quantized, non-exponential, state-dependent, and biologically plausible forms of discounting to both observable choice behavior and neural architecture, these models bridge gaps between theory, implementation, and empirical phenomenon across multiple domains in cognitive science, machine learning, and clinical research (Tee et al., 2020, Schultheis et al., 2022, Tiganj et al., 2018, Wang et al., 7 May 2026).