Time-Since-Last-Reward (TSLR) Dynamics

• TSLR is a discrete, history-dependent counter per arm that resets on a reward event, analogous to age-of-information in scheduling and queuing.
• It informs reward regularity by quantifying the variability of inter-reward intervals, thus balancing fairness and exploration in bandit algorithms.
• TSLR principles extend to diverse applications including autoregressive models, TD learning credit assignment, and stochastic process record statistics.

Searching arXiv for the cited TSLR-related papers to ground the article in published work. {"queries":[{"query":"Time-Since-Last-Reward combinatorial multi-armed bandit (Wu et al., 15 Sep 2025)"},{"query":"GenARM Autoregressive Reward Model (Xu et al., 2024)"},{"query":"Demystifying the Recency Heuristic in Temporal-Difference Learning (Daley et al., 2024)"}]} Time-Since-Last-Reward (TSLR) denotes a history-dependent temporal variable that measures how long it has been since a reward event last occurred. In its explicit combinatorial multi-armed bandit formulation, TSLR is a per-arm discrete counter with reset-on-success dynamics and an interpretation closely aligned with age-of-information (Wu et al., 15 Sep 2025). Closely related temporal constructions also appear in autoregressive test-time alignment, sequential recommendation, temporal-difference credit assignment, quantitative reward monitoring for non-Markovian reinforcement learning, non-stationary bandits with last-switch dependence, and stochastic-process record statistics, although several of those works use analogous quantities rather than the exact term “TSLR” (Xu et al., 2024, Chen et al., 2022, Daley et al., 2024, Adalat et al., 16 Nov 2025, Laforgue et al., 2021, Martin et al., 2018).

1. Formal definition and state dynamics

In the explicit formulation of TSLR, the quantity is defined per arm. For each arm $n \in \{1,\dots,N\}$ and round $t$, the TSLR is denoted by $Z_n(t)$, where $S_n(t)\in\{0,1\}$ indicates whether the arm is pulled and $X_n(t)\in\{0,1\}$ is the realized Bernoulli reward with unknown mean $\mu_n$. Its dynamics are

$$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$

Thus, if an arm is not rewarded in round $t$—either because it was not pulled or because it was pulled and failed—its age increases by one; if it receives a successful reward, the counter resets to $1$ rather than $0$ (Wu et al., 15 Sep 2025).

This reset convention is a recurrent source of confusion. The paper notes that one could also use $t$0, but the analysis is written with $t$1 and reset-on-success to $t$2. TSLR is also strictly per arm: arms may be activated in combinatorial subsets, but no separate TSLR is defined for a subset or joint action. Regularity is assessed at the arm level and then aggregated across arms (Wu et al., 15 Sep 2025).

The same paper makes clear that TSLR is not merely a bookkeeping device. It is a state variable whose dynamics are intentionally parallel to age processes in scheduling and queuing. That interpretation matters because it determines both the objective being optimized and the analytical tools used to study the algorithmic consequences of reward recency (Wu et al., 15 Sep 2025).

2. Reward regularity, age-of-information, and what TSLR measures

TSLR is explicitly connected to age-of-information (AoI): it is “essentially the same as the time since the last service” and “age of information,” but with the notion of service replaced by receipt of reward. The principal regularity metric is the running average of total expected TSLR,

$t$3

and the paper states that smaller TSLR implies more regular reward arrivals (Wu et al., 15 Sep 2025).

This interpretation is sharpened by a renewal-theoretic identity: if $t$4 denotes the inter-reward time of arm $t$5, then

$t$6

Accordingly, controlling mean TSLR controls not only average inter-reward time but also its variability. TSLR therefore measures short-term temporal regularity rather than cumulative reward alone (Wu et al., 15 Sep 2025).

That distinction separates TSLR objectives from ordinary regret minimization. A standard CMAB algorithm seeks to maximize

$t$7

which can produce starvation of low-mean arms and bursty reward sequences even when fairness constraints require minimum average service. TSLR penalizes such irregularity because long gaps without reward cause $t$8 to grow, which in turn increases the priority of the neglected arm. A common misconception is therefore to treat TSLR as a proxy for reward magnitude; in the formulation above, it is a proxy for reward spacing and regularity (Wu et al., 15 Sep 2025).

3. TSLR in regular-and-fair combinatorial bandits

The most direct algorithmic use of TSLR appears in the Regular and Fair Learning (RFL) algorithm. RFL combines three signals: virtual queues $t$9 for fairness, TSLR $Z_n(t)$0 for regularity, and UCB weights $Z_n(t)$1 for exploration and exploitation. The per-arm UCB term is defined from the pull count

$Z_n(t)$2

the sample mean

$Z_n(t)$3

and the truncated confidence bound

$Z_n(t)$4

Fairness is encoded by virtual queues

$Z_n(t)$5

with $Z_n(t)$6 chosen so that $Z_n(t)$7. RFL then selects the feasible activation vector solving

$Z_n(t)$8

where $Z_n(t)$9 and $S_n(t)\in\{0,1\}$0 tune the regularity–regret tradeoff (Wu et al., 15 Sep 2025).

The key structural lemma is

$S_n(t)\in\{0,1\}$1

assuming $S_n(t)\in\{0,1\}$2. This sample-path inequality links fairness debt to reward age: large TSLR necessarily implies a large queue up to scaling. The paper uses this connection inside Lyapunov drift arguments with

$S_n(t)\in\{0,1\}$3

to obtain finite-time fairness and regularity guarantees (Wu et al., 15 Sep 2025).

The resulting performance statements make the tradeoff explicit. Proposition 1 states that, under $S_n(t)\in\{0,1\}$4, there exists

$S_n(t)\in\{0,1\}$5

after which zero cumulative fairness violation holds. Proposition 2 bounds the running average of total expected TSLR by the minimum of a queue-based term and an intrinsic regularity term; the paper summarizes the second as

$S_n(t)\in\{0,1\}$6

so increasing $S_n(t)\in\{0,1\}$7 improves regularity and increasing $S_n(t)\in\{0,1\}$8 worsens it. Proposition 3 gives a cumulative regret bound

$S_n(t)\in\{0,1\}$9

which makes the same tradeoff visible on the regret side: larger $X_n(t)\in\{0,1\}$0 improves reward regularity but worsens regret, while larger $X_n(t)\in\{0,1\}$1 improves regret but worsens regularity (Wu et al., 15 Sep 2025).

4. Token-level and representation-level analogues of TSLR

Several recent sequence-modeling papers introduce temporally structured quantities that are not named TSLR but are directly analogous to it.

In GenARM, the original paper does not define TSLR explicitly, but the autoregressive reward model (ARM) yields a natural token-level reward process because

$X_n(t)\in\{0,1\}$2

The supplied formulation therefore defines a thresholded reward event

$X_n(t)\in\{0,1\}$3

and a token-level TSLR

$X_n(t)\in\{0,1\}$4

This quantity can be used to modulate reward guidance through a dynamic weight $X_n(t)\in\{0,1\}$5, yielding

$X_n(t)\in\{0,1\}$6

The proposal is explicitly marked as a natural extension rather than a definition from the original GenARM paper, but it shows how dense token-level reward signals make “time since last meaningful reward” operational at decoding time (Xu et al., 2024).

TLSRec provides a different but structurally similar mechanism. Its central temporal quantity is the lag

$X_n(t)\in\{0,1\}$7

discretized as

$X_n(t)\in\{0,1\}$8

embedded by

$X_n(t)\in\{0,1\}$9

and fed to a neural time gate

$\mu_n$0

The final fusion is

$\mu_n$1

The paper is about time since last interaction rather than reward, but it explicitly states that in a TSLR setting the same machinery can be applied by replacing the last interaction with the last reward event. The case study further reports that the average gate value decreases with lag, so larger lag shifts weight from short-term to long-term information (Chen et al., 2022).

Setting	Temporal quantity	Functional role
GenARM	$\mu_n$2	Adaptive reward temperature in decoding
TLSRec	$\mu_n$3	Time-lag-sensitive fusion of long/short-term preferences

A plausible implication is that TSLR can serve either as a hard counter with reset-and-increment dynamics, as in bandits, or as a learned latent control signal, as in neural sequence models. The distinction is methodological rather than conceptual: in both cases, elapsed time since a reward-relevant event modulates present decisions.

5. Credit assignment and non-Markovian reward specification

TSLR also has a direct interpretation as a temporal credit-assignment kernel. In the generalized TD return

$\mu_n$4

the weight sequence $\mu_n$5 measures how much a future TD error $\mu_n$6 credits an earlier stimulus at time $\mu_n$7. The weak recency heuristic requires

$\mu_n$8

and the strong recency heuristic requires

$\mu_n$9

TD$$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$0 is the canonical strong-recency case because

$$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$1

so the influence of a TD error decays exponentially with the time lag between the state and the error. The paper proves that any return estimator satisfying the weak recency heuristic is exactly a convex combination of $$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$2-step returns, hence yields a contraction mapping with the correct fixed point in the on-policy tabular setting; it also provides a counterexample where a delayed, non-monotone weighting diverges (Daley et al., 2024).

This places TSLR-like constructions inside a broader theory of admissible temporal weighting. Monotone nonnegative lag-weighting is theoretically safe; non-monotone weighting may be intuitive in delayed-reward settings, but it can destroy the contraction property. The same paper shows that long-tailed convex mixtures can retain a long window of effective credit assignment while keeping worst-case variance bounded through the contraction modulus $$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$3 (Daley et al., 2024).

A complementary line of work approaches the issue from temporal logic rather than TD weighting. Quantitative reward monitors synthesized from $$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$4 maintain real-valued registers over finite traces and emit dense stepwise rewards for non-Markovian specifications. The paper does not define TSLR explicitly, but it states that TSLR is a particular kind of history-dependent reward mechanism: a signal that depends on how long ago the agent last received a reward. Its monitor framework naturally supports such constructions because rewards are functions of monitor state and registers, and the product MDP $$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$5 converts non-Markovian reward objectives into Markovian control over an extended state space. The paper therefore provides a principled route to implementing TSLR-like reward shaping as a quantitative monitor with a reset-on-event register and a reward function decreasing in elapsed time since that event (Adalat et al., 16 Nov 2025).

6. Generalizations, analogues, and open tradeoffs

Beyond the explicit CMAB formulation, TSLR-like quantities appear in several adjacent settings. In Last Switch Dependent (LSD) bandits, each arm has a signed state $$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$6: positive values encode time since the arm was last played, while negative values encode the length of the current play streak. The expected reward when playing arm $$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$7 at time $$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$8 is

$$Z_n(t+1)= \begin{cases} Z_n(t)+1, & \text{if } S_n(t)X_n(t)=0,\ 1, & \text{if } S_n(t)X_n(t)=1. \end{cases}$$9

The positive branch behaves like a time-since-last-play variable, while the negative branch captures satiation under repeated use. The paper proves that computing the optimal policy is NP-hard, proposes the block-based ISI-CombUCB1 algorithm, and obtains regret bounds of order $t$0 by balancing approximation and estimation. This suggests that TSLR-like dependence can be generalized from reward regularity to novelty, recovery, satiation, and seasonality, but at a substantial algorithmic cost (Laforgue et al., 2021).

An even broader analogue appears in stochastic-process record statistics. For a process $t$1 on $t$2, the drawdown time

$t$3

is the time since the process last achieved its running maximum. The paper explicitly identifies this as a TSLR-type quantity. For Brownian motion with drift and for completely asymmetric Lévy processes, it derives exact density formulas and factorization results of the form

$t$4

This shows that “time since last event” variables can be studied independently of control or learning, as intrinsic functionals of path-dependent stochastic dynamics (Martin et al., 2018).

Across these literatures, the main open issues are tradeoff and structure rather than definition. In the CMAB setting, the exact optimality of the regret–regularity tradeoff remains open, and low-complexity variants of RFL are identified as future work. In LSD bandits, the optimal cyclic-policy problem is itself NP-hard, and the structure of optimal non-stationary policies is unresolved. In TSLR-aware neural and monitor-based systems, the supplied formulations suggest several natural constructions, but in GenARM and quantitative reward monitoring they remain extensions of the underlying framework rather than original named definitions (Wu et al., 15 Sep 2025, Laforgue et al., 2021, Xu et al., 2024, Adalat et al., 16 Nov 2025).

Taken together, these results support a unified view. TSLR is a temporally local summary of reward history whose formal role depends on the problem class: an age process for regularity in bandits, a lag-weighting kernel in TD learning, a latent gating signal in sequence models, a monitor register in non-Markovian RL, a signed recovery-or-satiation state in non-stationary bandits, or a last-record functional in stochastic processes. The common invariant is the same: elapsed time since the most recent reward-relevant event is treated as a state variable with algorithmic, statistical, or probabilistic consequences.