Temporal-Sensitivity Reward

• Temporal-sensitivity rewards are reward constructions that incorporate temporal characteristics like order, delay, and duration to evaluate processes beyond static outcomes.
• They are implemented via techniques such as latent state consistency, temporal localization objectives, and bootstrapped process values to reflect continuous progress.
• Applied in robotics, video analysis, language model reasoning, and bandit problems, these rewards improve credit assignment, sample efficiency, and interpretability.

Temporal-sensitivity reward denotes a family of reward constructions in which value depends not only on whether a desirable outcome occurs, but also on the temporal structure through which it occurs: order, delay, duration, phase progression, temporal localization, or sensitivity of intermediate reasoning to perturbations in time. The term does not name a single canonical formalism across the literature. Instead, it appears as a recurring design principle in reinforcement learning, imitation and reward learning, video understanding, language-model reasoning, temporal logic monitoring, bandits, and intertemporal decision-making. Across these settings, the common move is to replace purely terminal, static, or temporally invariant scoring with reward signals that are explicitly conditioned on temporal evolution, whether by latent dynamics constraints, temporal localization objectives, bootstrapped process values, time-indexed masks, deadline-sensitive utility, or quantitative finite-trace semantics (Wu et al., 2022, Fu et al., 2024, Wang et al., 12 May 2026, Zhang et al., 18 Sep 2025, Adalat et al., 16 Nov 2025).

1. Conceptual scope and historical background

A temporally sensitive reward differs from a conventional reward in that it treats time as part of the evaluative object rather than merely an index over transitions. In some formulations, this means that reward depends on whether events occur in the correct sequence; in others, that it depends on how quickly they occur, whether they persist, whether local temporal evidence supports the reasoning that produced an answer, or whether a trajectory prefix already exhibits partial satisfaction of a temporally extended objective. This suggests that temporal sensitivity is best understood as a structural property of reward design rather than a single algorithmic technique.

Several distinct research lines motivate this shift. In robotic manipulation, sparse success/failure rewards are described as too weak for efficient exploration, while hand-crafted dense rewards are brittle and prior methods often fail to directly address multimodal contact signals such as force/torque; the response is to learn a reusable dense reward from demonstrations and a simulator, but to make the latent representation respect one-step temporal evolution (Wu et al., 2022). In imitation and proxy-reward learning from video, standard optimal-transport reward labeling is criticized because it is largely insensitive to temporal order, so two trajectories containing similar frames but in different orders may receive similar OT values (Fu et al., 2024). In generated-video anomaly modeling, holistic quality rewards are insufficient because anomalies are “highly sparse in both space and time,” and the reward model must reason over when the anomaly occurs, not only whether the entire video is acceptable (Wang et al., 12 May 2026). In LLM reasoning, the problem is temporal inconsistency of process scores across adjacent reasoning states, which can make policy updates unstable and inference-time verification unreliable (Zhang et al., 18 Sep 2025).

The broadest conceptual formulation predates these machine-learning instantiations. In TIMERR, delay sensitivity is derived from bounded reward-rate optimization over a finite temporal integration window rather than introduced as an independent discounting bias. The subjective value of a delayed reward is

$$SV(r,t)= \frac{r - a_{\mathrm{est}} t}{1+\frac{t}{T_{\mathrm{ime}}}},$$

and subjective time is

$ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$

Here temporal sensitivity alters reward computation through both opportunity cost and nonlinear subjective time, with the same parameter $T_{\mathrm{ime}}$ governing discounting steepness and temporal compression (Namboodiri et al., 2013).

2. Core design patterns

Across the literature, temporal-sensitivity reward is implemented through a small number of recurring mechanisms.

First, some methods make reward depend on temporally structured latent progress. In robotic dense reward learning, progress is defined by normalized latent distance to a goal,

$$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$

but the latent space itself is trained to satisfy adjacent-state temporal consistency,

$$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$

so the resulting reward is sensitive to phase-wise progression and regressions rather than to static visual similarity alone (Wu et al., 2022).

Second, some methods attach reward to temporal localization accuracy. In CaC, the key temporal reward is frame-set IoU: $$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$ This reward supervises the first turn of a two-turn video anomaly localization process and is explicitly described as an intermediate localization reward. The paper reports that removing Temporal IoU reward causes the largest performance drop among the process-level rewards, from $0.817$ to $0.629$ overall accuracy (Wang et al., 12 May 2026).

Third, some methods impose temporal priors on alignment itself. TemporalOT modifies OT reward by replacing framewise cost with a context cost

$$\hat c(o_i, o^E_j) = \frac 1 {k_c} \sum_{h=0}^{k_c - 1} \left( 1 - \frac{\langle f(o_{i+h}), f(o^E_{j+h}) \rangle}{\Vert f(o_{i+h}) \Vert \Vert f(o^E_{j+h}) \Vert} \right),$$

and by applying a temporal mask

$$M(i, j) = \begin{cases} 1, & \text{if } j \in [i-k_m, i+k_m], \ 0, & \text{otherwise}. \end{cases}$$

This biases transport toward near-synchronous alignments. The paper is explicit that the method does not impose strict monotonicity and is not dynamic time warping; it encodes temporal order through proximity in indices under a similar-speed assumption (Fu et al., 2024).

Fourth, some methods smooth process reward over time by Bellman-style coupling. TDRM treats reasoning steps as ordered states and trains a process reward model with TD-soft targets,

$ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$0

rather than with independent hard labels. The paper stresses that its “TD regularization” is not a separate additive penalty, but is realized by replacing hard targets with temporally bootstrapped soft targets (Zhang et al., 18 Sep 2025).

Fifth, some methods score reasoning by perturbing temporally critical evidence. In TaRO, a reasoning trace $ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$1 is rescored on the original video $ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$2 and a perturbed video $ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$3 whose frames near the ground-truth start and end are locally shuffled: $ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$4

$ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$5

$ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$6

The temporal-sensitive reward is then assigned by group-relative thresholding: $ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$7 The underlying intuition is that high-quality reasoning should become less plausible when event-boundary evidence is disrupted (Zheng et al., 8 Jun 2026).

A common misconception is that temporal sensitivity always means per-step dense reward in the RL sense. CaC explicitly distinguishes its reward as rollout-level but decomposed into intermediate structured components, with status reward, Temporal IoU, Spatial IoU, Attribution IoU, and format reward organized hierarchically rather than as framewise environmental feedback (Wang et al., 12 May 2026).

3. Robotics, imitation, and control

In contact-rich robotics, temporally sensitive reward has been developed primarily as a remedy for sparse or brittle hand-designed shaping. “Learning Dense Reward with Temporal Variant Self-Supervision” learns a latent representation from RGB-D observations, force/torque, and velocity, then converts latent distance-to-goal into dense task progress. The method’s distinctive claim is that a reward representation should not merely distinguish start from goal statically but encode how the system progresses over time, especially in manipulation phases involving contact initiation, insertion, or handle rotation (Wu et al., 2022).

The sampling procedure is central. Temporal Variant Forward Sampling uses a time-dependent variance control function $ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$8 so that early and late stages are constrained while intermediate stages permit more deviation. Concretely, the reported experimental settings are sampling interval $ST(t)= \frac{t}{1+\frac{t}{T_{\mathrm{ime}}}}.$9, number of branches $T_{\mathrm{ime}}$0, number of steps per branch $T_{\mathrm{ime}}$1, and temporal variance controlled with an angle parameter

$T_{\mathrm{ime}}$2

The model trains on adjacent observation pairs $T_{\mathrm{ime}}$3, not on explicit temporal ranking labels, and the loss combines reconstruction with temporal enforcement: $T_{\mathrm{ime}}$4 with $T_{\mathrm{ime}}$5 (Wu et al., 2022).

The reward learned in this way is qualitatively temporally sensitive. In lap-joint assembly, reward rises gradually from $T_{\mathrm{ime}}$6 to $T_{\mathrm{ime}}$7 on successful trajectories but can decrease below $T_{\mathrm{ime}}$8 on failed ones when the latent state becomes farther from the goal than the initial state. In door opening, the learned reward fluctuates during an “inexpert” handle-rotation phase where a hand-crafted distance reward gives mostly analogous signals. The paper reports that, when used with Soft Actor-Critic for 500 epochs on the door-opening task, the learned reward yields “faster convergence, and more training stability” than a hand-crafted distance reward $T_{\mathrm{ime}}$9 and a sparse boolean reward (Wu et al., 2022).

TemporalOT addresses a different defect: proxy rewards derived from optimal transport over expert and agent trajectories can be temporally invariant because the transport plan is not constrained to preserve order. The per-step OT reward

$$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$0

can therefore reward trajectories that visit visually similar states in the wrong sequence. TemporalOT introduces local temporal context in the cost and a diagonal temporal mask in the transport plan. On 9 Meta-World tasks, the reported average success rates are TaskReward 20.8, BC 2.5, GAIfO 5.2, OT0.99 27.8, OT0.9 35.5, ADS 35.9, and TemporalOT 61.1, with TemporalOT outperforming ADS on 8 of 9 tasks (Fu et al., 2024).

The paper also clarifies what temporal sensitivity does and does not mean in this setting. The diagonal mask encourages near-synchronous matches, but does not enforce a globally monotone alignment path. Performance drops as demonstration speed mismatch grows, confirming that the approach relies on the assumption that agent and expert progress at roughly the same speed (Fu et al., 2024).

4. Language, video, and multimodal reasoning

In contemporary multimodal systems, temporally sensitive reward is often attached to structured reasoning rather than only to an end result.

CaC formulates generated-video anomaly evaluation as hierarchical spatiotemporal concentration. A first turn scans the whole video at 4 fps and predicts abnormality, a temporal window, and anomaly type; a second turn crops the predicted interval, resamples it at 8 fps, and predicts abnormal/normal status again plus frame-indexed bounding boxes and anomaly type. The full reward is

$$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$1

with weights

$$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$2

The temporal component directly evaluates frame-index interval overlap, and the paper reports that removing Temporal IoU reward causes the largest ablation drop, “because it directly determines the decision range of the second turn” (Wang et al., 12 May 2026).

TaRO addresses a related but distinct failure mode in video temporal grounding. Existing rewards such as

$$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$3

reward formatting and answer overlap but not whether the reasoning path is truly anchored to temporally critical evidence. TaRO’s Temporal-Sensitivity Reward tests whether the average token log-probability of a generated reasoning path falls when frames near the ground-truth start and end are locally shuffled. The final RL reward is

$$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$4

with reported hyperparameters $$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$5, $$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$6, $$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$7, and $$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$8. On Charades-STA, adding the temporal reward alone improves [email protected] from $$p = 1 - \frac{d(h_\phi(s), h_\phi(s_g))}{d(h_\phi(s_0), h_\phi(s_g))},$$9 to $$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$0 (Zheng et al., 8 Jun 2026).

In LLM reasoning, TDRM treats temporal inconsistency as excessive adjacent-state score variance. It defines TD error

$$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$1

value-change magnitude

$$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$2

and a local smoothness metric

$$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$3

The reported average local Lipschitz constant decreases from 0.3331 for ScalarPRM to 0.2741 for TDRM, and abstracted inference gains include Best-of-$$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$4 improvements up to 6.6% and tree-search gains up to 23.7% (Zhang et al., 18 Sep 2025).

A related but domain-specific example appears in image-to-video generation. VCD defines a conditioning-image-relative temporal consistency signal: $$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$5 The term is strongest for early frames and gradually relaxes. The paper reports that, on VBench-I2V for Open-Sora, fine-tuning with VCD improves Subject Consistency from 96.27 to 97.94 and Temporal Flickering from 98.98 to 99.21, while lowering Dynamic Degree from 26.59 to 17.56, indicating a tradeoff between stronger consistency and weaker large-motion generation (Aoshima et al., 22 Oct 2025).

5. Formal specifications, monitoring, and temporally extended tasks

Another major strand treats temporal sensitivity as a specification property. “Expressive Temporal Specifications for Reward Monitoring” uses quantitative $$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$6 to synthesize reward monitors whose outputs equal the quantitative valuation of a temporal formula over the current finite trace. The semantics are real-valued: $$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$7

$$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$8

$$h_\phi(s_t) + \Delta h_{\psi}(s_t) = h_\phi(s_{t+1}),$$9

The monitor construction is online, algorithm-agnostic, and linear in formula size. This yields dense, history-aware rewards for non-Markovian objectives such as safety, order, and persistence (Adalat et al., 16 Nov 2025).

Timed reward machines extend this perspective by incorporating explicit clocks and guards. A TRM is

$$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$0

with clock constraints of the form

$$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$1

In the induced MDP, an action includes both a delay and an environment action,

$$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$2

This allows reward to depend on whether events happen before a deadline, after sufficient dwell time, or with costly delay. The paper distinguishes digital-time and real-time semantics and reports that corner-point abstraction gives the best discounted return in experiments relative to digital clocks, uniform discretization, and ordinary reward machines (Majumdar et al., 19 Dec 2025).

Bandit models with temporally partitioned rewards show a related principle in a non-RL setting. In TP-MAB, a pull at time $$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$3 generates a reward vector

$$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$4

whose cumulative reward is

$$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$5

Under $$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$6-smoothness, the reward is spread evenly enough over time that partial feedback becomes statistically useful. The lower bound improves by a factor $$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$7, and TP-UCB-FR attains a dominant regret term of order

$$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$8

This suggests that temporal smoothness itself can be a complexity parameter for reward learning (Romano et al., 2022).

6. Incentives, fairness, and recurring limitations

Temporal-sensitivity reward is not confined to learning systems. In crowdsourcing, REFORM studies PBMs in “temporal settings,” meaning settings that prefer early reports. The generic reward is

$$R_{\mathrm{temp}}(y_i) = \frac{|\mathcal{F}_{\mathrm{pred}} \cap \mathcal{F}_{\mathrm{gt}}|}{|\mathcal{F}_{\mathrm{pred}} \cup \mathcal{F}_{\mathrm{gt}}|}.$$9

where $0.817$0 is a time-decay factor. REFORM adds a Temporal Reputation Model

$0.817$1

with round score

$0.817$2

Time therefore enters both immediately through $0.817$3 and dynamically through reputation, which affects future pairing chances. The simulations report $0.817$4 values after 200 rounds of about 0.09 for REFORM and 0.05 for RPTSC, though the paper also notes that the decay factor $0.817$5 is ignored in experiments, so empirical validation of temporal sensitivity is indirect (Kanaparthy et al., 2021).

A broader implication is that temporal sensitivity often introduces strong assumptions. Robotics reward learning assumes access to a simulator, multimodal sensing, and tasks with a relatively deterministic goal state (Wu et al., 2022). TemporalOT assumes similar-speed progress between expert and agent (Fu et al., 2024). CaC depends on expensive human annotations of temporal windows and per-frame boxes (Wang et al., 12 May 2026). TaRO requires ground-truth temporal boundaries and adds an extra forward pass on perturbed video, increasing per-step training time by 13.8% (Zheng et al., 8 Jun 2026). VCD improves consistency but lowers dynamic-degree metrics, so stronger temporal anchoring can suppress large motion (Aoshima et al., 22 Oct 2025). TDRM works best as a supplement to verifiable reward rather than as a standalone dense replacement, with the best reported RL mixture coefficient at $0.817$6 (Zhang et al., 18 Sep 2025).

A second recurring limitation is that “temporal sensitivity” is not equivalent to one specific mathematical object. It may refer to delay sensitivity in intertemporal choice, temporal order sensitivity in transport alignment, localization sensitivity in video anomaly detection, perturbation sensitivity of reasoning traces, or online quantitative valuation of temporal logic. A plausible implication is that the unifying question is not whether time appears in the reward, but how the reward discriminates among trajectories that would otherwise look equivalent under static or terminal scoring.

A third recurring lesson is that temporal shaping is often most valuable when outcome-only reward is underconstrained. This is explicit in robotics with sparse contact-rich tasks (Wu et al., 2022), in VTG and anomaly localization where answer correctness alone does not guarantee grounded reasoning (Wang et al., 12 May 2026, Zheng et al., 8 Jun 2026), in LLM reasoning where hard step labels do not propagate long-term credit (Zhang et al., 18 Sep 2025), and in temporally partitioned bandits where partial reward fragments can already constrain the value of an action (Romano et al., 2022).

Taken together, the literature supports a general characterization: temporal-sensitivity reward is any reward construction that makes evaluation depend on temporally meaningful structure—delay, order, duration, localization, persistence, or temporal evidence dependence—rather than on static state similarity or terminal success alone. Its practical role is to improve credit assignment, interpretability, and sample efficiency in settings where the temporal organization of behavior is itself part of the task definition (Wu et al., 2022, Fu et al., 2024, Wang et al., 12 May 2026, Zhang et al., 18 Sep 2025, Adalat et al., 16 Nov 2025).