Temporal Skipping: Mechanisms and Impact

• Temporal skipping is a design pattern that selectively processes time steps by skipping redundant or less informative inputs, enhancing temporal abstraction.
• It is applied across domains like reinforcement learning, video analysis, speech recognition, and networking to improve efficiency and learning speed.
• Empirical studies demonstrate significant computation savings and faster convergence, with performance gains dependent on proper skip parameter tuning.

Temporal skipping denotes a family of techniques that alter the temporal granularity at which a model senses, predicts, updates, or acts. Rather than processing every time step uniformly, these methods either repeat an action for multiple steps, skip state updates, subsample frames or snapshots, attach delayed skip connections, or deliberately ignore arrivals, handovers, or parameter updates when they are deemed redundant or harmful. Across reinforcement learning, recurrent sequence modeling, video understanding, speech recognition, temporal knowledge graphs, spiking neural networks, recommender training, and wireless systems, the common objective is to trade dense step-by-step processing for a temporally selective mechanism that preserves task-relevant information while reducing decision points, computation, or control overhead (Biedenkapp et al., 2021, Campos et al., 2017, Neitz et al., 2018, Habibian et al., 2021).

1. Scope and canonical formulations

The literature does not present temporal skipping as a single formalism. In reinforcement learning, it appears as explicit action repetition over a learned skip length; in recurrent models, as learned omission of hidden-state updates; in video and speech, as frame subsampling or blank-guided frame removal; in graph and spiking models, as skip information flow or temporally delayed skip connections; and in networking, as deliberate suppression of update transmissions or handovers (Biedenkapp et al., 2021, Campos et al., 2017, Yang et al., 2023, Wang et al., 2023, Malettira et al., 2024, Wang et al., 2018, Tokuyama et al., 2020).

Domain	Skipped unit	Representative mechanism
Reinforcement learning	Decision points / actions	Learned skip length $k$ for repeating $a_t$ (Biedenkapp et al., 2021)
Recurrent models	Hidden-state updates	Binary gate $u_t\in\{0,1\}$ decides UPDATE or COPY (Campos et al., 2017)
Dynamical prediction	Prediction intervals	Matching to any future frame up to horizon $H$ (Neitz et al., 2018)
Video / speech	Frames	Uniform frame skipping, random temporal skipping, or blank-guided removal (Muhammad et al., 2023, Zhu et al., 2018, Yang et al., 2023)
Temporal graphs / SNNs	Historical snapshots / delayed paths	Relation-aware skip flow or explicit temporal delays $\Delta t>1$ (Wang et al., 2023, Malettira et al., 2024)
Networks / information systems	Updates or handovers	Skip-or-switch, periodic skipping, time-based skipping (Wang et al., 2018, Tokuyama et al., 2023, Tokuyama et al., 2020)

A central distinction is whether the skip variable is explicit. TempoRL defines a joint choice $(a_t,k_t)$ and optimizes a skip-policy $\pi_J(s,a)\to j$ (Biedenkapp et al., 2021). Skip RNN defines a binary skip gate $u_t$ that either computes a new state or copies the previous one (Campos et al., 2017). By contrast, Adaptive Skip Intervals does not learn an explicit “action” or “skip-policy” module; the skip decision is implicit in the matching supervisor (Neitz et al., 2018). In speech, blank-regularized CTC does not predict a skip action directly either; it regularizes alignments so that blank symbols can be used to discard frames (Yang et al., 2023).

This suggests that temporal skipping is best understood as a design pattern for temporal abstraction rather than as a single algorithmic family. The operative question varies by domain: when to act, when to update, which historical states to retrieve, which frames to process, or which transmissions to suppress.

2. Action repetition and temporal abstraction in control and prediction

In "TempoRL: Learning When to Act" (Biedenkapp et al., 2021), a standard discounted MDP $M=\langle S,A,P,R,\gamma\rangle$ is augmented with a skip-length $k\in J=\{1,2,\dots,J_{\max}\}$. The agent chooses pairs $a_t$0 and commits to repeating $a_t$1 for $a_t$2 consecutive environment steps. The induced skip return is the usual $a_t$3-step return, and the objective is to learn a joint policy $a_t$4 and $a_t$5 that maximizes the expected $a_t$6-discounted return while reducing the total number of decision points (Biedenkapp et al., 2021).

TempoRL learns two value functions: a behavior value $a_t$7 and an action-conditioned skip value

$a_t$8

At decision time, the agent first picks $a_t$9 and then $u_t\in\{0,1\}$0, with $u_t\in\{0,1\}$1-greedy variants for exploration (Biedenkapp et al., 2021). The method was reported to reach halfway-to-optimal reward $u_t\in\{0,1\}$2 faster and to use $u_t\in\{0,1\}$3 fewer decision points than vanilla Q in tabular grid-worlds; in the 6×10 Cliff task, normal Q needs $u_t\in\{0,1\}$4 episodes to reach average reward $u_t\in\{0,1\}$5, whereas TempoRL needs only $u_t\in\{0,1\}$6, and decision-step count is reduced from $u_t\in\{0,1\}$7 to $u_t\in\{0,1\}$8 (Biedenkapp et al., 2021).

Adaptive Skip Intervals (ASI) addresses a related but distinct temporal-abstraction problem for recurrent dynamical models (Neitz et al., 2018). Given a trajectory $u_t\in\{0,1\}$9, the model predicts $H$0, but instead of matching only to $H$1, a matching supervisor searches over $H$2 and picks

$H$3

The skip interval is therefore discovered through future-frame matching, not emitted by a control head (Neitz et al., 2018). ASI uses two curricula: exploration of temporal matching via a decaying probability $H$4, and scheduled sampling via a decaying temperature $H$5 (Neitz et al., 2018). On Room Runner, fixed-step $H$6 or $H$7 baselines top out at roughly $H$8–$H$9 validation accuracy after $\Delta t>1$0k model evaluations, whereas ASI with $\Delta t>1$1 converges in half the number of forward passes and reaches $\Delta t>1$2 accuracy; with exploration it reaches $\Delta t>1$3 (Neitz et al., 2018). On Funnel Board, baselines achieve $\Delta t>1$4–$\Delta t>1$5 accuracy after $\Delta t>1$6k steps, while ASI climbs to $\Delta t>1$7 in under $\Delta t>1$8k steps (Neitz et al., 2018).

A more recent manipulation-oriented formulation, SkiP, operationalizes temporal skipping through action relabeling rather than a learned skip planner (Dai et al., 15 May 2026). Demonstration timesteps are partitioned into key and skip segments by Motion Spectrum Keying (MSK), and if a timestep $\Delta t>1$9 lies in a skip segment, the target is relabeled to the entrance of the next key segment: $(a_t,k_t)$0 The same backbone then learns both skip mode and refine mode within one network (Dai et al., 15 May 2026). Across 72 simulated tasks and 3 real-robot tasks, SkiP reduces executed control steps by $(a_t,k_t)$1–$(a_t,k_t)$2 while matching or improving success rates (Dai et al., 15 May 2026).

3. Selective state updates, sparse reminders, and long-range temporal credit

Skip RNN formulates temporal skipping as learned omission of recurrent state updates (Campos et al., 2017). A base RNN cell $(a_t,k_t)$3 is augmented with a binary skip gate $(a_t,k_t)$4. When $(a_t,k_t)$5, the model updates with $(a_t,k_t)$6; when $(a_t,k_t)$7, it copies $(a_t,k_t)$8. The actual hidden state is

$(a_t,k_t)$9

An auxiliary budget loss

$\pi_J(s,a)\to j$0

encourages fewer updates (Campos et al., 2017). Reported reductions are $\pi_J(s,a)\to j$1–$\pi_J(s,a)\to j$2 in RNN updates and FLOPs, with equal or better accuracy in most operating points (Campos et al., 2017). On Sequential MNIST, accuracy rises from $\pi_J(s,a)\to j$3 for LSTM to $\pi_J(s,a)\to j$4–$\pi_J(s,a)\to j$5 for Skip-LSTM/GRU while using only $\pi_J(s,a)\to j$6 of the pixels; on UCF-101, Skip-LSTM/GRU reaches $\pi_J(s,a)\to j$7–$\pi_J(s,a)\to j$8 using only $\pi_J(s,a)\to j$9–$u_t$0 of frames (Campos et al., 2017).

Sparse Attentive Backtracking (SAB) uses temporal skipping for credit assignment rather than forward execution (Ke et al., 2018). At each time $u_t$1, a learned attention mechanism scores a memory buffer of past hidden states,

$u_t$2

retains at most $u_t$3 memories by top-$u_t$4 sparsification, and forms a retrieval summary $u_t$5 (Ke et al., 2018). The final hidden state is $u_t$6, so the gradient can flow directly from $u_t$7 to selected memories: $u_t$8 This “teleport[s]” credit arbitrarily far back in time through a sparse set of temporal skip connections (Ke et al., 2018). On the copying memory task with sequence length $u_t$9, SAB with $M=\langle S,A,P,R,\gamma\rangle$0 reaches $M=\langle S,A,P,R,\gamma\rangle$1, compared to $M=\langle S,A,P,R,\gamma\rangle$2 for LSTM + BPTT; when trained on $M=\langle S,A,P,R,\gamma\rangle$3 and tested at $M=\langle S,A,P,R,\gamma\rangle$4, SAB attains $M=\langle S,A,P,R,\gamma\rangle$5, while LSTM yields $M=\langle S,A,P,R,\gamma\rangle$6 and LSTM+self-attention is out of memory (Ke et al., 2018).

The earlier Skipping Recurrent Neural Network (S-RNN) applies skipping to latent subsequence discovery in visual albums rather than to hidden-state update frequency (Sigurdsson et al., 2016). A storyline is defined as an ordered subset $M=\langle S,A,P,R,\gamma\rangle$7 of an album $M=\langle S,A,P,R,\gamma\rangle$8, and the model maximizes the marginal likelihood over all ordered subsets: $M=\langle S,A,P,R,\gamma\rangle$9 Because the sum over $k\in J=\{1,2,\dots,J_{\max}\}$0 subsets is intractable, the method uses an EM-style sampling procedure (Sigurdsson et al., 2016). Reported long-term next-image prediction accuracy is $k\in J=\{1,2,\dots,J_{\max}\}$1 for S-RNN versus $k\in J=\{1,2,\dots,J_{\max}\}$2 for LSTM/RNN, and AMT preference for S-RNN storylines is $k\in J=\{1,2,\dots,J_{\max}\}$3–$k\in J=\{1,2,\dots,J_{\max}\}$4 against the strongest baseline (Sigurdsson et al., 2016).

A common misconception is that temporal skipping in sequence models simply means downsampling the input. The recurrent literature shows several distinct variants: skipping may omit computation while retaining every input symbol in principle, as in Skip RNN; it may create sparse long-range reminders for backward credit assignment, as in SAB; or it may model only an ordered latent subset, as in S-RNN.

4. Frame, snapshot, and delayed-path skipping in perception and structured temporal data

Video, speech, and temporal-graph models implement temporal skipping primarily as selective observation. In face anti-spoofing, "Deep Ensemble Learning with Frame Skipping for Face Anti-Spoofing" divides a video of $k\in J=\{1,2,\dots,J_{\max}\}$5 frames into non-overlapping segments of fixed size $k\in J=\{1,2,\dots,J_{\max}\}$6, sets $k\in J=\{1,2,\dots,J_{\max}\}$7, and selects exactly one frame per segment: $k\in J=\{1,2,\dots,J_{\max}\}$8 The authors report that sampling $k\in J=\{1,2,\dots,J_{\max}\}$9–$a_t$00 frames per video clip preserves enough temporal information for anti-spoofing while reducing computation by $a_t$01 compared to processing every frame (Muhammad et al., 2023). DenseNet-201 extracts a 1,920-dimensional feature vector from each selected frame, and three recurrent sub-models—LSTM, BiLSTM, and GRU—are stacked by a meta-model (Muhammad et al., 2023). In cross-dataset testing, the meta-model reports HTERs of $a_t$02 on MSU-MFSD, $a_t$03 on Replay-Attack, and $a_t$04 on OULU-NPU (Muhammad et al., 2023).

Random Temporal Skipping (RTS) addresses multirate videos by randomizing inter-frame strides during training (Zhu et al., 2018). For a clip of $a_t$05 frames, one draws $a_t$06 and samples

$a_t$07

With $a_t$08 and $a_t$09, the clip can span up to $a_t$10 original frames (Zhu et al., 2018). On UCF101, when test clips are sampled with random strides in $a_t$11, accuracy is $a_t$12 without RTS and $a_t$13 with RTS; on the same benchmark, the full RTS two-stream model reaches $a_t$14, and on HMDB51 it reaches $a_t$15 (Zhu et al., 2018).

In speech recognition, blank-regularized CTC exploits the blank symbol to remove redundant frames in a neural Transducer (Yang et al., 2023). The standard CTC objective sums over alignments $a_t$16, and the paper introduces two regularizers on non-blank self-loops. The soft restriction adds a penalty $a_t$17 to

$a_t$18

producing

$a_t$19

A hard restriction instead limits the maximum run length of identical non-blank labels (Yang et al., 2023). On LibriSpeech, the baseline Transducer has WER $a_t$20 / $a_t$21, FRR $a_t$22, and RTF $a_t$23; the soft $a_t$24 model attains WER $a_t$25 / $a_t$26, FRR $a_t$27, and RTF $a_t$28, which is $a_t$29 faster (Yang et al., 2023).

Temporal knowledge graph completion supplies a snapshot-level analogue. Re-Temp inserts a relation-aware skip information flow after each timestamp representation (Wang et al., 2023). After CompGCN computes $a_t$30, the model averages the relations incident on the query entity at the target timestamp,

$a_t$31

constructs attention scores over the previous $a_t$32 inputs, and forms

$a_t$33

When $a_t$34 is small, the model effectively skips snapshot $a_t$35 (Wang et al., 2023). Reported MRR improvements include $a_t$36 versus $a_t$37 on ICEWS14, $a_t$38 versus $a_t$39 on ICEWS05-15, and $a_t$40 versus $a_t$41 on GDELT (Wang et al., 2023).

A separate architectural variant appears in TSkips for spiking neural networks (Malettira et al., 2024). Standard feedforward SNNs connect layer $a_t$42 at time $a_t$43 to layer $a_t$44 at the same time $a_t$45; TSkips adds explicit multi-step temporal delays $a_t$46 on both forward and backward skip connections: $a_t$47 and, with trainable mixing,

$a_t$48

A training-free NAS procedure, NASWOT-SAHD, ranks candidate architectures by spike-pattern diversity at initialization (Malettira et al., 2024). Reported gains include AEE $a_t$49 on DSEC-flow for the base SNN, $a_t$50 top-1 accuracy on DVS128 Gesture for F+B TSkips, and $a_t$51 on SSC for the 4-layer model with backward TSkip (Malettira et al., 2024).

5. Temporal skipping as systems optimization: convolution, embeddings, updates, and mobility

Some temporal-skipping methods operate at the systems level rather than the task-policy level. Skip-Convolutions reformulates video convolutions on residual frames $a_t$52, using the identity

$a_t$53

A binary gate $a_t$54 then decides whether residual computation is needed at each spatial location: $a_t$55 Gates may be norm-based, learned with Gumbel-Softmax, or structured block-wise for hardware efficiency (Habibian et al., 2021). On UA-DETRAC with EfficientDet D0–D3, Skip-Conv reduces compute from $a_t$56–$a_t$57 GMAC to $a_t$58–$a_t$59 GMAC with equal or slightly better AP; on JHMDB with HRNet-w32, it reduces $a_t$60 GMAC to $a_t$61 GMAC while increasing PCK from $a_t$62 to $a_t$63 (Habibian et al., 2021). On Intel Xeon CPU, HRNet-w32 conv layers take $a_t$64 ms/frame, Skip-Conv achieves $a_t$65 ms, and up to $a_t$66 ms with additional compression (Habibian et al., 2021).

Slipstream applies temporal skipping to recommender-model training by dynamically skipping stale embeddings (Maboud et al., 2024). For hot embeddings $a_t$67, periodic snapshots $a_t$68 are compared via

$a_t$69

If $a_t$70, row $a_t$71 is deemed stale; batches that touch only stale hot embeddings can be skipped (Maboud et al., 2024). Reported end-to-end speedups for Slipstream over 4-GPU XDL are $a_t$72 on Alibaba, $a_t$73 on Criteo-Kaggle, $a_t$74 on Criteo-Terabyte, and $a_t$75 on Avazu, with sampling and classification overhead under $a_t$76 of training time for large models (Maboud et al., 2024).

In Age of Information scheduling, temporal skipping appears as a skip-or-switch decision over packet arrivals (Wang et al., 2018). Updates arrive according to a Bernoulli process $a_t$77, transmission takes exactly $a_t$78 slots, and no buffer is available. When a new update arrives during service, the source chooses $a_t$79: switch to the new arrival or skip it (Wang et al., 2018). The optimal policy is shown to be a renewal policy with a sequential switching property and a multiple-threshold structure $a_t$80 (Wang et al., 2018). With $a_t$81 and $a_t$82, one finds $a_t$83, and for $a_t$84 one never switches (Wang et al., 2018). The multi-threshold policy typically reduces the time-average AoI by $a_t$85–$a_t$86 compared to both never-switch-until-completion and always-switch-to-the-freshest rules (Wang et al., 2018).

Wireless mobility management yields an analogous trade-off. Periodic handover skipping prohibits handovers for a period $a_t$87, then allows one HO at the end of the block (Tokuyama et al., 2023). Time-based handover skipping uses a fixed skipping time $a_t$88 during which all HOs are skipped (Tokuyama et al., 2020). Both lines derive analytical expressions for HO rate and expected data rate via stochastic geometry, and both report that skipping can outperform no skipping particularly when the UE moves fast (Tokuyama et al., 2023, Tokuyama et al., 2020). In the periodic setting, a utility $a_t$89 can have an optimal skipping period; in the time-based setting, there is a unique optimal skipping time maximizing transmission performance approximately (Tokuyama et al., 2023, Tokuyama et al., 2020).

These systems-oriented papers emphasize that temporal skipping need not skip semantic events or predictions; it can skip only computation, communication, or protocol transitions.

6. Empirical regularities, design trade-offs, and limitations

Several regularities recur across the literature. First, skipping usually helps when adjacent temporal states are redundant or when only a subset of temporal locations carry task-relevant information. TempoRL is designed for environments that need various degrees of fine and coarse control (Biedenkapp et al., 2021). ASI focuses on “easy-to-predict transitions” and on skipping over “inconsequential chaos” (Neitz et al., 2018). S-RNN is motivated by strong short-term correlations from near-duplicate frames (Sigurdsson et al., 2016). Frame-skipping anti-spoofing retains coarse temporal motion cues while reducing computational cost (Muhammad et al., 2023). Blank-regularized CTC exploits the fact that many frames should emit blank symbols (Yang et al., 2023).

Second, performance gains are typically non-monotonic in the skip parameter. TempoRL reports that larger $a_t$90 speeds up learning up to a point, with $a_t$91–$a_t$92 working well, while too many redundant skip-lengths slow early learning (Biedenkapp et al., 2021). TSkips reports that larger $a_t$93 such as $a_t$94–$a_t$95 consistently yields higher accuracy, but too small or too large delays degrade performance (Malettira et al., 2024). RTS shows gains that saturate beyond $a_t$96 (Zhu et al., 2018). In wireless handover skipping, the utility metric exhibits an optimal skip period or skipping time (Tokuyama et al., 2023, Tokuyama et al., 2020).

Third, many papers distinguish reactive skipping from structured, context-conditioned skipping. TempoRL states that, unlike DAR/FiGAR, it conditions skip-length on the chosen action and the current state, learning per-action, per-state skip behavior rather than a global average (Biedenkapp et al., 2021). Re-Temp reports that removing relation awareness causes a large MRR drop, from $a_t$97 to $a_t$98 on ICEWS14, and removing skip flow causes a drop to $a_t$99 (Wang et al., 2023). SAB relies on learned attention to a small set of reminders rather than uniform truncation (Ke et al., 2018).

Limitations also recur. Skip-Convolutions note that large camera motion invalidates the simple frame-difference prior because residual sparsity decreases (Habibian et al., 2021). ASI warns that greedy temporal matching can lose global alignment and that all experiments were in fully observable deterministic simulators with pixel-space losses (Neitz et al., 2018). Skip RNN notes that the straight-through estimator is biased and that extreme skipping budgets can cause the model to miss critical inputs (Campos et al., 2017). Blank-regularized CTC depends on well-calibrated CTC blank probabilities and requires a warmup before skipping (Yang et al., 2023). Re-Temp reports that on WIKI, simple one-step history can slightly outperform skip flow because $u_t\in\{0,1\}$00 of WIKI facts simply repeat from the previous year (Wang et al., 2023). These results indicate that temporal skipping is most effective when redundancy is genuine and harmful, not when repeated observations themselves are predictive.

A final misconception is that temporal skipping necessarily reduces accuracy. The reported results are mixed but frequently favorable: TempoRL learns successful policies up to an order of magnitude faster than vanilla Q-learning (Biedenkapp et al., 2021); Skip-Conv reports $u_t\in\{0,1\}$01–$u_t\in\{0,1\}$02 compute savings with no drop in accuracy (Habibian et al., 2021); blank-regularized CTC attains $u_t\in\{0,1\}$03 faster inference without sacrificing performance (Yang et al., 2023); and TSkips reports up to $u_t\in\{0,1\}$04 reduction in AEE and up to $u_t\in\{0,1\}$05 points higher classification accuracy depending on the dataset (Malettira et al., 2024). This suggests that, in many settings, temporal skipping is not merely a compression heuristic but a bias toward the temporal scales that matter for the objective.