TimeSHAP: Explaining Sequence Models

• TimeSHAP is a model-agnostic explainer that extends KernelSHAP to sequential data by assigning Shapley-value attributions to features across timesteps.
• It employs sequence perturbations and a temporal coalition pruning strategy to reduce computation and variance in explanations for long histories.
• Empirical results demonstrate that TimeSHAP delivers actionable insights at feature, event, and cell levels, aiding applications like fraud detection.

to=arxiv_search 彩神争霸平台json {"query":"TimeSHAP explaining recurrent models through sequence perturbations", "max_results": 10} to=search_arxiv 】【。】【”】【json {"query":"TimeSHAP explaining recurrent models through sequence perturbations", "max_results": 10} TimeSHAP is a post-hoc, model-agnostic explainer for recurrent and other sequence models that extends KernelSHAP to the sequential domain by assigning Shapley-value attributions to temporally structured inputs rather than to a single static feature vector. In its original formulation, it produces importance scores at three granularities—features across a sequence, timesteps or events, and individual feature-at-timestep cells—and introduces a pruning procedure for long histories so that explanations remain tractable and less variable (Bento et al., 2020). Subsequent literature has largely treated TimeSHAP as a local, perturbation-based method for explaining a particular prediction of a recurrent time-series model, in contrast to later methods that target global temporal dependence or multivariate-temporal structure (Tunyi, 25 Jun 2026).

1. Formal setting and explanatory objective

TimeSHAP is designed for settings in which a model prediction depends on an ordered history rather than on an isolated input. The original paper denotes a sequence as $x=(x_1,\dots,x_T)$, with per-timestep feature vectors $x_t \in \mathbb{R}^F$, or equivalently as a matrix $X \in \mathbb{R}^{d \times l}$, where $d$ is the number of features and $l$ the number of events. A predictive model $f$ consumes the whole sequence and returns a score $f(X)$ (Bento et al., 2020).

The explanatory goal is to distribute the prediction across structurally meaningful units of the sequence. At the feature level, TimeSHAP produces $\{\phi_f : f=1,\dots,F\}$, which summarize the contribution of each feature across the whole history. At the timestep or event level, it produces $\{\phi_t : t=1,\dots,T\}$, which quantify the contribution of each event. At the cell level, it produces $\{\phi_{t,f}\}$, isolating the contribution of a particular feature at a particular timestep. Conceptually, event-level and feature-level scores can be related through aggregation such as $x_t \in \mathbb{R}^F$0 and $x_t \in \mathbb{R}^F$1, although in the original method feature- and event-level attributions are computed directly rather than recovered only from cell scores (Bento et al., 2020).

This formulation distinguishes TimeSHAP from standard instance-centric SHAP usage. Rather than explaining only the most recent input, it aims to explain the full sequence dependence of a recurrent predictor, including effects mediated by hidden state. Later papers consistently describe the method as a Shapley-value extension for recurrent models that assigns importance to timesteps and variables for a specific forecast (Tunyi, 25 Jun 2026).

2. KernelSHAP extension to sequential perturbations

TimeSHAP inherits the additive surrogate view of KernelSHAP. For a coalition vector $x_t \in \mathbb{R}^F$2, where the “players” may be features, timesteps, or grouped cells, it fits a local linear explainer

$x_t \in \mathbb{R}^F$3

with $x_t \in \mathbb{R}^F$4 as the base score and $x_t \in \mathbb{R}^F$5 as the attribution for unit $x_t \in \mathbb{R}^F$6. The weighting kernel is the standard KernelSHAP kernel,

$x_t \in \mathbb{R}^F$7

and the surrogate is fit by minimizing the weighted loss

$x_t \in \mathbb{R}^F$8

The Shapley value itself retains the canonical form

$x_t \in \mathbb{R}^F$9

The distinctive element is the perturbation map $X \in \mathbb{R}^{d \times l}$0, which must construct valid sequence-shaped inputs for a recurrent model (Bento et al., 2020).

TimeSHAP defines a background matrix $X \in \mathbb{R}^{d \times l}$1 by repeating per-feature averages across time. For feature-axis perturbations, the coalition acts on rows globally over all events:

$X \in \mathbb{R}^{d \times l}$2

Here, $X \in \mathbb{R}^{d \times l}$3 masks feature $X \in \mathbb{R}^{d \times l}$4 at every timestep by replacing its row with the corresponding background row. For event-axis perturbations, the coalition acts on columns globally over all features:

$X \in \mathbb{R}^{d \times l}$5

In this representation, $X \in \mathbb{R}^{d \times l}$6 masks the entire event $X \in \mathbb{R}^{d \times l}$7 by replacing its column with the background column. Operationally, the full sequence length and order are preserved because the perturbation replaces values rather than deleting positions (Bento et al., 2020).

This construction gives TimeSHAP its recurrent focus. It does not attempt to explain the hidden state directly; instead, it perturbs the observed sequence, reruns the model, and uses the resulting output differences to estimate Shapley contributions. Later application papers preserve this general logic even when adding practical modifications such as Monte Carlo permutation sampling, specialized background sets, or attention-guided prioritization (Zheng et al., 30 Jul 2025).

3. Attribution granularities and grouped cell explanations

Feature-level and event-level explanations are the most direct uses of TimeSHAP. Feature-level explanations fit the surrogate over coalitions of rows, so the players are features across the whole sequence. Event-level explanations fit the surrogate over coalitions of columns, so the players are timesteps or events. Both satisfy the usual SHAP additivity property: the sum of the attributions equals $X \in \mathbb{R}^{d \times l}$8 (Bento et al., 2020).

Direct cell-level Shapley attribution over all $X \in \mathbb{R}^{d \times l}$9 feature–timestep cells is computationally infeasible because it would require a coalition space of size $d$0. TimeSHAP therefore introduces a grouping strategy that isolates highly relevant cells while aggregating less relevant or older regions. The procedure first uses temporal coalition pruning to identify a pruned prefix $d$1 of older events. It then uses feature- and event-level attributions to identify sets of relevant events $d$2 and relevant features $d$3, based on whether absolute attribution exceeds a user threshold $d$4. From these, it defines the cell intersections

$d$5

together with complementary groups

$d$6

and

$d$7

where $d$8 is the set of all cells. The coalition universe is then

$d$9

with

$l$0

groups instead of $l$1 individual cells. Grouped-cell perturbations use a coalition matrix $l$2 and the masked input

$l$3

where $l$4 is the all-ones matrix. This preserves exact cell resolution only where relevance justifies it, while keeping the optimization manageable (Bento et al., 2020).

The grouped-cell design is one of the reasons TimeSHAP occupies an intermediate position in the sequence-explanation literature. It is finer than purely event-level methods when local detail matters, yet it remains fundamentally organized around perturbation coalitions rather than an explicit probabilistic model of temporal dependence.

4. Temporal coalition pruning and computational behavior

The pruning procedure is central to TimeSHAP’s practical use on long histories. The algorithm scans the sequence backward and repeatedly collapses the input into two “features”: a prefix $l$5 and the remaining suffix $l$6. For each split, it evaluates the full set of $l$7 coalitions $l$8 under the event-axis perturbation and obtains two Shapley weights $l$9. If $f$0, where $f$1 is a tolerance, the prefix is deemed safely aggregable and the algorithm returns $f$2 as the pruning index. If no such $f$3 is found, no prefix is pruned (Bento et al., 2020).

The scan itself costs $f$4 calls, each over $f$5 coalitions, hence $f$6 for the pruning stage. If the pruning index is $f$7, event-level Shapley estimation is reduced from $f$8 to $f$9; for recurrent models whose forward pass scales linearly with sequence length, the total runtime becomes $f(X)$0. The original paper further reports that pruning lowers variance in addition to runtime, because the effective coalition space is smaller and less noisy (Bento et al., 2020).

The empirical case study used $f(X)$1 coalition samples, a pruning tolerance $f(X)$2, and a cell-level relevance threshold $f(X)$3. On 1,000 positive fraud sequences, the average length dropped from 182.1 to 32.9 with $f(X)$4, and to 19.7 with $f(X)$5; the relative standard deviation of Shapley values across runs dropped from 1.71 on the original sequences to approximately 0.98 at $f(X)$6 and approximately 0.68 at $f(X)$7. The fraction of sequences short enough for exact Shapley feasibility, using the paper’s $f(X)$8 rule of thumb, rose from 10.0% without pruning to 58.3% at $f(X)$9 and 78.8% at $\{\phi_f : f=1,\dots,F\}$0 (Bento et al., 2020).

These results are methodologically important because they show that TimeSHAP is not merely a direct transplantation of KernelSHAP to sequence data. Its practicality depends on a sequence-specific compression heuristic that trades temporal granularity on older events for computational tractability and lower attribution variance.

5. Empirical interpretation and domain use

The original TimeSHAP paper evaluates the method on a bank account takeover fraud detection model consisting of embeddings for categorical variables, a GRU layer, and a feed-forward classifier, trained on approximately 20 million events of three types—login, transaction, and enrollment. The predictive model achieved recall 84.3% at 1% false positive rate on validation and recall 79.9% at 0.89% false positive rate on test. Explanations were generated for the first positive prediction in each sequence (Bento et al., 2020).

Four empirical findings are emphasized. First, the model’s important features and events aligned with cues known to fraud analysts, including an enrollment–login–transaction pattern. Second, positive sequences could often be pruned to roughly 10% of their original length, indicating that older events frequently had residual attribution. Third, the most recent event contributed on average only 41% of the model’s score, while preceding events accounted for 59%, which directly illustrates sequential dependence rather than purely recency-based behavior. Fourth, client age received notably high attribution; a later audit confirmed higher false positive rates for older clients, so TimeSHAP functioned not only as a mechanistic explainer but also as a fairness diagnostic (Bento et al., 2020).

The paper also reports aggregate attribution patterns across positive sequences. The current transaction at $\{\phi_f : f=1,\dots,F\}$1 had the highest average attribution, approximately 0.28, but events from $\{\phi_f : f=1,\dots,F\}$2 to $\{\phi_f : f=1,\dots,F\}$3 still contributed materially, with average values approximately 0.03 to 0.13, and a long tail of distant events occasionally remained influential. At the feature level, major positive contributors included transaction type at approximately 0.29, event type at approximately 0.092, client age at approximately 0.090, and several IP and location features in the approximate range 0.03 to 0.08; some authentication and security-related features contributed negatively, consistent with lowering risk (Bento et al., 2020).

Later application papers adapted the same explanatory template to other sequential tasks. In a multi-hazard agricultural early warning system, TimeSHAP was combined with attention-based BiLSTM models to generate featural-temporal attributions $\{\phi_f : f=1,\dots,F\}$4, with temporal aggregation $\{\phi_f : f=1,\dots,F\}$5 and feature aggregation $\{\phi_f : f=1,\dots,F\}$6. The implementation used mean imputation in standardized feature space, background sequences from the same region, Monte Carlo permutation sampling, and attention-guided pruning; the authors describe these as their TimeSHAP configuration rather than as theoretical requirements. The resulting heatmaps highlighted, for example, TMIN and TMAX before frost alerts, precipitation and soil-moisture proxies before flood alerts, and sustained TMAX with suppressed PRCP before heat events (Zheng et al., 30 Jul 2025).

A trip prediction study similarly used TimeSHAP to explain LSTM and attention-based LSTM forecasts of next-trip time and distance. There, the explainer was applied at trip level or feature level through the local surrogate $\{\phi_f : f=1,\dots,F\}$7, with masking details delegated to Bento et al. The best parallel attention-based LSTM achieved a 3.99% prediction error, reported as 23.89% better than the baseline LSTM, and TimeSHAP visualizations qualitatively showed that the stronger model assigned high positive contributions to historically relevant long-distance trips in cases where the plain LSTM did not (Balouji et al., 2023).

6. Critiques, methodological relatives, and later developments

Later work has scrutinized both the assumptions and the scope of TimeSHAP. The most direct critique appears in the KARMA paper, which argues that perturbation-based methods such as TimeSHAP struggle with temporal autocorrelation because random perturbations break sequential structure and induce off-manifold inputs. In that framing, SHAP-style baseline marginalization is structurally incoherent for time series because $\{\phi_f : f=1,\dots,F\}$8 is not independent of $\{\phi_f : f=1,\dots,F\}$9; TimeSHAP therefore inherits the baseline selection problem, provides no certified attributions, and remains local rather than yielding a global model of learned temporal dependence. On synthetic VAR data with known causal edges, KARMA was tied with TimeSHAP on tiny and small graphs but outperformed it as graph density grew, with Kendall’s $\{\phi_t : t=1,\dots,T\}$0 rising to 0.90, 0.92, and 0.77 on medium, large, and xlarge cases respectively, versus 0.79, 0.68, and 0.64 for TimeSHAP (Tunyi, 25 Jun 2026).

A different criticism concerns the choice of explanatory units. GroupSegment-SHAP positions TimeSHAP as a method whose players are individual timesteps and whose perturbations mask or restore complete time points. Its argument is that such time-step players decouple the feature and time axes, so multivariate-temporal structure is fragmented across isolated steps rather than represented as coherent cross-variable regimes. Under matched perturbation budgets, that paper reports that GS-SHAP improves deletion-based faithfulness, measured by $\{\phi_t : t=1,\dots,T\}$1AUC, by about 1.7x on average over time-series SHAP baselines and reduces wall-clock runtime by about 40 percent on average; TimeSHAP is described there as generally the slowest of the compared explainers (Kim et al., 3 Jan 2026).

A related line of work studies how SHAP-style temporal grouping affects explanation quality. An empirical evaluation of segmentation strategies for time-series classification does not modify TimeSHAP itself, but it addresses a core design choice shared by TimeSHAP-like workflows: grouping consecutive time points into explanatory units. Across both univariate and multivariate tasks, that study finds that the number of segments has a greater impact on explanation quality than the specific segmentation algorithm, that equal-length segmentation consistently outperforms most custom segmentation methods, and that length-aware normalization improves attribution quality (Serramazza et al., 3 Sep 2025). This suggests that some practical difficulties attributed to “time-series SHAP” may arise as much from player construction as from Shapley estimation.

Other sequence-SHAP variants pursue different remedies. A 2022 paper explicitly describes TimeSHAP as the closest related work and introduces VARSHAP, closed-form SHAP values for linear time-series models such as AR, MA, ARMA, and VARMAX, and Time Consistent SHAP. The latter reduces coalition complexity from $\{\phi_t : t=1,\dots,T\}$2 to $\{\phi_t : t=1,\dots,T\}$3 by explaining lagged subgames and imposing a time-consistency axiom, thereby addressing scalability and temporal stability from a different angle than TimeSHAP’s pruning heuristic (Villani et al., 2022).

Taken together, these developments place TimeSHAP in a well-defined methodological niche. It is a sequence-aware SHAP explainer whose core contribution is to make recurrent-model explanations operational at feature, event, and selected cell resolution through sequence perturbations and pruning. Later literature has preserved its relevance as a baseline and practical tool, while also identifying limitations in baseline design, causal faithfulness, global interpretability, and multivariate-temporal player construction.