OrdShap: Order-Aware Feature Attribution

• OrdShap is a framework for post-hoc explainability that separates a feature’s value from its positional impact in sequential models.
• It quantifies the marginal effect of permuting feature positions using a rigorous, game-theoretic foundation and sampling-based approximations.
• Empirical validation in healthcare, NLP, and synthetic data shows that OrdShap provides more nuanced insights than traditional, value-only attributions.

OrdShap is a post-hoc feature attribution framework designed to elicit position-sensitive explanations for sequential black-box models. Unlike prior attribution methods that assume fixed feature ordering, OrdShap quantifies the marginal effect of permuting feature positions, thereby separating the influence of feature value ("what is present") and feature position ("where it occurs") in the input sequence. The approach is rooted in an axiomatic, game-theoretic foundation, providing both theoretical guarantees and practical algorithms for order-aware model explanation, with demonstrated empirical benefits across domains such as healthcare, NLP, and synthetic data (Hill et al., 16 Jul 2025).

1. Motivation and Problem Formulation

Feature attribution in sequential architectures—including Transformers, RNNs, and their variants—traditionally conflates two sources of importance: the magnitude or identity of a value and its position in the sequence. Most post-hoc techniques (e.g., KernelSHAP, LIME, gradient-based saliency) perturb only feature values under a fixed index, treating the position as invariant. This assumption is violated in domains where shuffling elements induces significant prediction changes. For example, reordering medical measurements or text tokens may flip the prediction of a clinical outcome or sentiment, respectively.

OrdShap seeks to decompose the overall feature attribution into:

• Value Importance (VI): The marginal effect on prediction of injecting a feature value, averaged over possible positions.
• Position Importance (PI): The marginal effect on prediction of altering a feature's position, holding the value fixed.

This separation supplies diagnosis of whether a model's reliance on a sequence element arises due to its content, its location, or both.

2. Mathematical Definitions and Formalism

OrdShap models sequential attribution as a cooperative game on ordered coalitions. Let $x = (x_1, ..., x_d)$ be a sequence, $f(x)$ the model output, $N = \{1, ..., d\}$ the feature set, and $\mathfrak{S}_N$ the group of permutations. The central ingredient is a permutation-aware characteristic function:

$$\omega_{f,x}(S,\sigma) = \mathbb{E}_{x' \sim \mathcal{X}} \left[ f(\text{permute}(x',\sigma)) \mid x'_{\sigma(i)} = x_i \; \forall\, i \in S \right]$$

where “permute(x’,σ)” reorders coordinates according to σ, and features not in S are ablated (typically via sampling from a reference distribution).

The OrdShap value $\gamma_{i, \ell}$ assigns to each input coordinate-position pair the marginal expected effect, averaged over all subsets and permutations where feature i occupies position $\ell$:

$$\gamma_{i,\ell}(N,\omega) = \sum_{\substack{S \subseteq N,\, i \in S}} \sum_{\substack{\sigma \in \mathfrak{S}_N\\sigma^{-1}(i) = \ell}} \frac{(|S|-1)!\,(d-|S|)!}{(d-1)!\,d!} \left[ \omega(S, \sigma) - \omega(S \setminus \{i\}, \sigma) \right]$$

This construction produces a $d \times d$ matrix of attributions. Aggregating $\gamma_{i,\ell}$ yields summary measures:

• OrdShap-VI ($f(x)$0): Feature i's average effect across all positions,

$f(x)$1

• OrdShap-PI ($f(x)$2): The signed slope from regressing attributions on position,

$f(x)$3

where $f(x)$4 indicates growing importance in later positions.

3. Axiomatic Foundation: Sánchez-Bergantiños (SB) Value

The classical Shapley value applies axiomatic fairness to unordered feature sets. Sánchez-Bergantiños (1997) generalizes this to ordered coalitions, defining an SB value sensitive to insertion order:

$f(x)$5

where $f(x)$6 inserts i into position $f(x)$7 in $f(x)$8.

A critical result (Theorem 3.1 of (Hill et al., 16 Jul 2025)) establishes that the average OrdShap value for each feature ($f(x)$9) coincides with the SB value $N = \{1, ..., d\}$0 for an appropriately symmetrized game $N = \{1, ..., d\}$1. This connects OrdShap to a unique, order-aware extension of Shapley values that satisfies efficiency, symmetry, null-player, and additivity axioms in the context of input sequences.

4. Approximation Algorithms and Computational Considerations

Exact OrdShap computation is intractable for moderate $N = \{1, ..., d\}$2, requiring $N = \{1, ..., d\}$3 model evaluations. Two model-agnostic estimators are proposed:

Sampling-Based Approximation

For each feature-position pair $N = \{1, ..., d\}$4, random subsets of features $N = \{1, ..., d\}$5 are sampled, along with permutations placing $N = \{1, ..., d\}$6 at location $N = \{1, ..., d\}$7. The corresponding marginal contributions are evaluated and averaged, using Shapley-style combinatorial weights.

• Complexity: $N = \{1, ..., d\}$8, where $N = \{1, ..., d\}$9 is the number of sampled subsets, $\mathfrak{S}_N$0 the permutation samples, and $\mathfrak{S}_N$1 the cost of a model call.

Least-Squares Estimation for VI and PI

A more efficient estimator solves for the $\mathfrak{S}_N$2 parameters ($\mathfrak{S}_N$3, $\mathfrak{S}_N$4) by regressing model response (relative to a baseline) against presence and positional offset of features over sampled sequences:

$\mathfrak{S}_N$5

The optimal coefficients are obtained by solving the corresponding normal equations on weighted sampled data.

• Complexity: $\mathfrak{S}_N$6, practical for low hundreds of features with thousands of samples.

5. Experimental Validation

Empirical results demonstrate OrdShap's practical relevance and superiority over traditional value-only attributions.

Datasets

• Health/EHR: MIMIC-III and eICU clinical cohorts, using BERT-based models for mortality and length-of-stay prediction.
• NLP: IMDB reviews with sentence-level DistilBERT sentiment analysis.
• Synthetic: Controlled 10-token sequences across two regimes: a purely value-driven linear model and a position-sensitive nonlinear model.

Evaluation and Baselines

Metric	Objective	Observation
Position Importance test	PI's effect via permutation	OrdShap-PI maintains/increases (AUC > 0.50), others degrade
Value Importance test (IncAUC, ExcAUC)	VI with/without random reordering	OrdShap-VI achieves high IncAUC on all models, competitive ExcAUC

Comparative methods: KernelSHAP, LIME, LOCO, Integrated Gradients, DeepLIFT, and Random.

Results

• OrdShap-PI outperforms all baselines in PI-based permutation tests and separates token types in the synthetic regime, whereas conventional attributions intermingle types.
• OrdShap-VI reliably yields highest inclusion AUC in value tests.
• Qualitative case studies in clinical settings show OrdShap-VI and PI distinguish value-driven from position-driven features (e.g., potassium chloride's early vs. bedside glucose's late importance), whereas value-only methods confound the two.

6. Insights, Limitations, and Future Perspectives

Key Insights

• OrdShap is the first post-hoc explainer to robustly disentangle value and position effects in sequential models.
• The approach generalizes established game-theoretic attributions (SB value) to capture ordering.
• The two-dimensional attribution (VI and PI) enables nuanced diagnosis of model reliance on content versus temporal location.

Limitations

• Exact solution complexity is prohibitive (factorial in sequence length), necessitating approximation with many model queries.
• Assumes arbitrary permutation is permissible; does not immediately transfer to irregular series or constrained domains.

Future Directions

• Leverage stratified sampling and low-rank approximations for scalability.
• Extend framework to continuous or irregular position spaces in time-series.
• Integrate with causal analysis to reflect structural constraints such as non-leakage in temporal prediction.
• Explore grouped feature attributions and their interplay with positional effects.
• Conduct user-facing evaluations for interpretability in clinical and financial applications.

OrdShap furnishes a principled, axiomatically justified, and empirically validated mechanism for order-sensitive attributions, elucidating both content and temporal effects in modern sequential modeling (Hill et al., 16 Jul 2025).