Hi-Fi: High-Order Feature Interaction Decomposition

• The paper introduces Hi-Fi, which extends LOCO by decomposing a feature’s predictive role into unique, redundant, and synergistic contributions.
• It uses an adaptive, optimization-based subset search to quantify higher-order interactions within model predictions.
• Applications range from GEANT simulations to health studies, underscoring its practical impact in revealing context-dependent feature interplay.

High-order Interactions for Feature Importance (Hi-Fi) is a regression-based feature-importance framework that extends Leave One Covariate Out (LOCO) so that a feature is not summarized by a single scalar relevance score. Instead, Hi-Fi decomposes a feature’s predictive role into a unique or two-body contribution, a redundant contribution, and a synergistic contribution. In this formulation, higher-order effects are quantified by examining how the predictive gain of a focal feature changes across subsets of companion variables, thereby separating standalone predictive utility from overlap and cooperative effects (Ontivero-Ortega et al., 2024, Ontivero-Ortega et al., 30 Jul 2025).

1. Conceptual basis

Hi-Fi was introduced to address a limitation of standard global feature-importance scores: a single value does not distinguish whether a feature is important because it uniquely predicts the target, because it duplicates information already present in other covariates, or because it participates in higher-order interactions that become useful only jointly. The framework is explicitly rooted in LOCO, but it imports the language of redundancy and synergy from the literature on multivariate interactions among random variables and then re-expresses those ideas in a predictability or regression setting rather than an information-theoretic one (Ontivero-Ortega et al., 2024).

In Hi-Fi, the object of interest is a driver feature $X$, a target $Y$, and a set of remaining inputs

$\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$

The central move is to replace a fixed conditioning set by a search over subsets $\mathbf z \subseteq \mathbf Z$. This makes feature importance explicitly context-dependent: the same feature can appear weak when conditioned on a redundant subset and strong when conditioned on a synergistic subset. The resulting decomposition is therefore feature-centric and subset-optimized, not a full combinatorial attribution over all interaction atoms (Ontivero-Ortega et al., 2024).

The framework is also careful about terminology. Its notions of redundancy and synergy are not exact PID atoms. They are defined through prediction-error reduction. This means that Hi-Fi is designed to characterize how a predictor uses a feature for a task, not to provide a universal information decomposition independent of model class or loss (Ontivero-Ortega et al., 2024).

2. Global predictability decomposition

The global Hi-Fi construction starts from standard LOCO: $$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$ where $\epsilon(\cdot)$ is the mean squared prediction error. The corresponding pairwise predictive power of $X$ alone is

$$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$

Hi-Fi then evaluates

$L_{\mathbf z}(X\to Y)$

over subsets $\mathbf z \subseteq \mathbf Z$ and defines a minimizing subset $Y$0 and a maximizing subset $Y$1. These two extremal subsets induce three quantities: $Y$2

$Y$3

$Y$4

The decomposition identity is

$Y$5

Under this interpretation, $Y$6 is the pure two-body influence of $Y$7 on $Y$8, $Y$9 is the part of $\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$0's apparent importance that can be explained away by overlap with other variables, and $\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$1 is the additional predictive utility that emerges only when $\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$2 is considered together with an appropriate subset of other variables. The framework also gives operational conditions: $\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$3 indicates redundancy, whereas

$\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$4

indicates synergy (Ontivero-Ortega et al., 2024).

This decomposition is additive, but it is not a Möbius inversion over all subsets and not a complete order-by-order interaction tensor. It is instead an optimization-based decomposition. The maximizing and minimizing subsets both define the scalar scores and identify which companion features participate in the redundant or synergistic effect alongside the driver feature. That is one of Hi-Fi’s distinctive traits: it does not only say that a feature has synergy or redundancy, but also points to the subsets that create those effects (Ontivero-Ortega et al., 2024).

3. Estimation by adaptive subset search

Exact optimization over all subsets $\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$5 is exponential in the number of companion variables, so the practical Hi-Fi procedure uses a greedy strategy. The search first evaluates all single variables, then adds variables one at a time to the current subset, choosing the addition that most decreases LOCO for the minimizing subset or most increases LOCO for the maximizing subset. This is a forward-selection heuristic for approximating $\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$6 and $\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$7 (Ontivero-Ortega et al., 2024).

The stopping rule is significance-based and uses surrogates obtained by permuting the selected $\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$8 variable. The search terminates when the observed increase or decrease in $\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.$9 is compatible with what would be expected from adding a variable with the same individual statistical properties but otherwise uncoupled from $\mathbf z \subseteq \mathbf Z$0 and $\mathbf z \subseteq \mathbf Z$1. This prevents unbounded growth of the selected multiplets and turns Hi-Fi into an adaptive subset-construction procedure rather than an exhaustive one (Ontivero-Ortega et al., 2024).

The method is model-dependent because $\mathbf z \subseteq \mathbf Z$2 is model-dependent. In the reported work, the framework is discussed for linear regression and is applied mainly with the hypothesis space induced by the inhomogeneous polynomial kernel of degree 2, that is, linear regression over all monomials of degree $\mathbf z \subseteq \mathbf Z$3. The paper also notes that LOCO is nonnegative for a broad class of predictors whose risk minimizer is invariant to adding statistically independent variables, specifically mentioning linear models, hypothesis spaces induced by inhomogeneous polynomial kernels, and Gaussian kernel functions (Ontivero-Ortega et al., 2024).

This dependence on the hypothesis space is not incidental. If the predictor cannot represent the relevant nonlinear or interaction structure, synergy may not be detected. Conversely, a richer predictor can expose cooperative effects that are invisible under simpler regressors. A plausible implication is that Hi-Fi is best understood as a decomposition of model-relative predictive structure, not of an abstract population mechanism.

4. Local Hi-Fi and comparison with Shapley effect

A later development extends Hi-Fi from global scores to individual patterns. Let

$\mathbf z \subseteq \mathbf Z$4

and let $\mathbf z \subseteq \mathbf Z$5 and $\mathbf z \subseteq \mathbf Z$6 be fitted regression models using $\mathbf z \subseteq \mathbf Z$7 and $\mathbf z \subseteq \mathbf Z$8, respectively. The local LOCO for pattern $\mathbf z \subseteq \mathbf Z$9 is

$$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$0

The global LOCO is the empirical average: $$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$1

Local Hi-Fi defines

$$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$2

$$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$3

$$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$4

with ensemble averages

$$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$5

This yields a local analogue of the global decomposition in which each observation receives a unique, redundant, and synergistic score (Ontivero-Ortega et al., 30 Jul 2025).

The local formulation changes the explanatory role of Hi-Fi. Global Hi-Fi describes what a feature does on average across a dataset; local Hi-Fi describes how that feature functioned for a specific pattern. The paper explicitly notes that, unlike the global scores, local scores can be negative. A negative local score indicates that, for that particular observation, the feature worsened the prediction or was “mis-informative” (Ontivero-Ortega et al., 30 Jul 2025).

The same paper compares local Hi-Fi with the Shapley effect. The global Shapley effect is

$$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$6

and the local version is

$$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$7

Both are built from subset-conditioned predictive gains, but they differ in what they summarize. Shapley effect averages over all contexts into one number, whereas Hi-Fi focuses on three reference points: the empty set, the minimizing subset, and the maximizing subset. Consequently, Shapley effect is an all-subsets allocation rule, while Hi-Fi is an interaction-typing decomposition that separates unique, redundant, and synergistic roles (Ontivero-Ortega et al., 30 Jul 2025).

5. Applications and empirical interpretation

The original global formulation is illustrated by a toy example in which the target is

$$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$8

with inputs

$$L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),$$9

Using degree-2 polynomial-kernel regression, the framework identifies $\epsilon(\cdot)$0 and $\epsilon(\cdot)$1 as synergistic because of dependency, $\epsilon(\cdot)$2 and $\epsilon(\cdot)$3 as redundant, $\epsilon(\cdot)$4 as purely unique, and $\epsilon(\cdot)$5 and $\epsilon(\cdot)$6 as purely synergistic because of the nonlinear interaction term $\epsilon(\cdot)$7. This example is methodologically important because it shows that the framework can capture both dependence-driven and interaction-driven synergy (Ontivero-Ortega et al., 2024).

A large-scale scientific application uses GEANT-based simulation for proton/pion discrimination with

$\epsilon(\cdot)$8

samples and six z-scored inputs: velocity $\epsilon(\cdot)$9, momentum $X$0, scattering angle $X$1, number of emitted photoelectrons $X$2, inner detector response $X$3, and outer detector response $X$4. With the hypothesis space induced by the inhomogeneous polynomial kernel of degree 2, the decomposition finds that $X$5 has a large unique contribution, synergy with $X$6, and redundancy with $X$7; $X$8 has small unique contribution, synergy with $X$9, and redundancy with $$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$0 and $$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$1; $$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$2 is synergistic with $$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$3; and $$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$4 is almost irrelevant. The paper’s main domain interpretation is that the synergistic cooperation between $$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$5 and $$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$6 likely corresponds to particle mass, a discriminative factor not captured by either variable alone (Ontivero-Ortega et al., 2024).

The local extension applies Hi-Fi to a One-Health study of air pollutants and Alzheimer’s disease mortality. The dataset comprises 32 features for 107 Italian provinces over 5 years, giving $$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$7 patterns after pooling years. Global Hi-Fi finds that air-pollution variables are the most synergistic group, with $$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$8, $$L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).$$9, and $L_{\mathbf z}(X\to Y)$0 showing high synergy, while some “other pathologies” variables, especially circulatory mortality, are highly redundant. The local analysis then shows that Bergamo and Brescia stand out as provinces where pollutant influence is most synergistic, whereas some provinces such as Pescara, Piacenza, and Siracusa show higher local unique scores for $L_{\mathbf z}(X\to Y)$1, and Pescara also for $L_{\mathbf z}(X\to Y)$2 (Ontivero-Ortega et al., 30 Jul 2025).

These applications illustrate the intended interpretive use of Hi-Fi. A large unique score points to a feature that remains predictive even under the most adverse conditioning. A large redundancy score indicates overlap or substitutability with other variables. A large synergy score indicates that the feature’s predictive value is mostly cooperative and becomes visible only in appropriate contexts. In empirical work, this often alters the ranking produced by one-number importance measures because a feature can rank highly in total relevance yet be mostly redundant rather than uniquely or synergistically informative.

6. Extensions, neighboring methods, and limitations

Hi-Fi occupies a specific position within the broader literature on interaction-aware feature importance. It is feature-centric, because it decomposes the role of a focal variable; optimization-based, because it uses extremal subsets rather than a full subset lattice decomposition; and predictability-based, because its primitive object is prediction-error reduction. A closely related but distinct line is iLOCO, which defines interaction importance for a subset $L_{\mathbf z}(X\to Y)$3 by inclusion–exclusion: $L_{\mathbf z}(X\to Y)$4 with the pairwise case

$L_{\mathbf z}(X\to Y)$5

iLOCO is model-agnostic, is formulated directly through risk differences under refitting, provides distribution-free / assumption-light confidence intervals, and extends formally to higher-order subsets, whereas Hi-Fi decomposes the importance of a single feature into unique, redundant, and synergistic parts (Little et al., 10 Feb 2025).

A further development, Stochastic Hi-Fi, reframes the method as a post-hoc, retraining-free interventional decomposition. For a coalition $L_{\mathbf z}(X\to Y)$6, it defines the interventional coalition loss

$L_{\mathbf z}(X\to Y)$7

and the coalition-specific LOCO gain

$L_{\mathbf z}(X\to Y)$8

It then sets

$L_{\mathbf z}(X\to Y)$9

with

$\mathbf z \subseteq \mathbf Z$0

This later work also proves, on a minimal 3-way XOR structural causal model, that signed pairwise interaction scores fundamentally conflate uniqueness, redundancy, and synergy: faithful pair indices return zero per pair, while projective pair indices spread the third-order effect into pair scalars; by contrast, the LOCO decomposition returns $\mathbf z \subseteq \mathbf Z$1 for each active feature. Stochastic Hi-Fi further introduces coupled diamond sampling for strict variance reduction and gives uniform finite-vocabulary convergence results (Aghilar et al., 17 Jun 2026).

Hi-Fi should also be distinguished from intrinsic interaction-learning architectures in recommendation systems. For example, models such as FiiNet learn the importance of explicit multi-order feature combinations through model-internal attention or gating over crossed representations, and treat those learned weights as interaction saliency. That is closely related in theme but methodologically different: Hi-Fi is a decomposition of feature importance, whereas such architectures are predictive models whose importance signals are internal to the forward pass (Wang et al., 2024).

Several limitations recur across the Hi-Fi literature. The subset search is heuristic rather than exhaustive; the decomposition depends on the chosen regression model and therefore on the expressive capacity of the hypothesis space; the higher-order contribution is summarized through optimized subsets rather than decomposed into all $\mathbf z \subseteq \mathbf Z$2-way, $\mathbf z \subseteq \mathbf Z$3-way, and higher-order atoms; and the framework is not identical to PID. In the local and stochastic variants, interpretation also depends on the masking or background distribution and on the degree of subset coverage. These constraints do not negate the method’s central contribution, but they delimit its scope: Hi-Fi is best understood as a structured, interaction-aware decomposition of predictive feature importance, not as a universal theory of multivariate dependence (Ontivero-Ortega et al., 2024, Ontivero-Ortega et al., 30 Jul 2025, Aghilar et al., 17 Jun 2026).