---
title: 'Hi-Fi: High-Order Feature Interaction Decomposition'
url: https://www.emergentmind.com/topics/high-order-interactions-for-feature-importance-hi-fi
type: topic
---

# Hi-Fi: High-Order Feature Interaction Decomposition

High-order Interactions for Feature Importance (Hi-Fi) is a regression-based feature-importance framework that extends Leave One Covariate Out (LOCO) so that a feature is not summarized by a single scalar relevance score. Instead, Hi-Fi decomposes a feature’s predictive role into a **unique** or **two-body** contribution, a **redundant** contribution, and a **synergistic** contribution. In this formulation, higher-order effects are quantified by examining how the predictive gain of a focal feature changes across subsets of companion variables, thereby separating standalone predictive utility from overlap and cooperative effects [2412.09964][2508.00930].

## 1. Conceptual basis

Hi-Fi was introduced to address a limitation of standard global feature-importance scores: a single value does not distinguish whether a feature is important because it uniquely predicts the target, because it duplicates information already present in other covariates, or because it participates in higher-order interactions that become useful only jointly. The framework is explicitly rooted in LOCO, but it imports the language of redundancy and synergy from the literature on multivariate interactions among random variables and then re-expresses those ideas in a predictability or regression setting rather than an information-theoretic one [2412.09964].

In Hi-Fi, the object of interest is a **driver** feature \(X\), a target \(Y\), and a set of remaining inputs
\[
\mathbf{Z}=\{z_\alpha\}_{\alpha=1,\ldots,n}.
\]
The central move is to replace a fixed conditioning set by a search over subsets \(\mathbf z \subseteq \mathbf Z\). This makes feature importance explicitly context-dependent: the same feature can appear weak when conditioned on a redundant subset and strong when conditioned on a synergistic subset. The resulting decomposition is therefore **feature-centric** and **subset-optimized**, not a full combinatorial attribution over all interaction atoms [2412.09964].

The framework is also careful about terminology. Its notions of redundancy and synergy are not exact PID atoms. They are defined through prediction-error reduction. This means that Hi-Fi is designed to characterize how a predictor uses a feature for a task, not to provide a universal information decomposition independent of model class or loss [2412.09964].

## 2. Global predictability decomposition

The global Hi-Fi construction starts from standard LOCO:
\[
L_{\mathbf Z}(X\to Y)=\epsilon(Y\mid \mathbf Z)-\epsilon(Y\mid X,\mathbf Z),
\]
where \(\epsilon(\cdot)\) is the mean squared prediction error. The corresponding pairwise predictive power of \(X\) alone is
\[
L_{\emptyset}(X\to Y)= \sigma_Y^2-\epsilon(Y\mid X).
\]

Hi-Fi then evaluates
\[
L_{\mathbf z}(X\to Y)
\]
over subsets \(\mathbf z \subseteq \mathbf Z\) and defines a minimizing subset \(\mathbf z_{\min}\) and a maximizing subset \(\mathbf z_{\max}\). These two extremal subsets induce three quantities:
\[
U=L_{\mathbf z_{\min}}(X\to Y),
\]
\[
R=L_{\emptyset}(X\to Y)-L_{\mathbf z_{\min}}(X\to Y),
\]
\[
S=L_{\mathbf z_{\max}}(X\to Y)-L_{\emptyset}(X\to Y).
\]
The decomposition identity is
\[
L_{\mathbf z_{\max}}(X\to Y)=S+R+U.
\]

Under this interpretation, \(U\) is the **pure two-body influence** of \(X\) on \(Y\), \(R\) is the part of \(X\)'s apparent importance that can be explained away by overlap with other variables, and \(S\) is the additional predictive utility that emerges only when \(X\) is considered together with an appropriate subset of other variables. The framework also gives operational conditions:
\[
L_{\emptyset}(X\to Y) > L_{\mathbf Z}(X\to Y)
\]
indicates redundancy, whereas
\[
L_{\emptyset}(X\to Y) < L_{\mathbf Z}(X\to Y)
\]
indicates synergy [2412.09964].

This decomposition is additive, but it is not a Möbius inversion over all subsets and not a complete order-by-order interaction tensor. It is instead an optimization-based decomposition. The maximizing and minimizing subsets both define the scalar scores and identify which companion features participate in the redundant or synergistic effect alongside the driver feature. That is one of Hi-Fi’s distinctive traits: it does not only say that a feature has synergy or redundancy, but also points to the subsets that create those effects [2412.09964].

## 3. Estimation by adaptive subset search

Exact optimization over all subsets \(\mathbf z \subseteq \mathbf Z\) is exponential in the number of companion variables, so the practical Hi-Fi procedure uses a greedy strategy. The search first evaluates all single variables, then adds variables one at a time to the current subset, choosing the addition that most decreases LOCO for the minimizing subset or most increases LOCO for the maximizing subset. This is a forward-selection heuristic for approximating \(\mathbf z_{\min}\) and \(\mathbf z_{\max}\) [2412.09964].

The stopping rule is significance-based and uses surrogates obtained by permuting the selected \(z\) variable. The search terminates when the observed increase or decrease in \(L_{\mathbf z}(X\to Y)\) is compatible with what would be expected from adding a variable with the same individual statistical properties but otherwise uncoupled from \(X\) and \(Y\). This prevents unbounded growth of the selected multiplets and turns Hi-Fi into an adaptive subset-construction procedure rather than an exhaustive one [2412.09964].

The method is model-dependent because \(\epsilon(\cdot)\) is model-dependent. In the reported work, the framework is discussed for linear regression and is applied mainly with the hypothesis space induced by the **inhomogeneous polynomial kernel of degree 2**, that is, linear regression over all monomials of degree \(\le 2\). The paper also notes that LOCO is nonnegative for a broad class of predictors whose risk minimizer is invariant to adding statistically independent variables, specifically mentioning linear models, hypothesis spaces induced by inhomogeneous polynomial kernels, and Gaussian kernel functions [2412.09964].

This dependence on the hypothesis space is not incidental. If the predictor cannot represent the relevant nonlinear or interaction structure, synergy may not be detected. Conversely, a richer predictor can expose cooperative effects that are invisible under simpler regressors. A plausible implication is that Hi-Fi is best understood as a decomposition of **model-relative predictive structure**, not of an abstract population mechanism.

## 4. Local Hi-Fi and comparison with Shapley effect

A later development extends Hi-Fi from global scores to **individual patterns**. Let
\[
\mathcal{D}=\{x_k,\mathbf{Z}_k,y_k\}_{k=1}^N
\]
and let \(f_{\mathbf Z}\) and \(f_{x\mathbf Z}\) be fitted regression models using \(\mathbf Z\) and \((x,\mathbf Z)\), respectively. The local LOCO for pattern \(i\) is
\[
\mathcal{L}_{\mathbf{Z}_i}(x_i \to y_i)
=
\left(y_i-f_{\mathbf Z}(\mathbf Z_i)\right)^2
-
\left(y_i-f_{x\mathbf Z}(x_i,\mathbf Z_i)\right)^2.
\]
The global LOCO is the empirical average:
\[
L_{\mathbf Z}(x\to y)=\frac{1}{N}\sum_{i=1}^N \mathcal{L}_{\mathbf{Z}_i}(x_i\to y_i).
\]

Local Hi-Fi defines
\[
U_i=\mathcal{L}_{\mathbf z_{\min\mid i}}(x_i\to y_i),
\]
\[
R_i=\mathcal{L}_{\emptyset}(x_i\to y_i)-\mathcal{L}_{\mathbf z_{\min\mid i}}(x_i\to y_i),
\]
\[
S_i=\mathcal{L}_{\mathbf z_{\max\mid i}}(x_i\to y_i)-\mathcal{L}_{\emptyset}(x_i\to y_i),
\]
with ensemble averages
\[
\frac{1}{N}\sum_{i=1}^N U_i=U,\qquad
\frac{1}{N}\sum_{i=1}^N R_i=R,\qquad
\frac{1}{N}\sum_{i=1}^N S_i=S.
\]
This yields a local analogue of the global decomposition in which each observation receives a unique, redundant, and synergistic score [2508.00930].

The local formulation changes the explanatory role of Hi-Fi. Global Hi-Fi describes what a feature does on average across a dataset; local Hi-Fi describes how that feature functioned for a specific pattern. The paper explicitly notes that, unlike the global scores, local scores can be negative. A negative local score indicates that, for that particular observation, the feature worsened the prediction or was “mis-informative” [2508.00930].

The same paper compares local Hi-Fi with the **Shapley effect**. The global Shapley effect is
\[
\Phi(x)= \sum_{\mathbf z\subseteq \mathbf Z}
\frac{|\mathbf z|!(n-|\mathbf z|)!}{(n+1)!}\,
L_{\mathbf z}(x\to y),
\]
and the local version is
\[
\Phi(x_i)= \sum_{\mathbf z\subseteq \mathbf Z}
\frac{|\mathbf z|!(n-|\mathbf z|)!}{(n+1)!}\,
\mathcal{L}_{\mathbf z_i}(x_i\to y_i).
\]
Both are built from subset-conditioned predictive gains, but they differ in what they summarize. Shapley effect averages over all contexts into one number, whereas Hi-Fi focuses on three reference points: the empty set, the minimizing subset, and the maximizing subset. Consequently, Shapley effect is an all-subsets allocation rule, while Hi-Fi is an interaction-typing decomposition that separates unique, redundant, and synergistic roles [2508.00930].

## 5. Applications and empirical interpretation

The original global formulation is illustrated by a toy example in which the target is
\[
Y=a_1+b_1+c+d_1d_2+\eta,
\]
with inputs
\[
X_1=a_2,\quad X_2=a_3,\quad X_3=b_2,\quad X_4=b_3,\quad X_5=c,\quad X_6=d_1,\quad X_7=d_2.
\]
Using degree-2 polynomial-kernel regression, the framework identifies \(X_1\) and \(X_2\) as synergistic because of dependency, \(X_3\) and \(X_4\) as redundant, \(X_5\) as purely unique, and \(X_6\) and \(X_7\) as purely synergistic because of the nonlinear interaction term \(d_1d_2\). This example is methodologically important because it shows that the framework can capture both dependence-driven and interaction-driven synergy [2412.09964].

A large-scale scientific application uses GEANT-based simulation for proton/pion discrimination with
\[
n=4{,}752{,}682
\]
samples and six z-scored inputs: velocity \(\beta\), momentum \(p\), scattering angle \(\theta\), number of emitted photoelectrons \(nphe\), inner detector response \(ein\), and outer detector response \(eout\). With the hypothesis space induced by the inhomogeneous polynomial kernel of degree 2, the decomposition finds that \(\beta\) has a large unique contribution, synergy with \(p\), and redundancy with \(\theta\); \(p\) has small unique contribution, synergy with \(\beta\), and redundancy with \(\theta\) and \(ein\); \(eout\) is synergistic with \(\beta\); and \(nphe\) is almost irrelevant. The paper’s main domain interpretation is that the synergistic cooperation between \(\beta\) and \(p\) likely corresponds to particle mass, a discriminative factor not captured by either variable alone [2412.09964].

The local extension applies Hi-Fi to a One-Health study of air pollutants and Alzheimer’s disease mortality. The dataset comprises 32 features for 107 Italian provinces over 5 years, giving \(N=535\) patterns after pooling years. Global Hi-Fi finds that air-pollution variables are the most synergistic group, with \(O_3\), \(PM_{10}\), and \(NO_2\) showing high synergy, while some “other pathologies” variables, especially circulatory mortality, are highly redundant. The local analysis then shows that Bergamo and Brescia stand out as provinces where pollutant influence is most synergistic, whereas some provinces such as Pescara, Piacenza, and Siracusa show higher local unique scores for \(O_3\), and Pescara also for \(NO_2\) [2508.00930].

These applications illustrate the intended interpretive use of Hi-Fi. A large unique score points to a feature that remains predictive even under the most adverse conditioning. A large redundancy score indicates overlap or substitutability with other variables. A large synergy score indicates that the feature’s predictive value is mostly cooperative and becomes visible only in appropriate contexts. In empirical work, this often alters the ranking produced by one-number importance measures because a feature can rank highly in total relevance yet be mostly redundant rather than uniquely or synergistically informative.

## 6. Extensions, neighboring methods, and limitations

Hi-Fi occupies a specific position within the broader literature on interaction-aware feature importance. It is **feature-centric**, because it decomposes the role of a focal variable; **optimization-based**, because it uses extremal subsets rather than a full subset lattice decomposition; and **predictability-based**, because its primitive object is prediction-error reduction. A closely related but distinct line is **iLOCO**, which defines interaction importance for a subset \(S\) by inclusion–exclusion:
\[
\mathrm{iLOCO}_S=\sum_{T\subseteq S}(-1)^{|S|-|T|}\Delta_T,
\]
with the pairwise case
\[
\mathrm{iLOCO}_{j,k}=\Delta_j+\Delta_k-\Delta_{j,k}.
\]
iLOCO is model-agnostic, is formulated directly through risk differences under refitting, provides distribution-free / assumption-light confidence intervals, and extends formally to higher-order subsets, whereas Hi-Fi decomposes the importance of a **single feature** into unique, redundant, and synergistic parts [2502.06661].

A further development, **Stochastic Hi-Fi**, reframes the method as a post-hoc, retraining-free interventional decomposition. For a coalition \(S\subseteq[n]\), it defines the interventional coalition loss
\[
L(S;x,y)= E_{Z\sim p_{bg}}\!\left[ \ell\left(f(m_S\odot x+(1-m_S)\odot Z),y\right)\right],
\]
and the coalition-specific LOCO gain
\[
LOCO(i\mid C)=L(C;x,y)-L(C\cup\{i\};x,y).
\]
It then sets
\[
U_i=\min_C LOCO(i\mid C),\qquad
L_i^{\max}=\max_C LOCO(i\mid C),\qquad
\pi_i=LOCO(i\mid \emptyset),
\]
with
\[
R_i=\pi_i-U_i,\qquad
S_i=L_i^{\max}-\pi_i,\qquad
L_i^{\max}=U_i+R_i+S_i.
\]
This later work also proves, on a minimal 3-way XOR structural causal model, that signed pairwise interaction scores fundamentally conflate uniqueness, redundancy, and synergy: faithful pair indices return zero per pair, while projective pair indices spread the third-order effect into pair scalars; by contrast, the LOCO decomposition returns \((U_i,R_i,S_i)=(0,0,1/2)\) for each active feature. Stochastic Hi-Fi further introduces coupled diamond sampling for strict variance reduction and gives uniform finite-vocabulary convergence results [2606.19410].

Hi-Fi should also be distinguished from intrinsic interaction-learning architectures in recommendation systems. For example, models such as FiiNet learn the importance of explicit multi-order feature combinations through model-internal attention or gating over crossed representations, and treat those learned weights as interaction saliency. That is closely related in theme but methodologically different: Hi-Fi is a decomposition of feature importance, whereas such architectures are predictive models whose importance signals are internal to the forward pass [2405.08852].

Several limitations recur across the Hi-Fi literature. The subset search is heuristic rather than exhaustive; the decomposition depends on the chosen regression model and therefore on the expressive capacity of the hypothesis space; the higher-order contribution is summarized through optimized subsets rather than decomposed into all \(3\)-way, \(4\)-way, and higher-order atoms; and the framework is not identical to PID. In the local and stochastic variants, interpretation also depends on the masking or background distribution and on the degree of subset coverage. These constraints do not negate the method’s central contribution, but they delimit its scope: Hi-Fi is best understood as a structured, interaction-aware decomposition of predictive feature importance, not as a universal theory of multivariate dependence [2412.09964][2508.00930][2606.19410].

Source: https://www.emergentmind.com/topics/high-order-interactions-for-feature-importance-hi-fi