Filter-Modulated Multi-Head Self-Attention

• FM-MSA is a feature selection mechanism that integrates classical filter methods with multi-head self-attention using normalized statistical priors.
• It modulates per-head attention weights with filter-derived scores, capturing nonlinear interactions while reducing sensitivity to random initialization.
• The approach aggregates multi-head outputs and applies a Random Forest reordering step, yielding robust and interpretable feature rankings in biomedical applications.

Filter-Modulated Multi-head Self-Attention (FM-MSA) is a mechanism for feature selection in high-dimensional data, integrating classical statistical filter methods with multi-head self-attention architectures. Introduced as the core component of the MAFS (Multi-head Attention-based Feature Selection) framework, FM-MSA leverages normalized filter-derived priors to guide attention weight formation, thereby uniting the interpretability and stability of statistical feature scoring with the expressive capacity of deep learning. This approach is engineered specifically for scenarios requiring scalable, stable, and interpretable feature ranking, such as genomics and biomedical predictive modeling (Sun et al., 6 Jan 2026).

1. Filter-Based Priors and Normalization

FM-MSA operates on an input feature matrix $X \in \mathbb{R}^{n \times d}$ with response $y \in \mathbb{R}^n$ (real-valued or one-hot-encoded for classification). It begins by computing M classical filter-based importance scores for each feature using a set of scalar-valued filters $\mathcal{F} = \{ f_1, \ldots, f_M \}$, examples being marginal Pearson correlation, distance correlation, and Kendall’s $\tau$. Each filter $f_j$ produces a vector of raw feature importances $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$. These are standardized per filter as: $$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$ where $\mu_j$, $\sigma_j$ are the mean and standard deviation of $w^{(j)}$. The normalized priors $y \in \mathbb{R}^n$0 are used to modulate the attention mechanisms in subsequent stages, biasing each head toward features statistically predictive of $y \in \mathbb{R}^n$1.

2. Per-Head Attention with Filter Modulation

In FM-MSA, the input $y \in \mathbb{R}^n$2 is first linearly embedded: $y \in \mathbb{R}^n$3 For each head $y \in \mathbb{R}^n$4, where $y \in \mathbb{R}^n$5 (number of filter methods), queries, keys, and values are computed: $y \in \mathbb{R}^n$6 with $y \in \mathbb{R}^n$7. The canonical Transformer attention logit is: $y \in \mathbb{R}^n$8 FM-MSA introduces filter-score modulation by adding a scaled prior: $y \in \mathbb{R}^n$9 where $\mathcal{F} = \{ f_1, \ldots, f_M \}$0 and $\mathcal{F} = \{ f_1, \ldots, f_M \}$1 (default $\mathcal{F} = \{ f_1, \ldots, f_M \}$2). Attention weights are recovered via softmax, and each head outputs

$\mathcal{F} = \{ f_1, \ldots, f_M \}$3

This approach allows each attention head to focus on features both relevant under its statistical filter and informative in the nonlinear deep embedding, capturing complex relationships and reducing sensitivity to random initialization.

3. Multi-Head Aggregation and Feature Reordering

FM-MSA departs from canonical multi-head architectures by aggregating per-feature attention strengths rather than concatenating latent outputs. For each head:

• Compute average attention per feature:

$\mathcal{F} = \{ f_1, \ldots, f_M \}$4

• Extract the top $\mathcal{F} = \{ f_1, \ldots, f_M \}$5 features per head: $\mathcal{F} = \{ f_1, \ldots, f_M \}$6
• Form the candidate pool $\mathcal{F} = \{ f_1, \ldots, f_M \}$7

Final feature ranking is performed by training a lightweight Random Forest (with $\mathcal{F} = \{ f_1, \ldots, f_M \}$8 by default) on $\mathcal{F} = \{ f_1, \ldots, f_M \}$9, evaluating each feature's mean Gini-importance $\tau$0, and selecting the top $\tau$1 features via descending $\tau$2. This "reordering module" ensures that multiple statistical and learned attention perspectives are fused for robust, conflict-resolved feature prioritization.

4. Initialization and Training Objectives

The attention module’s only deterministic inputs are the normalized filter-based priors $\tau$3; other parameters ($\tau$4, $\tau$5, $\tau$6, $\tau$7, $\tau$8) are randomly initialized following Glorot-uniform schemes. For each head, the predictive task uses the attention outputs $\tau$9 in a lightweight feed-forward prediction head. The loss per head is:

• Cross-entropy for classification: $f_j$0
• Mean squared error for regression: $f_j$1

An adaptive $f_j$2 penalty is imposed on the feature biases $f_j$3: $f_j$4 where $f_j$5, $f_j$6, with typical values $f_j$7, $f_j$8, $f_j$9. Total loss is summed over heads and used for joint backpropagation. Early stopping is performed on a validation split.

5. Algorithmic Workflow and Hyperparameters

A concise FM-MSA training and inference protocol is:

1. Compute and normalize M filter scores $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$0
2. Initialize projection matrices
3. For each epoch:
   • Embed $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$1 to $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$2
   • For each head, compute $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$3, $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$4, $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$5, attention with filter modulation, and prediction, then form $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$6
   • Sum losses, backpropagate, update weights
4. After convergence, aggregate per-head top-K, pool features, and rerank with a random forest

Empirically validated hyperparameter defaults:

• $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$7 number of filters (typically 3: Pearson, distance correlation, Kendall’s $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$8)
• Embedding/projection size $w^{(j)} = f_j(X, y) \in \mathbb{R}^d$9, per-head size $$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$0
• $$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$1
• $$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$2
• $$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$3 2% of $$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$4 (or fixed K if $$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$5 is small)
• $$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$6 for Random Forest; Gini criterion
• Optimizer: Adam with $$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$7, batch size 32, up to 100 epochs, early stopping with patience 10
• Dropout $$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$8–$$w^{\text{norm},(j)} = \frac{w^{(j)} - \mu_j}{\sigma_j}$$9, batch normalization after each linear

6. Interpretability, Stability, and Applicability

By modulating standard self-attention mechanisms with normalized statistical filter priors, FM-MSA explicitly incorporates domain-informed, stable feature signal as a bias, enabling improved reproducibility and interpretability relative to standard deep learning-based selectors. The multi-head mechanism captures diverse interaction patterns and guards against the pitfalls of single-head approaches, such as initialization sensitivity and limited dependency modeling. The reordering module’s explicit use of ensemble decision tree importance further fuses statistical, learned, and ensemble perspectives in the final ranking.

In high-dimensional biomedical applications, FM-MSA has demonstrated consistent gains in feature coverage and ranking stability versus traditional filters and neural attention models, allowing for robust, information-preserving feature selection (Sun et al., 6 Jan 2026). A plausible implication is its suitability for ultra-high-dimensional problems where statistical interpretability and nonlinear interaction modeling are simultaneously required.

7. Related Methods and Context

FM-MSA emerges in response to limitations of both classical filter-based and deep learning-based feature selection approaches. Whereas filter methods offer scalability and transparency but lack the ability to capture nonlinear or high-order dependencies, standard deep models may model complex signals yet are frequently unstable, non-interpretable, and non-scalable for ultra-high-dimensional domains. Hybrid methods such as FM-MSA represent an overview: statistical filter guidance biases deep attention structures, maximizing the retention of informative signals while ensuring that the final ranking reflects both statistical significance and learned nonlinear associations.

The explicit architectural departure from classical self-attention—injecting normalized priors as additive biases, multi-head feature aggregation via statistical selection, and the final decision-tree-based reordering—differentiates FM-MSA from standard transformer or plain neural network selectors. These advances are particularly salient for interpretability and robustness in sensitive domains such as genomics and precision medicine feature selection (Sun et al., 6 Jan 2026).