Online Group Feature Selection

• Online Group Feature Selection (OGFS) is a method that selects relevant feature groups from a data stream by leveraging group structure to maximize between-class variance and minimize within-class variance.
• It employs a two-stage process using spectral analysis for intra-group selection and Lasso regression to eliminate redundant features across groups.
• OGFS has demonstrated robust performance in image recognition and bioinformatics, showing stability under varying group arrival orders and improving classification accuracy with fewer features.

Online Group Feature Selection (OGFS) refers to a class of algorithms specifically designed for selecting relevant features from a dynamically arriving data stream, where features arrive in groups rather than individually. OGFS maintains the structural integrity of feature groups, such as those encountered when multi-modal or multi-descriptor features are computed in batches (e.g., SIFT descriptors in images, sensor arrays, biological pathway-based genes). The OGFS methodology emphasizes discriminative power and redundancy control under real-time or streaming constraints, utilizing principled statistical and optimization criteria as new groups are acquired and incrementally incorporated into the selected subset (Wang et al., 2016, Jing et al., 2014).

1. Problem Setting and Formalization

Let $X\in \mathbb{R}^{d\times n}$ denote the data matrix with $n$ instances and $d$ total features, and $y\in \{1,\ldots,c\}^n$ the class labels. In OGFS, features are not assumed to be available at once; instead, they arrive as a sequence of groups, $G_j = \{f_{j1},\ldots,f_{jd_j}\}$ at each timestep $j$. Each $f_{jk}$ is a feature vector of length $n$. The algorithm, unware of future groups, incrementally updates the selected subset $U = \bigcup_j g_j$ where $g_j \subseteq G_j$. Typical stopping criteria include reaching a target cardinality $|U|\geq K$, achieving a desired classification accuracy on $U$, or exhausting the feature stream.

The selection objective is threefold:

1. Maximize between-class variance,
2. Minimize within-class variance,
3. Reduce redundancy among selected features across all groups.

2. Intra-Group Feature Selection via Spectral Analysis

At each group arrival, OGFS applies a spectral criterion to decide, in an online manner, whether a feature should be provisionally included. Affinity matrices $S_b$ (between-class) and $S_w$ (within-class) are constructed as follows:

$$(S_b)_{ij} = \begin{cases} \frac{1}{n} - \frac{1}{n_\ell} & \text{if } y_i = y_j = \ell \ \frac{1}{n} & \text{if } y_i \neq y_j \end{cases}$$

$$(S_w)_{ij} = \begin{cases} \frac{1}{n_\ell} & \text{if } y_i = y_j = \ell \ 0 & \text{otherwise} \end{cases}$$

With degree matrices $D_b, D_w$ and Laplacians $L_b = D_b - S_b$, $L_w = D_w - S_w$, for any current feature set $U$, the spectral ratio is defined as: $$F(U) = \frac{\operatorname{tr}(X_U L_b X_U^\top)}{\operatorname{tr}(X_U L_w X_U^\top)}$$ where $X_U$ is the submatrix of $X$ restricted to features selected in $U$.

For each incoming feature $f$, the increment $F(U\cup\{f\}) - F(U)$ is evaluated. If $|F(U\cup\{f\}) - F(U)| > \epsilon$ (e.g., $\epsilon = 10^{-3}$), $f$ is included in the provisional intra-group set $G_j'$. This criterion enforces the selection of features that enhance between-class discrimination or compact within-class scatter as measured spectrally (Wang et al., 2016, Jing et al., 2014).

3. Inter-Group Feature Selection via Sparse Regression

Following the intra-group pass, OGFS seeks to further prune redundancy both within and across groups using Lasso regression. For the current selection $U$ and interim group $G_j'$, an augmented feature matrix $A$ is constructed from $U \cup G_j'$. The algorithm solves:

$$\min_{\beta} \| y - A^\top \beta \|_2^2 + \lambda \|\beta\|_1$$

Features corresponding to nonzero entries in $\beta$ are retained; features with zero coefficient are discarded. This step leverages the sparsity-inducing $\ell_1$ penalty to encourage a compact, non-redundant selection for subsequent group arrivals (Wang et al., 2016, Jing et al., 2014).

4. Algorithmic Summary and Complexity

The overall OGFS pipeline consists of repeated, two-stage processing for each arriving group:

1. Intra-group selection: For each feature in group $G_j$, compute $F(U\cup \{f\}) - F(U)$ and include $f$ in $G_j'$ if the increment exceeds $\epsilon$.
2. Inter-group Lasso: Apply sparse regression over $U \cup G_j'$; keep only those features with nonzero regression weights, updating $U$.
3. Stopping check: Terminate if the selected feature set reaches cardinality $K$, achieves the desired accuracy, or the stream ends.

The intra-group computation involves efficient eigen-trace updates of the spectral ratio, $O(p)$ per feature with $p$ small, while the inter-group Lasso regression is empirically linear or low polynomial in current feature count. Over $T$ groups, with $m_j$ features per group, overall time complexity is $O(\sum_j (m_j + |U_j|))$ and space is governed by $O(dn)$ and Laplacian storage (Wang et al., 2016, Jing et al., 2014).

5. Experimental Results and Empirical Characteristics

OGFS has been benchmarked on both image and biological data where group structure is natural or synthesized:

• Image benchmarks:
  • Cifar-10 (color image classification): OGFS achieved 49.6% accuracy with ≈2k features; using only intra-group step achieves 51.2% (≈5k features), offering a controllable trade-off.
  • Caltech-101 (object recognition): Outperformed all online baselines by an absolute 6–13% margin with 1–2k features.
  • LFW (face verification): Achieved ~81% accuracy versus 77% for Grafting and 66% for OSFS, using ~1.5k features.
• UCI/microarray benchmarks:
  • OGFS surpassed other online methods on 6/8 tasks and achieved comparable or better accuracy than offline LARS/MI selection with fewer features.
• Robustness: OGFS exhibited empirical stability under random permutation of group arrival order, with accuracy variance $\lt$1% (Wang et al., 2016, Jing et al., 2014).

Compared baselines included Alpha-investing, OSFS, Grafting (online) and mutual information, LARS, GBFS (offline). Metrics tracked were accuracy, number of features (compactness), and CPU time.

6. Theoretical Analysis and Guarantees

No formal convergence or regret-type guarantees are provided for OGFS. However, empirical evaluation demonstrates algorithmic stability concerning group order and efficacy in both discriminative power and redundancy pruning. The approach systematically combines local spectral discriminability with global sparsity control at each group addition. The performance and output subset depend explicitly on group quality and the parameters $\epsilon$ and $\lambda$, which require tuning (Wang et al., 2016, Jing et al., 2014).

7. Applications, Limitations, and Extensions

OGFS is applicable where feature groups arrive dynamically or possess inherent structure: multi-descriptor visual analysis, sensor networks with grouped measurements, bioinformatics with grouped genes, or any streaming data pipeline where "early, small-footprint" discriminative models are needed. The method assumes meaningful group boundaries and requires $\epsilon$ and $\lambda$ selection for spectral and Lasso steps. There is no theoretical guarantee of global optimality. A plausible implication is that further advances in theoretical understanding and adaptive parameter selection could extend OGFS to domains with weaker or less defined structural grouping.

---

References

• "Online Feature Selection with Group Structure Analysis" (Wang et al., 2016)
• "Online Group Feature Selection" (Jing et al., 2014)