---
title: 'Overlapping Group LASSO: Structured Feature Selection'
url: https://www.emergentmind.com/topics/overlapping-group-lasso
type: topic
---

# Overlapping Group LASSO: Structured Feature Selection

Overlapping Group LASSO is a structured sparsity-inducing regularization framework for regression and classification that enforces selection at the level of predefined groups of features, allowing groups to share features arbitrarily. Such models are essential in domains where feature structure is inherently multi-faceted (e.g., biological pathways, spatial patches, hierarchical covariates), but the non-separability introduced by overlapping groups poses distinct statistical and computational challenges. The core technical mechanism is a latent-variable decomposition: for each group, a latent coefficient vector specific to that group is introduced, and the total effect for each feature is the sum of its contributions across all groups to which it belongs. Penalization is then performed over the groupwise latent vectors via an $\ell_2$ norm. This article covers the mathematical formulation, convexity and dual properties, optimization strategies, screening and acceleration methods, statistical guarantees, and real-world applications in biomedical and signal processing contexts.

## 1. Mathematical Formulation and Latent Variable Structure

The canonical form of the overlapping group LASSO problem is:
\[
\min_{\beta_0, \{u_g\}}\, L(\beta_0, \beta_{-0}) + \lambda\sum_{g\in G} w_g \|u_g\|_2,\quad \text{subject to}\quad \beta_{-0} = \sum_{g\in G} A_g u_g,
\]
where:
- $L(\beta_0, \beta_{-0})$ is a convex loss (e.g., negative logistic log-likelihood $-\frac{1}{n}\sum_{i=1}^n [y_i x_i^\top\beta - \log(1 + e^{x_i^\top \beta})]$),
- $G = \{g_1,\ldots,g_J\}$ is a collection of groups (possibly overlapping subsets of ${1,\dots,p}$),
- $u_g \in \mathbb{R}^{|g|}$ are latent coefficients for group $g$,
- $A_g \in \{0,1\}^{p\times |g|}$ maps $u_g$ into the full coefficient vector $\beta_{-0}$,
- $w_g = \sqrt{|g|}$ (or other positive weights) scales the penalty per group,
- $\lambda>0$ is the global sparsity tuning parameter.

By eliminating $\beta_{-0}$, this constraint reduces the problem to a convex minimization in the expanded (latent) space:
\[
\min_{\{\ u_g\ \}}\ L\left(\beta_0, \sum_g A_g u_g\right) + \lambda\sum_g w_g \|u_g\|_2,
\]
which is isomorphic to non-overlapping group LASSO on a design matrix with duplicated columns matching the group overlap structure [1510.05144].

The latent group norm is formalized as:
\[
\Omega(\beta) = \inf_{\{v^g\}\ \text{with}\ \text{supp}(v^g) \subseteq g,\, \sum_g v^g = \beta}\ \sum_g d_g \|v^g\|_2,
\]
and this norm is convex, positively homogeneous, and ensures that the support is a union of selected groups [1110.0413].

## 2. Optimization Algorithms and Computational Enhancements

Block coordinate descent is standard for overlapping group LASSO, exploiting the group-separable nature of the latent variable representation. The group-wise update involves weighted soft-thresholding (proximal operator) for each group:
\[
u_g^{(t)} \leftarrow \frac{1}{H_g}\left(1 - \frac{n\lambda w_g}{\|g_g\|_2}\right)_+ g_g,
\]
where $g_g$ is the gradient and $H_g$ is a group-wise Hessian or Lipschitz constant [1510.05144, 1402.4512].

Accelerated proximal-gradient methods (e.g., FISTA) can be applied, but the non-separable proximal operator is computed by projecting onto the intersection of norm balls defined by the overlapping groups. For $\ell_2$ penalties, dual-based projected Newton methods solve the key subproblem efficiently by exploiting active set strategies [1209.0368, 1009.0306].

ADMM frameworks are especially suited when the groups are large or highly overlapping. Each ADMM step alternates between updating the full coefficient vector and independent group-wise latent variables, with closed-form soft-thresholding for the $\ell_2$ group norm [2310.19012].

Dimension reduction and screening techniques are crucial for scalability. Recent adaptive schemes compute dual certificates for group support:
- If $\|\beta^*_{G_t}\|_2 < w_t$ (LASSO certificate) or $\|u^\dagger_{J_t}\|_2<1$ (OGN certificate), group $G_t$ is safely screened as inactive [2601.20697, 1410.6880].
These strategies dramatically reduce the active group set and accelerate convergence.

## 3. Statistical Theory, Support Recovery, and Oracle Bounds

Overlapping group LASSO achieves structured sparsity: the support of the estimated vector is a union of groups. Model selection error bounds, estimation rates, and prediction guarantees depend on the group sparsity structure, overlap degree, and mutual incoherence properties:
- The basic estimation error scales with the number and size of active groups [1402.4512]:
  \[
  n \geq C\,\sigma_f^2\,\epsilon^{-2}\,k\,\min\{(1+\mu)^2[\log K + L],\,((1+\mu)/\mu)^2 l \log p\}
  \]
  for $(k,l)$ group- and within-group sparsity, $\mu = \lambda_1/\lambda_2$ trade-off.
- Oracle bounds (RE-type) extend to the overlapping case, with finite sample rates deteriorating as the number of groups and degree of overlap grow [1103.4614]:
  \[
  (1/n)\|X(\hat\beta-\beta^0)\|_2^2 \leq (64\sigma^2)/(\kappa(s)^2 n) [ \max_g |g| + A\sqrt{\max_g |g|} \log M ]
  \]
- Exact asymptotic selection is guaranteed under further non-nesting group conditions, with adaptive weights (e.g., $\lambda_g = 1/\|v_g^{OLS}\|_2^\gamma$) serving as key to support recovery [1103.4614].

For sparse overlapping group LASSO variants (SOGlasso, SOSlasso), theoretical error bounds interpolate between those for $\ell_1$ and pure group LASSO, achieving minimax rates under sub-Gaussian designs [1402.4512, 1311.5422].

Statistical equivalence between overlapped and their tightest non-overlapping separable relaxations has been established; the latter attain the same error and support recovery guarantees, but allow scaling to very large problems [2211.09221].

## 4. Variants: Hierarchical and Sparse Overlapping Group LASSO

Several generalizations extend the core framework:
- Sparse Overlapping Group LASSO (SOGlasso) mixes $\ell_1$ sparsity across and within overlapping groups, with the estimator:
  \[
  \min_{w}\ L(w;X,y) + \lambda_1\|w\|_1 + \lambda_2\sum_{G\in\mathcal{G}}\|w_G\|_2
  \]
  and equivalent latent variable decompositions [1402.4512, 1311.5422].
- Hierarchical Overlapping Group LASSO (HOGL) structures groups and their overlaps along hierarchies or chains, enforcing additional logical constraints (e.g., variable selection implies degree selection in polynomial models) [2510.19311, 1512.01631].
- Latent Overlapping Group LASSO (LOG) is particularly effective for hierarchical sparse modeling (HSM), where the latent variable acts as a group-specific carrier allowing precise enforcement of context- or graph-based selection rules. LOG ensures uniform shrinkage across group nesting depth, as opposed to classical GL which over-penalizes deep parameters [1512.01631, 2206.05337].

## 5. Algorithmic Implementation and Practical Tools

Implementations like grpregOverlap (R) build the expanded design matrix only virtually, index efficiently, and delegate the sequence of group coordinate descent problems to proven engines [1510.05144]. Warm starts, strong rules for active set screening, and group decomposition strategies are explicitly supported, with performance scaling to tens of thousands of effective variables.

Divide-and-conquer methods (DC-ogLasso) split large datasets, solve overlapping group LASSO subproblems locally, aggregate support via majority voting, and refit active features, yielding near-linear scalability and model selection consistency with minimal communication overhead [1612.02222].

Dimension reduction frameworks (AdaDROPS) utilize adaptive certificate-based support expansion, permitting primal-dual splitting, ADMM, and overparameterized variable projection methods. In high overlap regimes, AdaDROPS delivers order-of-magnitude speedups relative to baseline solvers [2601.20697].

Screening rules employing dual polytope projection (DPP) and safe group-level tests efficiently discard inactive groups prior to solving, crucial for large-scale statistics and machine learning applications [1410.6880].

## 6. Applications and Empirical Performance

Overlapping group LASSO and its variants have demonstrated significant empirical benefit in high-dimensional biomedical data, neuroimaging, genomics, and signal processing:
- In gene pathway selection, group-structured regularization accounts for biological overlap, yielding improved classifier accuracy and more parsimonious, interpretable pathway selection compared to marginal hypothesis tests (e.g., GSEA) or plain LASSO [1510.05144].
- In fMRI and MEG studies, overlapping spatial groups facilitate joint modeling across subjects or brain regions, with SOGlasso and SOSlasso outperforming non-overlapping methods in both predictive risk and anatomical interpretability [1402.4512, 1311.5422].
- In growth curve models (GMANOVA) and spline-based spectrum cartography, hierarchical and overlapping group LASSO penalties recover variable and degree structures reliably, outperforming non-hierarchical methods, especially at higher complexity [2510.19311, 1010.0274].

Simulations consistently show lower coefficient RMSE, improved support recovery frequency, and classification error reductions on real and synthetic datasets. For instance, OGLasso yielded up to 10% lower misclassification and selected distinct gene pathways not identified by GSEA or ordinary lasso [1510.05144]. Separable approximations match statistical rates of classical overlapping LASSO but afford 5–20× faster computation [2211.09221].

## 7. Design Considerations, Limitations, and Future Directions

The choice of group structure and weights is critical for identifiability, support recovery, and finite-sample efficiency. Excessive overlap or poorly calibrated weights degrade statistical rates and selection consistency [1103.4614, 1110.0413]. The LOG/latent approach provides a principled route to encode complex selection rules, including hierarchical, hereditary, and mutually exclusive relationships among features, with guaranteed feasibility via group design [2206.05337].

Safe screening, adaptive weighting, dimension reduction, and acceleration techniques have evolved to address the computational bottlenecks of overlapping sparsity. Future research directions include scaling algorithms for ultra-high-dimensional settings, refining support recovery under complex combinatorial selection rules, and integrating domain-specific hierarchical regularization in emerging fields (e.g., multi-modal signal analysis, clinical coherence modeling).

Overlapping group LASSO presents a mathematically coherent, statistically solid, and practically scalable framework for structured feature selection—particularly when available prior knowledge is rich, groups are complexly interconnected, and model interpretability is central.

Source: https://www.emergentmind.com/topics/overlapping-group-lasso