Hierarchical Variable Selection (HVS) Algorithm

• Hierarchical Variable Selection (HVS) is a method that systematically selects a parsimonious set of predictors from multi-level, tree-structured data.
• The algorithm employs a three-step process—group-wise stepwise regression using AIC, pooled variable selection, and ridge regression for regularization—to balance interpretability and predictive performance.
• Empirical validations in ESG risk assessment demonstrate that HVS outperforms standard methods by achieving lower AIC/BIC scores and improved out-of-sample performance.

A Hierarchical Variable Selection (HVS) algorithm is a principled methodology for identifying a parsimonious set of explanatory variables from data structures endowed with a multi-level or tree-like organization. HVS methodologies are especially relevant when the data exhibit a much higher number of potential predictors than observations and when the predictors are naturally grouped, such as in Environmental, Social, and Governance (ESG) risk assessment, genomics, or multi-domain industrial datasets. The HVS paradigm exploits the inherent grouping to yield models that are both statistically powerful and interpretable, providing improved explanatory power and generalization compared to standard variable selection techniques (Chen et al., 26 Aug 2025).

1. Algorithmic Structure and Hierarchical Principles

The HVS algorithm proceeds in three principal steps designed to respect the multi-level grouping of variables:

1. Category-level Selection: Within each predefined variable group (e.g., ESG “categories”), stepwise regression is performed using only the raw variables in that group. Model selection is governed by the Akaike Information Criterion (AIC), retaining predictors that contribute significantly to the response variable (e.g., log-volatility).
2. Pooled Selection Across Categories: Variables selected in the first step (across all groups) are aggregated. A second stepwise regression is run on this collection, resulting in additional parsimony and addressing multi-collinearity that may not be resolved in group-wise isolation.
3. Final Model Regularization: Ridge regression is applied to the predictors selected in step two, imposing an L₂ penalty to shrink coefficients, thereby controlling overfitting, especially in high-dimensional regimes where the ratio of predictors to observations is large.

This sequential structure strictly enforces the tree or hierarchy present in many data sets: predictors are first filtered and validated within their native group, then only those with demonstrated signal are allowed to survive to a global selection, and, finally, the regularization ensures stability and interpretability of the final model.

2. Application to ESG Data: Structure and Preprocessing

ESG datasets are characterized by deep hierarchical structure, typically with raw features (e.g., emission metrics, workforce attributes) grouped into categories (e.g., “Emission Reduction,” “Employment Quality”), which are themselves aggregated into higher-level pillars and ultimately an overall ESG score.

Distinctive aspects of the ESG application include:

• Variable Surfeit and Missingness: The number of raw ESG metrics can exceed the number of samples by an order of magnitude or more, and many features exhibit substantial missingness.
• Category-based Preprocessing: The algorithm pragmatically addresses data sparsity by discarding numeric variables with less than 80% non-missing values and treating missing Boolean variables as zero.
• Parsimony and Interpretability: By constraining the selection process to operate first within categories and subsequently in aggregate, HVS produces more interpretable models that can be mapped directly back to specific raw ESG measures.

3. Quantitative Performance and Model Selection Criteria

The HVS algorithm’s performance is systematically benchmarked against models that use aggregated ESG scores and standard variable selection techniques (e.g., single-step stepwise, PCA, Lasso). Key evaluation metrics include:

Step	Metric Used	Selection/Regularization Criterion
Category-level regression	Adjusted R², AIC	Stepwise, penalized by AIC
Pooled regression	Adjusted R², AIC	Stepwise, penalized by AIC
Full model (post-Ridge)	% Deviance Explained (%dev), BIC, MSE	Ridge regression (L₂ penalty), BIC for parsimony

• HVS achieves substantially higher out-of-sample %deviance explained compared to models using aggregated scores or non-hierarchical selection.
• Lower BIC and AIC values for HVS indicate superior parsimony given predictive performance.
• In-sample and out-of-sample MSEs are reduced relative to baseline and alternative selection approaches, with matched pairs tests confirming the statistical significance of improvements.

4. Mathematical Underpinnings and Model Formulation

The final model in an HVS framework for ESG risk takes the form: $$\log(\mathrm{Volatility}) = \beta_0 + \sum_i \beta_i x_i + \epsilon$$ where $x_i$ are standardized predictors arising from the hierarchical selection pipeline.

Critical operations and criteria:

• Ridge penalty: Minimization objective becomes

$$\min \left\{ \sum_i \left(y_i - (\beta_0 + \sum_j \beta_j x_{ij})\right)^2 + \lambda \sum_j \beta_j^2 \right\}$$

with λ as regularization parameter.

• Category Importance: For a given ESG category C_j, its aggregate contribution is quantified by

$$\mathrm{Score}_{C_j} = \sum_{i \in I_j} | \beta_i |,\qquad \%\ \mathrm{Importance}_{C_j} = \frac{\mathrm{Score}_{C_j}}{\sum_k | \beta_k |} \times 100\%$$

where $I_j$ is the index set of predictors in category $C_j$.

• Response Transformation: Logarithmic or Box–Cox transformations are applied to risk/volatility to approximate Gaussianity, ensuring that linear model assumptions hold to a reasonable degree.

5. Empirical Validation and Industry Insights

Sector-specific analyses using US company datasets reveal the following:

• Sectoral Differentiation: HVS uncovers that ESG categories most predictive of risk differ across sectors (e.g., “Emission Reduction” impactful in Energy, “Compensation Policy” in Finance), insights not discernible from aggregate ESG scoring alone.
• Fine-grained Factor Discovery: Within-category selection allows identification of raw variables (e.g., specific emission metrics) that drive sector risk, enabling more actionable guidance for ESG risk management.
• Model Robustness: Out-of-sample evaluation, using both leave-one-out and time-based train/test splits, shows the HVS algorithm’s generalization, with post–Ridge regularization models consistently outperforming or matching more complex and less interpretable alternatives (e.g., Lasso with higher dimensionality).

6. Broader Implications, Limitations, and Future Directions

The HVS strategy generalizes to any context with tree-structured, high-dimensional predictors and limited sample size. It enhances both statistical robustness and substantive interpretability in domains where grouping is intrinsic—be that ESG analytics, genomics, or clustered industrial process data.

Limitations include:

• The reliance on stepwise regression may be suboptimal in extremely high-dimensional settings where greedy optimization can be inconsistent.
• Variable selection is constrained by the initial preprocessing (e.g., exclusion of highly missing numerics), potentially omitting informative variables.

Suggested future work, as described, includes: experimenting with nonstandard segmentation schemas (e.g., beyond capitalization splits), improving imputation procedures, and extending the methodology to other response variables such as returns and to domains beyond ESG.

A plausible implication is that HVS can support regulatory bodies or practitioners aiming to define sector-specific ESG reporting or compliance standards, since it yields explicit, interpretable rankings of risk-relevant variables, unlike opaque aggregate models. The hierarchical variable selection design thus stands as a robust solution for high-dimensional, grouped predictor selection with strong empirical and interpretive guarantees (Chen et al., 26 Aug 2025).