Support-Set Aggregator Selection

• Support-set-based aggregator selection is a method to determine optimal aggregation functions for variable-sized support sets, ensuring robustness and model selection consistency.
• It utilizes a range of methods—from numerical and threshold-based to learnable neural aggregators—to handle sparsity and variability in data inputs.
• The approach is applied in sparse regression, distributed inference, and deep set architectures, providing both theoretical guarantees and empirical performance improvements.

Support-set-based aggregator selection refers to the principled determination of aggregation functions that synthesize information from variable-sized sets of features, parameters, or related observations ("support sets") in statistical, probabilistic, and machine learning models. The choice of aggregator controls properties such as sparsity, model selection consistency, robustness to varying support-set sizes, and computational or communication efficiency. This topic underlies multiple domains, including sparse regression, distributed inference, deep set architectures, and relational probabilistic models.

1. Support Set Definition and Contexts

The support set is formally defined as the set of indices or elements corresponding to relevant, potentially nonzero, variables or observations for a particular estimation, prediction, or inference task. In sparse parametric models, such as generalized vector autoregressions (GVAR), the parameter support set $S=\{j:\beta_j\neq0\}$ identifies the active subnetwork or feature subset (Ruiz et al., 2023). In relational probabilistic models, the support set associated with a target variable may comprise a varying multiset of associated predictors (e.g., movies rated by a user, neighbors in a graph) (Kazemi et al., 2017). For functions over set-valued inputs, as in Deep Set learning, support sets are arbitrary-sized unordered collections whose aggregation must respect permutation invariance (Soelch et al., 2019).

Support sets often exhibit substantial variability in cardinality across entities or subsamples, which prompts the need for robust aggregation strategies that avoid overconfidence, spurious sparsity, or loss of informative signals.

2. Aggregator Mechanisms and Formulations

Aggregator selection encompasses several families of mechanisms depending on model architecture and problem class:

• Numerical aggregators: These include linear functionals such as count, proportion, mean, and sum. For instance, logistic regression style aggregators use $P(Y=1|S)=\sigma(w_0 + w_1\cdot\mathrm{Count}(S))$ or its normalized variant (Kazemi et al., 2017). In Deep Sets, sum aggregation underpins universal approximators for permutation-invariant mappings: $f(X)=\rho\left(\sum_{i=1}^N \phi(x_i)\right)$ (Soelch et al., 2019).
• Threshold and frequency-based support aggregation: Candidate supports obtained via LASSO on subsamples are aggregated by computing selection frequencies $\pi_j$ and thresholding at $\tau$: $\hat S_{\mathrm{agg}}=\{j:\pi_j>\tau\}$ (Ruiz et al., 2023). Monotonicity as $\tau$ varies and stability-selection style bounds characterize this approach.
• Robust and informed aggregators: For relational models, per-item empirical rates (e.g., $P_1$ and $P_2$ formulas) and pseudo-count smoothing absorb information about population heterogeneity, while k-neighbor sampling or dropout capping yield bounded confidence estimates (Kazemi et al., 2017).
• Communication-efficient distributed aggregators: The message algorithm aggregates feature inclusion indicators via median selection across parallel LASSO fits, minimizing communication costs and retaining oracle estimation rates (Wang et al., 2014).
• Learnable and adaptive neural aggregators: Deep Set learning extends aggregation beyond fixed commutative reductions to recurrent attention-based modules, e.g., r-LSE or r-sum, which enhance expressivity and generalization under input set size shifts (Soelch et al., 2019).

3. Theoretical Guarantees and Consistency

Support-set-based aggregation methods frequently enjoy rigorous theoretical properties under standard sparsity and identifiability conditions:

• Model selection consistency: Under s-sparse parameter vectors, compatibility or restricted eigenvalue conditions, and sufficiently large subsample size ($n\gtrsim s\log p$), aggregated support estimators asymptotically recover the true support set with probability tending to one (Ruiz et al., 2023).
• Finite-sample bounds: For frequency-thresholded aggregators, expected false positive and negative rates are functions of threshold, number of subsamples, and error rates of individual fits; these bounds mirror those derived for stability selection (Ruiz et al., 2023).
• Oracle estimation rates: Median subset aggregation and communication-efficient algorithms such as message match or surpass the oracle coefficient estimation rate $\sigma^2 s / n$, conditional on valid tuning and irrepresentable conditions (Wang et al., 2014).
• Permutation invariance and universal approximation: Aggregators built from commutative reductions guarantee permutation invariance, with sum-isomorphism extensions to mean and log-sum-exp forms; for recurrent attention-based aggregators, universal approximator status remains an open question (Soelch et al., 2019).

4. Empirical Performance and Evaluations

Empirical studies across diverse domains quantify the impact of support-set-based aggregator selection:

Method/Class	Domain	Key Outcomes
Support-set aggregation	Sparse GVAR, ecological networks	0–2% avg. error, reduced false positives, sparse interpretable networks (Ruiz et al., 2023)
Median selection/message	Regression/classification, dist.	Consistent selection, efficient coeff. estimation, communication cost $O(mp)$ (Wang et al., 2014)
P₁/P₂, relational dropout	Relational probabilistic models	Lower MSE/LL, overconfidence mitigation, strong in large/skewed support sets (Kazemi et al., 2017)
r-LSE/r-sum aggregators	Deep sets, point clouds	Higher OOD accuracy, lower hyperparameter sensitivity (Soelch et al., 2019)

Simulation studies emphasize that support aggregation reduces error, especially false positives, relative to direct selection or refit-based benchmarks (Ruiz et al., 2023). Dropout or capped aggregators avoid the pathological overconfidence of classical sum/product forms in settings with extreme variability of support size (Kazemi et al., 2017). In distributed regimes, the median selection framework achieves full-data selection accuracy with minimal communication rounds (Wang et al., 2014). Deep Set empirical tests demonstrate marked gains in robustness and sensitivity when switching from non-learnable to learnable recurrent aggregators (Soelch et al., 2019).

5. Guidelines and Principles for Aggregator Selection

Selecting appropriate support-set-based aggregators is governed by the statistical properties of the support sets, intended interpretability, and computational constraints:

• For small, uniform support sets, simple sum or product aggregators (logistic regression, naive Bayes) suffice.
• For large or highly variable support sets, size-normalized (P₂), per-item empirical (P₁), capped neighbor (dropout, k-sampling), or thresholded-frequency aggregation are recommended to control overconfidence and maintain stability (Kazemi et al., 2017, Ruiz et al., 2023).
• When distributed computation or communication efficiency is crucial, median selection over parallel subsets (message) balances selection accuracy and resource use (Wang et al., 2014).
• In set-valued neural models, max-pooling or log-sum-exp are effective for classification with large N, whereas recurrent attention aggregators yield superior generalization and reduced hyperparameter sensitivity (Soelch et al., 2019).

Pseudo-count and dropout hyperparameters should be tuned to balance bias and variance, matching typical informativeness or neighborhood size per entity.

6. Applications and Implications

Support-set-based aggregator selection is foundational for problems such as sparse network recovery, high-dimensional regression, relational learning, deep permutation-invariant architectures, and ecological interaction inference.

For example, aggregation of LASSO-selected supports across resampled GVAR datasets enables robust identification of active edges in Granger-causal networks, with demonstrated utility for uncovering changes in ecological connectivity across epochs (Ruiz et al., 2023). In large-scale machine learning systems, the message algorithm makes feature selection practically feasible under bandwidth constraints, providing theoretical guarantees for distributed environments (Wang et al., 2014). In relational domains, context-aware aggregators and regularization strategies are necessary to ensure calibrated uncertainty and avoid the pitfalls of naive pooling (Kazemi et al., 2017). Deep Set aggregator selection is integral to the design of point cloud classifiers, mixture estimators, and unsupervised spatial attention modules, directly affecting generalization and stability (Soelch et al., 2019).

No single aggregator family is universally optimal; rather, the structure and distribution of support sets, the nature of the signals, and the application context must jointly inform aggregator selection. The empirical recipes and mathematical formulations in the referenced works present a robust framework for adapting aggregation strategies to task requirements, ensuring sparsistency, calibration, and computational tractability.