Progressive Parameter Selection (PPS)

• Progressive Parameter Selection (PPS) is a method that decomposes global parameter spaces into incremental subsets for targeted model adaptation and improved inference.
• It is applied in interval analysis, continual learning, and penalized model selection, enabling sharper uncertainty quantification, reduced catastrophic forgetting, and consistent variable selection.
• PPS frameworks utilize iterative, derivative-guided parameter subdivision to achieve empirical gains in computational efficiency and model performance.

Progressive Parameter Selection (PPS) refers to a family of algorithms and strategies for adaptive, data-driven allocation, subdivision, or selection of parameters across inference, learning, or optimization tasks. At its core, PPS decomposes the global parameter space or solution path into subsets or progressive increments, guiding model adaptation, feature selection, or uncertainty analysis via strategic evaluation and subdivision. PPS unifies diverse instantiations across interval analysis for inverse problems (Shary et al., 2020), continual learning under distribution shift (Li et al., 17 Dec 2025), and statistical model selection over penalization paths (Liu et al., 2016). This concept underpins methods for maximizing inference accuracy, minimizing catastrophic forgetting, and improving variable selection consistency by leveraging incremental or partitioned parameter updates.

1. Foundational Concepts and Formal Definitions

PPS entails decomposing the parameter set—whether coefficients in a model, trainable weights per task, or solution path grid points—into subsets or intervals. Each subset is then processed progressively: either isolated for individual adaptation, subdivided for sharper enclosures, or assessed for relevance at varying levels of penalization.

• In interval least-squares (Shary et al., 2020), PPS is formalized as Partitioning of the Parameter Set, systematically splitting uncertain elements of $A$ and $b$ within $Ax=b$ to confine the set of solutions $\Xi_{lsq}([A],[b])$ as tightly as possible.
• In continual learning (Li et al., 17 Dec 2025), PPS means allocating a fresh, task-specific vector $P_m$ per task $T_m$, with all prior $P_{1},...,P_{m-1}$ frozen to preserve subspace isolation.
• In penalized model selection (Liu et al., 2016), PPS is realized as Selection by Partitioning the Solution Paths (SPSP): features are classified at each value along the penalty path, aggregating relevance decisions rather than relying on a single global $\lambda$.

Objective functions formalize the PPS principle:

• For continual learning, the task $T_m$ is associated with loss:

$$L_P(\theta_{P_m}) = -\sum_{(x, y) \in T_m} \log p(y\,|\,[P_m, ..., P_1, x]\,; \theta)$$

with total loss $L = L_{QA} + \lambda_P L_P$ over both real and synthetic data (Li et al., 17 Dec 2025).

• For model selection, SPSP seeks the set

$\hat{S} = \bigcup_{k=1}^K \{j: R_j(\lambda_k)=1\}$

selecting all variables marked "relevant" at any $\lambda_k$ along the solution path (Liu et al., 2016).

2. Algorithmic Procedures and Implementations

PPS algorithms are defined by a repeatable loop: at each stage, a subset/parameter vector is selected, assessed, potentially updated, and then fixed, with further refinements incrementally improving the estimate or task performance.

PPS in Interval Least Squares: ILSQ-PPS

The ILSQ-PPS algorithm (Shary et al., 2020) operates as follows:

• Maintain a list of interval systems ([Q], [r]) with computed lower bounds.
• At each step, the current system is subdivided along the parameter (matrix entry or right-hand side) whose uncertainty most affects the target variable—measured via interval derivatives.
• Stopping criterion: subdivision continues until all widths are smaller than $\epsilon$, yielding a sharp enclosure.

PPS for Continual Learning: PPSEBM

Within PPSEBM (Li et al., 17 Dec 2025):

• A fresh set of parameters $P_m$ is allocated for each new task $T_m$, trained only on that task.
• Previous $P_j$ ($j < m$) are frozen and concatenated to the model's input.
• EBM-based generative replay injects pseudo-samples from earlier tasks into $T_m$'s training mix, enforcing that $P_m$ learns new knowledge without catastrophic forgetting.
• Pseudocode outlines this process, including parameter allocation, data augmentation, and optimization restricted to the active subspace.

PPS in Penalized Model Selection: SPSP

SPSP (Liu et al., 2016) systematically:

• Evaluates coefficients at each $\lambda$ along the penalization path.
• Sorts and identifies significant adjacent gaps among coefficient magnitudes.
• Partitions variables into relevant/irrelevant per $\lambda_k$ and finally aggregates selection decisions across all $\lambda$.

3. Theoretical Properties and Statistical Guarantees

PPS-based frameworks establish desirable convergence, accuracy, and consistency properties:

• In interval analysis, inclusion monotonicity ensures outer enclosures only shrink as subdivisions progress, and convergence to the minimal enclosures follows under exact solvers as $\epsilon\to0$; however computational effort grows exponentially in the worst case (Shary et al., 2020).
• For continual learning, PPS ensures parameter isolation: each $P_m$ encodes task-specific knowledge, preventing destructive interference. The EBM-based generative replay couples with PPS by aligning $P_m$ updates with distributions of past data (Li et al., 17 Dec 2025).
• For SPSP, under compatibility or restricted eigenvalue conditions, selection along the solution path is consistent, requiring weaker assumptions than the irrepresentable condition. SPSP selects all true signals with high probability, and achieves favorable tradeoffs between false positives (FP) and false negatives (FN) (Liu et al., 2016).

4. Empirical Evaluation and Benchmark Comparisons

PPS variants have demonstrated empirically robust performance across diverse domains:

• Interval Least Squares: ILSQ-PPS yields sharper outer enclosures for the solution set than Gay's, Bentbib's, or HBR methods under varying uncertainty shapes and problem sizes. For example, in the Rohn's 3×2 benchmark, ILSQ-PPS delivers $x_1 \in [-0.0375, 0.0363]$, $x_2 \in [0.9467, 1.0543]$ compared to the HBR's looser bounds (Shary et al., 2020).
• Continual Learning: PPSEBM with PPS achieves only 0.8% maximum forgetting over five tasks, outperforming Fine-tune (21.6% forgetting), EWC, MAS, GEM, and LAMOL, and approaches the multitask upper bound on decaNLP benchmarks (Li et al., 17 Dec 2025).
• Model Selection: SPSP methods avoid over-selection seen in CV or AIC/BIC tuning, achieving substantially lower FP and model error in high-dimensional (p≫n) regimes and retaining minimal true features (e.g., only 4 genes in TCGA data), with smallest mean prediction error (Liu et al., 2016).

PPS Method	Domain	Main Strength
ILSQ-PPS	Interval analysis	Sharp outer enclosure of solution sets
PPSEBM (PPS)	Continual learning	Robustness to catastrophic forgetting
SPSP	Model selection	Consistency, low FP/FN, minimal model error

5. Extensions and Generalizations

PPS methodology generalizes beyond canonical settings:

• SPSP applies directly to strictly convex penalties (e.g., ridge regression), nonconvex penalties (SCAD, MCP), generalized linear models, Gaussian graphical models, and Cox models, by leveraging the same partitioning and selection framework (Liu et al., 2016).
• The separation of parameter subspaces in continual learning is compatible with prompt-tuning and soft prompting for NLP transformers.
• PPS in interval analysis leverages adaptive splitting heuristics, e.g., splitting along parameters with largest $$|\partial x_\nu / \partial s| \cdot \mathrm{wid}(s)$$ to accelerate local convergence (Shary et al., 2020).

A plausible implication is that the common structure of sequentially subdividing or partitioning parameter spaces underpins algorithmic advances in uncertainty quantification, feature selection, and lifelong learning.

6. Computational Complexity and Practical Considerations

• In interval least squares, each subdivision requires $O((m+n)^3)$ operations, with the number of subdivisions $N$ scaling exponentially in the worst case but reduced by derivative-guided splitting (Shary et al., 2020).
• SPSP incurs minimal overhead: partitioning at each $\lambda_k$ is $O(p\log p)$, and no cross-validation is needed, resulting in an order-of-magnitude speedup over stability selection in practice (Liu et al., 2016).
• In continual learning, PPS limits parameter growth (only $O(Mp_\text{len})$ extra parameters for $M$ tasks), and EBM replay further regularizes without requiring task labels at test time (Li et al., 17 Dec 2025).

7. Impact and Outlook

PPS fundamentally changes how parameters are managed in learning and inference:

• By enabling sharper uncertainty quantification for interval problems,
• By protecting learned representations from forgetting in continual learning settings,
• By providing data-adaptive, consistent variable selection without reliance on a single penalty tuning parameter.

PPS continues to inform algorithmic approaches in robust statistics, interpretable machine learning, and incremental model updating, and its partitioning principles stand as a unifying concept for sequentially-structured, resource-efficient learning and inference across domains (Shary et al., 2020, Li et al., 17 Dec 2025, Liu et al., 2016).