Inferring Change Points in Regression via Sample Weighting

Abstract: We study the problem of identifying change points in high-dimensional generalized linear models, and propose an approach based on sample-weighted empirical risk minimization. Our method, Weighted ERM, encodes priors on the change points via weights assigned to each sample, to obtain weighted versions of standard estimators such as M-estimators and maximum-likelihood estimators. Under mild assumptions on the data, we obtain a precise asymptotic characterization of the performance of our method for general Gaussian designs, in the high-dimensional limit where the number of samples and covariate dimension grow proportionally. We show how this characterization can be used to efficiently construct a posterior distribution over change points. Numerical experiments on both simulated and real data illustrate the efficacy of Weighted ERM compared to existing approaches, demonstrating that sample weights constructed with weakly informative priors can yield accurate change point estimators. Our method is implemented as an open-source package, weightederm, available in Python and R.

• The paper introduces WeightedERM, which integrates sample weighting into empirical risk minimization for detecting change points in high-dimensional GLMs.
• It provides a precise asymptotic analysis under Gaussian designs, demonstrating strong alignment between theoretical predictions and empirical performance.
• WeightedERM effectively incorporates prior information and uncertainty quantification, showcasing practical utility in financial, biomedical, and macroeconomic applications.

High-Dimensional Change Point Regression via Sample Weighting

Introduction

The paper "Inferring Change Points in Regression via Sample Weighting" (2604.11746) presents a unified framework for offline detection and inference of change points in high-dimensional generalized linear models (GLMs). The authors introduce a procedure, Weighted Empirical Risk Minimization (WeightedERM), which leverages prior information about change point locations through the use of sample-level weights in standard estimators (e.g., M-estimators, likelihood estimators). The theoretical analysis is conducted under high-dimensional asymptotics with Gaussian designs, and the empirical effectiveness is demonstrated across both simulated and real high-dimensional datasets.

Methodological Overview

The core contribution is WeightedERM, a versatile change point regression framework for sequences of observations $\{(y_i, \mathbf{x}_i)\}_{i=1}^n$, where the conditional distribution of $y_i$ may change at unknown positions (change points) in the observation stream. The model allows for $L^*-1$ unknown change points partitioning the sequence into $L^*$ regimes, each with its own regression parameter vector in a GLM.

WeightedERM operates by assigning to each observation $i$ and regime $\ell$ a weight $\pi_i^{(\ell)}$, representing the marginal prior probability that the $i$-th sample arises from regime $\ell$. These weights are generated based on the user's prior beliefs or constraints about change point configurations. For each regime $\ell$, a weighted empirical risk minimization is performed:

$y_i$0

where $y_i$1 is the loss associated with the GLM of interest (e.g., squared error, logistic loss). The change points themselves are then estimated by partitioning the sequence so as to minimize a composite risk over the fitted regime parameters.

Crucially, the method can incorporate arbitrarily informative or uninformative priors over change points, which directly translate into nontrivial weight patterns. Figures provided in the paper (see below) demonstrate these weights for different settings and priors.

Figure 1: Example of sample weights $y_i$2 based on regime marginals under a uniform location prior on two change points; $y_i$3, $y_i$4.

WeightedERM’s approach stands in contrast to prior methods that either fail to leverage prior information about change locations or focus primarily on sparse signals or sparse regime shifts.

Theoretical Insights: High-Dimensional Asymptotics

The authors provide a detailed asymptotic analysis in the proportional regime ($y_i$5, $y_i$6), accommodating general Gaussian design matrices with arbitrary covariance. Under weak regularity conditions, they derive a precise characterization of the joint distribution of WeightedERM regime estimates via a system of nonlinear fixed-point equations defined in terms of the sample weight profiles and the GLM loss function.

Concretely, the asymptotic joint risk and bias/covariance structure for the $y_i$7 regime estimators $y_i$8 can be explicitly calculated from these fixed-point equations. This yields sharp prediction formulas for finite-sample errors, and the analysis extends to derived quantities such as the limiting distribution of change point estimators under the (normalized) Hausdorff metric. The theory is validated by a close empirical match between theoretical and observed quantity distributions even at moderate dimension.

Uncertainty Quantification

Beyond point estimates, the paper develops an inferential framework for uncertainty quantification in change point locations. The key technical device is a "debiased" estimator for the natural parameters (via a one-step correction to the weighted M-estimate), whose distribution is asymptotically tractable. Leveraging this, the authors construct a posterior over possible change point configurations, which is shown to be asymptotically valid.

Figure 2: Posterior distributions on change point location generated by WeightedERM (left) versus theory (right), for a linear model with two change points. The close match validates asymptotic predictions.

A notable achievement is the computable nature of these posteriors in high dimensions, incorporating data-driven estimation of key quantities (e.g., signal strength) and demonstrating practical tractability.

Experimental Evaluation

The empirical results systematically compare WeightedERM (abbreviated as WERM) to state-of-the-art change point methods across multiple setups:

• Synthetic Sparse Linear Models: WERM outperforms or matches leading alternatives in localization error and number-of-change-point recovery across varying noise models (including heavy-tailed).
• Sparse-Difference Models: For cases where regime changes are sparse differences in the signal (as with the CHARCOAL or McScan methods), WERM remains competitive, despite not assuming such sparsity.
• High-Dimensional Logistic Regression: WERM demonstrates lower mean and maximum localization error relative to recent $y_i$9-penalized segmenting methods, with correct model selection performance across varying signal-to-noise settings.

In all considered scenarios, WERM is not only statistically accurate but also practically efficient, with computational runtime comparable to baselines.

Figure 3: Sample weights used in the logistic regression setting and theoretical/empirical error convergence for the detection of a single change point.

Additionally, real-world data experiments on macroeconomic (FRED-MD) and biomedical (myocardial infarction) datasets confirm WERM's practical utility and its ability to recover change points aligned with meaningful structural shifts recognized in subject-matter literature.

Numerical Highlights and Contradictory Claims

• Strong Empirical Match to Asymptotic Theory: Across both MSE and normalized Hausdorff error for change point detection, the observed finite-sample performance aligns closely with theoretical predictions, demonstrating quantitative reliability even at moderate $L^*-1$0.
• Competitive Without Sparsity: WERM achieves low localization error without requiring sparsity or sparsity in differences, challenging the conventional wisdom that such structure is necessary for high-dimensional change point estimation.
• Accurate Posterior Inference: The asymptotic posterior approximation approach enables valid uncertainty quantification for change points, with empirical posterior distributions robustly matching theoretical expectations.

Implications and Future Directions

WeightedERM provides a scalable and flexible solution to high-dimensional change point detection with rigorous performance guarantees under broad Gaussian design conditions. Its sample-weighting paradigm enables principled integration of prior knowledge or weak information about change points, facilitating practical deployment in scientific, financial, and biomedical applications where such prior cues commonly arise.

The theoretical results open the door to several promising directions:

• Penalized/Structured Extensions: Incorporation of structured priors or penalties (e.g., sparsity, group structure) directly in the WeightedERM optimization to further enhance performance in domain-specific settings.
• Non-Gaussian and Dependent Designs: Extension of the methodology and asymptotic guarantees to encompass designs with heavy tails, more general dependencies, or time series structure.
• Online/Streaming Change Point Detection: Adaptation of the sample-weighting principle for online settings, leveraging the posterior framework for rapid detection and uncertainty quantification in data streams.
• Fully Bayesian Nonparametric Approaches: Development of nonparametric priors (e.g., Dirichlet process mixtures) over both the number and locations of change points, with WeightedERM-inspired computational machinery.

Conclusion

This work rigorously establishes a flexible, theoretically grounded methodology for high-dimensional change point regression that seamlessly incorporates sample-level prior information via principled weighting. The combination of precise asymptotics, practical uncertainty quantification, and strong empirical results suggest that the WeightedERM paradigm will be valuable for both methodological research and applied inference in high-dimensional, heterogeneous data environments.