Replicable Random Design Regression

• Replicable random design regression is a framework that guarantees consistent, reproducible estimation across experiments with random data design.
• It leverages operator-based variable selection, penalty-driven estimators, and unbiased volume-rescaled sampling to mitigate randomness-induced bias.
• The methodology extends to autoreplicable algorithms in reinforcement learning and sequential design strategies for improved model stability.

Replicable random design regression refers to frameworks, algorithms, and statistical methodologies that guarantee consistent, reproducible, and stable estimation when learning models from data generated under random experimental or observational designs. The goal is to ensure that regression estimators, variable selection procedures, and uncertainty quantification are robust to the randomness inherent in both the design and subsequent data sampling, so that independent analyses on fresh data or re-analysis of the same experiment yield equivalent results, within quantifiable margins. Key contributions in replicable random design regression span operator-based variable selection, penalty-driven estimator design, unbiasedness constructions via determinantal point processes, PAC-Bayes risk guarantees, replicable rounding, and design-based inference for complex random outcomes.

1. Regression Models and Variable Selection with Random Designs

Random design regression settings typically involve models of the form

$Y = BX + \varepsilon$

where $X \in \mathbb{R}^p$ is a vector of random predictors, $Y \in \mathbb{R}^q$ is the response, $B$ is a $q \times p$ coefficient matrix, and $\varepsilon$ is random error independent of $X$. Variable selection in such frameworks is inherently more challenging than fixed design due to the randomness of $X$ and the need to identify relevant predictors whose coefficient vector is nonzero across responses.

The operator-based selection method (Mbina et al., 2015) reduces this to estimating a permutation $\sigma$ and a dimension $s$, based on the criterion

$\xi_K = \| V_{12} - V_1 \Pi_K V_{12} \|$

with $V_1 = \mathbb{E}[X \otimes X]$, $V_{12} = \mathbb{E}[Y \otimes X]$, and $\Pi_K$ is an orthogonal projector onto coordinates in $K$. Predictors are ranked so that

$$\xi_{K_{\sigma(1)}} \geq \cdots \geq \xi_{K_{\sigma(s)}} > 0,\quad \xi_{K_{\sigma(s+1)}} = \cdots = \xi_{K_{\sigma(p)}} = 0$$

and relevant indices are given by $\{ \sigma(1), \dots, \sigma(s) \}$. Empirical estimates $\hat{\xi}_K^{(n)}$ and penalized criteria

$$\hat{\phi}_i^{(n)} = \hat{\xi}_{K_i}^{(n)} + f_n(i),\quad \hat{\psi}_i^{(n)} = \hat{\xi}_{J_i}^{(n)} + g_n(\sigma(i))$$

are used, with consistent estimators for variable permutation and dimensionality. Simulation studies demonstrate lower prediction errors than prior methods for moderate or large sample sizes.

2. Unbiasedness and Replicability via Joint Sampling Constructs

Standard least squares estimators in random design are biased due to the randomness of the design matrix. By volume-rescaled sampling and determinantal point processes (DPP), unbiased estimators can be constructed regardless of the response model (Dereziński et al., 2019). The probability measure for volume-rescaled sampling is

$$P(X \in A) = \frac{ \mathbb{E}_{X \sim D^k} (\det(X^\top X)\ \mathbb{1}_{X \in A}) }{ \mathbb{E}_{X \sim D^k} \det(X^\top X) }$$

Such samples have diversity-enhancing properties, and critically

$\mathbb{E}[X^+ y] = w^*$

for the least squares estimator $w^*$, eliminating bias from design variability. Efficient algorithms are devised for practical generation of such designs with strong statistical guarantees and scalability.

3. PAC-Bayesian and Robust Generalization in Random Design Regression

Non-asymptotic PAC-Bayesian bounds for Gram matrix and least squares regression provide robust control of empirical risk (Catoni, 2016). Decomposition by truncation and tail bounding leads to guarantees of the form

$$\frac{1}{n}\sum W_i \leq \mathbb{E} W + \sqrt{u^{2-q} \frac{\mathbb{E} W^q}{n \epsilon}} + \frac{ (q-1)^{q-1} }{ q^q } u^{-(q-1)} \frac{ \mathbb{E} W^q }{ \epsilon }$$

with quantification via moment bounds and optimization of the truncation threshold. These results enable robust design of regression estimators under heavy-tailed or adversarial data distributions, enhancing replicability and stability with finite samples.

4. Replicable Algorithms and Complexity Measures

Replicable algorithms are those that, given independent samples from the design, produce identical or near-identical outputs (model parameters, predictions, certificates). List replicability requires outputs to lie in a small candidate set $L$; certificate replicability leverages auxiliary strings to stabilize outputs (Dixon et al., 2023). For regression tasks, rounding-based procedures and deterministic selection are used to constrain variability. Lower bounds establish the optimality of list size (e.g., no $k$-list replicable algorithm for coin bias estimation with $k < d+1$). These notions underpin rigorous frameworks for quantifying and achieving replicability in machine learning.

5. Sequential, Batch, and Design-Based Replication Strategies

Replication in simulation experiments and controlled optimization involves dynamic trade-offs between exploration (diversity of sampled conditions) and exploitation (replicating noisy conditions for variance reduction). Gaussian process surrogate modeling with non-uniform replication achieves computational savings and improved uncertainty quantification; adaptive sequential strategies (e.g., lookahead, rollout, horizon control) balance mean-squared prediction error globally (Binois et al., 2017). Batch Bayesian optimization further allows adaptive allocation of replication in parallel evaluation settings under heteroscedastic noise, with regret bounds and risk-sensitive extensions (Dai et al., 2023).

Rerandomization—a strategy to ensure covariate balance via quadratic forms—controls design-induced variance through accept/reject constraints of the form $v^\top A v \leq a$ where $v$ is the covariate mean difference and $A$ is a scaling matrix. Choice of $A$ (e.g., Euclidean or Mahalanobis) optimizes different aspects of design precision, with Euclidean rerandomization shown to be minimax optimal unless outcome-covariate relationships are exactly known (Schindl et al., 19 Mar 2024).

Design-based regression adjustment methods, including generalized regression estimators and Horvitz-Thompson extensions, provide variance bounding and estimation tools for arbitrary designs, including clustered or blocking structures. These frameworks enable precise, replicable inference for treatment effect estimation with explicit variance bounds (Middleton, 2018, Li et al., 2019).

6. Replicable Regression in Reinforcement Learning and Function Approximation

Recent advances extend replicable regression principles into reinforcement learning with linear function approximation (Eaton et al., 10 Sep 2025). The core tools are

• Replicable ridge regression, achieved via hypergrid-based rounding for weight vectors after fitting,
• Replicable uncentered covariance estimation, which ensures replicable matrix computation for exploration bonuses.

Sample complexity bounds quantify error rates: $$N \in \Omega\left( \frac{(B+Y)^2 d^3}{\lambda^2 \varepsilon^2 (\rho - 2\delta)^2} \log\left(\frac{1}{\delta}\right) \right)$$ where parameters are chosen to guarantee replicability (controlled deviation probability $\rho$) and statistical accuracy ($\varepsilon$). Such designs yield identical learned policies in RL for independent random samples and fixed internal randomness, resolving instability and reproducibility issues prevalent in classical RL implementations.

7. Optimism, Regularization, and Complexity in Model Selection

Asymptotic optimism—gap between test and training error—serves as a predictive complexity measure for regression models under random design (Luo et al., 18 Feb 2025). Classical estimators decompose optimism into signal and noise-dependent components. Regularized estimators (ridge, kernel ridge regression) admit precise asymptotic formulas for optimism, facilitating comparisons among linear, NTK, and deep neural network methods. Empirical studies indicate differing optimism trajectories among these model classes, especially under mis-specification or overparameterization, informing model selection and design for replicability and predictive reliability.

---

In conclusion, replicable random design regression synthesizes a set of principled approaches—operator criteria, penalty-driven selection, unbiased sampling, complexity measures, and design-aware strategy—to ensure stable, reproducible, and robust inference in regression analysis. These methodologies address challenges from estimator bias and variance under random design, algorithmic variability, experimental design-induced randomness, and complex dependencies in modern data settings. They underpin current standards for empirical reliability and methodological transparency in statistical learning, causal inference, and reinforcement learning.