--- title: Subsample-and-Aggregate Methods url: https://www.emergentmind.com/topics/subsample-and-aggregate type: topic --- # Subsample-and-Aggregate Methods Subsample-and-aggregate denotes a family of procedures in which a dataset is broken into smaller subsamples, a local statistic is computed on each subsample, and those local outputs are combined by averaging, noise addition, or another aggregation rule. In one canonical big-data formulation, for i.i.d. observations \(Z_1,\dots,Z_N\sim F_Z\), \(m_N\) subsamples of size \(k_N\ll N\) are drawn and the aggregate estimator is \[ \hat\theta_{k_N,m_N}=\frac1{m_N}\sum_{j=1}^{m_N}\hat\theta_{k_N,s^{(j)}}. \] This same pattern appears in subbagging for memory-constrained estimation [2103.00631], in block-based subagging for scalable inference [2112.06434], in adaptive data analysis through subsampling queries that output few bits [2302.08661], in time-series differential privacy through subsampling and noisy re-stitching in the time domain [2201.04762], and in recent private aggregation schemes based on robust winsorization [2501.14095]. Across these settings, subsample-and-aggregate is not a single algorithm but a methodological template for trading full-sample computation against memory, privacy, robustness, and generalization. ## 1. Canonical forms and notation In the random-subsample formulation studied by Zou et al., the full dataset has size \(N\), the parameter \(\theta_0\in\mathbb R^d\) is defined by estimating equations \(E[\psi_{\theta_0}(Z)]=0\), and the analyst draws \(m_N\) subsets of size \(k_N\) from the collection \(S_N^{k_N}\) of all \(\binom{N}{k_N}\) subsets. Each subset is sampled uniformly without replacement within the subset and with replacement across subsets, so overlap between drawn subsamples is permitted. For each subsample \(s\), one computes the Z-estimator \(\hat\theta_{k_N,s}\) solving \(\sum_{i=1}^{k_N}\psi_\theta(Z_{s_i})=0\), and then averages the resulting estimators. When all \(\binom{N}{k_N}\) subsets are used, the construction is a complete U-statistic; when only \(m_N\) subsets are used, it is an incomplete U-statistic of infinite order [2103.00631]. A distinct but related formulation replaces random subsets by non-random block subsamples. Politis considers \(X_1,\dots,X_n\), a subsample size \(b\), an overlap parameter \(h\ge 1\), and the blocks \[ B_j=\{X_{(j-1)h+1},\dots,X_{(j-1)h+b}\},\qquad q=\Bigl\lfloor\frac{n-b}{h}\Bigr\rfloor+1. \] The subagging estimator is then the simple average \(\hat\theta_{b,n}^{\rm subagg}=\frac1q\sum_{j=1}^q\hat\theta_{b,j}\), where \(\hat\theta_{b,j}=T_b(B_j)\). The same block collection also supports subsampling-based distribution estimation through \[ L_{n,b,h}(x)=\frac1q\sum_{j=1}^q 1\!\left\{T_b(B_j)-T_n(X_1^n)\le x\right\}. \] This shows that subsample-and-aggregate can be instantiated with either random or deterministic subsamples [2112.06434]. A third canonical description appears in the privacy literature: classical subsample-and-aggregate partitions the rows of a dataset into random subsets, computes a local statistic on each, then aggregates them with noise. The time-domain mechanism of Huang et al. takes an analogous view in the time dimension rather than the row dimension, replacing row subsampling by random selection of time indices [2201.04762]. This suggests that the essential object is the decomposition-and-recombination pattern, not a fixed sampling geometry. ## 2. Statistical subbagging under memory constraints The subbagging theory in Zou et al. is built around the asymptotic regime \(k_N\to\infty\), \(m_N\to\infty\), and \(k_N\ll N\), with regularity conditions C1–C4 on smoothness, moments, and nonsingularity. Defining \[ V_0=E\!\left[\frac{\partial\psi_{\theta_0}(Z)}{\partial\theta^\top}\right],\qquad \Sigma_0=\operatorname{Var}[\psi_{\theta_0}(Z)],\qquad \alpha=\lim_{N\to\infty}\frac{k_Nm_N}{N}\in(0,\infty], \] their main asymptotic result states that if \[ \frac{k_N}{N}\to 0,\qquad \frac{k_N}{\sqrt N}\to\infty,\qquad \frac{k_Nm_N}{N}\to\alpha, \] then \[ \sqrt N\bigl(\hat\theta_{k_N,m_N}-\theta_0\bigr)\xrightarrow{d} N\!\Bigl(0,\,(1+\tfrac1\alpha)\,V_0^{-1}\Sigma_0(V_0^{-1})^\top\Bigr). \] Relative to the full-sample estimator, the asymptotic variance is inflated by the factor \(1+1/\alpha\); when \(\alpha=\infty\), this inflation vanishes and the full-sample asymptotic variance is recovered [2103.00631]. The tuning parameters \(k_N\) and \(m_N\) encode the central trade-off. Larger \(k_N\) reduces the \(O(1/k_N)\) bias of each subsample estimator but increases memory use. Larger \(m_N\) decreases the variance inflation factor \(1/\alpha\) but increases CPU time. Algorithm 1 in the paper requires \(k_N\gg\sqrt N\), for example \(k_N\approx N^\beta\) with \(\beta\in(1/2,1)\), together with \(m_N\approx \alpha N/k_N\). Algorithm 2 adds first-order bias correction, reducing bias from \(O(1/k_N)\) to \(O(1/k_N^2)\), and thereby permits \(k_N\approx N^\beta\) with \(\beta\in(1/3,1/2)\) while retaining \(\sqrt N\)-consistency. The computational profile is explicitly favorable to out-of-core analysis. The procedure loads one subsample at a time, so memory is \(O(k_N)\); runtime is \(O(m_N\,T(k_N))\), where \(T(k_N)\) is the cost of solving the estimating equations on \(k_N\) points; subsampling itself costs \(O(m_Nk_N\log k_N)\); and the \(m_N\) solves can be performed independently. In the American airline example with \(N=118{,}914{,}459\) flight records, full-sample logistic regression required memory of approximately \(12\) GiB, whereas Algorithm 1 with \(k_N\approx 11{,}109\), \(\alpha=0.01\), and \(m_N\approx 2{,}140\) used about \(2.6\) MiB and produced estimates almost identical to the full-sample fit, with slightly larger standard errors by approximately the \(1/\sqrt\alpha\) factor predicted by theory. The example makes explicit that subbagging is not only asymptotically justified but practically motivated by memory constraints. ## 3. Scalable subagging and subsampling-based inference Subsampling was developed in the 1990s as a general method for estimating the sampling distribution of a statistic \(\hat\theta_n\), especially for confidence intervals and hypothesis tests, and Politis revisits it in the big-data regime where even drawing one large random subsample can be costly [2112.06434]. The key observation is that a carefully chosen deterministic set of block subsamples can replace random subset generation while preserving the inferential role of subsampling. Under the assumptions \(b\to\infty\), \(b/n\to 0\), \(T_n=n^\alpha\ell(n)\) with \(\alpha>0\), and \(h=O(b)\), the distribution estimator \(L_{n,b,h}(x)\) converges in probability to the limiting distribution \(F(x)\) of \(T_n(\hat\theta_n-\theta)\) at each continuity point \(x\), and \[ \operatorname{Var}\bigl(L_{n,b,h}(x)\bigr)=O(b/n). \] Thus \(O(n/b)\) block subsamples suffice to estimate the distribution of the full-sample statistic at essentially the same computational order as computing the statistic once. The same block family yields a subagging estimator whose behavior can be analyzed by bias-variance decomposition. When \(h\ge b\), the block statistics are independent and \[ E[\hat\theta_{b,n}^{\rm subagg}]=E[\hat\theta_b(B_1)],\qquad \operatorname{Var}[\hat\theta_{b,n}^{\rm subagg}]=\frac1q\operatorname{Var}[\hat\theta_b(B_1)]. \] Under Assumption B, with \(b\sim C_2n^\delta\) and \(h\sim C_3n^\eta\) for \(0<\eta<\delta<1\), the mean-squared error satisfies \[ \operatorname{MSE}\bigl(\hat\theta_{b,n}^{\rm subagg}\bigr) =O\!\Bigl(n^{-2\beta\delta}+n^{-(1-\eta-2\alpha\delta)}\Bigr). \] This can be tuned either to match the full-sample rate \(n^{-2\alpha}\) or, when the bias exponent \(\beta\) exceeds \(\alpha\), to exceed the full-sample convergence rate. A plausible implication is that aggregation over smaller estimators can improve not only computational feasibility but also the statistical rate, provided the bias structure is favorable. The paper also develops inference directly for the subagging estimator. Under a uniform \((2+\varepsilon)\)-moment condition, \[ n^{(1+\eta-2\alpha\delta)/2}\bigl(\hat\theta_{b,n}^{\rm subagg}-\theta\bigr)\rightsquigarrow N(C_\delta,\sigma^2), \] and \(\sigma^2\) can be estimated by the empirical variance of the block statistics. This yields confidence intervals centered at the subagged estimator rather than at the original \(\hat\theta_n\). One common misconception is that random subsampling is essential to subagging; Politis’ construction shows that, at least in this framework, appropriately chosen non-random blocks are sufficient. ## 4. Adaptive data analysis and information-theoretic aggregation In adaptive data analysis, the central difficulty is that an analyst may choose query \(t\) after observing the answers to queries \(1,\dots,t-1\), so standard nonadaptive concentration arguments no longer control generalization. Blanc studies a subsampling-based framework in which each query \(\phi:X^w\to Y\) operates on a random subsample of size \(w\) drawn without replacement from the dataset and returns a value in a small output space \(Y\). The main information-theoretic statement is \[ I\bigl(\mathbf S;(\mathbf Y_1,\dots,\mathbf Y_T)\bigr) \le \frac{4}{n}\,\mathbb E\!\Bigl[\sum_{t=1}^T w_t(|Y_t|-1)\Bigr], \] where \(\mathbf S\sim\mathcal D^n\), query \(t\) has arity \(w_t\), and \(\mathbf Y_t\sim \phi_t(\mathbf S)\). The paper then converts this mutual-information control into generalization guarantees for adaptively chosen test queries [2302.08661]. Two mechanisms are developed in detail. For statistical queries \(\phi_t:X\to[0,1]\), the mechanism repeatedly draws \(k\) points uniformly from the sample, flips Bernoulli coins with means \(\phi_t(x_i)\), and returns their average. With \[ k=\Theta\!\bigl(\ln(T/\delta)/\tau^2\bigr),\qquad n=O\!\Bigl(\frac{\sqrt{T\ln(T/\delta)\ln(1/\delta)}}{\tau^2}\Bigr), \] the answers satisfy, with probability at least \(1-\delta\), the simultaneous error bound \[ \bigl|y_t-\mathbb E_{x\sim\mathcal D}[\phi_t(x)]\bigr| \le \max\bigl(\tau^2,\tau\sqrt{\operatorname{Var}(\phi_t(\mathcal D))}\bigr). \] For approximate median queries, the sample is split into \(k\) disjoint blocks, each block is queried, and a binary-search procedure aggregates the resulting votes. This framework differs from classical estimator averaging, but it remains recognizably subsample-and-aggregate: subsampling introduces limited dependence on any single data point, and aggregation converts many low-information local outputs into a global answer. The paper’s limitations are equally instructive. It assumes the queries are subsampling queries, requires \(w=1\) for the high-confidence \(\log(1/\delta)\) bounds, and studies generalization rather than runtime. A common misconception is that subsampling is merely a computational heuristic; in this setting it is the primary mechanism controlling information leakage and hence adaptivity-induced bias. ## 5. Time-domain subsample-and-aggregate under differential privacy Huang et al. adapt subsample-and-aggregate to time-series data, where the sensitive object is not a row but an individual’s participation pattern across time. The data model observes \(n\) individuals over \(T\) discrete time steps, with participation vectors \(d_i\in\{0,1\}^T\) and aggregate count signal \[ x=\operatorname{count}(D)\in\mathbb N^T,\qquad x_t=\sum_{i=1}^n d_{i,t}. \] Adjacent datasets differ in one individual’s entire participation vector, and the paper assumes that no individual can contribute at more than \(I\) time steps: \[ \sup_i \sum_{t=1}^T d_{i,t}=I