Sequential Subspace Noise Injection

• Sequential Subspace Noise Injection is a method that injects structured noise sequentially into orthogonal subspaces to balance privacy, accuracy, and stability.
• It decomposes high-dimensional parameters into orthogonal blocks and applies Gaussian noise blockwise, preserving total variance while reducing per-step distortion.
• Empirical studies in certified unlearning, diffusion-based inverse problems, and distributed optimization demonstrate superior trade-offs between privacy, utility, and convergence.

Sequential subspace noise injection is a methodology for injecting carefully structured stochastic perturbations into high-dimensional optimization or learning procedures, with the goal of preserving rigorous privacy/security guarantees or improving stability and utility during constraint-driven updates. In contrast to isotropic or all-at-once noise addition, sequential subspace noise injection decomposes the target space into orthogonal subspaces and systematically schedules noise into separate blocks or low-dimensional subspaces on each sequential step. This approach has been applied across certified unlearning (Dolgova et al., 8 Jan 2026), diffusion-based inverse problem solving (Su et al., 24 Oct 2025), and privacy-preserving distributed optimization (Li et al., 2020), and is characterized by provable utility–privacy and accuracy–stability trade-offs superior to non-structured noise alternatives.

1. Core Principles and Definitions

Sequential subspace noise injection (SSNI) involves several key design steps:

• Subspace decomposition: A target parameter space $\mathbb{R}^d$ (e.g., neural net parameters, distributed multipliers, or state vectors) is partitioned into $k$ orthogonal subspaces $V_1, \ldots, V_k$, typically via orthonormal basis matrices $A_i \in \mathbb{R}^{d \times r_i}$ such that $A^\top A = I_d$ and $\mathbb{R}^d = V_1 \oplus \cdots \oplus V_k$.
• Block-wise noise scheduling: At each iteration or cycle, noise is injected into one specific subspace $V_i$ by perturbing the corresponding coordinates $B_i = A_i^\top \theta$ with a Gaussian sample $\xi_i \sim \mathcal{N}(0, \sigma^2 I_{r_i})$. All other blocks are held fixed for that cycle.
• Total variance preservation: Over $k$ cycles, each subspace receives an aggregate noise budget equal to what would have been assigned under isotropic noise injection, preserving the overall privacy or robustness certification.
• Sequential composition: By leveraging properties of Rényi differential privacy (RDP) or mutual-information-based obfuscation, the method guarantees additive privacy amplification or security certification through the summing of per-block privacy budgets (Dolgova et al., 8 Jan 2026, Li et al., 2020).

In applications to diffusion models or distributed optimization, the “subspace” is often structurally defined: localized in parameter blocks, low-dimensional stochastic codebooks, or the nullspace of communication topologies.

2. Methodological Instantiations

Certified Unlearning

In noisy fine-tuning for certified unlearning, standard approaches inject isotropic Gaussian noise $k$0 at each optimization step, often collapsing downstream accuracy. SSNI improves this by distributing the noise budget as follows:

• Orthogonal decomposition: $k$1 with each $k$2
• At each step, sample $k$3 and add only to $k$4, cycling through $k$5
• Per-step distortion is reduced, yet the cumulative $k$6-DP guarantee (under RDP analysis and the Shift Reduction Lemma) remains intact (Dolgova et al., 8 Jan 2026).

Diffusion-Based Inverse Problems

Noise Combination Sampling (NCS), interpreted as sequential subspace noise injection, operates within Denoising Diffusion Probabilistic Models:

• At each reverse step $k$7, a codebook $k$8 of $k$9 independent noise vectors defines a K-dimensional subspace $V_1, \ldots, V_k$0
• The optimal noise injection is determined via closed-form maximization of alignment to the measurement score $V_1, \ldots, V_k$1, leading to $V_1, \ldots, V_k$2 where $V_1, \ldots, V_k$3
• Sequential subspace projection maintains generative fidelity—especially with small step counts $V_1, \ldots, V_k$4—by preserving stochasticity along the learned data manifold (Su et al., 24 Oct 2025).

Privacy-Preserving Distributed Optimization

Subspace perturbation applies SSNI by identifying non-convergent orthogonal dual subspaces:

• For a consensus system, the dual variable $V_1, \ldots, V_k$5 is decomposed into a convergent subspace $V_1, \ldots, V_k$6 and orthogonal complement $V_1, \ldots, V_k$7.
• Additive noise $V_1, \ldots, V_k$8 is introduced at each step only into $V_1, \ldots, V_k$9: $A_i \in \mathbb{R}^{d \times r_i}$0, $A_i \in \mathbb{R}^{d \times r_i}$1
• Privacy of node-level data is guaranteed since only $A_i \in \mathbb{R}^{d \times r_i}$2 influences primal updates; convergence is unaffected, and mutual information leakage is bounded (Li et al., 2020).

3. Privacy and Utility Guarantees

Differential Privacy and Mutual Information

SSNI retains the overall differential privacy certificate by careful partitioning of noise variance:

• The sum of per-block Rényi divergences yields the same $A_i \in \mathbb{R}^{d \times r_i}$3 under $A_i \in \mathbb{R}^{d \times r_i}$4-block sequential composition as in isotropic NFT, provided $A_i \in \mathbb{R}^{d \times r_i}$5 for all $A_i \in \mathbb{R}^{d \times r_i}$6
• Shifted RDP and the Shift Reduction Lemma formalize how privacy is maintained across sequential, block-wise Gaussian noise application (Dolgova et al., 8 Jan 2026)
• In distributed optimization, per-iteration mutual information between private data and publicly visible trajectories can be suppressed arbitrarily by increasing variance in $A_i \in \mathbb{R}^{d \times r_i}$7—perfect privacy in the infinite noise limit is achieved without loss of convergence or accuracy (Li et al., 2020)

Accuracy and Stability

Empirical results demonstrate clear utility advantages:

• In certified unlearning, all-at-once NFT causes dramatic accuracy collapse (e.g., on CIFAR-10, test accuracy falls below 20%), while SSNI recovers accuracy close to full retraining even with large privacy budgets (Dolgova et al., 8 Jan 2026)
• For diffusion inverse problems, NCS/SSNI achieves large PSNR and FID improvements at small $A_i \in \mathbb{R}^{d \times r_i}$8 due to subspace guidance preserving the manifold structure and measurement consistency (Su et al., 24 Oct 2025)
• In distributed consensus and optimization, the primal recovery remains unaffected while privacy increases, unlike standard DP approaches that directly perturb output trajectories and degrade both speed and final accuracy (Li et al., 2020)

4. Algorithmic Specification and Implementation

A generic SSNI algorithm involves:

1. Input: decomposition parameters (e.g., $A_i \in \mathbb{R}^{d \times r_i}$9, basis matrices $A^\top A = I_d$0, noise variance $A^\top A = I_d$1), pretrained or initial state, privacy budget
2. Preprocessing: compute noise and step budgets to match privacy or utility constraints, decompose parameters into blocks
3. Main loop: for each subspace (block or codebook),
   • Freeze coordinates outside the targeted subspace (or operate only on dual variables in $A^\top A = I_d$2)
   • Update block with projected gradient and noise
   • Aggregate for next step
4. Postprocessing: (optional) unrestricted fine-tuning or further post-processing to maximize recovery

Specific instantiations (pseudocode, step-wise details) have been formalized for certified unlearning (Dolgova et al., 8 Jan 2026), diffusion models (Su et al., 24 Oct 2025), and distributed optimization (Li et al., 2020).

5. Applications and Empirical Observations

Application Domains

Domain	Noise Subspace Choice	Main Utility
Certified Unlearning (Dolgova et al., 8 Jan 2026)	Orthogonal parameter blocks	Preserves DP certificates; limits accuracy loss
Diffusion-based Inverse Problems (Su et al., 24 Oct 2025)	Gaussian codebook at each step	Improves generation stability and robustness
Distributed Optimization (Li et al., 2020)	Non-convergent dual nullspace	Guarantees privacy with no convergence penalty

• In image classification (MLP, ResNet-18, ViT-Tiny), SSNI achieves near-retraining-level accuracy and 100% unlearning (no membership inference signal), outperforming both pure NFT and alternative baselines such as influence functions or fine-tuning (Dolgova et al., 8 Jan 2026).
• In diffusion problem solving, especially with small denoising step counts, NCS-SSNI methods outperform classical score-guidance and stepwise-tuned solvers, with negligible additional computation and without manual parameter tuning (Su et al., 24 Oct 2025).
• In consensus and LASSO estimation over random graphs, introducing noise only in the dual’s non-convergent subspace yields the same mean-squared-error decay and final performance as in the non-private case, with privacy determined solely by noise variance and minimal communication overhead (Li et al., 2020).

6. Comparison to Alternative Approaches and Trade-Offs

Direct isotropic noise injection or DP perturbation:

• Causes substantial per-update distortion, frequently destroying solution utility or slowing convergence
• Requires higher noise variance for the same privacy level, with cumulative negative impact on learning or estimation

Sequential subspace noise injection:

• Spreads disturbance over multiple dimensions and iterations, preserving cumulative privacy or security
• Achieves privacy–utility or privacy–convergence trade-offs that are unattainable for classical DP or all-at-once perturbative approaches

A plausible implication is that SSNI plays a central role in future secure and private optimization methodologies where problem structure—be it spatial, spectral, or network-based—can be exploited for efficiency and fidelity.

7. Limitations and Implementation Considerations

• Orthogonal decomposition and subspace identification may incur additional computational or architectural cost, though these are typically outweighed by downstream utility gains
• For privacy against eavesdroppers, a single initial secure exchange of subspace-encoded noise coefficients may be required (Li et al., 2020)
• Hyperparameters (such as block size $A^\top A = I_d$3 or codebook size $A^\top A = I_d$4) should balance between flexibility and computational tractability, with $A^\top A = I_d$5 or $A^\top A = I_d$6 chosen so as to capture major directions of the score or gradient while minimizing overhead (Su et al., 24 Oct 2025, Dolgova et al., 8 Jan 2026)
• Empirical validation remains necessary for each new application to verify that orthogonal subspace partitioning aligns with the problem’s most sensitive or effective directions

Sequential subspace noise injection thus provides a rigorous, broadly applicable, and empirically validated tool for distributing stochasticity in algorithmic updates, balancing strict privacy or robustness guarantees with optimal or near-optimal downstream performance.