Poisoning Attacks on Linear Regression

• The paper demonstrates how adversaries craft poisoning points using bilevel optimization to significantly boost prediction error measured by MSE.
• The study outlines attack strategies including availability and integrity (backdoor) attacks, with methods ranging from statistical heuristics to eigen-decomposition techniques.
• The work also reviews robust defenses such as iterative trimming, N-LID weighting, and certified regression to mitigate the impact of poisoned data in critical applications.

Poisoning attacks on linear regression refer to adversarial manipulations of the training data intended to subvert or degrade the accuracy, reliability, or integrity of regression models. In these attacks, the adversary crafts and injects specific training points—potentially with arbitrarily chosen feature vectors and response values—so that the trained model’s predictions deviate from those learned on uncontaminated data. Linear regression, despite its simplicity and foundational status in statistical learning, is highly susceptible to such attacks, which can induce large increases in prediction error, bias coefficient estimates, or even introduce backdoor behaviors under minimal poisoning budgets.

1. Attack Strategies and Optimization Frameworks

Poisoning attacks in linear regression have been formulated using various adversarial objectives, the two most prominent being:

• Availability Attacks: The adversary seeks to maximally increase the overall prediction error, typically measured by the Mean Squared Error (MSE) across the test set. For example, in (Jagielski et al., 2018), the attack is operationalized as a bilevel optimization problem with the outer objective maximizing validation set loss through carefully chosen poisoning points.
• Integrity (Backdoor) Attacks: The adversary aims to manipulate the model’s prediction on specific instances or subpopulations (backdoors) while keeping general performance largely unaffected. (Peinemann et al., 7 Aug 2025) rigorously demonstrates that a single poison sample—strategically crafted along a direction unused by the benign data—can inject a backdoor into linear regression without noticeable impact on benign performance.

The underlying optimization frameworks include:

• Bilevel Optimization: The adversary solves

$$\max_{\mathcal{D}_p} L(\mathcal{D}_{val}, \theta^*) \quad \text{subject to} \quad \theta^* = \arg\min_\theta J(\mathcal{D} \cup \mathcal{D}_p, \theta),$$

where $L$ is the outer loss and $J$ is typically the regularized training objective (Jagielski et al., 2018, Li et al., 2020).

• Closed-form and SDP Relaxations: When targeting specific coefficients, a Sherman-Morrison formula expresses the poisoned estimate, and the adversarial problem is reduced to a ratio of quadratic or quartic forms, often solved via eigen-decomposition or SDP relaxations (Li et al., 2020).
• Heuristic and Statistical Attacks: Fast, lightweight attacks can be constructed by sampling features from estimated data distributions and assigning responses near feasible boundaries (Jagielski et al., 2018); density-based heuristics that avoid bilevel computation are also effective (Cinà et al., 2021).

2. Attack Efficacy: Bounds, Trade-offs, and One-Poison Backdoors

The effectiveness and stealth of poisoning attacks are quantifiable in terms of induced risk, model deviation, and detectability:

• Error Amplification: Even injection of a small fraction (as low as 2%) of poisoned data can cause the MSE to increase by 50% or more (Müller et al., 2020), or, in adversarially sensitive domains such as healthcare (Warfarin dosage), shift clinical decisions by over 100% for most patients (Jagielski et al., 2018).
• Robustness Conditions: Linear learners are inherently robust to indiscriminate poisoning when class-wise data is well-separated with low variance and the constraint set for poisoning is small. Theoretical results in (Suya et al., 2023) explicitly link robustness to the ratio $|\gamma_1 - \gamma_2|/\sigma$ (class separation to variance) and the projected size of the feasible set.
• Backdoor Minimality: (Peinemann et al., 7 Aug 2025) proves the “one-poison hypothesis” for linear regression: a single poison sample, constructed along a direction unused by the benign data and with appropriate strength, can inject a high-accuracy backdoor with negligibly small effect on benign test error.

Attack Class	Optimization Strategy	Required Knowledge
Availability (MSE incr)	Bilevel/SDP/Statistical	Full to limited (black-box) access
Integrity (backdoor)	Directional insertion	Feature distribution moments
Indiscriminate	Constraint-bound extremal pts	Gaussian data, feasible set shape

3. High-dimensional and Multi-party Challenges

High-dimensionality and distributed learning present new challenges and vulnerabilities:

• Curse of Dimensionality: As established in (Hoang, 25 Sep 2024), when the number of features $D \gtrsim 169 H^2/P^2$ (with $H$ honest and $P$ poisoned samples), robust aggregation methods such as geometric median or clipped mean fail, allowing attackers to arbitrarily manipulate model parameters.
• Low-rank Structure: (Liu et al., 2016) shows that relaxing unrealistic sub-Gaussian/independence assumptions and relying on low-rank approximations yields efficient and provably robust regression, provided the number of adversarial examples remains below specific subspace-related thresholds.
• Multi-party Poisoning: Universal $(k,p)$-poisoning attacks (Mahloujifar et al., 2018) demonstrate that if an adversary can corrupt $k$ out of $m$ data sources and each modified data block departs from its honest distribution by at most $p$ in total variation, the induced risk can be amplified by $\Omega(p \cdot k/m)$. Universality ensures algorithm-agnostic vulnerability in federated or distributed settings.

4. Defenses: Robust Regression, Trimming, Certification, and Detection

A spectrum of defenses has been proposed, often trading computation, detection capabilities, and guarantees:

• Trimming and Iterative Selection: Methods such as TRIM (Jagielski et al., 2018, Wen et al., 2020, Müller et al., 2020) iteratively remove points with the highest residuals and retrain, providing MSE guarantees relative to the contamination rate ($\leq 1 + \alpha/(1-\alpha)$ times clean MSE for poisoning rate $\alpha$).
• Probabilistic Defenses: Proda (Wen et al., 2020) samples random subsets, estimating the likelihood of containing only clean points and retrains on the subset with minimum MSE, achieving logarithmic runtime and high resilience.
• LID-based Sample Weighting: The N-LID defense (Weerasinghe et al., 2020) calculates a local intrinsic dimensionality-based anomaly score to downweight suspected poison samples, outperforming robust regression and iterative trim in both accuracy and speed.
• Certified Regression: Using reductions to voting-based robust classification (Hammoudeh et al., 2022), regression prediction robustness can be numerically certified even without model-specific or data distributional assumptions. The use of the median as a robust aggregator enables pointwise guarantees under adversarial training set modification.
• Spectral Signatures and Hessian-based Detection: (Granziol et al., 21 May 2025) demonstrates that poisoned data induce a “spike” in the Hessian with respect to model input; spiked eigenvalues are used both for detection (without retraining) and for constructing remedial projections to eliminate the poison effect.

Defense Mechanism	Key Property	Computational Regime	Guarantees
Trimming/Proda	High MSE inlier selection	Iterative/logarithmic	MSE $\leq (1+\alpha/(1-\alpha))$
N-LID weight	LID-based anomaly scoring	Parallelizable	Up to 76% MSE reduction vs. naive
Certified Regression	Voting median, ensembling	Ensemble/partition	Pointwise attack-tolerance
Hessian spike detect	Input-Hessian spectral outlier	Lanczos, quick	Poison presence and location

5. Practical Ramifications and Applications

Poisoning attacks and defenses in linear regression have significant consequences:

• High-stakes Automation: Systems in medicine, finance, and power grid management, where regression predictions underpin safety and resource allocation, are particularly vulnerable: shifts in regression models due to adversarial data can have direct, measurable impacts on health and economic outcomes (Jagielski et al., 2018, Müller et al., 2020).
• Database Systems: Learned index structures, which rely on regression to approximate cumulative distribution functions (CDFs), are uniquely susceptible to global poisoning: even a single poisoned key can have cascading effects on query latency and correctness (Kornaropoulos et al., 2020).
• Mission-critical Controls: In linear dynamical systems, stealthy attacks constructed to avoid detection by standard statistical tests can dramatically increase parameter estimation error, undermining data-driven control (Russo, 2022).
• Data Provenance and Cleaning: The need for careful monitoring, provenance tracking, and (potentially) adversarial training emerges as essential—especially as large, untrusted, or distributed datasets become the norm.

6. Future Research Directions

The literature identifies several open problems and promising lines of exploration:

• Extension to Nonlinear Models: While linear regression admits closed-form and analytically tractable models for both attack and defense, corresponding techniques must be adapted or generalized for neural networks and other non-convex learners (Jagielski et al., 2018, Granziol et al., 21 May 2025).
• Adaptive, Online, and Federated Settings: Countermeasures effective in offline or batch settings may be circumvented in online, streaming, or federated learning. Online certification, robust incremental update rules, and real-time anomaly detection are active areas (Mahloujifar et al., 2018, Hoang, 25 Sep 2024).
• Combining Certification, Detection, and Heuristic Defenses: Integrated approaches—combining theoretical pointwise certification, dense detection via spectral signatures, and practical inlier selection—may yield tighter robustness and lower computational cost (Hammoudeh et al., 2022, Granziol et al., 21 May 2025).
• Fundamental Limits: The curse of dimensionality phenomenon (Hoang, 25 Sep 2024) suggests that as model size and data dimension increase, attacker power grows superlinearly, imposing intrinsic trade-offs between expressivity and robustness.

7. Summary Table: Taxonomy of Attacks and Defenses

Dimension	Attack/Defense	Mechanism	Notable Guarantee/Impact
Attack	Bilevel opt.	Joint feature+target design	Up to 100–300× MSE under real-world constraints
	Directional	Unused space backdoor	One-poison backdoor; negligible benign impact
	Statistical	Covariance/mean matching	Fast, black-box; less effective
Defense	Trimming/Proda	Iterative/ensemble selection	$(1+\alpha/(1-\alpha))$ factor on clean MSE; fast
	N-LID	LID-based reweighting	76% lower MSE vs. naive ridge; efficient
	Certification	Median + ensemble	Pointwise certifiable robustness
	Spectral	Hessian eigenvalue spike	Detection and remediation without model retraining

These collective findings establish that poisoning attacks pose fundamental threats to linear regression, but also that theoretically rigorous and practically viable defense mechanisms—decoupled from strong data assumptions—can offer substantial mitigation. The remaining research frontier focuses on closing expressivity-versus-robustness gaps, efficient defense scaling, and adversarial resilience in high-dimensional, adaptive, and distributed environments.