Adversarial Co-Evolution of Malware and Detection Models: A Bilevel Optimization Perspective

Abstract: Machine learning-based malware detectors are increasingly vulnerable to adversarial examples. Traditional defenses, such as one-shot adversarial training, often fail against adaptive attackers who use reinforcement learning to bypass detection. This paper proposes a robust defense framework based on bilevel optimization, explicitly modeling the strategic interaction between a defender and an attacker as an adversarial co-evolutionary process. We evaluate our approach using the MAB-malware framework against three distinct malware families: Mokes, Strab, and DCRat. Our experimental results demonstrate that while standard classifiers and basic adversarial retraining often remain vulnerable, showing evasion rates as high as 90 %, the proposed bilevel optimization approach consistently achieves near-total immunity, reducing evasion rates to 0 - 1.89 %. Furthermore, the iterative framework significantly increases the attacker's query complexity, raising the average cost of successful evasion by up to two orders of magnitude. These findings suggest that modeling the iterative cycle of attack and defense through bilevel optimization is essential for developing resilient malware detection systems capable of withstanding evolving adversarial threats.

• The paper develops a bilevel optimization approach modeling adversarial co-evolution between malware generators and detection systems to drastically reduce evasion rates.
• Experiments demonstrate that iterative adversarial retraining reduces evasion to near-zero while increasing attack query complexity by orders of magnitude.
• The methodology exposes the limitations of static defenses, guiding the shift towards robust, game-theoretic equilibrium approaches in malware detection.

Adversarial Co-Evolution of Malware and Detection Models via Bilevel Optimization

Problem Context and Motivation

Machine learning-based malware detection systems are increasingly challenged by rapidly evolving adversarial threats, exploiting distributional shifts and leveraging adaptive attack strategies. Classical defenses, including one-shot adversarial training, fail to sustain robustness against attackers utilizing reinforcement learning or other sequential optimization methodologies. The asymmetry between static learning-based defenders and continuously adapting attackers underlines persistent vulnerabilities, particularly in the era of polymorphic and evasive malware variants. This paper develops a formal bilevel optimization perspective to model the adversarial co-evolutionary interaction between an adaptive malware generator and a detection model, emphasizing iterative re-optimization and strategic equilibrium.

Technical Framework

Adversarial Attack Modeling

Adversarial modification of Portable Executable (PE) binaries is constrained by semantic preservation: generated adversarial samples must retain malicious payload functionality. The perturbation space comprises discrete, syntactic actions such as section injection, benign overlay addition, import manipulation, and header field modification. MAB-malware [song2021mabmalware] serves as the attack agent, operationalized as a multi-armed bandit RL agent discovering minimal and effective perturbation sets under strict execution integrity constraints.

Bilevel Optimization Formulation

The defender–attacker interaction is expressed as a bilevel optimization problem:

• Outer minimization (defender): Optimize classifier parameters $\theta$ over cumulative adversarial and benign samples to minimize detection loss.
• Inner maximization (attacker): For fixed $\theta$, maximize evasion success via optimal transformations $\delta^*$ drawn from the discrete space $\mathcal{S}(x)$.

This process simulates iterative best-response (IBR) dynamics: at each cycle, the defender retrains on past adversarial samples, while the attacker adapts by optimizing against the updated model. Iterative convergence is monitored via stability of the adversarial evasion rate and query complexity.

Experimental Methodology

Three defensive strategies are evaluated:

• (1) Baseline: Static Random Forest classifier trained solely on non-adversarial data.
• (2) Adversarial Retraining: One-shot inclusion of adversarially perturbed training samples followed by retraining.
• (3) Bilevel Optimization: Iterative defender–attacker co-adaptation, where adversarial samples generated in each round are incrementally incorporated into training, approximating game-theoretic equilibrium.

Evaluation spans three families (Mokes, Strab, DCRat), utilizing balanced subsets (250 benign, 250 malware) from EMBER and RawMal-TF datasets [anderson2018ember, balik2025rawmal]. Metrics are reported on both clean and adversarial test sets: accuracy, precision, recall, evasion rate (ER), and average attack queries.

Empirical Results

Robustness and Performance

• Baseline classifiers exhibit pronounced vulnerability; ERs reach up to 96%, with most adversarial samples evading detection in fewer than 32 queries.
• Adversarial retraining offers partial improvement for certain families (e.g., Mokes, ER drops to 6.25%) but fails for others (DCRat, ER persists at 90%), confirming the insufficiency of static defenses for complex adversarial surfaces.
• Bilevel optimization systematically hardens detection boundaries:
  • Achieves near-zero ER (0–1.89%) for all tested families.
  • Maintains or improves performance on clean data; no degradation in accuracy or recall.
  • Significantly escalates attack complexity: average queries increase by up to two orders of magnitude (e.g., Strab: 3118 queries vs. 31 for baseline).
  • Rapid convergence observed (typically 3–5 iterations), with the adversarial augmentation process saturating and yielding stable $\epsilon$-Nash equilibrium.

Family-Specific Dynamics

• Mokes reaches zero evasion after four iterations with cumulative adversarial augmentation stabilizing at 278 samples.
• DCRat, initially highly vulnerable, requires broad adversarial sample coverage (465 total) to achieve perfect immunity.
• Strab demonstrates a diverse adversarial spectrum, demanding deeper iterative cycles (371 samples over five iterations) to restrict evasion opportunities.

Theoretical and Practical Implications

Strategically modeling malware detection as an adversarial bilevel game fundamentally alters defense capabilities. The co-evolutionary framework exposes the limitations of one-shot adversarial training and enables the defender to anticipate and neutralize adaptive threat modalities, including RL-driven malware generators. Practically, the reduction of ER to negligible values and escalation of attacker operational costs indicate enhanced real-world resilience, particularly in settings where distribution shifts and attacker-defender information asymmetries are prominent.

Theoretically, this approach delivers actionable approximations to Nash equilibria in nonconvex, discrete malware perturbation spaces, and invites extension to more complex architectures (e.g., DNNs, GBDTs) and broader threat models (black-box, partial observability, dynamic allocation of defense resources).

Future Directions

Potential research avenues include:

• Extension of iterative bilevel optimization to deep learning-based detectors and integration with online learning paradigms.
• Scalability studies across larger, multi-family malware datasets to further stress-test convergence properties.
• Inclusion of attacker reward function heterogeneity and mixed-strategy optimization, to anticipate increasingly sophisticated evasion tactics.
• Game-theoretic exploration of constrained resource allocation and multi-defender scenarios.

Conclusion

This work rigorously demonstrates that classical and static adversarial training approaches are inadequate against strategic and adaptive malware attacks. Iterative bilevel optimization offers a principled framework for achieving robust defense, maximizing model immunity, and elevating attacker query complexity. The findings underscore the necessity of adopting adversarial co-evolutionary approaches in modern malware detection and pave the way for future explorations in adversarial learning and security games (2604.22569).