Dynamic multi-agent interactions as a remedy for benchmark saturation

Establish whether dynamic multi-agent attack-and-defense interactions provide a viable mechanism for counteracting cybersecurity benchmark saturation without continually retiring or engineering new challenge environments.

Background

The paper compares solo performance with contested multi-round play for GPT-5.6 Sol, Claude Opus 4.8, and GLM-5.3. Although all three agents solve 30 of 40 challenges in the solo setting, their capture rates decline across successive contested rounds as defenders patch vulnerabilities and attackers must adapt.

Based on this observed decline, the paper proposes—but does not establish—that dynamic interactions between attackers and defenders could preserve discriminative evaluation power as static benchmarks approach their performance ceiling. The unresolved issue is whether this mechanism can reliably mitigate saturation without requiring the continual replacement or redesign of challenge environments.