A Benchmark for Strategic Auditee Gaming Under Continuous Compliance Monitoring

Abstract: Continuous post-deployment compliance audits, mandated by emerging regulations such as the EU AI Act and Digital Services Act, create a class of strategic gaming distinct from the one-shot input/output gaming studied in prior work. Regulated systems can delay outcome reporting, drift their reports within plausible noise envelopes, exploit longitudinal sample attrition, and cherry-pick among ambiguous metric definitions. We formalize continuous auditing as a $T$-round Stackelberg game between an auditor that commits to a temporal policy and an adaptive auditee, and identify a structural feature of any noise-aware static-auditor design: a cover regime in which coverage gaps and granularity gaps cannot be closed simultaneously. We make this formal as Observation 1 and show that two minimal extension policies, each derived from the observation, close the regime along orthogonal axes: a sample-size-aware static rule (Periodic-with-floor) closes the granularity-failure case, while a history-conditioned suspicion-escalation policy closes the coverage-failure case for the naive Drift strategy -- and neither closes both, exactly as the observation predicts; an audit-aware OffAuditDrift strategy that exploits Stackelberg commitment defeats both. To support empirical study we contribute a non-additive harm decomposition (welfare loss $W$, coverage loss $C$) that exposes how attrition shifts harm from the regulator-accountable surface to a regulator-invisible one; an initial library of five auditee strategies (Delay, Drift, Cherry-pick, Attrition, OffAuditDrift) and five auditor policies, calibrated to summary statistics from published audits of the DSA Transparency Database; and a reproducible simulator with a small, extensible Python interface.

• The paper formalizes continuous compliance monitoring as a repeated Stackelberg game with evolving metrics, strategic reporting, sample-size manipulation, and auditor-selected audit rounds.
• The paper identifies a cover regime in which static audit schedules face an unavoidable coverage–granularity trade-off, with a drift interval of 0.031–0.098 under the benchmark’s parameters.
• The paper’s experiments show that OffAuditDrift evades every tested policy, while minimum sample-size floors and suspicion escalation address different failure modes; the reproducible 60-cell benchmark runs in under a minute.

Continuous post-deployment compliance obligations under the EU AI Act and Digital Services Act create a longitudinal audit relationship in which a regulated firm reports outcome metrics over many rounds while an auditor chooses which rounds to scrutinize. The paper "A Benchmark for Strategic Auditee Gaming Under Continuous Compliance Monitoring" (2605.06340) formalizes this relationship as a repeated game, identifies a structural trade-off inherent in a natural class of auditor designs, and contributes a runnable benchmark calibrated to published audits of the DSA Transparency Database. This essay summarizes the formalization, the theoretical observation, the empirical results, and the artifact, and states the paper's principal limitations.

Motivation and positioning

Prior formal work on strategic behavior in machine learning—strategic classification, performative prediction, and multi-resource audit games—models essentially one-shot interactions: a single classifier facing strategic agents, or a single audit allocation across one population. Applied audit-ecosystem research clarifies what auditors need in terms of access but typically assumes an honest-but-curious auditee. The paper argues that none of this work captures the discretion a continuously audited firm has over the timing, definition, and population coverage of its own reports. Empirical motivation comes from documented inconsistencies in DSA Transparency Database reporting that persist across six-month cycles and from platform-side API restrictions that create audit blind-spots on major platforms.

The paper's contribution is therefore a translation effort: it ports the continuous-auditing intuition from the accounting literature into a game-theoretic, ML-friendly, runnable environment, in the format of a benchmark contribution (documented environment, strategy taxonomy, baselines, reproducibility scaffolding).

Formal setup

The benchmark models continuous compliance auditing as a $T$-round Stackelberg game. A latent fairness metric evolves by clipped Gaussian drift; each round the auditee reports a metric value and a sample size $n_t$; the auditor audits a subset of rounds according to a committed policy. Detection on audited rounds uses a noise-aware Wald-style threshold: a gap is flagged when it exceeds $\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$ with $z = 1.96$. The rule's key property is that the tolerance band scales with the sample size the auditee itself claims: approximately $0.031$ at $n_t = 1000$ versus $0.098$ at $n_t = 100$ (at $\hat m = 0.5$).

Three information regimes are distinguished. Under R1 (public deterministic schedule, the headline setting), the auditee observes the committed audit set at $t=0$; under R2 the policy distribution is public but the realized draw is private; under R3 neither is disclosed. The headline experiments sit in R1, so the strongest adversarial result is best read as a worst-case-auditee bound.

Five auditee strategies are provided: Delay (report a lagged metric), Drift (add a fixed bias $n_t$0), Cherry-pick (report the maximum of five noisy candidate estimates), Attrition (restrict to a small verified sub-sample below an outcome threshold, widening the noise band, plus a small drift), and OffAuditDrift (drift only on rounds known to be unaudited). Five auditor policies span three static temporal-coverage baselines (One-shot, Periodic, Scheduled-random) and two extensions along orthogonal axes: a Min-sample floor (sample-size-conditioned static policy) and Suspicion-escalation (history-conditioned adaptive policy).

A distinctive methodological contribution is the non-additive harm decomposition: welfare loss $n_t$1 accumulates $n_t$2 over unaudited rounds (harm inside the counted-but-unverified report), while coverage loss $n_t$3 accumulates $n_t$4 (harm to the excluded population). The authors are explicit that $n_t$5 and $n_t$6 are disjoint accountability surfaces, not summands of a single total; the object of study is the reallocation of harm between a regulator-accountable and a regulator-invisible surface.

The cover regime

The paper's central structural result is stated as an Observation rather than a theorem, since the proof is immediate from the Wald threshold. Define the cover regime as the interval of drift magnitudes that exceed the full-sample threshold but fit within the small-sample threshold—at the benchmark parameters this is $n_t$7, which contains the default $n_t$8. Within this regime, any static temporal-coverage policy faces an unavoidable trade-off between two failure modes:

• Coverage failure: if any round is unaudited, an evading auditee drifts on an unaudited round and is never detected, accumulating positive welfare loss.
• Granularity failure: if every round is audited, an auditee reporting $n_t$9 widens the noise band beyond its drift and is again never detected.

Auditing more often cannot fix granularity failure, and no static policy fixes coverage failure; a minimum-sample-size floor lies outside the class and defeats Case 2 directly, while only adaptive, history-conditioned policies address Case 1. The observation is conditional, not universal: outside the regime it is silent, and the regime closes near metric boundaries where $\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$0 shrinks.

Empirical results

The headline grid (30 seeds, $\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$1) confirms the observation along orthogonal axes and yields several notable quantitative patterns.

Detection works only for above-envelope, non-adaptive drift. Against Drift, Scheduled-random detects in $\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$2 rounds, Periodic in $\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$3, One-shot in $\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$4, yet welfare losses of 400–550 person-rounds accumulate on unaudited rounds under the paper's no-remediation convention.

OffAuditDrift defeats every policy, including both extensions. By reporting honestly exactly on audited rounds, the audit-aware strategy achieves zero audited-round gap and $\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$5 (never detected) against all five policies, with welfare loss monotone in audit coverage ($\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$6). Suspicion-escalation fails here because its base-cadence audits never observe a non-zero gap, so escalation never triggers. This is the direct empirical witness for coverage failure and validates the paper's characterization of the static class's limits. Under R2, the authors note, this strategy would lose its zero-gap guarantee.

Attrition reallocates harm off the accountability surface. Against Attrition, all three static policies never detect despite a $\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$7 gap: welfare loss $\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$8 collapses from 400 (Drift) to 39, while coverage loss $\epsilon + z\sqrt{\hat m_t(1-\hat m_t)/n_t}$9 rises from 0 to 534. The nominal arithmetic sum rises by 43%, but the substantive point is that the regulator-observable component shrinks by roughly 90% while the off-record component is created—more individuals are exposed while the auditor sees a smaller problem. The Min-sample floor closes exactly this case, detecting on its first audited round ($z = 1.96$0) when $z = 1.96$1; Suspicion-escalation does not ($z = 1.96$2), because attrition keeps gaps inside the noise envelope.

Suspicion-escalation closes coverage failure only against naive drift. It detects Drift in $z = 1.96$3 rounds and halves $z = 1.96$4 to 200, but neither closes Attrition nor survives OffAuditDrift—exactly the orthogonal-axes prediction of the observation.

The Bonferroni-correction tax is small but visible on stochastic gaps. Cherry-pick's mean gap of $z = 1.96$5 is detected in $z = 1.96$6 rounds uncorrected but slips to $z = 1.96$7 under Bonferroni correction across $z = 1.96$8 audits—a roughly 1.5-round latency tax for family-wise error control that any strategy with gaps just above the full-sample threshold would incur. Fixed-gap Drift is unaffected.

False-positive validation. Under an honest-but-noisy auditee, per-round uncorrected FPR matches the nominal $z = 1.96$9 within Monte-Carlo error, uncorrected family-wise rates track $0.031$0 (empirical $0.031$1 at $0.031$2 against an analytical $0.031$3), and Bonferroni restores family-wise control ($0.031$4–$0.031$5).

Calibration and reproducibility. Simulator parameters are anchored to summary statistics from published DSA-TDB audits (a $0.031$6pp drift envelope, a $0.031$7 sample-size ratio, a 12-round horizon corresponding to six years of six-month cycles); the benchmark does not ingest DSA-TDB records directly. The artifact is pip-installable, runs the full 60-cell grid in under a minute on a laptop, includes a 21-test suite with regression tests pinning headline values, and exposes minimal Auditee/Auditor protocols for community-contributed strategies and policies.

Limitations and open questions

The paper is explicit about scope. The headline results sit in information regime R1, making the OffAuditDrift result a worst-case bound rather than a claim about randomized or secret schedules; an R2/R3 evaluation is a one-configuration extension but is not run. Strategies are canonical policy probes, not equilibrium best responses, and multi-game reputation effects are deliberately bracketed. The DSA calibration uses summary statistics rather than per-platform trace ingestion. The Min-sample floor exposes a structurally binary TPR/FPR trade-off—it cannot distinguish strategic attrition from a legitimately small operating population—and the paper identifies why precision floors are regulatorily available yet rarely prescribed: false-positive inflation on small populations, legal contestability of any specific floor value, and methodological discretion in existing DSA/AI Act obligations. Auditor-side reference noise is deferred to a v2 extension. The named open questions are concrete: whether an adaptive-auditor frontier (Bayesian posteriors over $0.031$8, gap) trajectories, SPRT, learned policies) can jointly close both failure modes and Pareto-dominate on the (detection, FPR) plane, and how the benchmark transfers to algorithmic hiring, credit scoring, and clinical decision support.

Conclusion

The paper contributes a formalization of continuous compliance auditing as a repeated Stackelberg game, a conditional structural result delineating exactly which static auditor class faces an unavoidable coverage–granularity trade-off, a non-additive harm decomposition showing how attrition shifts harm from a regulator-accountable to a regulator-invisible surface, and a calibrated, reproducible benchmark whose empirical results confirm the theory along orthogonal axes while demonstrating that a fully audit-aware adversary escapes all provided policies. Its principal value lies in making the failure modes of current cadence-centric audit designs precise and runnable, and in framing adaptive, sample-size-aware auditor design as the well-posed next problem.