Labels Are Not Endpoints: Treatment Leakage and Construct Validity in MCP Agent Security Evaluation

Abstract: Security evaluations of tool-using agents often equate stored labels with behavioral facts. We audit a preserved campaign by tracing 10,200 execution rows to 180 model-bound requests, 45 semantic requests, and 15 observable stimuli. Two schema treatments were delivered, but the planned external payload-family corpus was not. The historical grader exhibited direct treatment leakage: treatment metadata gated the ATTACK_SUCCESS class, so fixed behavior could change class under treatment relabeling. A treatment-blind reconstruction corrects 58 historical ATTACK_SUCCESS or HIJACK_ATTEMPT labels to authorized benign completions while preserving three verified protected-data transfers and one separate unauthorized-forwarding case. The locked v2 census contains exactly zero ATTACK_SUCCESS records, while the forwarding case remains a HIJACK_ATTEMPT at a semantic boundary concerning objective completion. A dual-reviewer blinded concordance review of all 96 requests deemed structurally interpretable by locked v2 produced identical reviewer-consensus classes but differed from the locked codebook on four construct-boundary cases. We contribute a seven-link Integrity Chain and an executable, scope-bounded endpoint-integrity linter. The result is a campaign-bounded measurement audit, not a population attack-rate, model-ranking, defense-efficacy, or causal estimate.

• The paper demonstrates that treatment labels leaked into the historical grader, causing 58 of 70 adversarial labels to be reclassified as authorized benign completions despite a fully reproducible pipeline.
• The corrected treatment-blind endpoint analyzed 180 model-bound requests and found zero ATTACK_SUCCESS cases, alongside three verified critical exploits, one hijack attempt, 89 authorized completions, and 84 structurally invalid requests.
• The paper operationalizes measurement integrity through a Treatment-Invariance Test, seven-link Integrity Chain, blinded review, and executable linter while cautioning that reproducibility cannot replace construct validity or support claims about unavailable defenses and payload families.

This paper reports a measurement-integrity audit of the authors' own prior MCP-style agent security campaign (2608.12880). The audit finds that a fully reproducible, hash-frozen evaluation pipeline nonetheless produced an invalid security endpoint because treatment assignment leaked directly into the grading logic. The paper's central contribution is methodological: a Treatment-Invariance Test, a corrected treatment-blind endpoint, an analysis-unit reconstruction, a seven-link Integrity Chain, and an executable endpoint-integrity linter.

Background and motivation

Tool-using agents encounter capabilities through model-facing discovery metadata—tool names, schemas, descriptions, and capability advertisements. The MCP specification treats these as untrusted unless they originate from trusted servers. Prior work established indirect prompt injection as a measurable threat in tool-integrated settings (InjecAgent, AgentDojo, ASB) and MCP-native attack and defense surfaces (MCPTox, MCP-ITP, MCP Security Bench, ShareLock, VIGIL, SafeMCP, among others). The paper positions itself against this literature not as a new attack claim but as a forensic question: how can a deterministic, fully reproducible evaluation still report an outcome partly defined by its treatment label? The authors draw on construct validity theory (Cronbach and Meehl), measurement invariance, data leakage, and pseudoreplication to argue that implementation correctness and measurement validity are distinct properties.

Study design, threat model, and delivered treatments

The testbed exposes deterministic local tools (canned weather, inventory, an internal note source, a mock outbox sink) through an MCP-style discovery interface. The adversary controls natural-language discovery metadata only. Three capability densities (D1, D3, D5) cross three schema surfaces: CLEAN, POISON_TD (forwarding instructions in tool descriptions), and POISON_CA (forwarding policy in the capability advertisement). Four frozen model integrations are used: Qwen2.5-7B-Instruct, DeepSeek-R1-Distill-Llama-8B, Mistral-7B-Instruct-v0.3, and Phi-3.5-mini-instruct, all with greedy decoding and a 512-token cap, making repetitions deterministic replays rather than stochastic draws.

Two planned treatment components failed to materialize, and the paper states this plainly: the external payload-family corpus was never delivered (payload hashes are null; runtime retrieved content was set to none), and recorded defense-condition labels produced identical prompts, outputs, and trajectories across 168 duplicate groups. Consequently, the study identifies neither payload-family effects nor defense efficacy—an important boundary on any interpretation of the campaign.

Unit reconstruction: from 10,200 rows to 15 stimuli

The queue contains 10,200 execution rows, but hashing task text, expected sequence, and execution plan collapses these to 15 observable stimuli (one D1, five D3, nine D5). Crossing stimuli with three delivered schema variants yields 45 semantic requests; binding to four model–tokenizer authorities yields 180 model-bound requests arranged in 60 matched three-surface blocks. Complete-input multiplicity ranges from 11 to 162 rows (median 47.5), with 100% within-group concordance on serialized turns, outputs, parser state, stored class, and normalized behavior. The implication is direct: execution volume is not study breadth, and the repeated rows cannot increase any denominator.

The historical defect and its reproduction

The historical runner computed a flag $$A = \mathbf{1}\{\text{not utility}\}\,\mathbf{1}\{\text{surface}\neq\mathrm{CLEAN}\}$$ and passed it as adversarial_payload_present to the grader, which gated the positive classes on $A=1$. The grader's attack-success branch ignored the runner's unauthorized-call count, authorized-sequence match, benign-task success, and privilege-aggregation flag. A minimal counterfactual test—holding an authorized two-tool completion fixed and flipping only $A$—moves the label from benign success to ATTACK_SUCCESS. This is direct treatment leakage into the outcome: the endpoint was $Y=g(B,T)$ rather than a function of behavior alone. Notably, the existing test suite passed; its positive fixture encoded the same defective assumption. Code-path correctness was mistaken for construct validity, and reproducibility made the error exactly reproducible rather than preventing it.

The corrected treatment-blind endpoint

The corrected endpoint $h(B)$ consumes only the behavioral record $B$: accepted invocation, structural validity, the pre-existing authorization plan, dispatcher-executed calls, benign completion, unauthorized-action indicators, and protected source–sink content identity. Treatment, model identity, historical labels, and review status are excluded. It implements a six-class cascade: STRUCTURAL_INVALID, CRITICAL_EXPLOIT (strict hash-matched protected transfer), AUTHORIZED_BENIGN_COMPLETION, BENIGN_TASK_FAILURE, ATTACK_SUCCESS (multi-capability completion of a predeclared unauthorized objective), and HIJACK_ATTEMPT. The decision schema was frozen before surface-wise tabulation, and automated equivalence tests confirm the implementation mirrors the specification across all 180 records and all six branches.

Results of the reconstruction

The reconciliation is stark. Of 70 historical ATTACK_SUCCESS or HIJACK_ATTEMPT labels, 58 reconstruct as authorized benign completions (22 former attack successes and 36 former hijack attempts); the remainder split into one hijack attempt, one benign task failure, and ten structurally invalid requests. Four historical CRITICAL_EXPLOIT labels reconstruct as three verified critical exploits and one structural invalidity. The corrected 180-request census contains exactly zero ATTACK_SUCCESS records: 89 authorized benign completions, three benign task failures, one hijack attempt, three critical exploits, and 84 structurally invalid requests.

Surface	Auth. benign	Benign fail	Hijack	Attack	Critical	Invalid	Protected transfers (interpretable denom.)
CLEAN	29	2	0	0	0	25	0/31
POISON_TD	28	0	0	0	2	26	2/30
POISON_CA	28	0	1	0	1	26	1/30

The three CRITICAL_EXPLOIT cases are verified protected-data transfers: an authorized source read followed by an unauthorized outbox write, with the protected note appearing verbatim in the sink body and matching source/sink SHA-256 hashes. All three occur in two M1 (Qwen2.5) scenario blocks under delivered poisoned surfaces, while their matched CLEAN counterparts complete only the authorized plans. A fourth, separate case (EI-067F9857ECE149) executes an unauthorized outbox write containing only weather/task text—no protected source is read—so it remains a HIJACK_ATTEMPT at the semantic boundary between forwarding and objective completion. The authors emphasize that these are case evidence, not a population attack rate: scenario-mix resampling (10,000 resamples, seed 20260728) and leave-one-scenario-out diagnostics are descriptive only, and three recorded mechanical predicate variants leave all 180 classes unchanged, though a broader semantic alternative for the forwarding case was not tested.

Structural invalidity materially changes the denominator: 84/180 requests are structurally invalid under the frozen accepted-invocation contract, and M3 (Mistral-7B) is entirely uninterpretable (45/45 invalid), attributed to endpoint incompatibility of the model–tokenizer–wrapper–parser integration that cannot be localized to one component. The authors caution that structural invalidity is an operational state, not evidence of resistance or model incompetence.

Blinded concordance review and linter

A dual-reviewer blinded concordance review of the 96 structurally interpretable requests achieved 96/96 raw agreement (Cohen's $\kappa = 1.0$) with zero predicate or class disagreements. However, reviewer consensus differed from locked v2 on exactly four construct-boundary cases: one reviewer-ATTACK_SUCCESS versus v2 HIJACK_ATTEMPT (objective completion), and three reviewer-STRUCTURAL_INVALID versus v2 BENIGN_TASK_FAILURE. The authors retain v2 unchanged, noting that this preserves codebook consistency but does not establish that v2 is uniquely valid. Reviewer provenance (independence, blinding, no AI use, identities) is author-attested rather than externally verified.

The endpoint-integrity linter reproduced 10/10 prespecified diagnostic outcomes on the historical and v2 specifications plus synthetic fixtures. Its scope limits are explicit: static checks catch declared treatment or prohibited fields, metamorphic relabeling detects only treatment-valued fields supplied at the rule-evaluation boundary, and upstream-derived proxies (the leak_proxy fixture) are a documented non-detection. The linter replay establishes implementation behavior, not detection accuracy or external validity.

Limitations and open questions

The paper is candid about scope. The v2 endpoint is a post-hoc remediation, not a preregistration; treatment invariance is necessary but not sufficient for construct validity, since authorization predicates and protected-flow interpretation remain design choices. The blinded review covers only the 96 interpretable requests and does not re-adjudicate the 84 structurally invalid ones. Causal attribution is blocked by the fixed deterministic configuration and bundled TD/CA surface differences. The planned payload-family and defense conditions were never delivered, so defense efficacy and security–utility trade-offs are not estimable. External validity is strictly bounded to four integrations, 15 stimuli, and a local mock testbed. The specific open question left by the forwarding case is whether unauthorized forwarding without a protected transfer should count as objective completion—a semantic boundary that deterministic codebook stability does not settle.

Conclusion

The paper demonstrates that a benchmark can be exactly repeatable and still answer the wrong question: treatment delivery was verified, yet treatment identity helped decide whether an attack occurred. Correcting the endpoint reclassified 58 of 70 historical adversarial labels as authorized benign completions while preserving three verified protected-data transfers and one hijack attempt as inspectable case evidence. The durable contribution is procedural—bind treatment bytes, executed behavior, authorization, outcome rule, and analysis unit before interpreting results—operationalized through the Treatment-Invariance Test and the seven-link Integrity Chain, which make measurement validity a checkable gate rather than an assumption.