Generalizability of placement and protocol effects to production-scale specifications

Determine whether the observed rule-placement effect and the null effect of the specification-consumption protocol transfer to a production specification comprising hundreds of documents.

Background

The study evaluates a single, fictional Oracle-to-PostgreSQL migration specification organized into 18 documents. Its experiments find that the placement of a rule can affect implementer compliance, whereas adding the tested consumption protocol produces no measurable improvement.

The authors identify the transfer of these findings to a production specification with hundreds of documents as untested. Establishing whether the effects persist at that scale is important because larger specifications may introduce different retrieval, context-closure, organizational, and implementation behaviors. The issue is also connected to the study's single-case design and limited production realism.

References

Whether the placement effect and the protocol null transfer to a production specification of hundreds of documents is untested, and is the most important open question for the programme.

Measuring What a Specification Determines: A Formal Semantic-Block Model and an Execution-Judged Benchmark  (2608.19475 - Grynets et al., 19 Aug 2026) in Section XII, Threats to Validity