- The paper introduces deferred binding abstract syntax, reifying properties so users can implement custom generation, shrinking, fuzzing, feedback, and parallel-testing runners.
- The Rocq and Racket implementations re-create multiple runner designs with no observable performance overhead, while the Rocq runner outperformed QuickChick in 9 of 12 strategy-workload pairs.
- The approach enables practical experimentation: external shrinking produced counterexamples 2.66× smaller on average in one System F workload, and parallel testing achieved roughly 3× speedup on non-trivial tasks.
Motivation: runners are baked into frameworks
Property-based testing (PBT) frameworks expose two components: a property language (the API for writing universally quantified executable predicates) and a property runner (the loop that generates inputs, checks the predicate, shrinks counterexamples, and reports results). The authors observe that while modern frameworks offer extensive configuration of generators, shrinkers, printers, statistics, and test counts, one aspect is fundamentally unconfigurable: the runner itself. In QuickCheck and its descendants, the property type is an opaque function wrapper produced by a shallow embedding, so the generate-and-check and shrink-and-check loops cannot be reinterpreted by users. Consequently, alternative runners from the literature—coverage-guided fuzzing (FuzzChick, Crowbar, JQF, HypoFuzz), integrated shrinking (Hedgehog, Hypothesis, Falsify, RackCheck), targeted feedback-guided testing (Target, FuzzFactory, SlowFuzz, PerfFuzz, SQLancer), combinatorial coverage, and parallel testing (QuickerCheck)—each required a new framework or major redesign of library internals, with no reuse of components or properties across frameworks.
Deferred binding abstract syntax
The core proposal is a new mixed embedding for properties, termed deferred binding abstract syntax (DBAS). A property is reified as an inductive data structure indexed by a context of quantified types:
Forall extends the context with a type and carries annotations (generator, mutator, shrinker, printer) as functions from the current environment;Check takes a function binding the entire environment and returning a boolean;Implies gates the remainder of the property on a precondition.
The key move is to relocate the host-language binder from the quantification site to every use site. This is viable precisely because properties exhibit many quantifications and few variable uses, and it resolves the central obstacle of HOAS-style embeddings: without a concrete value for a quantified variable, one cannot recurse into a function-typed subterm, whereas DBAS exposes the structure of the "rest" of the property without needing generated values. Pure deep embeddings are rejected as impractical (users would have to learn a new language), and pure shallow embeddings are rejected as uninterpretable.
Dependently typed implementation in Rocq
The Rocq implementation extends QuickChick. Contexts are encoded as inductive lists of types with an interpretation function mapping a context to the tuple of its types; the property inductive type is indexed by Ctx, and Check receives a function from the interpreted context to bool. Dependent pattern matching enables user-level runners such as a genAndRun fixpoint, written with the standard convoy pattern to assist type inference. To recover ergonomics, the authors build a surface notation (ForAll e :- Expr, Check ...) backed by recursive Rocq notations, a typeclass Untuple that associates a predicate's type with its context plus a correctness proof, and a Derive Property command that synthesizes a DBAS property from a plain Rocq predicate. This shows the deeper embedding need not sacrifice the typeclass-driven, near-zero-effort experience of the shallow API.
Dynamically typed implementation in Racket
The Racket implementation encodes properties as structs (Forall, Implies, Check) with a dictionary of augments (#:contract, #:gen, #:shrink) attached per quantified variable; generators are treated as opaque values sampled via a user-supplied function, easing interoperability with RackCheck generators. Variable transformer macros and the isomorphism between properties and a list of quantifiers terminated by one Check allow a surface syntax nearly identical to shallow-embedding style, hiding environment plumbing. Contracts enforce typed boundaries on generated values where desired. The trade-off relative to Rocq is that well-formedness is enforced only by dynamic errors, but the language gains the ability to invoke functions on statically unknown types.
Evaluation
The evaluation has three parts, using the ETNA benchmark workloads (binary search trees, red-black trees, simply-typed lambda calculus, System F, IFC), measured with bucket charts averaged over repeated trials.
Expressiveness. The authors re-implement all runners surveyed—generational, integrated shrinking, coverage-guided fuzzing, targeted feedback, combinatorial, and parallel—entirely in user code, in both Rocq and Racket. The fuzzing runner is parameterized over a SeedPool/Utility typeclass interface abstracting search strategy and power schedule, generalizing classic coverage-guided fuzzing in the spirit of FuzzFactory.
Performance. Against QuickChick (Rocq) and RackCheck (Racket), the DBAS-based runners show no observable performance overhead: the Rocq DBAS runner outperforms QuickChick in 9 of 12 strategy/workload pairs with no notable differences in mean solve time. This is the paper's strongest practical claim: reified properties cost essentially nothing. Incidentally, porting workloads exposed and led to a fix of a bug in RackCheck's core.
Experiments enabled by programmability.
- Seed pool design: exploring six pool representations and four energy schedules (21 configurations) on the IFC workload, heap-based pools consistently solved tasks that other pools solved only intermittently; requiring all 5 of 5 campaigns to succeed, only heap pools consistently found counterexamples. The authors intend to submit this configuration as a QuickChick default.
- Integrated vs. external shrinking: on the System F workload, a simple type-based external shrinker reduced counterexamples to an average of 2.66× smaller (σ = 1.23), while RackCheck's integrated shrinker achieved only 1.04× (σ = 0.37), failing to shrink at all in 149 of 360 trials and growing inputs in 68. The paper explains this via integrated shrinking's size metric being defined over generator structure rather than the value, so "smaller randomness" need not yield smaller inputs. The authors are careful to note this is a pathological case and that integrated shrinking remains more user-friendly in typical settings; the contribution is that DBAS lets users switch per-context.
- Parallelism: a naive port of QuickerCheck's shared-size-counter algorithm to the Racket DBAS library took hours (versus a two-year, 50+-commit effort for the original Haskell implementation) and achieved roughly 3× speedup on the non-trivial tasks, with a 20% slowdown on tasks solvable in under a millisecond due to threading overhead—honest about where parallelism does not pay.
The paper situates DBAS against free generators (orthogonal: they reify generation, not the loop), finally-tagless and other mixed embeddings (Carette–Kiselyov–Shan; Tlön embeddings; embedding-by-unembedding; deeper/shallow hybrids), and surveys the shrinking and feedback capabilities of existing frameworks, noting that no widely used framework exposes a reified property representation supporting user-defined runners.
Limitations and open questions
The paper concedes several constraints. Ergonomics in the Rocq encoding depend on dependent pattern-matching artifacts (convoy patterns) that are hidden but present. The Racket encoding sacrifices static well-formedness guarantees. The central open problem is portability: Haskell is conspicuously unsupported because representing contexts requires either Rocq-style dependent types or Racket-style dynamic looseness, and mainstream languages lack both; the authors note that role-dependent types in Haskell may suffice but this is unverified. The seed-pool study is explicitly a demonstration of feasibility rather than a general empirical claim—its "clear winner" is established on a single workload with a single generator/mutator pairing, and generalization to other domains remains open. The shrinking comparison likewise rests on one workload and one generator, so it does not establish a general superiority of external shrinking.
Conclusion
The paper argues, and demonstrates, that the property runner—previously an immutable component of every PBT framework—can be moved into user space by reifying properties with deferred binding abstract syntax. Implementations in Rocq and Racket recover the ergonomics of shallow embeddings via dependent types, typeclasses, notations, and macros, incur no measurable performance cost, and enable rapid experimentation (seed-pool design, shrinking strategy selection, parallelization) that previously required new frameworks. The main unresolved question is how to bring the same programmability to languages with conventional type systems.