Interactive Test-Driven Formalization

• Interactive test-driven formalization is a method that iteratively refines informal intent into formal, verifiable specifications using tests, counterexamples, and feedback loops.
• It spans applications from software unit testing and theorem proving to educational logic exercises, addressing the intent gap between natural language and precise behavior.
• Implementations like TiCoder and ACL2s demonstrate its practical impact by integrating automated test generation, dynamic candidate refinement, and stability metrics for improved reliability.

Interactive test-driven formalization is a methodology for refining informal intent into a checkable formal specification through iterative interaction with tests, counterexamples, models, or proof obligations. In the coding setting, it is motivated by the “intent gap” between natural-language requirements and precise program behavior; in educational logic systems and interactive theorem proving, closely related workflows use automatically generated feedback to help users converge on correct formulas, conjectures, or behavioral descriptions. Across these settings, the common structure is an interactive loop in which candidate formalizations are proposed, checked mechanically, and revised in response to failure cases or ambiguity (Lahiri, 17 Mar 2026).

1. Conceptual framework

A general formulation models intended behavior as a set $I \subseteq B$, candidate code as $C$, and a formal specification as $S$ with semantics $\mathrm{sem}(S)\subseteq B$. Satisfaction is written $C \models S$ iff $\mathrm{sem}(C)\subseteq \mathrm{sem}(S)$, and $S \models I$ iff $\mathrm{sem}(S)\subseteq I$. The discrepancy between intent and implementation is the intent gap,

$$\mathrm{gap}(I,C)=(I\setminus \mathrm{sem}(C))\cup(\mathrm{sem}(C)\setminus I).$$

Intent formalization is then the process $F:I\;(\text{informal}) \to S\;(\text{formal})$ such that $C$0 is checkable by tests or proofs and aims at high completeness with respect to intent and high soundness once code is generated. The rationale for making this process interactive is that fully automatic formalization is described as brittle, whereas test-driven interaction concentrates user effort on ambiguous behaviors (Lahiri, 17 Mar 2026).

In the program-synthesis formulation, the initial intent is $C$1, where $C$2 is a prefix, $C$3 a natural-language description, and $C$4 a function header. A generator produces a set $C$5 of candidate programs and a pool $C$6 of candidate tests, and the interactive objective is to identify a small approved set $C$7 such that the surviving candidates

$C$8

contain at least one correct implementation. This makes tests not merely an evaluation device but an explicit formalization artifact (Lahiri et al., 2022).

System	Formalized artifact	Automatic check
Diproche “Math Dictations”	first-order predicate logic over the real numbers	tableau-based implication checks against target formulas
Diproche “Game of Def”	$C$9 in $S$0 over a $S$1 grid	exhaustive model evaluation on 441 grid points
TiCoder	approved input/output tests from doc-string, header, and optional prefix	pruning and ranking of candidate code by validated tests
ACL2s	conjectures of the form $S$2	checkpoint testing with backtracking on false generalizations
Dualization-based testers	QAC or ITree specifications	derived validators and testers with formal soundness/completeness notions

The breadth of these systems shows that interactive test-driven formalization is not confined to one representation language. It appears in first-order logic exercises, executable unit-test selection, theorem proving, trace-based I/O specification, and nondeterministic protocol validation.

2. Educational and logic-oriented origins

Carl’s “Math Dictations” and “Game of Def,” implemented in the Diproche framework, provide a clear educational instance of the paradigm. Both are web-style exercises backed by Prolog programs, each with a single text-entry field and a “Check” button. In Math Dictations, students formalize a natural-language sentence such as “The real function $S$3 is strictly increasing” in a custom syntax close to LaTeX, using constructs such as Ax: φ, Ex: φ, (~φ), (φ&ψ), (φ v ψ), (φ->ψ), (φ<->ψ), and atomic formulas like a<b or f(x)=0. The checker parses the submission, performs a free-variable check, and then uses a tableau-based first-order prover to test $S$4 and $S$5 against each target formula, with a hard limit of 3 instantiations for each universal quantifier. In Game of Def, students write a single-free-variable formula in the geometric logic $S$6 over a finite $S$7 grid using predicates such as rechts, links, ueber, unter, nachbar, and dist(·,·)=dist(·,·), after which the system evaluates the formula over all 441 grid points and colors each cell green, red, or yellow to display overlap and miss. Both modules employ a four-fold necessity/sufficiency feedback regime, and the note characterizes the overall approach as reminiscent of unit testing in software engineering (Carl, 2020).

Westphal & Voigtländer present a related but more explicitly semantic framework for student-written interactive Haskell programs. Their specification language resembles regular expressions for lexical analysis, but augments them with global variables that store both current values and histories of prior inputs. Its semantics is defined by acceptance of execution traces over read and write actions ending in a distinguished stop symbol. Because the set of valid traces is usually infinite, the checker samples generalized traces, extracts the input subsequence, runs the student program in an instrumented rep-monad, normalizes the observed trace, and checks a coverage relation between the concrete execution and the sampled generalized trace. When coverage fails, the system reports the first mismatch. The same specification therefore acts as generator, oracle, and feedback source, and the paper explicitly notes further pedagogical uses such as more helpful feedback, sample-solution generation, and random exercise generation (Westphal et al., 2020).

These educational systems establish an important point: test-driven formalization did not originate only as a response to LLMs. It already existed as a method for teaching formal language use and for specifying interactive behavior in a way that supports immediate, localized correction.

3. LLM-mediated code generation

Interactive test-driven code generation makes the paradigm explicit for AI-assisted programming. The abstract algorithm begins by generating candidate code $S$8, candidate tests $S$9, and then refining both through StaticMutate, DynamicMutate, test ranking, user queries, and code ranking. TiCoder is the concrete implementation described in this line of work. It uses OpenAI Codex code-davinci-002, samples 100 code candidates and 50 tests with temperature=0.8 and top_p=0.95, and ranks tests by a discriminative heuristic that partitions the current code set into passing and failing subsets $\mathrm{sem}(S)\subseteq B$0 and $\mathrm{sem}(S)\subseteq B$1. The test score is

$\mathrm{sem}(S)\subseteq B$2

so the preferred test is the one that most evenly splits the candidate set. User answers are three-valued—YES, NO, or UNK—and the remaining code is reranked by the number of tests it passes. On MBPP and HumanEval, the 2022 evaluation reports pass@1@1 values of 70.73% on MBPP and 55.28% on HumanEval for TiCoder, compared with 48.24% and 30.49% for Codex@0; as the number of queries grows from $\mathrm{sem}(S)\subseteq B$3 to $\mathrm{sem}(S)\subseteq B$4, pass@1@m rises from 70.7% to 85.9% on MBPP and from 55.3% to 84.5% on HumanEval. The same paper reports that a user-approved test is found within at most five queries for 87.1% of MBPP with average queries approximately 1.7 and 95.7% of HumanEval with average queries approximately 1.5, and its ablations show performance drops when TestRanking, DynamicMutate, or the pass prompt strategy are removed (Lahiri et al., 2022).

The 2024 TiCoder study sharpens the notion of partial formalization by defining it as refinement of informal natural-language intent into a checkable specification fragment, typically a small suite of unit tests. Its workflow interleaves LLM-based test synthesis, user validation of proposed tests, and pruning or reranking of LLM-generated code. In the illustrative “LowerUnderscore” interaction, the system proposes tests such as assert text_lowercase_underscore("aa_bb_cc")==True, the user answers whether the proposed output is expected, and the system eliminates all candidate implementations inconsistent with that answer before selecting the next most discriminative test. In a mixed-methods user study with 15 programmers, correctness in the TiCoder-PassFail condition was 84% versus 40% in the control condition with $\mathrm{sem}(S)\subseteq B$5, time showed no significant slowdown at approximately 250–320 sec/task, and NASA-TLX cognitive-load was mean 28 versus 45 with $\mathrm{sem}(S)\subseteq B$6. In large-scale experiments on MBPP and HumanEval with text-davinci-003, code-davinci-002, CodeGen-6B, and CodeGen2.5-7B, the paper reports an average absolute improvement of 45.97% in pass@1 within 5 user interactions, along with automatic generation of accompanying unit tests; the reported average absolute improvements are +19.57% and +24.08% at $\mathrm{sem}(S)\subseteq B$7, and +38.43% and +41.44% at $\mathrm{sem}(S)\subseteq B$8, for TiCoder-PassFail and TiCoder-Output respectively (Fakhoury et al., 2024).

In these systems, tests serve two roles simultaneously: they are a clarification interface for the user and a ranking signal for the machine. The formalization remains lightweight, but it becomes executable and therefore operationally useful.

4. Integration with theorem proving and tester synthesis

Chamarthi, Dillinger, Kaufmann, and Manolios integrate testing into the ACL2 Sedan by coupling ACL2’s waterfall engine with random-and-bounded-exhaustive testing and a hint mechanism that allows test results to influence proof search. Conjectures are represented as implications $\mathrm{sem}(S)\subseteq B$9, and testing is applied not only at the top level but also to checkpoint subgoals produced after simplification, destructor elimination, case splitting, and related proof processes. Variable domains are inferred from defdata types, each with a corresponding enumerator $C \models S$0, and the default testing regime uses $C \models S$1 trials unless the user switches to bounded exhaustive enumeration. The engineering additions include override hints, a backtrack-hint mechanism, test-clause, test-gen-checkpoint, and global hook forms for automatically installing testing hints. A particularly important effect is testing-driven backtracking: if a generalization step yields a false conjecture, ACL2s can backtrack automatically and retry the original goal without generalization. The paper reports that this eliminated six bad generalizations in a single matrix-algebra proof, and it describes several years of freshman teaching use at Northeastern University in which automatic counterexamples helped novices identify missing hypotheses and false conjectures (Chamarthi et al., 2011).

Li’s work on “Testing by Dualization” generalizes the idea to interactive, nondeterministic systems. In the QAC setting, specifications are stateful server models over queries, answers, and internal choices, while validators process observed traces and are characterized by rejection-soundness and rejection-completeness. For asynchronous systems, the thesis uses Interaction Trees with Pure and Impure constructors. Dualization transforms a server-side specification into a tester by exchanging sends and receives and by translating internal choices into symbolic constraints maintained in a validation state. A second interpretation step handles unification and choice resolution, yielding a derived tester that can accept exactly the traces explainable by some symbolic assignment of nondeterministic choices. The framework also introduces a JSON-based intermediate representation and J-expressions for test-input generation from recorded traces, with constraint solving and backtracking over nondeterministic branches. The case studies include HTTP/1.1 conditional requests and a file synchronizer, and the resulting harnesses can shrink failures to minimal reproducible scripts (Li, 2022).

These systems move beyond ordinary unit testing. In ACL2s, tests become proof-search diagnostics. In dualization-based testing, a formal behavioral model is compiled into both validator and generator, so the interactive loop operates directly on protocol semantics rather than on isolated input/output examples.

5. Formal metrics and stability analysis

The survey treatment of intent formalization proposes explicit metrics for judging partial specifications. Given a test suite $C \models S$2 used as a proxy oracle, $C \models S$3 is the fraction of known valid examples accepted by $C \models S$4, and $C \models S$5 is the fraction of synthetic wrong outputs caught by $C \models S$6. With a distribution $C \models S$7 over candidate programs, $C \models S$8 is the probability that a randomly generated candidate satisfies $C \models S$9, while $\mathrm{sem}(C)\subseteq \mathrm{sem}(S)$0 is full inclusion of all intended behaviors, or an approximation thereof via a sufficiently large test set. Test-generation itself can be clause-guided and coverage-driven: the survey defines coverage over specification clauses and proposes a probabilistic fault model in which tests are chosen to maximize expected fault detection subject to a budget. Thresholds on these measures determine whether the system should ask the user to relax or strengthen the current specification (Lahiri, 17 Mar 2026).

The formal analysis of iterated TDD approaches the same territory from a different angle. It models a specification as a consistent set of input/output pairs $\mathrm{sem}(C)\subseteq \mathrm{sem}(S)$1, software as a computable total function $\mathrm{sem}(C)\subseteq \mathrm{sem}(S)$2, and tests as predicates $\mathrm{sem}(C)\subseteq \mathrm{sem}(S)$3 checking whether $\mathrm{sem}(C)\subseteq \mathrm{sem}(S)$4. It then derives equivalence classes from simple control-flow paths, defines coupling by path overlap, and studies the evolution of partitions $\mathrm{sem}(C)\subseteq \mathrm{sem}(S)$5 under successive specification chunks $\mathrm{sem}(C)\subseteq \mathrm{sem}(S)$6. The paper states three principal results: a correctness-production theorem when new classes do not overlap existing paths, a bounded-churn theorem expressed through the stability metric

$\mathrm{sem}(C)\subseteq \mathrm{sem}(S)$7

and a chaotic break-down result under non-trivial coupling, phrased in terms of a positive Lyapunov exponent. It also proposes concrete tool support for maintaining the current partition, recording coupling scores, generating one test for each newly created equivalence class, and warning when the stability number exceeds a threshold (Ruparel et al., 2024).

Taken together, these analyses indicate that interactive test-driven formalization can be studied both as a specification-validation problem and as a refinement-dynamics problem. The first concerns whether the emerging formal artifact actually captures intent; the second concerns whether iterative addition of tests and requirements preserves stability or induces churn.

6. Limitations, misconceptions, and research directions

A persistent misconception is that a small approved test suite already constitutes a complete formal semantics of intent. The TiCoder work explicitly defines partial formalization as a checkable specification fragment rather than a full specification, and the broader survey argues that there is no oracle for specification correctness other than the user. It follows that even when tests are executable and discriminative, they remain proxy artifacts whose adequacy must itself be validated; the same survey also characterizes fully automatic formalization of entire natural-language paragraphs as brittle (Fakhoury et al., 2024, Lahiri, 17 Mar 2026).

Several systems report concrete technical constraints. In Diproche, detailed user-study data, learning gains, satisfaction surveys, and run-time benchmarks had not yet been reported at publication time; the tableau prover had a known bug, universal instantiations were manually capped at 3, and Game of Def problems were deliberately designed so that two quantifiers suffice because deeper nesting leads to impractical runtimes. ACL2s reports extensive classroom and expert experience, but no large-scale controlled user study (Carl, 2020, Chamarthi et al., 2011).

For LLM-mediated code generation, the reported limitations include dependence on test-generation quality, idealized or simulated user feedback, non-trivial execution cost for LLM-to-code execution loops, benchmark intent descriptions that may be less open-ended than real specifications, and model-endpoint drift that complicates exact reproduction (Lahiri et al., 2022).

The stated research agenda is correspondingly broad. It includes scaling to side effects, mutable state, concurrency, and large codebases; supporting change-intent and compositionality over iterative development, merges, and library upgrades; improving cost-effective clarification by selecting the clauses or tests with the highest expected bug-prevention value; combining test-based, symbolic, mutation, and docstring-alignment metrics for automated specification validation; handling richer logics with recursive predicates and ghost state; and integrating the workflow into IDEs, issue trackers, code review, and CI/CD pipelines. In the educational setting, additional directions include extending Math Dictations to number theory and geometry, exploring decidable theories with o-minimality or quantifier elimination, and adopting LaTeX-style editing widgets.

Across these variants, interactive test-driven formalization is best understood not as a replacement for formal methods by testing, but as an incremental method for constructing, validating, and revising formal specifications under realistic ambiguity.