If-T: Benchmark for Type Narrowing

• If-T is a language-agnostic benchmark that defines type narrowing as the flow-sensitive refinement of union types through runtime tests.
• It categorizes narrowing into basic, compound, and advanced control flows, isolating behaviors like aliasing, nested conditionals, and custom predicates.
• Comparative evaluations across TypeScript, Flow, mypy, Pyright, and Typed Racket reveal trade-offs between precision, implementation complexity, and performance.

If-T is a language-agnostic design benchmark for type narrowing in gradually-typed languages. It is intended to characterize how a type system validates correct code and rejects incorrect code when programs rely on runtime tests to refine the usage of incoming data, rather than on datatype-driven design. Unlike a traditional performance-focused benchmark, If-T measures narrowing behavior; unlike a test suite, it does not require full conformance, and deviations are acceptable when justified by design considerations such as compile-time performance. The benchmark is guided by the literature on type narrowing, the documentation of gradual languages such as TypeScript, and experiments with typechecker implementations (Guo et al., 5 Aug 2025).

1. Motivation and problem setting

Type narrowing is central to gradual typing because dynamic languages such as JavaScript, Python, and Racket commonly use runtime tests to distinguish among possible value shapes. In these settings, a static checker must be flow-sensitive: after a test such as x is T, the checker should refine the type environment differently on the true and false branches. Without such a mechanism, the system is either too permissive, missing real errors, or too conservative, forcing manual casts throughout existing codebases (Guo et al., 5 Aug 2025).

If-T was introduced to address the absence of a shared, language-agnostic specification of desirable narrowing behavior. Existing systems, including TypeScript, Flow, mypy, Pyright, and Typed Racket, embody different mixtures of ad-hoc rules and formal mechanisms. Prior formalizations such as set-theoretic types and full occurrence typing are powerful but significantly more complex than a standard type system, and the practical payoff of that extra complexity had remained unclear in the absence of a common benchmark.

A recurring misconception that If-T explicitly rejects is that narrowing quality can be assessed purely through implementation complexity or throughput. Its purpose is instead comparative and diagnostic: it isolates concrete narrowing behaviors and lets designers decide whether supporting each behavior is worthwhile. A second misconception is that a benchmark of this kind must define a strict standard. If-T is deliberately framed as a design benchmark, not a normative conformance suite.

2. Formal basis: flow-sensitive typing, set-theoretic refinement, and occurrence typing

If-T takes as its conceptual baseline a standard typing environment $\Gamma$ that maps variables to possibly union types. A test x is T splits control flow and refines the environment asymmetrically:

• on the then-branch, $\Gamma(x)$ is refined to $\Gamma(x) \cap T$;
• on the else-branch, $\Gamma(x)$ is refined to $\Gamma(x) \setminus T$.

This is the core notion of flow-sensitive typing, or type narrowing. The benchmark is not itself a calculus, but it is organized around two canonical formal approaches that make these refinements explicit (Guo et al., 5 Aug 2025).

The first is the set-theoretic view of types, in which types are interpreted as sets of values and admit Boolean-algebra operators:

$$\tau ::= \text{Base} \mid \tau \cup \tau \mid \tau \cap \tau \mid \neg \tau \mid \tau_1 \setminus \tau_2$$

with subtyping defined by semantic inclusion,

$$\tau_1 \le \tau_2 \quad\text{iff}\quad \llbracket \tau_1 \rrbracket \subseteq \llbracket \tau_2 \rrbracket.$$

Under this interpretation, narrowing is set intersection on the positive branch and set difference on the negative branch.

The second is occurrence typing, exemplified by Typed Racket. Here a typing judgment for an expression $e$ has the form

$\Gamma\;\vdash\;e:\tau\;;\;\Phi^+\;|\;\Phi^-\;;\;o$

where $\tau$ is the ordinary type of $\Gamma(x)$0, $\Gamma(x)$1 and $\Gamma(x)$2 are sets of propositions describing refinements in the true and false branches, and $\Gamma(x)$3 is an objective or path identifying the part of a value that the expression inspects. The associated environment updates are expressed through erase, which replaces $\Gamma(x)$4 with either $\Gamma(x)$5 or $\Gamma(x)$6 depending on the proposition.

The benchmark reproduces a canonical conditional typing rule of this style:

$\Gamma(x)$7

and notes that logical connectives compose these propositions; for example, negation swaps $\Gamma(x)$8 and $\Gamma(x)$9. The significance of this formal backdrop is methodological: If-T uses minimal programs to ask which of these refinement capabilities an implementation actually realizes.

3. Benchmark structure and coverage

The core If-T suite contains 13 items. Each item has a “Success” example that should typecheck and a corresponding “Failure” example that should be rejected. The suite is divided into four categories that isolate distinct dimensions of narrowing behavior (Guo et al., 5 Aug 2025).

Category	Items
Basic Narrowing	positive, negative, connectives, nesting_body
Compound Structures	struct_fields, tuple_elements, tuple_length
Advanced Control Flow	alias, nesting_condition, merge_with_union
Custom Predicates	predicate_2way, predicate_1way, predicate_checked

The basic category tests whether systems refine on the true branch, on the false branch, through logical connectives such as and, or, and not, and through nested conditionals in the body. These programs represent the minimal expectations for a practical narrowing system. For example, positive asks whether a successful test such as x is String permits subsequent string-specific operations, while negative asks whether the complement information in the else-branch is propagated precisely enough to exclude non-members of the tested type.

The compound-structure category asks whether narrowing tracks paths into data structures. struct_fields tests whether field tests refine exactly that property. tuple_elements checks whether indexing and testing refine the selected element. tuple_length asks whether a test such as Tuple.length(x) is 2 narrows between alternative tuple types.

The advanced-control-flow category examines whether implementations preserve and compose refinement information beyond direct syntax. alias asks whether let-bound tests can carry through as guards when the variable is immutable. nesting_condition asks whether an inner test embedded in an if/else expression composes with the outer branch condition. merge_with_union requires that when two branches assign different refined types to the same variable, the join be the precise union rather than Top.

The custom-predicate category addresses user-defined narrowing functions. predicate_2way concerns symmetric predicates that refine positively and negatively. predicate_1way concerns asymmetric predicates of the form implies x is T, which refine only when true. predicate_checked asks whether the body of a user-defined predicate is itself checked against its declared narrowing annotation.

4. Implementations and comparative findings

The benchmark was implemented for five typecheckers: TypeScript, Flow, Typed Racket, mypy, and Pyright. The authors translated the 26 mini-programs into these systems and summarized the outcomes in a pass/fail table in which a dot indicates that the Success example typechecks and the Failure example is rejected soundly, while a cross denotes either unsound acceptance or over-conservative rejection (Guo et al., 5 Aug 2025).

The reported results show a common baseline. All five engines pass the four basic items. All also pass struct_fields and tuple_elements. The paper further reports that only the three JavaScript-based systems pass tuple_length, while Typed Racket requires a manual cdr/car style check instead of Tuple.length.

The advanced-control-flow items expose sharper divergence. Flow and mypy do not support alias, because they ignore tests bound to variables. Only Typed Racket supports nesting_condition; the JavaScript and Python checkers treat if (if e …): as a parsing statement rather than as an expression, and the benchmark notes that supporting this behavior would require moving to an expression-style typing judgment such as occurrence typing. On merge_with_union, mypy is described as overly conservative, collapsing the join to Top in one test.

The custom-predicate items distinguish systems even more clearly. All accept two-way predicates, but only Flow and Typed Racket check them thoroughly. Only Typed Racket and Pyright accept the one-way form implies x is T; TypeScript mandates that predicates refine both ways. TypeScript, mypy, and Pyright do not verify that a predicate’s body actually meets its annotation, so unsound definitions slip through, whereas Typed Racket rejects them.

These findings matter because If-T is not measuring language popularity or implementation maturity. It is measuring which refinement behaviors each checker chooses to support, and therefore where each design sits in the precision–complexity design space.

5. Design trade-offs exposed by If-T

If-T organizes narrowing implementations into three broad design levels. The first is ad-hoc narrowing, exemplified by TypeScript, Flow, Pyright, and mypy. Its advantages are minimal complexity and scalability in large existing codebases. Its limitations, according to the benchmark discussion, include missed support for alias-based tests, nested-condition expressions, one-way narrowing, and predicate-body soundness checks. In such systems, developers often compensate with boilerplate or duplicated logic (Guo et al., 5 Aug 2025).

The second is occurrence typing, represented by Typed Racket. Its principal advantage is uniform support across the benchmark’s dimensions, including custom asymmetric predicates, nested tests, and precise branch merges. Its cost is implementation complexity: the typing judgment must be extended with propositions $\Gamma(x) \cap T$0 and $\Gamma(x) \cap T$1 and objectives, together with the associated metatheory. If-T presents this as a direct precision-for-complexity trade-off rather than as an unconditional improvement.

The third is set-theoretic types, which If-T treats as an open frontier rather than a mainstream deployed solution in JavaScript or Python checkers. Their attraction lies in semantic completeness and in the ability to express arbitrarily refined predicates. Their drawback is computational: the benchmark notes that such systems are known to blow up compile times even on small examples, citing minutes on a flatten function.

A central contribution of If-T is therefore comparative calibration. It does not claim that every implementation should realize every narrowing feature. Instead, it provides concrete examples through which language designers can decide whether the additional annotation burden, implementation effort, and compile-time cost of a more expressive narrowing mechanism are justified.

6. Significance, misconceptions, and future directions

If-T provides a baseline for future research on narrowing systems. For researchers, it offers criteria for categorizing designs through a shared collection of positive and negative examples. For language designers, it makes the payoff of typechecker complexity visible in concrete programs. Both the benchmark and its implementations are described as freely available online, and the project includes a datasheet template for documenting how a system approaches narrowing (Guo et al., 5 Aug 2025).

Its significance is partly terminological and methodological. The benchmark clarifies that type narrowing is not merely a convenience feature for dynamic-language ergonomics; it is a foundational mechanism for validating gradually-typed programs whose structure is driven by runtime tests rather than by closed algebraic datatypes. It also clarifies that “better” narrowing is not a one-dimensional notion. The relevant axes include precision, annotation burden, and performance, and different systems occupy different points in that design space.

The paper outlines several future extensions. These include mutation and concurrency, where side effects or multithreading may invalidate assumptions established by earlier tests; subtyping tests, such as the distinction between isinstance and type(x) is T; deeper path-sensitive struct refinements, such as nested field tests; explicit performance benchmarks for particular features such as predicate_1way and nesting_condition; and broader language coverage, including Hack, Luau, Pyre, Sorbet, Static Python, and Typed Clojure. These directions suggest that If-T is intended not as a finished taxonomy but as an extensible benchmarking framework for the ongoing design of gradual type systems.