Sionnx: Automated ONNX Unit Test Generator

• Sionnx is an automatic unit test generator for ONNX operator conformance that uses formal specifications and systematic randomization to ensure comprehensive test coverage.
• It employs a dedicated Operator Specification Language (OSL) and LLVM TableGen to translate compact rules into exhaustive tests across types, dimensions, and boundary conditions.
• The system streamlines cross-framework validation by integrating Python reference algorithms, significantly improving coverage and reducing manual testing overhead.

Sionnx is an automatic unit test generator designed to verify Open Neural Network Exchange (ONNX) operator conformance using a rule-driven, specification-based approach. By coupling a formal specification language for operators with automatic test generation and reference algorithm integration, Sionnx delivers comprehensive and backend-agnostic validation pipelines for ONNX implementations (Cai et al., 2019).

1. Motivation for Automated ONNX Conformance Testing

Major machine-learning (ML) frameworks such as TensorFlow and PyTorch define their own semantics for core computational operators. ONNX provides a unified operator set for cross-framework portability, but the official conformance suite historically relies on a minimal set of manually-written test cases per operator. This deficiency leads to insufficient coverage of type, dimension, and boundary cases; makes it difficult to maintain tests for newly introduced operators; and burdens developers with lengthy, error-prone reference implementations.

Sionnx addresses these challenges by:

• Defining a compact yet complete rule set for each operator’s attributes and operand properties;
• Introducing a domain-specific language to declare these rules in a uniform manner;
• Automatically expanding those specifications, together with operator reference implementations, into extensive Python test suites;
• Employing a systematic randomization algorithm to guarantee broad coverage of data types, tensor shapes, and boundary cases.

2. Formal Specification of Operator Constraints

Every ONNX operator $\mathcal{O}$ is characterized by a collection of constraint rules:

$$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$

Where:

• $r_\text{attr}$ are rules over operator attributes;
• $r_\text{input}$ and $r_\text{output}$ are rules over tensor operands;
• $r_\text{prop}$ are implicit properties, e.g., broadcasting.

Each rule $r \in R$ is a conjunction of constraints on:

• Type: $$t \in \{ \mathrm{f16}, \mathrm{f32}, \mathrm{i32}, \ldots \}$$,
• Dimension: $d \in \mathbb{N}$ (dimension/rank),
• Value range: $v \in [v_\text{min}, v_\text{max}]$.

Attribute constraints are expressed as:

$$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$0

Input tensor constraints take the form:

$$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$1

This formalism provides the foundation for automatic, exhaustive test generation.

3. Operator Specification Language (OSL)

Sionnx introduces the Operator Specification Language (OSL), a concise DSL for declaring operator test requirements. OSL is formally defined using extended Backus–Naur Form (BNF):

$$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$9

Key OSL constructs:

• attr_name/type_list: attribute type and valid value intervals,
• basic_type_list/min_dim/max_dim: tensor element types and ranks,
• axis_bound: for constraints such as $$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$2,
• optional: probabilistic input omission,
• properties: e.g., multidirectional broadcasting.

A representative OSL example for a DepthToSpace operator:

$r_\text{attr}$0

4. Sionnx Architecture and Generation Pipeline

The Sionnx system is architected with the following core components:

• OSL Front-End: Utilizes LLVM TableGen to parse OSL files and emit an in-memory schema $$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$3 for each operator.
• Reference Algorithm Loader: Maps operator specifications to Python reference implementations.
• Test Compiler: Applies the TDBc-gen (Type, Dimension, Boundary case generator) randomization algorithm to generate comprehensive pytest-compatible test files.

The TDBc-gen algorithm comprises three phases:

1. Type Coverage: Systematically enumerates all valid type combinations across inputs:

$$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$4

where $$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$5 is the number of inputs, $$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$6 the allowed type set for input $$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$7.

1. Dimension Coverage: Ensures uniform sampling across all valid input ranks in $$R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \}$$8.
2. Boundary Coverage: Explicitly targets tests at minimum and maximum values for each operand and parameter.

All code generation is driven by LLVM TableGen backends (-gen-onnx-tests, -gen-onnx-smoke-tests), such that adding a new operator spec is as simple as dropping a .osl file and optionally a corresponding .algorithm reference implementation.

5. End-to-End Workflow Example

A typical development and verification workflow using Sionnx follows these steps:

1. Write an OSL spec (e.g., DepthToSpace.osl) capturing operator constraints.
2. Provide a Python reference algorithm (e.g.,

$r_\text{attr}$1

1. Invoke TableGen:
   • For smoke tests: $r_\text{attr}$2
   • For full tests: $r_\text{attr}$3
2. Test Execution: The generated test_depth_to_space.py imports ONNX and NumPy, constructs reference test nodes, populates randomized operands in accordance with the OSL constraints and TDBc-gen, and compares backend outputs to reference results via expect.

The following table illustrates the effectiveness of the automated pipeline:

Metric	Manual Tests	Sionnx Smoke	Sionnx Full
Tests/operator	5–10	200	1000
Type-combination coverage	20%	100%	100%
Dim-combination coverage	15%	100%	100%
Boundary-condition coverage	10%	100%	100%
Mean conformance pass-rate	85%	96%	98%

Source: (Cai et al., 2019), Table 1.

6. Integration and Cross-Framework Testing

To test or update an ONNX operator:

1. Write: <Op>.osl and its reference <Op>.algorithm under the respective directories.
2. Compile: Use TableGen (-gen-onnx-tests or -gen-onnx-smoke-tests) to generate Python files.
3. Run: Execute the ONNX test harness (e.g., pytest tests/) against any supported backend.

Sionnx-produced tests are fully backend-agnostic. Changing the test target requires only reconfiguring the harness, not the individual test cases or specifications.

7. Limitations and Ongoing Extensions

Current Sionnx limitations include:

• Complex inter-operand constraints (e.g., “channels divisible by blocksize²,” “scale ≠ 0”) are enforced by handwritten logic in the compiler, not natively expressible in OSL.
• Reference algorithm support is Python-only, with planned support for C++ and direct calls into optimized libraries (for example, cuDNN).
• Dynamic input shapes and sparse patterns are not fully handled; future versions may extend OSL to encode richer shape algebra and nonzero constraints.
• Symbolic and constraint-based generation may be adopted to ensure precise coverage of semantic corner cases, such as quantization overflows.

Sionnx provides the first pipeline for specification-driven, automatic conformance testing in ONNX, improving coverage from under 20% (manual tests) to near 100% on type, dimension, and boundary axes while significantly streamlining reference implementation burden (Cai et al., 2019).