--- title: 'Sionnx: Automated ONNX Unit Test Generator' url: https://www.emergentmind.com/topics/sionnx type: topic --- # Sionnx: Automated ONNX Unit Test Generator Sionnx is an automatic unit test generator designed to verify Open Neural Network Exchange (ONNX) operator conformance using a rule-driven, specification-based approach. By coupling a formal specification language for operators with automatic test generation and reference algorithm integration, Sionnx delivers comprehensive and backend-agnostic validation pipelines for ONNX implementations [1906.05676]. ## 1. Motivation for Automated ONNX Conformance Testing Major machine-learning (ML) frameworks such as TensorFlow and PyTorch define their own semantics for core computational operators. ONNX provides a unified operator set for cross-framework portability, but the official conformance suite historically relies on a minimal set of manually-written test cases per operator. This deficiency leads to insufficient coverage of type, dimension, and boundary cases; makes it difficult to maintain tests for newly introduced operators; and burdens developers with lengthy, error-prone reference implementations. Sionnx addresses these challenges by: - Defining a compact yet complete rule set for each operator’s attributes and operand properties; - Introducing a domain-specific language to declare these rules in a uniform manner; - Automatically expanding those specifications, together with operator reference implementations, into extensive Python test suites; - Employing a systematic randomization algorithm to guarantee broad coverage of data types, tensor shapes, and boundary cases. ## 2. Formal Specification of Operator Constraints Every ONNX operator $\mathcal{O}$ is characterized by a collection of constraint rules: $$ R = \{ r_\text{attr}, r_\text{input}, r_\text{output}, r_\text{prop} \} $$ Where: - $r_\text{attr}$ are rules over operator attributes; - $r_\text{input}$ and $r_\text{output}$ are rules over tensor operands; - $r_\text{prop}$ are implicit properties, e.g., broadcasting. Each rule $r \in R$ is a conjunction of constraints on: - **Type**: $t \in \{ \mathrm{f16}, \mathrm{f32}, \mathrm{i32}, \ldots \}$, - **Dimension**: $d \in \mathbb{N}$ (dimension/rank), - **Value range**: $v \in [v_\text{min}, v_\text{max}]$. Attribute constraints are expressed as: $$ r_\text{attr} : (\mathrm{type} \in \mathrm{TypeSet}) \wedge (\dim = 0) \wedge (\mathrm{value} \in [v_\text{min}, v_\text{max}]) $$ Input tensor constraints take the form: $$ r_{\text{in}(k)} : (\mathrm{dtype} \in \mathrm{ElemTypeSet}) \wedge (d_\text{min} \leq \dim \leq d_\text{max}) \wedge (\mathrm{axis\_bound} \Rightarrow \dim \geq \mathrm{axis} + 1) $$ This formalism provides the foundation for automatic, exhaustive test generation. ## 3. Operator Specification Language (OSL) Sionnx introduces the Operator Specification Language (OSL), a concise DSL for declaring operator test requirements. OSL is formally defined using extended Backus–Naur Form (BNF): ``` ::= [] ::= 'op_name' '=' String ::= 'op_code' '=' String ::= 'attributes' '{' { } '}' ::= '{' ... }' ::= 'inputs' '{' { } '}' ::= 'outputs' '{' { } '}' ::= '{' ... }' ::= 'properties' '{' { } '}' ::= 'broadcast' | 'nonzero' ``` Key OSL constructs: - **attr_name/type_list**: attribute type and valid value intervals, - **basic_type_list/min_dim/max_dim**: tensor element types and ranks, - **axis_bound**: for constraints such as $\text{rank} \geq \text{axis} + 1$, - **optional**: probabilistic input omission, - **properties**: e.g., multidirectional broadcasting. A representative OSL example for a DepthToSpace operator: ``` { op_name = "DepthToSpaceTest" op_code = "op_depth_to_space" attributes = { { attr_name = "blocksize" type_list = [i32] min_val_list = [1] max_val_list = [4] } } inputs = { { index = 0 basic_type_list = [f16,f32,i8,i16,i32,i64,u8,u16,u32,u64,bool] min_dim = 4, max_dim = 4 } } outputs = inputs } ``` ## 4. Sionnx Architecture and Generation Pipeline The Sionnx system is architected with the following core components: - **OSL Front-End:** Utilizes LLVM TableGen to parse OSL files and emit an in-memory schema $R$ for each operator. - **Reference Algorithm Loader:** Maps operator specifications to Python reference implementations. - **Test Compiler:** Applies the TDBc-gen (Type, Dimension, Boundary case generator) randomization algorithm to generate comprehensive pytest-compatible test files. The TDBc-gen algorithm comprises three phases: 1. **Type Coverage:** Systematically enumerates all valid type combinations across inputs: $$ n_\text{comb} = \prod_{i = 0}^{N-1} |T[i]| $$ where $N$ is the number of inputs, $T[i]$ the allowed type set for input $i$. 2. **Dimension Coverage:** Ensures uniform sampling across all valid input ranks in $[d_\text{min}, d_\text{max}]$. 3. **Boundary Coverage:** Explicitly targets tests at minimum and maximum values for each operand and parameter. All code generation is driven by LLVM TableGen backends (`-gen-onnx-tests`, `-gen-onnx-smoke-tests`), such that adding a new operator spec is as simple as dropping a `.osl` file and optionally a corresponding `.algorithm` reference implementation. ## 5. End-to-End Workflow Example A typical development and verification workflow using Sionnx follows these steps: 1. **Write an OSL spec** (e.g., `DepthToSpace.osl`) capturing operator constraints. 2. **Provide a Python reference algorithm** (e.g., ```python def DepthToSpace_Compute(x_0, blocksize): b, c, h, w = x_0.shape tmp = np.reshape(x_0, [b, blocksize, blocksize, c // (blocksize**2), h, w]) tmp = np.transpose(tmp, [0, 3, 4, 1, 5, 2]) return np.reshape(tmp, [b, c // (blocksize**2), h * blocksize, w * blocksize]) ``` 3. **Invoke TableGen**: - For smoke tests: ``` llvm-tblgen -gen-onnx-smoke-tests DepthToSpace.osl -I . -o tests/ ``` - For full tests: ``` llvm-tblgen -gen-onnx-tests DepthToSpace.osl -I . -o tests/ ``` 4. **Test Execution:** The generated `test_depth_to_space.py` imports ONNX and NumPy, constructs reference test nodes, populates randomized operands in accordance with the OSL constraints and TDBc-gen, and compares backend outputs to reference results via `expect`. The following table illustrates the effectiveness of the automated pipeline: | Metric | Manual Tests | Sionnx Smoke | Sionnx Full | |---------------------------------|-------------|--------------|-------------| | Tests/operator | 5–10 | 200 | 1000 | | Type-combination coverage | 20% | 100% | 100% | | Dim-combination coverage | 15% | 100% | 100% | | Boundary-condition coverage | 10% | 100% | 100% | | Mean conformance pass-rate | 85% | 96% | 98% | *Source: [1906.05676], Table 1.* ## 6. Integration and Cross-Framework Testing To test or update an ONNX operator: 1. **Write**: `.osl` and its reference `.algorithm` under the respective directories. 2. **Compile**: Use TableGen (`-gen-onnx-tests` or `-gen-onnx-smoke-tests`) to generate Python files. 3. **Run**: Execute the ONNX test harness (e.g., `pytest tests/`) against any supported backend. Sionnx-produced tests are fully backend-agnostic. Changing the test target requires only reconfiguring the harness, not the individual test cases or specifications. ## 7. Limitations and Ongoing Extensions Current Sionnx limitations include: - **Complex inter-operand constraints** (e.g., “channels divisible by blocksize²,” “scale ≠ 0”) are enforced by handwritten logic in the compiler, not natively expressible in OSL. - **Reference algorithm support** is Python-only, with planned support for C++ and direct calls into optimized libraries (for example, cuDNN). - **Dynamic input shapes** and **sparse patterns** are not fully handled; future versions may extend OSL to encode richer shape algebra and nonzero constraints. - **Symbolic and constraint-based generation** may be adopted to ensure precise coverage of semantic corner cases, such as quantization overflows. Sionnx provides the first pipeline for specification-driven, automatic conformance testing in ONNX, improving coverage from under 20% (manual tests) to near 100% on type, dimension, and boundary axes while significantly streamlining reference implementation burden [1906.05676]. Source: https://www.emergentmind.com/topics/sionnx