DevMuT: Developer-Guided DL Testing

• DevMuT is a mutation-based testing method that leverages developer expertise to guide mutations and enforce legality constraints for detecting framework defects.
• It employs seven mutation operators and nine constraints to simulate common development practices, covering model construction, training, and inference.
• Evaluations on PyTorch, JAX, and MindSpore show significant improvements in model diversity, legal rate, and defect detection across performance, resource, accuracy, and crash categories.

Searching arXiv for DevMuT and closely related DL framework testing papers to ground the article. DevMuT is a mutation-based testing method for deep learning frameworks that uses developer expertise to guide how models are mutated, how legality is constrained, and which defects are treated as significant. Its stated objective is to expose framework defects that are close to real development scenes and are of concern to developers, rather than defects induced primarily by rare or unrealistic mutated models. In the reported study, DevMuT is evaluated on PyTorch, JAX, and MindSpore with 29 deep learning models from nine kinds of industrial tasks, and is positioned as a lifecycle-aware framework-testing system spanning model construction, training, and inference rather than inference-only inconsistency checking (Mu et al., 6 Jul 2025).

1. Scope, motivation, and target defect classes

DevMuT is situated in the testing of deep learning frameworks rather than the testing of individual trained models. The underlying premise is that framework defects can profoundly cause disastrous accidents, and that prior mutation-based studies often detect defects that are technically valid but low-value from developers’ perspective because the triggering models are edge cases, are too unrealistic, or are ignored by framework maintainers. DevMuT addresses this by generating models through mutation operators and constraints derived from developer expertise, thereby simulating developers’ common operations in development and detecting defects within more stages of the DL model lifecycle, including model training and inference (Mu et al., 6 Jul 2025).

The method targets four broad defect classes. Performance defects include abnormal training loss such as NAN and inf, degraded inference evaluation, and low execution efficiency. Resource defects include abnormal memory or GPU growth, memory leaks, and unreasonable resource scheduling or allocation. Accuracy defects include inconsistent outputs across frameworks, outlier outputs, and forward/backward numerical discrepancies. Crash defects include crashes during mutation or model construction, training, and inference. The paper also reports documentation defects and framework functionality defects, which are counted separately in its confirmed-bug analysis.

A central distinction in the motivation is the critique of prior work. Earlier methods are described as focusing mainly on whether mutated models execute and whether outputs are consistent across frameworks during inference. This leaves many developer-relevant defect modes underexplored, especially defects tied to model construction, training-time behavior, resource management, and efficiency. DevMuT therefore reframes DL framework mutation testing as a developer-oriented testing problem rather than merely a model-diversification problem.

2. Developer expertise as the organizing principle

The defining design feature of DevMuT is that its mutation operators, legality constraints, and defect oracles are derived from structured interviews with six professionals from leading Internet companies. These participants specialize in deep learning framework quality assurance, are responsible for framework development, testing, and defect detection, and each have over four years of experience. The interviews lasted about 30 minutes each, were recorded with consent, and covered five areas: participant profile, scripts and operations frequently modified, experience in recognizing weird or unrealistic models, defect types regarded as important, and views on current frameworks and desired improvements (Mu et al., 6 Jul 2025).

The extracted expertise is encoded into three artifacts: 7 mutation operators, 9 mutation constraints, and 10 test oracles for 4 defect types. This design is intended to simulate how developers actually alter model structure scripts and execution scripts, how they judge whether a mutation is weird or illegal, and which runtime signals they consider meaningful during debugging and quality assurance.

The interview results identify two major classes of common development operations. In structure scripts, developers often replace structures, add structures, delete structures, and modify parameters. In execution scripts, developers often change data augmentation, loss functions, optimizers, and optimizer parameters. The same interviews also yielded tactics for constraining mutations, such as shape-dimension rules, permissible parameter ranges, legal positions for addition or deletion, and developer criteria for weird models, including models that are too time-consuming, have invalid output accuracy, or exhibit abnormal gradients.

This suggests that DevMuT is less a pure fuzzing system than a mutation platform whose search space is curated by developer practice. A plausible implication is that its realism claims derive not from semantic verification of model intent, but from the fact that its operators and filters are anchored in reported engineering behavior.

3. Mutation operators and legality constraints

DevMuT mutates two subjects: model structure scripts and execution scripts. It defines seven mutation operators, five of which are structure-oriented and two of which are execution-oriented.

Operator	Name	Effect
MO1	Existing Structure Replacement	Replace an existing backbone, cascade operator, or basic operator with an external structure of the same functional role
MO2	Shape and Dimension Change	Adjust input/output size or dimensionality of basic operators or first/last operators of cascade structures
MO3	Series Connection Addition/Deletion	Add or delete cascade/basic operators in serial order to change model depth
MO4	Parallel Connection Addition/Deletion	Add or delete branches to change model width
MO5	Parameter Value Change	Modify parameter values of basic operators
MO6	Loss Function Change	Change loss-function type or parameter values
MO7	Optimizer Change	Change optimizer type or optimizer parameters

The reported mention frequencies among interviewees are 100% for MO1, 50% for MO2, 83.3% for MO3, 83.3% for MO4, 66.7% for MO5, 50% for MO6, and 50% for MO7. These frequencies are used to support the claim that the operator set reflects routine development behavior rather than arbitrary mutation design (Mu et al., 6 Jul 2025).

The legality apparatus is divided into guidance constraints and filter constraints. Guidance constraints MC1–MC6 steer generation. MC1 enforces structure functionality consistency in replacement; the candidate pools include 5 backbone types, 7 cascade operator types, and 5 basic operator groups. MC2 limits shape and dimension changes to a scale from one-fourth of the original to four times the original. MC3 governs serial additions and deletions, requiring task-type consistency and stating that added or removed serial structures should not exceed 20% of original model depth for common image classification and object detection models. MC4 governs parallel additions and deletions and allows no more than two branches at the same position. MC5 restricts parameter mutation values to developer-recommended ranges; the paper gives Conv2d.kernel_size values of 3\times3, 5\times5, and 7\times7 as an example and states that 12 kinds of parameters from four basic operators were selected. MC6 restricts loss and optimizer changes to task-dependent candidate sets consisting of 4 optimizers and 4 loss functions.

Filter constraints MC7–MC9 reject weird or illegal mutants after generation. MC7 imposes execution-time constraints; developers suggested acceptable training-time increase below 2–3x and inference-time increase below 1–2x, while DevMuT uses a threshold of three times based on inference time. MC8 filters models whose outputs exceed numeric ranges or produce meaningless outliers such as NAN and inf; the paper states that accuracy ranges are used for float16, float32, and float64, and gives MindSpore float32 accuracy range under $1e38$ as an example. MC9 filters mutants whose output gradients contain outliers greater than $1e2$ or lower than $1e-3$.

Together, these operators and constraints define DevMuT’s central tradeoff: broad mutation diversity is explicitly subordinated to a notion of legal realism grounded in developer judgment.

4. Search strategy, mutation workflow, and defect oracles

DevMuT operates in two phases: Model Mutation and Defect Detection. In the mutation phase, it starts from seed models, uses two Q-networks to choose mutation operators and seed models, mutates under developer-derived constraints, filters illegal mutants, stores legal mutants, and updates the search policy with rewards tied to model diversity. In the defect-detection phase, it detects defects exposed during mutation, selects top-$k$ mutants, trains them, runs inference, compares behavior across frameworks, and records crashes, numerical anomalies, resource anomalies, performance anomalies, and output inconsistencies (Mu et al., 6 Jul 2025).

The search procedure is formulated as reinforcement learning. The state is the generated model, the action is the mutation operator, and the reward is the output distance between generated and original or current models. The paper states that the final layer is used as the basis for calculating reward. Operator selection follows an $\epsilon$-greedy policy over double-Q learning, and when mutation fails or yields illegal models, DevMuT reselects a seed model using a UCB policy. The paper prints the UCB definition in malformed form, but its intended components are the current reward of model $d$, the number of times $d$ has been selected, the number of times operator $s$ has been selected on $d$, and an exploration coefficient $c$.

The two Q-networks are updated in standard double-Q style. The pseudocode gives the update rules

$1e2$0

and

$1e2$1

This is intended to reduce local-optimum effects relative to random or simpler stochastic search.

DevMuT’s defect oracles are organized by defect type. For performance defects, one oracle checks abnormal loss values such as NAN and inf; another compares finite loss curves across frameworks using Dynamic Time Warping, with

$1e2$2

where $1e2$3, $1e2$4, and $1e2$5, and $1e2$6 is the total loss-series distance. Additional performance oracles check execution efficiency and inference evaluation differences across frameworks. For resource defects, DevMuT records memory and GPU usage and reports defects when usage increases anomalously, when maximum resource use exceeds a threshold, or when cosine similarity of memory or GPU usage vectors across frameworks falls below a threshold. For accuracy defects, it reports output-distance anomalies, one-sided outliers, or divergent outlier patterns across frameworks. For crash defects, it reports mutation, training, or inference crashes that occur in one framework but not another.

A notable design choice is that defect detection spans multiple lifecycle stages. The system explicitly tests construction or mutation, training, and inference. The paper later reports deployment-stage defects as well, particularly around pre-trained weight loading and backend behavior.

5. Experimental setting and quantitative results

The reported evaluation uses PyTorch 1.10.1, MindSpore 2.2.0, and JAX 0.4.27. The benchmark contains 29 DL models from nine kinds of industrial tasks, including image classification, object detection, semantic segmentation, text classification, anomaly detection, defect detection, key point detection, scene recognition, and regression prediction. Representative models listed in the paper include VGG16, ResNet50, MobileNetV2, ViT, YOLOv3, YOLOv4, SSD variants, U-Net, U-Net++, DeepLabV3, DeepLabV3++, TextCNN, PatchCore, SSIM-AE, OpenPose, and CRNN. An additional 12 models from prior work are included for fairness in baseline comparison (Mu et al., 6 Jul 2025).

Two baselines are compared: COMET, described as a state-of-the-art mutation-based testing method using coverage-guided model generation, and Muffin, described as a generation-based testing method focusing on training-stage inconsistency defects. A practical limitation is that COMET could not execute on the new industrial models not marked with *, so it was run only on the 12 prior-work models for fairness.

Generated-model diversity is evaluated with Layer Input Coverage (LIC), Layer Parameter Coverage (LPC), and Layer Sequence Coverage (LSC), and the paper uses the Scott Knott ESD test for comparison. Legality is measured by the legal rate of generated mutants under the developer-derived constraints. On these measures, DevMuT is reported to achieve at least 71.68% improvement on average in the diversity of generated models and 28.20% improvement on average in legal rates over baselines. Its overall legal rate is reported as 81.17%, and it generated 226 illegal models. By comparison, COMET produced 434 illegal models in 1183 models, corresponding to 36.69% illegal models.

The defect-detection results are the paper’s principal empirical claim. DevMuT is reported to detect 117 defects, of which 63 are confirmed, 24 are fixed, and 8 are high value confirmed by developers. In the comparative analysis discussed under RQ1, DevMuT detects 17 defects during execution, of which 14 are confirmed. The paper further states that DevMuT found three new kinds of defects that the baselines could not detect: 6 abnormal memory usage defects, 6 abnormal performance defects, and 10 train crash defects.

The search-strategy ablation compares DevMuT with DevMuT$1e2$7 and DevMuT$1e2$8. Reported legal rates are 81.17% for DevMuT, 73.83% for DevMuT$1e2$9, and 51.92% for DevMuT$1e-3$0. Average time per generated model is 95.08 s for DevMuT, 97.92 s for DevMuT$1e-3$1, and 244.917 s for DevMuT$1e-3$2. The paper interprets this as evidence that double-Q learning yields a better balance of diversity, legality, and efficiency than random or MCMC selection.

6. Defect distribution and representative bug cases

Among the 63 confirmed defects, the paper reports 7 in the construction stage, 21 in the training stage, 12 in the inference stage, and 14 in the deployment stage; the remaining 9 are documentation and framework functionality defects. This stage distribution is important because it directly supports the claim that DevMuT moves beyond inference-centric testing (Mu et al., 6 Jul 2025).

The paper presents several illustrative cases. In a TextCNN abnormal training loss case, after 20 mutations, MindSpore training loss changed from NAN to inf during the first epoch, whereas in PyTorch the loss changed from 0.171 to 5.969; after 200 batches, middle-layer output distances across frameworks ranged from $1e-3$3 to $1e-3$4, suggesting a weight-update or interface defect. In a LeNet-5 inference-evaluation and performance defect, MindSpore inference evaluation remained at 11.35% while loss hovered around 2.0–2.9 for 46 epochs, later rose to around 99% accuracy with loss in $1e-3$5 to $1e-3$6, and after epoch 164 the loss became -inf and then NAN, after which accuracy collapsed to 9.8%; developers traced this to a defect in CrossEntropyLoss, which was later fixed.

Resource defects are illustrated by a memory leak via ms.jit. For LeNet-5-1, memory usage began at 0.704 GB, reached 5.696 GB after one day, and then 9.408 GB the next day; the cause was continuous caching and lack of timely recycling by MindSpore’s ms.jit decorator. Another case involves a Pad operator memory allocation defect: after 50 mutations of VGG16-1, MindSpore threw a memory allocation exception while the equivalent PyTorch model executed normally, and the defect was attributed to incorrect secure-memory handling in the Pad operator.

Efficiency and numerical defects are also emphasized. A mutated OpenPose efficiency defect showed MindSpore to be 45.86% slower than PyTorch after 50 mutations, with the bottleneck traced to a Flatten operator lacking necessary optimization. In an accuracy defect involving VGG16-1, after six mutation rounds the model produced NAN on MindSpore and PyTorch, but when reproduced from GPU V100 to Ascend 910, outputs changed from NAN to normal values; developers traced this to output-design issues on Ascend 910 hardware, and the defect received the Main label. Crash cases include a Conv2d kernel/input-size inconsistency, where MindSpore executed normally while PyTorch failed when kernel size exceeded input size, and a ViT training forward crash, where a mutated model failed during forward propagation because lower optimization level prevented constant folding from eliminating a redundant layer, resulting in a slice error.

These examples are used to support the claim that DevMuT tends to expose defects tied to realistic engineering scenarios: backend-specific numerical behavior, memory growth over long execution, training-loss instability, and optimization regressions.

7. Deployment status, limitations, and place in DL framework testing

DevMuT has been deployed in the MindSpore community since December 2023 for continuous quality assurance. The paper treats this deployment, together with 63 confirmed defects, 24 fixed defects, 8 high-value defects, and a submitted pull request fixing MindSpore defects, as evidence that the method has practical utility beyond a one-off research prototype (Mu et al., 6 Jul 2025).

The system’s practical costs are also stated explicitly. Mutation time per generated model is 95.08 s for DevMuT, compared with 98.18 s for COMET and 14.45 s for Muffin. Defect detection requires around 45 minutes per model, because DevMuT executes training, inference, resource tracking, and cross-framework comparison. After the nine constraints and manual inspection, the reported false positive rate is 22.22%. The remaining false positives mainly stem from loss inconsistency without downstream effect, crashes due to wrong settings, and migration or configuration issues.

Several limitations are acknowledged. The mutation operators reflect the expertise of only six interviewed professionals. Not all frameworks and tasks are covered. Oracle thresholds are model-specific and require calibration. Defect detection is computationally expensive. Some formulas and metrics are inherited from prior work and are imperfectly formalized in the paper. Baseline comparison with COMET is restricted because COMET could not handle newer industrial models. The paper identifies future extension directions including more frameworks such as Jittor and TensorFlow, and more tasks such as audio recognition.

Within the broader landscape of DL framework testing, DevMuT is best characterized as a developer-expertise-guided mutation system that reorients testing from abstract model diversity toward realistic model evolution and developer-valued bug discovery. Its distinctive combination is the use of developer-derived mutation operators, developer-derived legality constraints, test oracles tied to performance, resource, accuracy, and crash defects, and lifecycle-aware execution across construction, training, and inference. This suggests that its principal contribution is methodological alignment: it attempts to make mutation testing for deep learning frameworks resemble actual framework-facing model development rather than synthetic perturbation alone.