Papers
Topics
Authors
Recent
Search
2000 character limit reached

DLMMM: Multi-Objective Deep Learning Testing

Updated 6 July 2026
  • DLMMM is a testing method for deep learning frameworks that uses multi-measurement fusion to guide model mutation and bug detection.
  • It improves on earlier approaches by measuring operator combination variety and execution time to overcome single-indicator limitations.
  • DLMMM employs a CRITIC-based fitness function to balance bug detection performance, operator diversity, and runtime for efficient testing.

Searching arXiv for papers and acronym usage around “DLMMM”. arxiv_search(query="DLMMM OR \"Deep Learning Framework Testing via Heuristic Guidance Based on Multiple Model Measurements\" OR \"Deep learning model merging\" OR \"Discriminative Latent Mixture Model\"", max_results=10) DLMMM denotes Deep Learning Framework Testing via Heuristic Guidance Based on Multiple Model Measurements, a model-based testing method for deep learning frameworks that treats generated deep learning models as test inputs, executes them across multiple frameworks such as TensorFlow, PyTorch, and MindSpore, and guides further model generation by jointly measuring bug detection performance, operator combination variety, and model execution time (Zou et al., 21 Jul 2025). It was introduced to address the limitations of prior single-indicator heuristic guidance, and is presented as the first deep learning framework testing method to include multiple model measurements into heuristic guidance and fuse these measurements to achieve their trade-off (Zou et al., 21 Jul 2025).

1. Definition and problem setting

Deep learning frameworks are the software substrates on which most modern deep learning applications are built. They provide operators, execution engines, and model construction abstractions such as computation graphs and automatic differentiation. Bugs in these frameworks can silently corrupt downstream applications in autonomous driving, medical diagnosis, and industrial monitoring; empirical studies cited in the DLMMM description report real-world bugs in TensorFlow and Keras that manifest as crashes, wrong numerical outputs, or NaN values (Zou et al., 21 Jul 2025).

Within this setting, DLMMM belongs to the line of work that uses DL models as test inputs. The general workflow is differential testing: generate a model, run it on multiple frameworks with the same input tensor or tensors, and detect crashes, NaNs, or output inconsistencies. Earlier systems in this model-based testing lineage include LEMON, COMET, Muffin, and Gandalf, which use heuristic guidance to bias generation toward promising models. DLMMM preserves this overall testing paradigm but replaces single-indicator guidance with a multi-measurement formulation (Zou et al., 21 Jul 2025).

2. Motivation and limitations of earlier heuristic guidance

DLMMM is motivated by three limitations identified in earlier framework-testing methods. First, prior work typically measured operator type variety rather than operator combination variety. This distinction matters because many framework bugs are triggered by specific local substructures rather than by isolated operator types. The DLMMM description emphasizes that sequence-based metrics such as COMET’s operator sequences cannot properly handle DAGs with branches, and therefore cannot quantitatively assess distinct subgraphs in architectures such as ResNet-style models (Zou et al., 21 Jul 2025).

Second, earlier methods largely ignored model execution time. Under a fixed testing budget, slow models reduce the number of executed tests, even if each individual model is somewhat more likely to expose a bug. DLMMM treats this as a first-class optimization variable rather than an incidental runtime artifact. The motivating observation is that shorter models can yield higher bug detection efficiency, measured as bugs per second, even when larger models sometimes find slightly more bugs individually (Zou et al., 21 Jul 2025).

Third, earlier methods neglected correlations and trade-offs among measurements. Bug detection performance, operator combination variety, and execution time are not independent. Higher operator combination variety can improve bug-finding potential, but it is also associated with longer execution time. DLMMM therefore frames heuristic guidance as a trade-off problem rather than as optimization of a single surrogate signal. This suggests a multi-objective view of framework testing in which utility must be assessed relative to a time budget as well as to structural diversity (Zou et al., 21 Jul 2025).

3. Measurement system

DLMMM assigns three measurements to each tested model. These are computed during differential testing and then used as the basis of heuristic guidance (Zou et al., 21 Jul 2025).

Measurement Meaning Core definition
performance Bug detection performance Crash/NaN placeholder or inconsistency score
variety_degree Operator combination variety Number of distinct motifs of size depth
time Model execution time Wall-clock execution time

Bug detection performance follows prior work such as Muffin and Gandalf. If a model triggers a crash or NaN bug, performance is set to the average value of all elements in the input tensor used for that model. If it does not trigger a crash or NaN, performance is based on cross-framework inconsistency. For frameworks ii and jj, DLMMM computes

result_diffij=∣resulti−resultj∣result\_diff_{ij} = |result_i - result_j|

and then

inconsistencyij=max⁡(result_diffij).inconsistency_{ij} = \max(result\_diff_{ij}).

Larger values indicate stronger evidence that a model is exposing anomalous framework behavior (Zou et al., 21 Jul 2025).

Operator combination variety is DLMMM’s main new metric. Each model is represented as a DAG ⟨VG,EG⟩\langle V_G, E_G\rangle, where vertices are tensors and directed edges are labeled by operator types. For a fixed depth, DLMMM enumerates all connected subgraphs with exactly that many operator edges, treats them as motifs, and removes duplicates via labeled directed-graph isomorphism. The resulting count is

variety_degree=∣{distinct motifs of size depth}∣.variety\_degree = |\{\text{distinct motifs of size } depth\}|.

When depth=1depth = 1, this reduces to operator type variety; when depth>1depth > 1, it measures operator combination variety and naturally accommodates branch structures (Zou et al., 21 Jul 2025).

Model execution time, denoted time, is the wall-clock time needed to run the model with the chosen input tensor on the tested frameworks. DLMMM treats this as part of heuristic guidance because test-round runtime is dominated by model execution, and fixed-budget testing requires explicit throughput-awareness (Zou et al., 21 Jul 2025).

4. Correlation-aware fusion and fitness

The core of DLMMM is a fitness function that fuses the three measurements while accounting for scale differences and inter-measurement correlation. To do this, DLMMM stores tested models in a judge_matrix whose columns are performance, variety, and time (Zou et al., 21 Jul 2025).

The fusion uses the CRITIC method. Because execution time should contribute negatively to heuristic value, DLMMM first transforms it by reciprocity:

xp,time′=1xp,time.x'_{p,time} = \frac{1}{x_{p,time}}.

Each column is then normalized by L2 normalization:

zpq=xpq∑p=1nxpq2.z_{pq} = \frac{x_{pq}}{\sqrt{\sum_{p=1}^{n} x_{pq}^{2}}}.

For each measurement qq, DLMMM computes a contrast term

σq=∑p=1m(zpq−zˉq)2m−1\sigma_q = \sqrt{\frac{\sum_{p=1}^{m} (z_{pq} - \bar{z}_q)^2}{m - 1}}

and a conflict term

fq=∑(1−rq),f_q = \sum (1-r_q),

where rqr_q denotes Pearson correlations between measurement qq and the others. Their product defines the information carrying capacity,

Cq=σq⋅fq,C_q = \sigma_q \cdot f_q,

which is normalized into weights

wq=Cq∑q′=1nCq′.w_q = \frac{C_q}{\sum_{q'=1}^{n} C_{q'}}.

The model fitness is then

fitness=∑q=1nwq⋅xq.fitness = \sum_{q=1}^{n} w_q \cdot x_q.

Because time is inverted before normalization, higher fitness corresponds to better bug detection performance, higher operator combination variety, and shorter actual execution time (Zou et al., 21 Jul 2025).

This design is explicitly intended to replace arbitrary hand-tuned weighting with a correlation- and contrast-aware scheme. In DLMMM’s framing, a measurement should receive more influence when it both discriminates strongly across models and contributes information not already captured by the other measurements (Zou et al., 21 Jul 2025).

5. Multi-level heuristic guidance and model generation

DLMMM uses the fused fitness in multi-level heuristic guidance. The two stated levels are model-level guidance, which selects seed models to mutate, and operator-level guidance, which biases the choice of mutation operators (Zou et al., 21 Jul 2025).

Models are represented as DAGs with exactly one source vertex and one sink vertex, and a valid model must be connected and satisfy type and shape constraints. Mutation is implemented by replacing edges. DLMMM randomly samples a predecessor vertex ii that is not the sink, randomly samples a successor vertex jj with j>ij > i to preserve acyclicity, and then samples an operator oo according to its current weight:

p(o)=weighto∑k=1nweightk.p(o) = \frac{weight_o}{\sum_{k=1}^{n} weight_k}.

The operator on edge (i,j)(i,j) is replaced by oo; invalid graphs are discarded and retried. In the provided description, this low-level operation can correspond to higher-level insert, remove, or change-operator mutations (Zou et al., 21 Jul 2025).

DLMMM does not depend on pre-existing public models for initialization. Instead, it builds a seed model pool by starting from a trivial model consisting of a single chain of identity operators from source to sink, adding that model to the pool, and repeatedly mutating randomly selected pool members until a predefined seed-pool size is reached. As testing proceeds, fitness and operator weights are updated, so later mutations become more strongly guided by prior measurements (Zou et al., 21 Jul 2025).

For seed selection during test input generation, DLMMM uses a tournament algorithm, described as a standard evolutionary algorithm technique. This places DLMMM within mutation-based evolutionary testing, but with guidance informed by a fused multi-measurement fitness rather than by a single heuristic indicator (Zou et al., 21 Jul 2025).

6. Empirical rationale and significance

The empirical rationale for DLMMM is based on observed correlations among the three measurements. The reported Pearson correlation between operator combination variety and bug detection performance is greater than 0.2 for Muffin and COMET, and the paper gives the specific values 0.21 and 0.32 for variety versus performance. It also reports positive correlation between variety and execution time, in the range 0.42–0.50 across methods (Zou et al., 21 Jul 2025).

These numbers motivate the central trade-off. More distinct motifs tend to improve bug-finding potential, but they also tend to make models slower. The empirical study further states that “larger” models, defined as the top 50% by time, sometimes find slightly more bugs, whereas “smaller” models, the bottom 50% by time, have better bugs-per-second efficiency because they are faster. DLMMM therefore targets both effectiveness and efficiency, seeking more unique bugs and more bugs per unit time rather than maximizing only one of these criteria (Zou et al., 21 Jul 2025).

In methodological terms, DLMMM extends prior framework-testing systems in three directions at once: it replaces operator-type counting with motif-based operator combination counting, incorporates runtime directly into heuristic guidance, and models measurement interaction through CRITIC-based fusion. This suggests a shift from single-signal heuristic design toward explicit trade-off management in deep learning framework testing (Zou et al., 21 Jul 2025).

7. Acronym usage and disambiguation

The acronym DLMMM is not unique across adjacent literatures. In the framework-testing sense, it refers to Deep Learning Framework Testing via Heuristic Guidance Based on Multiple Model Measurements (Zou et al., 21 Jul 2025). In a distinct literature on model consolidation, the phrase deep learning model merging (DLMMM) is used to denote the broader problem of combining trained models without returning to raw data; within that usage, DMM is introduced as a data-free model merging framework for highly divergent models (Liu et al., 6 Mar 2026).

DLMMM should also be distinguished from DMML, which denotes Deep Mamba Multi-modal Learning, a multimodal fusion framework that uses Mamba for per-modality semantic refinement, additive multimodal fusion, and CNN-based deep fusion; its multimedia-retrieval instantiation is DMMH (Zhu et al., 2024). In older clustering literature, related acronyms such as DLM and sometimes DLMM/DLMMM refer to the Discriminative Latent Mixture Model, a Gaussian mixture model in a latent orthonormal discriminative subspace estimated by the Fisher-EM algorithm (Bouveyron et al., 2011).

Accordingly, in current arXiv usage the meaning of DLMMM is context-dependent. In the setting of deep learning framework quality assurance, however, the term denotes the multi-measurement, CRITIC-fused, mutation-guided testing method introduced in 2025 (Zou et al., 21 Jul 2025).

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to DLMMM.