MalNet-Tiny-Common Benchmark

• MalNet-Tiny-Common is a benchmark that evaluates graph-based Android malware classifiers under covariate shift by using disjoint malware types within the same families.
• It employs a controlled formation of training and test sets to simulate realistic within-family variant evolution and expose overestimated in-distribution accuracy.
• The benchmark highlights the benefits of semantic enrichment via metadata and LLM embeddings to improve classification robustness despite severe distribution shifts.

Searching arXiv for the cited MalNet papers and benchmark context. to=arxiv_search 天天中彩票提款 大发快三是不是 ,大香蕉?_rgctxassistant to=arxiv_search tool code 天天中彩票软件ి{"query":"MalNet Tiny Common (Tran et al., 8 Aug 2025, Freitas et al., 2021) malware", "max_results": 10} Reviewing the most relevant arXiv matches. MalNet-Tiny-Common is a distribution-shifted benchmark for Android malware classification derived from MalNet and introduced to evaluate robustness under covariate shift, specifically the setting in which training and test data share malware families but use disjoint malware types (Tran et al., 8 Aug 2025). It is motivated by limitations in standard MalNet-Tiny evaluation, where conventional splits can produce overly optimistic results exceeding 94% accuracy, while failing to capture the practically important case of previously unseen variants from known families (Tran et al., 8 Aug 2025). In the broader MalNet lineage, the benchmark inherits the dataset tradition of large-scale malware representation originating in MalNet-Image, which framed malware analysis as a visual learning problem over Android APK-derived artifacts and emphasized public, reproducible benchmarking at scale (Freitas et al., 2021). MalNet-Tiny-Common instead serves graph-based Android malware classification, with function call graph representations enriched by metadata and, optionally, code embeddings from LLMs (Tran et al., 8 Aug 2025).

1. Position within the MalNet family of benchmarks

MalNet-Tiny-Common is defined relative to two antecedents: MalNet and MalNet-Tiny. In the benchmark paper, MalNet is described as a large-scale dataset of Android malware, represented as Function Call Graphs (FCGs), with approximately 1.5M samples (Tran et al., 8 Aug 2025). MalNet-Tiny is a manageable, balanced subset of MalNet whose classes are defined by malware family/type pairs and which has become the de facto GNN benchmark; under its traditional splits, graph-based malware classifiers can achieve over 94% accuracy (Tran et al., 8 Aug 2025).

This benchmark should be distinguished from MalNet-Image, a separate but related resource that represents malware as images rather than graphs. MalNet-Image contains over 1.2 million malware images across 47 types and 696 families, created from Android APK files via DEX extraction and 1D-to-2D byte transformation, with a public “Tiny” subset designed for rapid experimentation (Freitas et al., 2021). The naming similarity between MalNet-Image Tiny and MalNet-Tiny-Common reflects shared provenance in the MalNet ecosystem, but the benchmark roles are different. MalNet-Image Tiny is a reduced image-classification subset, whereas MalNet-Tiny-Common is a graph benchmark constructed to probe generalization under shift (Freitas et al., 2021, Tran et al., 8 Aug 2025).

A plausible implication is that the MalNet program has bifurcated into complementary benchmark families: image-based malware classification aimed at CV-style evaluation and graph-based malware classification aimed at structural and semantic reasoning. MalNet-Tiny-Common belongs squarely to the latter.

2. Construction methodology and label regime

The defining construction principle of MalNet-Tiny-Common is that train and test sets contain malware from the same families but different types, with types within a family mutually exclusive across train and test (Tran et al., 8 Aug 2025). This is intended to simulate covariate shift while preserving family-level continuity. From the original MalNet, for each target family matching those in MalNet-Tiny, new and unused types are selected per family to populate the test set (Tran et al., 8 Aug 2025).

For each selected family/type pair, 1,000 samples are randomly drawn (Tran et al., 8 Aug 2025). The benign class is sampled from a disjoint set (Tran et al., 8 Aug 2025). Where some families or types have insufficient data, additional subtypes may be pooled under “mixed” when necessary, although overlap is minimized for fairness (Tran et al., 8 Aug 2025). Each data point retains a $(\text{family}, \text{type})$ label, but evaluation may collapse predictions to the family level because the benchmark is specifically designed around within-family type shift (Tran et al., 8 Aug 2025).

The paper’s appendix-level summary states that the families are the same as in MalNet-Tiny, while the types are deliberately distinct from those used in MalNet-Tiny (Tran et al., 8 Aug 2025). Example families include addisplay, adware, benign, downloader, and trojan; example type differences illustrate that MalNet-Tiny and MalNet-Tiny-Common preserve family compatibility while changing the concrete type composition (Tran et al., 8 Aug 2025).

Property	MalNet-Tiny-Common
Families	Same set as MalNet-Tiny
Types	Disjoint from MalNet-Tiny
Samples per class	1,000
Benign class	Sampled from disjoint set
Purpose	Cross-type within-family generalization

This construction makes the benchmark neither a random resampling nor a simple balanced subset. It is a deliberately shifted evaluation protocol in which class identity is preserved at the family level while subtype realization changes between train and test.

3. Distribution shift scenario and evaluation objective

MalNet-Tiny-Common is expressly intended to evaluate robustness to distribution shift in a setting that resembles real deployment, where models encounter new malware variants from known families (Tran et al., 8 Aug 2025). The benchmark paper characterizes this as a covariate-shift scenario: test types are unseen during training, but they belong to families that are present in training (Tran et al., 8 Aug 2025). This differs from a domain-shift setting in which both families and types are disjoint across splits.

The associated graph representation is formalized as

$G = \phi(x) = (\mathbf{A}, \mathbf{X}),$

where $\mathbf{A} = \phi_T(x)$ denotes the adjacency matrix derived from FCG topology and $\mathbf{X} = \phi_F(x)$ denotes the node feature matrix, which may be enriched with semantic data (Tran et al., 8 Aug 2025). The robustness objective is to maintain predictive performance on both the training distribution $\mathcal{D}$ and the shifted test distribution $\mathcal{D}'$:

$$\mathbb{P}_{(x,y)\sim\mathcal{D}}[\hat{F}(G)=y] \quad \text{and} \quad \mathbb{P}_{(x,y)\sim\mathcal{D}'}[\hat{F}(G)=y].$$

This formulation makes explicit that the benchmark is not merely about in-distribution accuracy. It is about whether the classifier captures family-level semantic regularities that survive changes in subtype composition.

A common misconception is to treat strong performance on MalNet-Tiny as evidence of robust malware understanding. The benchmark evidence contradicts that interpretation: accuracy on standard splits can remain very high even when models fail substantially under within-family variant shift (Tran et al., 8 Aug 2025). MalNet-Tiny-Common was introduced precisely to expose that gap.

4. Relation to MalNet-Tiny and MalNet-Tiny-Distinct

MalNet-Tiny-Common is one of two shifted benchmarks introduced alongside MalNet-Tiny-Distinct (Tran et al., 8 Aug 2025). The comparison among the three evaluation regimes is central to understanding its purpose.

Benchmark	Train/test relationship	Simulated shift
MalNet-Tiny	Overlapping families/types	None
MalNet-Tiny-Common	Same families, disjoint types	Covariate shift
MalNet-Tiny-Distinct	Disjoint families and types	Domain shift

MalNet-Tiny-Common focuses on within-family, across-type generalization, whereas MalNet-Tiny-Distinct is a stricter benchmark for generalization to completely novel malware families (Tran et al., 8 Aug 2025). In this sense, MalNet-Tiny-Common occupies an intermediate position: harder than conventional MalNet-Tiny evaluation, but less extreme than the unseen-family setting.

This suggests that the benchmark is especially useful for separating two failure modes. One is memorization of idiosyncratic subtype structure within known training data; the other is failure to generalize even when coarse family membership remains stable. By holding families constant and shifting only types, MalNet-Tiny-Common isolates the second phenomenon more cleanly than either standard random splits or family-disjoint evaluation.

5. Experimental findings and robustness implications

The principal empirical result is a severe performance drop when standard graph-based malware classifiers are transferred from MalNet-Tiny evaluation to MalNet-Tiny-Common (Tran et al., 8 Aug 2025). The paper reports that accuracy can drop by over 45%, exemplified by Exphormer moving from 94.5% on MalNet-Tiny to 49.5% on MalNet-Tiny-Common (Tran et al., 8 Aug 2025). This result is used to demonstrate brittleness to distribution shift even within the same family.

The benchmark paper attributes this brittleness to the limitations of structure-only representations and corresponding model architectures, which often fail to capture deeper semantic patterns (Tran et al., 8 Aug 2025). To address this, it proposes semantic enrichment of FCG nodes through function-level metadata and, when available, LLM-derived code embeddings (Tran et al., 8 Aug 2025). Under MalNet-Tiny-Common, these enriched features consistently improve robustness relative to LDP-only models, and classification performance improves by up to 8% across several model-feature-collation configurations (Tran et al., 8 Aug 2025).

The paper gives a representative performance comparison in which a baseline reaches 49.5% on MalNet-Tiny-Common, a Zero+Meta variant reaches 50.5%, and Zero+Meta+LLM+LDP reaches 53.9% (Tran et al., 8 Aug 2025). It also reports that removing any semantic feature typically reduces robustness, and that LLM-based embeddings alone are insufficient; the best results arise from combined features (Tran et al., 8 Aug 2025). Adaptation-based methods such as Test-Time Adaptation and Domain Adaptation can further improve robustness when paired with semantic enrichment (Tran et al., 8 Aug 2025).

These findings are significant because they reframe the benchmark from a pure leaderboard instrument into a stress test for representation quality. MalNet-Tiny-Common reveals that top-line in-distribution accuracy can mask substantial fragility when subtype composition changes but family identity does not.

6. Feature design, collation strategies, and semantic enrichment

Each sample in MalNet-Tiny-Common is a graph whose nodes may be enriched with names, types, code statistics, and optionally LLM-based code embeddings (Tran et al., 8 Aug 2025). The enrichment framework is designed to operate under real-world constraints where feature availability is inconsistent, so the benchmark also tests feature collation strategies for incomplete node attributes (Tran et al., 8 Aug 2025).

Among the reported collation schemes, “Zero” and “Prune” perform best (Tran et al., 8 Aug 2025). The “Zero” strategy, which imputes missing node features with zeros, is reported as particularly effective on MalNet-Tiny-Common and supports resilience when features are incomplete (Tran et al., 8 Aug 2025). This is important because malware analysis pipelines often encounter partial observability, missing symbols, or unavailable metadata.

The benchmark therefore tests more than classifier architecture. It tests the interaction among graph topology, semantic side information, missing-feature handling, and shift robustness. A plausible implication is that MalNet-Tiny-Common is best understood as a benchmark for semantically attributed malware graphs under subtype shift, rather than as a generic GNN dataset.

The graph-centric design also contrasts with the image-centric MalNet-Image line. In MalNet-Image, malware binaries are transformed into grayscale or color-coded images via DEX extraction, byte reshaping, and resizing to $256 \times 256$ pixels, with the Tiny subset providing 87,430 images across 43 classes using fixed train/validation/test splits of 61,201 / 8,743 / 17,486 (Freitas et al., 2021, Abouelkhaire et al., 22 Apr 2026). Those image benchmarks study issues such as scale variation induced by resizing binaries of different lengths, pretraining, augmentation, and schedule-free optimization (Abouelkhaire et al., 22 Apr 2026). MalNet-Tiny-Common instead concentrates on semantic graph enrichment under covariate shift. The shared malware source domain does not imply interchangeable evaluation conclusions across these modalities.

7. Research significance, interpretation, and limitations

MalNet-Tiny-Common addresses a specific deficiency in prior Android malware benchmarks: the tendency of standard splits to overestimate generalization by allowing closely related samples, including variants from the same family, to appear across training and testing (Tran et al., 8 Aug 2025). Its core contribution is to make within-family variant generalization measurable and reproducible. In that respect it functions as a realism-oriented robustness benchmark.

Several interpretations follow directly from the reported results. First, strong performance on standard MalNet-Tiny should not be treated as sufficient evidence of robustness to evolving threats (Tran et al., 8 Aug 2025). Second, structure-only GNN pipelines appear inadequate for capturing the semantic continuity needed to generalize across unseen types within known families (Tran et al., 8 Aug 2025). Third, semantic feature enrichment, especially when combining metadata, LLM embeddings, and structural descriptors, provides measurable robustness gains under this shift (Tran et al., 8 Aug 2025).

At the same time, the benchmark does not eliminate all ambiguity. The paper notes that family/type structure may be collapsed to the family level for some experiments (Tran et al., 8 Aug 2025). This indicates that evaluation granularity can vary with experimental design. In addition, some family/type pairs may require pooling under “mixed” because of limited data, which suggests that perfect subtype purity is not always feasible (Tran et al., 8 Aug 2025). These are methodological constraints rather than flaws, but they are relevant when comparing models or interpreting absolute accuracy values.

Within the MalNet ecosystem, MalNet-Tiny-Common thus occupies a distinctive role. MalNet-Image opened large-scale public benchmarking for image-based malware analysis and highlighted research directions in multi-class imbalanced classification and interpretable security (Freitas et al., 2021). Later work on MalNet-Image Tiny showed that compact image classifiers benefit from ImageNet pretraining, Mixup, TrivialAugment, feature pyramid networks, and schedule-free AdamW, achieving $F1_{\mathrm{macro}} = 0.6927$ on 43-class malware type classification (Abouelkhaire et al., 22 Apr 2026). MalNet-Tiny-Common extends the broader program in a different direction: evaluating whether graph-based classifiers can remain reliable when malware evolves within known families (Tran et al., 8 Aug 2025). Its significance lies not in maximizing in-distribution accuracy, but in exposing the gap between nominal benchmark performance and operational robustness.