MalNet-Tiny-Distinct Benchmark

• The benchmark introduces a cross-family domain shift setup by training on original MalNet-Tiny families and testing on disjoint malware families.
• It uses precisely 5,000 function-call graphs (1,000 per class) to assess robustness of graph neural networks under unseen family conditions.
• Semantic enrichment and adaptation techniques, including test-time adaptation, are shown to recover significant accuracy losses, highlighting their practical value.

Searching arXiv for the specified paper to ground the article in the cited source. MalNet-Tiny-Distinct is a benchmark for graph-based Android malware classification under distribution shift, introduced in the context of a semantic enrichment framework for function-call graphs. It is constructed from the original MalNet corpus of Android malware APKs and is designed to stress cross-family generalization by evaluating models trained on the original MalNet-Tiny families against a disjoint set of previously unseen families. In the source study, MalNet-Tiny-Distinct is positioned as a domain-shift benchmark, in contrast to covariate-shift settings, and is used to assess whether graph neural networks can remain robust when family labels do not overlap between training and test domains (Tran et al., 8 Aug 2025).

1. Benchmark definition

MalNet-Tiny-Distinct consists of 5,000 function-call graphs sampled from five malware families or family/type pairs that are wholly disjoint from those in the original MalNet-Tiny base split. Each class contains exactly 1,000 samples, so the benchmark is uniformly balanced. The benchmark is explicitly intended to simulate a true domain shift in which the malware families seen during evaluation are absent from training.

The construction begins from the five “base” MalNet-Tiny families, $$F_{\mathrm{base}} = \{\text{addisplay}, \text{adware}, \text{benign}, \text{downloader}, \text{trojan}\}$$, and defines a disjoint test-family set $F_{\mathrm{distinct}}$ that contains five new families or family/type pairs. The defining property of the benchmark is that the training and testing family sets do not intersect.

This formulation makes MalNet-Tiny-Distinct a benchmark for cross-family generalization rather than for interpolation within a fixed label set. A plausible implication is that it probes whether learned representations capture malware semantics that transfer beyond family-specific structural regularities.

2. Data source and construction procedure

All samples come from the original MalNet corpus of Android malware APKs, comprising approximately 1.5 million samples. Each APK is decompiled with AndroGuard and represented as a Function-Call Graph (FCG), exactly as in the original MalNet and MalNet-Tiny releases. No code, strings, or metadata are removed during graph extraction; only the pure call-graph topology is retained. To remain compatible with MalNet-Tiny, every selected sample must have at most 5,000 nodes in its call graph (Tran et al., 8 Aug 2025).

For the Distinct split, the selected family/type pairs are sampled uniformly, with 1,000 APKs chosen at random from each class using seed $0$. This mirrors the MalNet-Tiny procedure of 1,000 samples per class.

Family/type class	Samples
spr / lootor	1,000
clicker + trojan / dowgin	1,000
riskware / nandrobox	1,000
malware / <various–mixed>	1,000
spyware / <various–mixed>	1,000

The presence of generic or mixed labels is not incidental. The source study states that, even in the large MalNet corpus, it was difficult to find large entirely disjoint families with at least 1,000 samples. As a result, some Distinct labels are generic or mixed, although all classes still contain 1,000 samples after cleaning by node count.

3. Formal split specification

The benchmark is defined by a family-wise partition in which the training set and the test set are unions over disjoint family sets. Let $F_{\mathrm{train}}$ denote the malware families used for training, namely the original MalNet-Tiny base families, and let $F_{\mathrm{test}} = F_{\mathrm{distinct}}$ denote the disjoint set of new families used for testing. For each family $f$, let $S_f$ denote its set of 1,000 samples. Then

$$S_{\mathrm{train}} = \bigcup_{f \in F_{\mathrm{train}}} S_f, \qquad S_{\mathrm{test}} = \bigcup_{f \in F_{\mathrm{test}}} S_f,$$

with

$$F_{\mathrm{train}} \cap F_{\mathrm{test}} = \emptyset.$$

This formalization is central to the benchmark’s semantics. The Distinct split is not merely a resampling of the original MalNet-Tiny label space; it is a disjoint-family partition intended to enforce non-overlap at the family level. This suggests that performance on MalNet-Tiny-Distinct reflects transfer across malware families rather than memorization of family-specific graph motifs.

4. Evaluation protocol and metrics

“Upstream” training is performed on the original MalNet-Tiny split, comprising 5,000 samples across the five base families, using its train, validation, and test folds as in prior work. “Downstream” testing for Distinct evaluates the frozen or adapted model on the 5,000 held-out Distinct samples, again using the standard 80/10/10 or 80/20 split by family/type. In practice, the study treats the full set of 5,000 Distinct samples as the target domain. When Test-Time Adaptation is applied, the 5,000 Distinct graphs are passed unlabelled for unsupervised adaptation or partially labelled for domain adaptation before final scoring (Tran et al., 8 Aug 2025).

The primary metric is classification accuracy, measured as percent correct. The study reports both absolute accuracy on Distinct and the drop relative to the standard MalNet-Tiny setting. $F_1$ scores are not reported.

The protocol is therefore tailored to robustness assessment under distribution shift. Because the benchmark is evaluated against a target domain composed of unseen families, accuracy drops relative to the standard split are treated as a direct indicator of failed generalization.

5. Relation to MalNet-Tiny-Common and observed difficulty

MalNet-Tiny-Distinct is paired in the study with MalNet-Tiny-Common, but the two benchmarks target different forms of shift. Common uses covariate shift, defined as the same families with new sub-types. Distinct uses domain shift, defined as completely new families. The source study reports that Distinct is markedly more challenging: baseline GNNs trained on the original Tiny split typically lose 30 to 45 points of accuracy when first tested on Distinct (Tran et al., 8 Aug 2025).

This distinction is methodologically important. In Common, family identity remains stable and the shift is intra-family. In Distinct, family identity itself changes, so the benchmark tests whether a model can generalize across family boundaries. A common misunderstanding would be to treat Distinct as a harder version of the same problem; the source description instead frames it as a different regime, with non-overlapping family labels and a stronger form of domain shift.

The challenge is compounded by label noise. Because some Distinct classes are generic or mixed, the study notes that peak performance is slightly degraded. Even so, the benchmark remains large, class-balanced, and family-disjoint, which is why it is presented as a durable test of cross-family robustness.

6. Role in semantic enrichment and adaptation studies

In the source study, MalNet-Tiny-Distinct serves as a principal evaluation setting for graph neural networks augmented with semantic node features. The proposed framework enriches function-call graphs with contextual features, including function-level metadata and, when available, code embeddings derived from LLMs. Under this evaluation regime, graph neural networks that incorporate the semantic node features recover most of the lost accuracy on the Distinct split, improving test accuracy by up to 8% over structural-only baselines. The study further reports that domain-adaptation methods, including finetuning and adapters, combined with the semantic graphs, yield final Distinct accuracies in the high-90s, and that the authors still achieve greater than 94% accuracy after adaptation despite label noise (Tran et al., 8 Aug 2025).

These results give the benchmark a dual significance. First, it exposes a limitation of structure-only graph representations: high standard-split accuracy does not ensure robustness to unseen malware families. Second, it provides a controlled setting for testing whether semantic augmentation and adaptation methods can mitigate that failure mode. A plausible implication is that MalNet-Tiny-Distinct is useful not only as a benchmark dataset, but also as an instrument for separating architectural gains from gains that arise from richer node semantics.

In summary, MalNet-Tiny-Distinct is a 5,000-sample, five-class, family-disjoint benchmark derived from MalNet function-call graphs, with exact class balance and evaluation based solely on classification accuracy and its degradation relative to the standard MalNet-Tiny split. Its defining contribution is to operationalize cross-family domain shift in a compact benchmark format, thereby enabling direct study of robustness, semantic enrichment, and adaptation in graph-based Android malware classification.