MalEval: Benchmark for Android Malware Auditing

• The paper introduces MalEval as a framework that shifts malware evaluation from binary detection to a structured, auditor-aligned process focused on traceability and evidence grounding.
• MalEval employs static reachability analysis and function-level representations to filter noise and ensure that only executable, sensitive functions are analyzed.
• The framework benchmarks LLM performance using metrics like Fidelity Score and Workload Reduction Score, highlighting both strengths and limitations in automating malware auditing.

Searching arXiv for the MalEval paper and closely related malware-evaluation work. MalEval is a benchmark and evaluation framework for fine-grained Android malware behavior auditing with LLMs. It is designed for the post-detection stage of security analysis rather than for malware detection alone: the central question is not merely whether a sample is malicious, but whether an LLM can help an analyst identify suspicious functions, ground behavioral claims in verifiable evidence, synthesize a behavior report, and distinguish true malware from false positives under realistic constraints (Zheng et al., 17 Sep 2025). In this formulation, malware evaluation shifts from label accuracy toward traceability, evidence authenticity, and analyst workload reduction, placing MalEval within a broader movement from classifier-centric security benchmarks to workflow-aligned auditing frameworks; the contrast with detector-oriented systems such as MalCVE and MeLeMaD is instructive in this regard (Cristea et al., 17 Oct 2025, K et al., 30 Dec 2025).

1. Definition, scope, and motivating problem

MalEval was introduced to address three practical barriers in malware auditing: ground-truth scarcity at function and evidence level, noise interference from large benign codebases, and fragile traceability and hallucination in LLM outputs (Zheng et al., 17 Sep 2025). Standard malware benchmarks typically evaluate final decision accuracy, such as malware versus benign labels or family labels. MalEval instead evaluates a multi-stage reasoning and auditing pipeline aligned with analyst practice. It works on reachable functions rather than full raw code, uses expert-verified reports and sensitive API evidence as ground truth, evaluates evidence grounding and report synthesis in addition to detection, includes false-positive correction as a first-class task, and penalizes outputs that are fluent but ungrounded or not traceable to input evidence (Zheng et al., 17 Sep 2025).

This design positions MalEval closer to malware auditing than malware classification. The benchmark assumes that a sample has already been flagged as suspicious and asks whether an LLM can support the downstream investigative workflow. That distinction is consequential. Detector-oriented systems such as MeLeMaD are framed as binary classification pipelines optimized for accuracy, precision, recall, F1-score, MCC, and AUC on large malware corpora (K et al., 30 Dec 2025). MalCVE extends this orientation by combining malware detection with CVE association for JAR binaries (Cristea et al., 17 Oct 2025). MalEval, by contrast, is centered on causal and verifiable explanation: what malicious behavior is present, which functions or APIs support the claim, whether the justification is traceable, and whether the model can correct a false alarm (Zheng et al., 17 Sep 2025). This suggests that MalEval is best understood as an auditing benchmark rather than a detection benchmark.

2. Ground truth construction and benchmark artifacts

MalEval builds ground truth at two levels. Its low-level ground truth is a curated sensitive API list serving as the atomic evidence base. Its high-level ground truth consists of expert-written or expert-verified behavior reports parsed into structured labels and summaries, normalized into a standardized JSON/Markdown-like form with behavior categories, extracted textual evidence, confidence, and concise summaries (Zheng et al., 17 Sep 2025). The combination is intended to make evaluation both fine-grained and auditable.

The sensitive API inventory begins from PScout and is then updated by crawling official Android developer documentation, extracting API descriptions, filtering deprecated APIs, using an LLM to identify successors of deprecated APIs, and adding APIs that require explicit permissions. The final list contains 20,337 sensitive APIs (Zheng et al., 17 Sep 2025). In MalEval, this updated inventory is not auxiliary metadata; it is the basis for low-level evidence attribution.

The report corpus is likewise curated for evaluability. Vendor reports are crawled, normalized to Markdown, stripped of boilerplate such as advertisements, and parsed into structured fields including behavior categories, evidence, confidence, and concise summaries (Zheng et al., 17 Sep 2025). The benchmark therefore links unstructured analyst-style reporting to structured evaluation targets. A plausible implication is that MalEval treats natural-language reporting as an object of formal evaluation only after constraining it by a verifiable evidence model.

The emphasis on expert-verified reporting distinguishes MalEval from malware benchmarks whose outputs are restricted to labels. MalCVE, for example, structures LLM output into JSON fields for verdict, confidence, rationale, behavior summary, activities, indicators of compromise, libraries used, MITRE ATT&CK mapping, and CVE search queries, but its target tasks remain malware detection and CVE association rather than function-linked behavior auditing (Cristea et al., 17 Oct 2025). MalEval’s artifact design is therefore tied specifically to evidentiary attribution and report verifiability (Zheng et al., 17 Sep 2025).

3. Reachability reduction and function-level representation

To reduce noise, MalEval does not feed raw APK code directly to LLMs. Instead, it performs static reachability analysis. An entrypoint is defined as a method invocable without a prior call to another user-defined method. Entrypoints are identified through manifest-declared lifecycle methods such as onCreate and overridden or implemented Android framework methods; from those entrypoints, MalEval performs BFS on the call graph to collect all transitively reachable functions (Zheng et al., 17 Sep 2025).

The quantitative effect of this preprocessing is substantial. The benchmark reports an average of 733 entrypoints per APK, 689,232 total reachable functions, and a 66.02% reduction from the original function count (Zheng et al., 17 Sep 2025). These figures establish reachability analysis as a core denoising mechanism rather than a minor implementation detail. The resulting benchmark input is restricted to code that can actually execute, which is especially important in Android applications where framework code, libraries, and benign scaffolding can swamp the malicious signal.

MalEval then maps each reachable function to a structural intermediate representation: $$IR(f) = \langle \mathcal{N}(f), \mathcal{D}_{c}(f), \mathcal{S}(f) \rangle$$ where $\mathcal{N}(f)$ is the function signature or name, $\mathcal{D}_{c}(f)$ is a context-driven description of the function, and $\mathcal{S}(f)$ is a sensitivity score (Zheng et al., 17 Sep 2025). The context-driven description is built by analyzing the one-hop caller-callee neighborhood $\mathcal{C}(f)$, summarizing neighboring functions, and combining that with the function’s own code and signature (Zheng et al., 17 Sep 2025).

This representation is the key unit for both explanation and evaluation. It is neither raw code nor an unconstrained summary; it is a function-level attribution object designed to support ranking, evidence coverage, report generation, and traceability checks. The benchmark’s ablation finding that removing inter-procedural context sharply reduces FS, reduces RQ and EAS, and hurts overall WRS indicates that the one-hop contextualization is operationally significant rather than decorative (Zheng et al., 17 Sep 2025). At the same time, the paper notes that context can add noise for some models, especially in sample discrimination, which suggests that representation quality and context budget remain coupled problems in LLM-based auditing (Zheng et al., 17 Sep 2025).

4. Analyst-aligned tasks and evaluation metrics

MalEval defines four tasks corresponding to stages of human malware auditing: function prioritization, evidence attribution, behavior synthesis, and sample discrimination (Zheng et al., 17 Sep 2025). The benchmark thus treats malware auditing as a staged process in which ranking, grounding, synthesis, and final decision support are evaluated separately before being combined.

Task	Output	Principal metrics
Function Prioritization	Sensitivity scores and ranking over reachable functions	FS
Evidence Attribution	Selected subset of functions covering sensitive APIs	CSR
Behavior Synthesis	Structured JSON report	RQ, EAS, SAS
Sample Discrimination	Malicious/benign decision and malware category	FPCR, TPMR, $F1_c$

For function prioritization, the goal is to identify which reachable functions are most likely to be maliciously relevant. Because there are no direct function labels, MalEval uses Fidelity Score: $$FS(k) = 1 - \frac{P(\hat{y}\,|\,\mathcal{F}_{r} \setminus \mathcal{F}_{k})}{P(\hat{y}\,|\,\mathcal{F}_{r})}$$ where $\hat{y}$ is the classifier’s predicted malware category, $P(\hat{y}\,|\,\mathcal{F}_{r})$ is classifier confidence using all reachable functions, and $\mathcal{F}_{k}$ is the top-$\mathcal{N}(f)$0 of functions selected as most sensitive (Zheng et al., 17 Sep 2025). Higher FS means the selected functions are genuinely important for the malware prediction.

For evidence attribution, the objective is to map low-level evidence to a compact set of functions. MalEval evaluates API-level evidence attribution through the Coverage-of-Selected Rate. The paper notes that the LaTeX is slightly malformed, and gives the intended interpretation as

$\mathcal{N}(f)$1

with $\mathcal{N}(f)$2 denoting functions selected above a sensitivity threshold $\mathcal{N}(f)$3, $\mathcal{N}(f)$4 the sensitive APIs invoked by those functions, $\mathcal{N}(f)$5 the ground-truth sensitive APIs, and $\mathcal{N}(f)$6 all reachable functions (Zheng et al., 17 Sep 2025). Higher CSR corresponds to broader evidence coverage at lower analyst cost.

For behavior synthesis, the model must generate a structured JSON report containing is_malicious, present_behaviors over 12 MalRadar behavior categories, function-linked evidence, a high-level summary, and a malware category (Zheng et al., 17 Sep 2025). Evaluation uses an LLM-as-a-judge configuration and three metrics. Report Quality is

$\mathcal{N}(f)$7

where $\mathcal{N}(f)$8, $\mathcal{N}(f)$9, and $\mathcal{D}_{c}(f)$0 are insight, comprehensiveness, and evidence quality scores, respectively (Zheng et al., 17 Sep 2025). Evidence Authenticity Score measures whether cited functions genuinely support the described behaviors, and Syntax Authenticity Score measures whether cited functions actually appear in the input set: $\mathcal{D}_{c}(f)$1 where $\mathcal{D}_{c}(f)$2 is the set of cited or supporting functions and $\mathcal{D}_{c}(f)$3 is the input function set (Zheng et al., 17 Sep 2025). Together, these metrics separate narrative fluency from evidentiary grounding and traceability.

For sample discrimination, the benchmark asks whether a sample is truly malware or a false positive. It uses False Positive Correction Rate, True Positive Maintenance Rate, and macro $\mathcal{D}_{c}(f)$4 on malware category classification (Zheng et al., 17 Sep 2025). This formalizes the practical security requirement that a model should reduce false alarms without sacrificing detection of real threats.

To summarize practical usefulness across stages, MalEval introduces Workload Reduction Score: $\mathcal{D}_{c}(f)$5

$\mathcal{D}_{c}(f)$6

$\mathcal{D}_{c}(f)$7

where $\mathcal{D}_{c}(f)$8 denotes analytical accuracy, $\mathcal{D}_{c}(f)$9 decision support, and the weights are set equally in experiments (Zheng et al., 17 Sep 2025). WRS estimates how much human analyst workload the model can reduce while remaining trustworthy. This suggests that MalEval’s unifying objective is not maximal raw capability on any single subtask, but balanced utility across an analyst workflow.

5. Dataset composition and experimental protocol

The malware corpus used by MalEval has two parts. The Archived Malware set consists of 222 samples from MalRadar, covering 51 families and 6 categories. These samples are well-documented, have vetted reports available, include families without known variants, and are deduplicated by identical .dex files (Zheng et al., 17 Sep 2025). The Latest Malware set contains 33 samples collected after July 2024, with reports from Malpedia and APKs downloaded from Koodous; it covers 10 families and 3 categories and is intended to test performance on newer threats outside most LLM training horizons (Zheng et al., 17 Sep 2025). The malware categories listed in the benchmark table are Trojan, Banker, Rootkits, Spyware, Adware, and Ransomware (Zheng et al., 17 Sep 2025).

For false-positive evaluation, MalEval includes 25 benign apps collected from Androzoo, all after July 2024, designed to emulate an approximately 10% false positive rate (Zheng et al., 17 Sep 2025). These samples are used in sample discrimination, making false-positive correction an explicit part of the benchmark rather than an incidental byproduct.

Seven LLMs are evaluated: Llama-3.1-8B-Instruct, Qwen2.5-Coder-14B-Instruct, Qwen3-32B, DeepSeek-V3-0324, GPT-4o-mini, Gemini-2.5-Flash, and Claude-3.7-Sonnet (Zheng et al., 17 Sep 2025). Open-source models are run locally using vLLM, official API models are used through their APIs, and temperature is fixed at 0 for deterministic outputs (Zheng et al., 17 Sep 2025).

Task-specific evaluation details are tightly specified. Task 1 FS uses an XGBoost classifier on TF-IDF features of function summaries, with an 80% train and 20% test malware split and top 10% most sensitive functions removed for fidelity computation (Zheng et al., 17 Sep 2025). Task 3 report evaluation uses LLM-as-a-judge with GPT-5-mini held out from the evaluated models (Zheng et al., 17 Sep 2025). Additional experiments remove inter-procedural context and add application metadata such as package information, components, and certificate information (Zheng et al., 17 Sep 2025).

The comparison between archived and latest malware is tested using Welch’s t-test, and most models show no significant differences between the two subsets for report quality and evidence authenticity (Zheng et al., 17 Sep 2025). The paper interprets this cautiously, attributing it likely to reuse of malware templates and frameworks, the summary-based evaluation abstraction, and the persistence of familiar behavioral patterns among new families (Zheng et al., 17 Sep 2025). It explicitly cautions that similar AM/LM performance should not be interpreted as full generalization to completely novel attack strategies (Zheng et al., 17 Sep 2025).

6. Empirical findings, failure modes, and relation to adjacent malware-evaluation work

MalEval’s principal empirical result is that contemporary LLMs remain limited as end-to-end malware auditors. Even the best model, Claude-3.7-Sonnet, achieves a WRS of 50.67 on the base setup, which the benchmark characterizes as still far from reliable malware auditing (Zheng et al., 17 Sep 2025). This result is notable because it tempers the apparent strength of LLMs in isolated subtasks with a workflow-level measure of trustworthiness and usefulness.

The task-level findings are differentiated. In function prioritization, all models have low FS, indicating difficulty in isolating the few truly decisive malicious functions; Claude-3.7-Sonnet is best at pinpointing important functions (Zheng et al., 17 Sep 2025). In evidence attribution, Qwen-family models show strong CSR, especially Qwen2.5-Coder and Qwen3-32B, but this is accompanied by weak precision in identifying the most decisive functions, which the paper interprets as sensitivity to surface API cues rather than deep causal reasoning (Zheng et al., 17 Sep 2025). In behavior synthesis, Gemini-2.5-Flash achieves the best report quality, Claude-3.7-Sonnet is strong in accurate categorization and sample discrimination, DeepSeek-V3 produces coherent reports but struggles to reject false positives, GPT-4o-mini is shallow and brittle, and Llama-3.1-8B performs poorly across nearly all reporting metrics (Zheng et al., 17 Sep 2025). In sample discrimination, Claude-3.7-Sonnet and Gemini-2.5-Flash are strongest, though some models improve false-positive correction at the cost of true-positive maintenance (Zheng et al., 17 Sep 2025).

The ablation results reinforce the importance of representation and context. Removing inter-procedural context sharply reduces FS, reduces RQ and EAS, and lowers WRS, although CSR may remain stable and sample discrimination can show model-dependent effects (Zheng et al., 17 Sep 2025). Adding application metadata has mixed effects: it can improve false-positive correction for some models, can harm true-positive maintenance, and may reduce report quality for code-centric models. The paper notes that DeepSeek-V3 and Claude-3.7-Sonnet can benefit, Gemini-2.5-Flash may be distracted by it, and smaller models often collapse under the extra information (Zheng et al., 17 Sep 2025). This suggests that auxiliary context is not uniformly beneficial; the utility of metadata is mediated by model capacity and task stage.

MalEval also identifies recurring failure modes. These include inability to synthesize global threat objectives, superficial technical interpretation, benign-by-association bias, and logical blindness to legal API abuse when malicious intent emerges only from the combination of individually legitimate calls (Zheng et al., 17 Sep 2025). The broader pattern is that models are relatively good at API-driven or surface-pattern behaviors but weak at multi-step reasoning, causal inference, and cross-function attack-chain reconstruction, with hallucination control remaining a major issue for smaller models (Zheng et al., 17 Sep 2025).

In relation to adjacent research, MalEval occupies a distinct niche. MalCVE demonstrates that LLM-based pipelines can achieve strong binary malware detection and nontrivial CVE association on decompiled and deobfuscated JAR malware, with 97% mean detection accuracy and recall@10 up to 65% on CVE attribution (Cristea et al., 17 Oct 2025). MeLeMaD reports very high detector performance on CIC-AndMal2020, BODMAS, and EMBOD through chunk-wise feature selection and MAML-based meta-learning (K et al., 30 Dec 2025). Those results indicate that malware detection and related attribution tasks can show high performance under task-specific formulations. MalEval’s contribution is to show that these capabilities do not directly translate into reliable fine-grained Android malware behavior auditing. A plausible implication is that audit-quality explanation, evidence grounding, and false-positive correction constitute a harder problem class than detection alone.

MalEval’s broader significance lies in reframing malware evaluation from final classification toward trustworthy analyst support. Its main contributions are a new benchmark for malware auditing beyond classification, a traceable workflow grounded in analyst practice, expert-verified reports and an updated sensitive API list, a function-level structural representation for verifiable evaluation, four analyst-aligned tasks, a workload-oriented metric, and a systematic evaluation of seven LLMs on recent malware and misclassified benign apps (Zheng et al., 17 Sep 2025). Within that framework, MalEval shows that current LLMs can assist with triage and summarization, but are not yet reliable enough for fully automated malware auditing (Zheng et al., 17 Sep 2025).