GDPR-Bench-Android: Compliance Benchmark

• GDPR-Bench-Android is a comprehensive, source-code-native benchmark containing 1951 manually annotated GDPR violation instances from 15 Android repositories covering 23 Articles.
• It defines two evaluation tasks—multi-granularity violation localization and snippet-level multi-label classification—employing metrics such as Accuracy@k and Macro-F1 to assess diverse methods.
• The benchmark compares heterogeneous approaches, including LLMs, retrieval-augmented and agentic methods, alongside a deterministic Formal-AST baseline for clear, reproducible legal reasoning.

GDPR-Bench-Android is a benchmark for evaluating automated GDPR compliance detection in Android applications. It was introduced as “the first comprehensive benchmark for evaluating diverse automated methods for GDPR compliance detection in Android applications,” and it contains 1951 manually annotated violation instances from 15 open-source repositories, covering 23 GDPR articles at file-, module-, and line-level granularities (Ran et al., 1 Nov 2025). The benchmark is source-code-native rather than APK-centric, defines two evaluation tasks, and includes Formal-AST, a deterministic formal baseline intended to support multi-paradigm comparison across rule-based, retrieval-augmented, agentic, and large-language-model approaches (Ran et al., 1 Nov 2025).

1. Motivation and problem setting

Modern Android apps routinely handle personal data, which triggers stringent GDPR requirements including lawfulness (Art 6), data minimization (Art 5), and security by design (Arts 25/32) (Ran et al., 1 Nov 2025). In this setting, manual audits are described as slow and error-prone, while existing static tools such as FlowDroid, Amandroid, and IccTA operate on APKs rather than source code and struggle with legal semantics (Ran et al., 1 Nov 2025). At the same time, LLMs and agentic or retrieval-augmented methods offer promise for code + law reasoning, but lacked a standard, source-code-native benchmark for fair comparison before GDPR-Bench-Android (Ran et al., 1 Nov 2025).

The benchmark is designed to fill that gap in three ways. First, it provides a large, manually annotated source-code corpus of real GDPR violations in Android apps. Second, it introduces a deterministic formal baseline directly on source code. Third, it defines two complementary tasks—multi-granularity localization and snippet-level multi-label classification—with explicit metrics (Ran et al., 1 Nov 2025). This design makes GDPR-Bench-Android not merely a dataset, but an evaluation framework for comparing heterogeneous paradigms under a common protocol.

A notable characteristic of the benchmark corpus is that it includes many “spy” or RAT apps to ensure unambiguous violations (Ran et al., 1 Nov 2025). This choice indicates an emphasis on annotation clarity and legal salience rather than on sampling neutrality across the Android ecosystem. A plausible implication is that the benchmark prioritizes diagnostic value for method comparison over population representativeness.

2. Dataset composition and annotation structure

GDPR-Bench-Android is built from 15 open-source Android repositories and contains 1951 distinct violation instances (Ran et al., 1 Nov 2025). Each instance is linked to (app_name, repo_url, commit_id), a code path / snippet, a single violated_article, and a human-written annotation_note (Ran et al., 1 Nov 2025). Violations span 23 distinct GDPR Articles, with a markedly skewed distribution: Art 6 (Lawfulness) contributes 442 instances (22.66%), Art 5 (Principles) contributes 430 (22.04%), Art 25 (Privacy by design) contributes 311 (15.94%), and Art 32 (Security) contributes 254 (13.02%); the remaining 19 Articles form a long tail, with Art 16 or Art 18 appearing only once (Ran et al., 1 Nov 2025).

The benchmark also characterizes snippet granularity and representation. Of the 1951 snippets, 957 are single-line (49.05%) and 994 are multi-line (50.95%) (Ran et al., 1 Nov 2025). The file-type distribution is heterogeneous: .js 27.06%, .json 24.30%, .java 15.27%, .kt 12.51%, .cs 8.92%, .php 6.46%, with the remainder categorized as others (Ran et al., 1 Nov 2025). Average lengths are reported as approximately 171 characters for code snippets with $\sigma \approx 206$, and approximately 224 characters for annotation notes with $\sigma \approx 90$ (Ran et al., 1 Nov 2025).

Two task-specific datasets are derived from these 1951 records (Ran et al., 1 Nov 2025). Task 1 uses 368 “file-centric” entries, each aggregating file-, module-, and line-level violations. Task 2 uses 887 unique snippet entries with multi-label arrays of all violated articles (Ran et al., 1 Nov 2025). The distinction is methodologically important: Task 1 evaluates ranked localization across hierarchical context windows, whereas Task 2 evaluates exhaustive label recovery from isolated snippets.

The benchmark’s annotation schema embeds both legal and program-analysis structure. Because each record includes a violated article and a human-written annotation note, the dataset supports both doctrinal interpretation and code-level evidence alignment. This suggests that GDPR-Bench-Android can be used not only for end-task benchmarking, but also for error analysis on article-specific and context-specific failure modes.

3. Tasks and evaluation protocol

GDPR-Bench-Android defines two tasks (Ran et al., 1 Nov 2025). Task 1, multi-granularity violation localization, requires a method to rank all 23 Articles by likelihood of violation for each file, module (class/package), or line snippet. Evaluation uses Accuracy@k, asking whether a correct article appears in the top-$k$ predictions. Let $N$ be the number of examples, and for example $i$ at granularity $g$ let the rank of its first correct article be $r_i^{(g)}$. Then

$$\mathrm{Accuracy}@k_g = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\{r_i^{(g)} \le k\}, \quad k=1,2,\dots,5.$$

The benchmark reports Accuracy@1 through Accuracy@5 (Ran et al., 1 Nov 2025).

Task 2, snippet-level multi-label classification, gives a method an isolated snippet and requires prediction of the full set of violated_articles (Ran et al., 1 Nov 2025). For this task, the benchmark uses Exact-match Accuracy, Macro-Precision, Macro-Recall, and Macro-F1, averaged over $C=23$ articles (Ran et al., 1 Nov 2025). The use of macro-averaged metrics is significant because the article distribution is long-tailed; it prevents performance on the dominant articles from completely obscuring performance on rarely occurring articles.

The two tasks probe different capabilities. Task 1 emphasizes ranked legal attribution under varying context sizes, from line-local cues to file-level architectural reasoning. Task 2 emphasizes exhaustive label recovery under a difficult multi-label regime. The benchmark’s own findings later show that these capabilities are not aligned across paradigms (Ran et al., 1 Nov 2025).

4. Evaluated methods and the Formal-AST baseline

The benchmark evaluates 11 methods: eight state-of-the-art LLMs, Formal-AST, a retrieval-augmented generation (RAG) method, and an agentic (ReAct) method (Ran et al., 1 Nov 2025). The baseline LLMs are GPT-4o, O1, Claude-3.7-Sonnet-20250219, Claude-Sonnet-4.5-20250929, Deepseek-r1, Qwen2.5-72B-instruct, Gemini-2.5-pro-preview-06-05-thinking, and Gemini-2.5-pro-thinking, all evaluated in zero-shot, single completion, temperature=0.0 mode (Ran et al., 1 Nov 2025).

The RAG method builds a vector store of all GDPR article texts plus 1900+ code + violation examples, retrieves top-$k$ relevant documents, and then invokes GPT-4o on the snippet (Ran et al., 1 Nov 2025). The ReAct method also uses GPT-4o as the “brain,” but places it inside a Reason+Act loop with three tools: gdpr_lookup(article_id) → definition, code_search(keyword) → file/line hits, and rule_check(snippet) → simple static-analysis verdict (Ran et al., 1 Nov 2025). The result is iterative planning with tool calls followed by a final violation list (Ran et al., 1 Nov 2025).

Formal-AST is the benchmark’s distinctive symbolic component. It is a source-code-native, rule-based analyzer that parses Java/Kotlin (and other) files into ASTs, extracts semantic predicates such as CollectsDataPredicate and HasConsentPredicate, and applies approximately 35 hand-crafted logical rules (Ran et al., 1 Nov 2025). Representative rules include:

• CollectsData $\sigma \approx 90$0HasConsent $\sigma \approx 90$1 Violation(Article 6)
• TransmitsOverHTTP $\sigma \approx 90$2UsesTLS $\sigma \approx 90$3 Violation(Article 32)

Formal-AST is described as deterministic and fully transparent, and it serves as the benchmark’s “floor” for comparison (Ran et al., 1 Nov 2025). In methodological terms, it anchors the evaluation space by providing a baseline with explicit semantics and reproducible behavior, in contrast to the stochastic and prompt-sensitive properties of LLM-based systems.

5. Empirical results and paradigm-dependent behavior

The benchmark’s central empirical conclusion is that no single paradigm excels across all tasks (Ran et al., 1 Nov 2025). On Task 1, the best method depends on granularity. At the file level, the ReAct agent achieves the highest Accuracy@1 at 17.38%, with Deepseek-r1 next at 11.55% (Ran et al., 1 Nov 2025). At the module level, ReAct again leads with 17.13%, compared with Formal-AST 9.73% and Claude-4.5 9.60% (Ran et al., 1 Nov 2025). At the line level, Qwen2.5-72B leads with 61.60%, ahead of GPT-4o 57.45% and RAG 56.87%, and in stark contrast to Formal-AST’s 1.86% (Ran et al., 1 Nov 2025).

On Task 2, which is described as a difficult multi-label problem, the strongest scores remain low in absolute terms (Ran et al., 1 Nov 2025). Claude-Sonnet-4.5 achieves the best Macro-F1 at 5.75% and the best Macro-Recall at 6.53%, while the RAG method achieves the highest Macro-Precision at 7.10% (Ran et al., 1 Nov 2025). These figures indicate that exhaustive snippet-level legal classification remains challenging even for the best-performing methods in the benchmark.

The reported error profile is paradigm-dependent (Ran et al., 1 Nov 2025). Baseline LLMs are described as strong at localized, snippet-level pattern matching, especially in Task 1 at the line granularity, but weaker on long contexts and on exhaustive multi-label prediction; they also suffer from hallucinations on ambiguous code (Ran et al., 1 Nov 2025). Formal-AST is described as perfectly precise when a rule fires, but with extremely low recall on snippets and semantic blindness to patterns not hand-coded (Ran et al., 1 Nov 2025). RAG yields the highest precision in multi-label classification, attributed to in-context examples, and moderately boosts the LLM base on Task 1 medium contexts (Ran et al., 1 Nov 2025). ReAct performs best for holistic, long-context file and module localization through tool-driven reasoning, with less gain at the line level where context is small (Ran et al., 1 Nov 2025).

The benchmark interprets the hierarchical file $\sigma \approx 90$4 module $\sigma \approx 90$5 line breakdown as revealing that neural methods rely heavily on local cues, while symbolic or agentic approaches are needed for architectural reasoning (Ran et al., 1 Nov 2025). This is an analysis claim reported by the benchmark itself rather than an external extrapolation.

6. Relation to earlier Android GDPR-compliance research

GDPR-Bench-Android sits within a broader line of work on Android privacy and GDPR analysis, but it occupies a distinct position by making source-code-native benchmark evaluation the primary object (Ran et al., 1 Nov 2025). Earlier work by Khedkar and Bodden proposed a static analysis approach for data protection in Android apps using Soot, SootFX, and Jicer, with a three-point lattice over Public, Personal, and Disguised data; that work explicitly identified publication of a standard benchmark suite of Android apps annotated with GDPR-relevant ground truth as a next step (Khedkar et al., 2024). In that sense, GDPR-Bench-Android can be read as a concrete realization of a benchmarking agenda that earlier static-analysis work had already articulated.

A second precursor is SliceViz, which analyzes an Android app through privacy-relevant program slicing over an Application Dependence Graph, visualizes the resulting slices, and describes how such slice extraction could seed a structured GDPR benchmark for Android apps (Khedkar et al., 20 Mar 2025). SliceViz emphasizes privacy-relevant data sources, sinks, and warning-scale visualization, whereas GDPR-Bench-Android formalizes benchmark tasks and comparative evaluation across automated paradigms.

Other related systems targeted narrower compliance subproblems. An earlier automated system for mHealth apps checked privacy-policy completeness, data-collection consistency, and transmission security on 796 apps, reporting substantial violation rates (Fan et al., 2020). A large-scale study of cross-border personal data transfers analyzed 10,080 Android apps and found that approximately 56% were potentially non-compliant with GDPR transfer requirements (Guamán et al., 2021). Work on privacy-related data reporting constructed UI- and API-based datasets to compare detected versus declared data collection in Android apps, highlighting both under-reporting and over-reporting (Khedkar et al., 2024). RADS-Checker extended the scope to Right of Access by the Data Subject, combining privacy-policy analysis, request submission, and completeness verification of returned data copies (Li et al., 2024).

Relative to these predecessors, GDPR-Bench-Android is broader in article coverage and more explicit in evaluation design. It covers 23 GDPR Articles, includes file-, module-, and line-level localization, and benchmarks 11 methods spanning formal, neural, retrieval-augmented, and agentic approaches (Ran et al., 1 Nov 2025). This suggests a shift from building single-purpose compliance detectors toward establishing a common experimental substrate for comparative research.

7. Limitations, interpretive boundaries, and future directions

The benchmark identifies several limitations (Ran et al., 1 Nov 2025). It is static, source-only, and Android-focused; it does not model runtime behavior or cross-file flows. The evaluation is zero-shot, so few-shot prompting or fine-tuning may improve LLMs. Additional limitations include prompt sensitivity, limited commit context, and annotation ambiguity in legal interpretation, since ground truth depends on expert judgment (Ran et al., 1 Nov 2025).

These limitations are substantive rather than incidental. A static, source-only benchmark can evaluate source-code reasoning, but it cannot directly capture dynamic consent flows, runtime data exfiltration, or backend processing. Likewise, expert judgment is unavoidable in legal annotation, but it introduces a dependence on interpretive choices. The benchmark therefore measures automated performance against a carefully constructed operationalization of GDPR violations, not against an uncontested legal oracle. This suggests that leaderboard results should be read as comparative diagnostics rather than as final statements about legal correctness.

The future directions proposed for the benchmark are correspondingly expansive (Ran et al., 1 Nov 2025). They include extension to iOS/web backends, incorporation of inter-procedural analysis, modeling of dynamic consent flows, exploration of chain-of-thought prompts and fine-tuning on GDPR data, development of richer Formal-AST rule sets, construction of hybrid neuro-symbolic pipelines, and longitudinal commit-history benchmarking to capture code evolution (Ran et al., 1 Nov 2025). The benchmark also states that all code, data, prompts, and the Formal-AST implementation are publicly available at https://github.com/Haoyi-Zhang/GDPR-Bench-Android (Ran et al., 1 Nov 2025).

The benchmark’s most important general insight is that there is no single “silver-bullet” approach (Ran et al., 1 Nov 2025). Hybrid systems that combine formal guarantees, retrieval, and LLM reasoning are identified as the most promising direction (Ran et al., 1 Nov 2025). Given the reported strengths and weaknesses of Formal-AST, RAG, ReAct, and baseline LLMs, that conclusion is not merely speculative; it follows directly from the benchmark’s task-dependent performance landscape.