---
title: 'GDPR-Bench-Android: Compliance Benchmark'
url: https://www.emergentmind.com/topics/gdpr-bench-android
type: topic
---

# GDPR-Bench-Android: Compliance Benchmark

GDPR-Bench-Android is a benchmark for evaluating automated GDPR compliance detection in Android applications. It was introduced as “the first comprehensive benchmark for evaluating diverse automated methods for GDPR compliance detection in Android applications,” and it contains **1951** manually annotated violation instances from **15** open-source repositories, covering **23 GDPR articles** at **file-, module-, and line-level granularities** [2511.00619]. The benchmark is source-code-native rather than APK-centric, defines two evaluation tasks, and includes **Formal-AST**, a deterministic formal baseline intended to support multi-paradigm comparison across rule-based, retrieval-augmented, agentic, and large-language-model approaches [2511.00619].

## 1. Motivation and problem setting

Modern Android apps routinely handle personal data, which triggers stringent GDPR requirements including **lawfulness (Art 6)**, **data minimization (Art 5)**, and **security by design (Arts 25/32)** [2511.00619]. In this setting, manual audits are described as slow and error-prone, while existing static tools such as **FlowDroid**, **Amandroid**, and **IccTA** operate on APKs rather than source code and struggle with legal semantics [2511.00619]. At the same time, LLMs and agentic or retrieval-augmented methods offer promise for code + law reasoning, but lacked a standard, source-code-native benchmark for fair comparison before GDPR-Bench-Android [2511.00619].

The benchmark is designed to fill that gap in three ways. First, it provides a large, manually annotated source-code corpus of real GDPR violations in Android apps. Second, it introduces a deterministic formal baseline directly on source code. Third, it defines two complementary tasks—**multi-granularity localization** and **snippet-level multi-label classification**—with explicit metrics [2511.00619]. This design makes GDPR-Bench-Android not merely a dataset, but an evaluation framework for comparing heterogeneous paradigms under a common protocol.

A notable characteristic of the benchmark corpus is that it includes many “spy” or RAT apps to ensure unambiguous violations [2511.00619]. This choice indicates an emphasis on annotation clarity and legal salience rather than on sampling neutrality across the Android ecosystem. A plausible implication is that the benchmark prioritizes diagnostic value for method comparison over population representativeness.

## 2. Dataset composition and annotation structure

GDPR-Bench-Android is built from **15 open-source Android repositories** and contains **1951 distinct violation instances** [2511.00619]. Each instance is linked to **(app_name, repo_url, commit_id)**, a **code path / snippet**, a single **violated_article**, and a human-written **annotation_note** [2511.00619]. Violations span **23 distinct GDPR Articles**, with a markedly skewed distribution: **Art 6 (Lawfulness)** contributes **442 instances (22.66%)**, **Art 5 (Principles)** contributes **430 (22.04%)**, **Art 25 (Privacy by design)** contributes **311 (15.94%)**, and **Art 32 (Security)** contributes **254 (13.02%)**; the remaining **19 Articles** form a long tail, with **Art 16** or **Art 18** appearing only once [2511.00619].

The benchmark also characterizes snippet granularity and representation. Of the **1951** snippets, **957** are **single-line (49.05%)** and **994** are **multi-line (50.95%)** [2511.00619]. The file-type distribution is heterogeneous: **.js 27.06%**, **.json 24.30%**, **.java 15.27%**, **.kt 12.51%**, **.cs 8.92%**, **.php 6.46%**, with the remainder categorized as others [2511.00619]. Average lengths are reported as approximately **171 characters** for code snippets with **$\sigma \approx 206$**, and approximately **224 characters** for annotation notes with **$\sigma \approx 90$** [2511.00619].

Two task-specific datasets are derived from these **1951** records [2511.00619]. **Task 1** uses **368 “file-centric” entries**, each aggregating file-, module-, and line-level violations. **Task 2** uses **887 unique snippet entries** with multi-label arrays of all violated articles [2511.00619]. The distinction is methodologically important: Task 1 evaluates ranked localization across hierarchical context windows, whereas Task 2 evaluates exhaustive label recovery from isolated snippets.

The benchmark’s annotation schema embeds both legal and program-analysis structure. Because each record includes a violated article and a human-written annotation note, the dataset supports both doctrinal interpretation and code-level evidence alignment. This suggests that GDPR-Bench-Android can be used not only for end-task benchmarking, but also for error analysis on article-specific and context-specific failure modes.

## 3. Tasks and evaluation protocol

GDPR-Bench-Android defines two tasks [2511.00619]. **Task 1**, multi-granularity violation localization, requires a method to rank all **23 Articles** by likelihood of violation for each **file**, **module (class/package)**, or **line snippet**. Evaluation uses **Accuracy@k**, asking whether a correct article appears in the top-$k$ predictions. Let $N$ be the number of examples, and for example $i$ at granularity $g$ let the rank of its first correct article be $r_i^{(g)}$. Then

$$
\mathrm{Accuracy}@k_g
=
\frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\{r_i^{(g)} \le k\},
\quad k=1,2,\dots,5.
$$

The benchmark reports **Accuracy@1** through **Accuracy@5** [2511.00619].

**Task 2**, snippet-level multi-label classification, gives a method an isolated snippet and requires prediction of the full set of **violated_articles** [2511.00619]. For this task, the benchmark uses **Exact-match Accuracy**, **Macro-Precision**, **Macro-Recall**, and **Macro-F1**, averaged over **$C=23$ articles** [2511.00619]. The use of macro-averaged metrics is significant because the article distribution is long-tailed; it prevents performance on the dominant articles from completely obscuring performance on rarely occurring articles.

The two tasks probe different capabilities. Task 1 emphasizes ranked legal attribution under varying context sizes, from line-local cues to file-level architectural reasoning. Task 2 emphasizes exhaustive label recovery under a difficult multi-label regime. The benchmark’s own findings later show that these capabilities are not aligned across paradigms [2511.00619].

## 4. Evaluated methods and the Formal-AST baseline

The benchmark evaluates **11 methods**: **eight state-of-the-art LLMs**, **Formal-AST**, a **retrieval-augmented generation (RAG)** method, and an **agentic (ReAct)** method [2511.00619]. The baseline LLMs are **GPT-4o**, **O1**, **Claude-3.7-Sonnet-20250219**, **Claude-Sonnet-4.5-20250929**, **Deepseek-r1**, **Qwen2.5-72B-instruct**, **Gemini-2.5-pro-preview-06-05-thinking**, and **Gemini-2.5-pro-thinking**, all evaluated in **zero-shot**, **single completion**, **temperature=0.0** mode [2511.00619].

The **RAG** method builds a vector store of all GDPR article texts plus **1900+ code + violation examples**, retrieves top-$k$ relevant documents, and then invokes **GPT-4o** on the snippet [2511.00619]. The **ReAct** method also uses **GPT-4o** as the “brain,” but places it inside a Reason+Act loop with three tools: **gdpr_lookup(article_id) → definition**, **code_search(keyword) → file/line hits**, and **rule_check(snippet) → simple static-analysis verdict** [2511.00619]. The result is iterative planning with tool calls followed by a final violation list [2511.00619].

**Formal-AST** is the benchmark’s distinctive symbolic component. It is a **source-code-native, rule-based analyzer** that parses **Java/Kotlin (and other) files** into ASTs, extracts semantic predicates such as **CollectsDataPredicate** and **HasConsentPredicate**, and applies approximately **35 hand-crafted logical rules** [2511.00619]. Representative rules include:

- **CollectsData $\land \neg$HasConsent $\Rightarrow$ Violation(Article 6)**
- **TransmitsOverHTTP $\land \neg$UsesTLS $\Rightarrow$ Violation(Article 32)**

Formal-AST is described as **deterministic** and **fully transparent**, and it serves as the benchmark’s “floor” for comparison [2511.00619]. In methodological terms, it anchors the evaluation space by providing a baseline with explicit semantics and reproducible behavior, in contrast to the stochastic and prompt-sensitive properties of LLM-based systems.

## 5. Empirical results and paradigm-dependent behavior

The benchmark’s central empirical conclusion is that **no single paradigm excels across all tasks** [2511.00619]. On **Task 1**, the best method depends on granularity. At the **file level**, the **ReAct agent** achieves the highest **Accuracy@1** at **17.38%**, with **Deepseek-r1** next at **11.55%** [2511.00619]. At the **module level**, **ReAct** again leads with **17.13%**, compared with **Formal-AST 9.73%** and **Claude-4.5 9.60%** [2511.00619]. At the **line level**, **Qwen2.5-72B** leads with **61.60%**, ahead of **GPT-4o 57.45%** and **RAG 56.87%**, and in stark contrast to **Formal-AST’s 1.86%** [2511.00619].

On **Task 2**, which is described as a difficult multi-label problem, the strongest scores remain low in absolute terms [2511.00619]. **Claude-Sonnet-4.5** achieves the best **Macro-F1** at **5.75%** and the best **Macro-Recall** at **6.53%**, while the **RAG** method achieves the highest **Macro-Precision** at **7.10%** [2511.00619]. These figures indicate that exhaustive snippet-level legal classification remains challenging even for the best-performing methods in the benchmark.

The reported error profile is paradigm-dependent [2511.00619]. Baseline LLMs are described as strong at localized, snippet-level pattern matching, especially in **Task 1** at the line granularity, but weaker on long contexts and on exhaustive multi-label prediction; they also suffer from hallucinations on ambiguous code [2511.00619]. **Formal-AST** is described as perfectly precise when a rule fires, but with extremely low recall on snippets and semantic blindness to patterns not hand-coded [2511.00619]. **RAG** yields the highest precision in multi-label classification, attributed to in-context examples, and moderately boosts the LLM base on Task 1 medium contexts [2511.00619]. **ReAct** performs best for holistic, long-context file and module localization through tool-driven reasoning, with less gain at the line level where context is small [2511.00619].

The benchmark interprets the hierarchical file $\rightarrow$ module $\rightarrow$ line breakdown as revealing that neural methods rely heavily on local cues, while symbolic or agentic approaches are needed for architectural reasoning [2511.00619]. This is an analysis claim reported by the benchmark itself rather than an external extrapolation.

## 6. Relation to earlier Android GDPR-compliance research

GDPR-Bench-Android sits within a broader line of work on Android privacy and GDPR analysis, but it occupies a distinct position by making **source-code-native benchmark evaluation** the primary object [2511.00619]. Earlier work by **Khedkar and Bodden** proposed a static analysis approach for data protection in Android apps using **Soot**, **SootFX**, and **Jicer**, with a three-point lattice over **Public**, **Personal**, and **Disguised** data; that work explicitly identified publication of a standard benchmark suite of Android apps annotated with GDPR-relevant ground truth as a next step [2402.07889]. In that sense, GDPR-Bench-Android can be read as a concrete realization of a benchmarking agenda that earlier static-analysis work had already articulated.

A second precursor is **SliceViz**, which analyzes an Android app through privacy-relevant program slicing over an **Application Dependence Graph**, visualizes the resulting slices, and describes how such slice extraction could seed a structured GDPR benchmark for Android apps [2503.16640]. SliceViz emphasizes privacy-relevant data sources, sinks, and warning-scale visualization, whereas GDPR-Bench-Android formalizes benchmark tasks and comparative evaluation across automated paradigms.

Other related systems targeted narrower compliance subproblems. An earlier automated system for **mHealth apps** checked **privacy-policy completeness**, **data-collection consistency**, and **transmission security** on **796** apps, reporting substantial violation rates [2008.05864]. A large-scale study of **cross-border personal data transfers** analyzed **10,080** Android apps and found that approximately **56%** were potentially non-compliant with GDPR transfer requirements [2103.07297]. Work on **privacy-related data reporting** constructed UI- and API-based datasets to compare detected versus declared data collection in Android apps, highlighting both under-reporting and over-reporting [2409.04167]. **RADS-Checker** extended the scope to **Right of Access by the Data Subject**, combining privacy-policy analysis, request submission, and completeness verification of returned data copies [2410.12463].

Relative to these predecessors, GDPR-Bench-Android is broader in article coverage and more explicit in evaluation design. It covers **23 GDPR Articles**, includes **file-, module-, and line-level** localization, and benchmarks **11 methods** spanning formal, neural, retrieval-augmented, and agentic approaches [2511.00619]. This suggests a shift from building single-purpose compliance detectors toward establishing a common experimental substrate for comparative research.

## 7. Limitations, interpretive boundaries, and future directions

The benchmark identifies several limitations [2511.00619]. It is **static**, **source-only**, and **Android-focused**; it does not model runtime behavior or cross-file flows. The evaluation is **zero-shot**, so few-shot prompting or fine-tuning may improve LLMs. Additional limitations include **prompt sensitivity**, **limited commit context**, and **annotation ambiguity in legal interpretation**, since ground truth depends on expert judgment [2511.00619].

These limitations are substantive rather than incidental. A static, source-only benchmark can evaluate source-code reasoning, but it cannot directly capture dynamic consent flows, runtime data exfiltration, or backend processing. Likewise, expert judgment is unavoidable in legal annotation, but it introduces a dependence on interpretive choices. The benchmark therefore measures automated performance against a carefully constructed operationalization of GDPR violations, not against an uncontested legal oracle. This suggests that leaderboard results should be read as comparative diagnostics rather than as final statements about legal correctness.

The future directions proposed for the benchmark are correspondingly expansive [2511.00619]. They include extension to **iOS/web backends**, incorporation of **inter-procedural analysis**, modeling of **dynamic consent flows**, exploration of **chain-of-thought prompts** and **fine-tuning on GDPR data**, development of richer **Formal-AST** rule sets, construction of **hybrid neuro-symbolic pipelines**, and **longitudinal commit-history benchmarking** to capture code evolution [2511.00619]. The benchmark also states that all code, data, prompts, and the Formal-AST implementation are publicly available at **https://github.com/Haoyi-Zhang/GDPR-Bench-Android** [2511.00619].

The benchmark’s most important general insight is that there is **no single “silver-bullet” approach** [2511.00619]. Hybrid systems that combine **formal guarantees**, **retrieval**, and **LLM reasoning** are identified as the most promising direction [2511.00619]. Given the reported strengths and weaknesses of Formal-AST, RAG, ReAct, and baseline LLMs, that conclusion is not merely speculative; it follows directly from the benchmark’s task-dependent performance landscape.

Source: https://www.emergentmind.com/topics/gdpr-bench-android