LLMAID: AI Discovery in Android Apps

• LLMAID is a framework that integrates static analysis with LLM-based semantic reasoning to automatically detect and classify AI capabilities within Android APKs.
• The system achieved over 90% precision and recall in detecting AI components, identifying 242% more AI applications than state-of-the-art rule-based methods.
• It generates concise AI service summaries that aid developers, regulators, and users in understanding embedded AI functionalities and auditing mobile apps.

Searching arXiv for the specified paper to ground the article and citation. LLMAID, short for LLM for AI Discovery, is a semantics-driven framework for automatically identifying and summarizing AI capabilities in Android apps using LLMs. It is designed to answer two operational questions that are increasingly salient for users, regulators, and researchers: whether an application incorporates AI capabilities, and what kinds of AI services it provides. The framework combines static program analysis with LLM-based semantic reasoning to detect AI-relevant components in APKs, classify those components by functionality, and generate concise summaries of embedded AI services. In the reported evaluation, LLMAID was applied to a dataset of 4,201 Android applications and identified 242% more real-world AI apps than a state-of-the-art rule-based approach, while achieving precision and recall above 90% in detecting AI-related components (Liu et al., 24 Nov 2025).

1. Problem setting and motivation

Modern mobile apps increasingly embed both on-device and cloud-based AI, including TensorFlow Lite models, Firebase ML Kit calls, and OpenAI GPT APIs, to support image recognition, NLP, recommendation, and generative services (Liu et al., 24 Nov 2025). This growth creates a practical auditing problem: AI functionality is often present, but not directly visible from store descriptions or conventional software metadata.

LLMAID is motivated by the observation that AI indicators in Android packages are fragmented across multiple artifact types. AI can appear as Java or Kotlin packages, native libraries in .so form, on-device model files such as .tflite and .caffemodel, or hard-coded HTTPS calls to cloud APIs. This heterogeneity complicates robust detection. Rule-based scanners can become stale as new SDKs and services appear, and they may miss lightly invoked APIs or obfuscated components. Manual inspection, although potentially precise, does not scale to large app ecosystems and cannot richly characterize functionality (Liu et al., 24 Nov 2025).

Within this setting, LLMAID defines its objective as automating AI Discovery across three linked subtasks: detecting all AI-relevant components in an APK, classifying them by functionality, and producing a concise, user-friendly summary of the app’s AI services. This framing distinguishes simple artifact matching from semantic identification of capability (Liu et al., 24 Nov 2025).

2. System architecture and pipeline

LLMAID is organized into four main tasks: candidate extraction, knowledge base interaction, AI capability analysis and detection, and AI service summarization (Liu et al., 24 Nov 2025). The pipeline begins with APK analysis and ends with a generated report oriented toward developers, end users, or regulators.

In candidate extraction, each APK is unpacked with AAPT2; all .dex files are decompiled to Jimple via Soot + Dexpler; native libraries are parsed with readelf and objdump; and assets are scanned. This stage extracts fine-grained components, including package names, API signatures, on-device model files, and hard-coded HTTPS endpoints. A coarse rule-based whitelist filter then removes known non-AI system and Android framework packages such as androidx.*, java.*, and android.*. The output is a large candidate set per app, with a reported median of approximately 433 packages, 296 APIs, and 53 HTTPS calls (Liu et al., 24 Nov 2025).

In knowledge base interaction, LLMAID maintains two caches: AI-KB, containing previously verified AI components and their functional descriptions, and non-AI-KB, containing previously verified non-AI components. For each candidate, membership in non-AI-KB triggers immediate discard; membership in AI-KB reuses the stored decision and skips the LLM; otherwise the candidate is forwarded for LLM-based analysis. Once labeled, it is inserted into the appropriate knowledge base. The paper states that this caching reduces redundant LLM calls and gradually improves throughput and precision (Liu et al., 24 Nov 2025).

In AI capability analysis and detection, LLMAID uses GPT-3.5-turbo in two sub-steps. First, a capability-analysis prompt asks the model to interpret the functionality of each component and produce a short description. Second, a component-determination prompt asks whether the component is AI-related and requests a brief justification. If the answer is affirmative, the component and its description are retained. The reported settings are a 4 K token context window, batch size of 3 components per request, temperature = 0.2, top_p = 0.95, and JSON-structured output (Liu et al., 24 Nov 2025).

In AI service summarization, LLMAID filters out low-level APIs whose intent is already clear from package context, then prompts the LLM to synthesize a concise, user-oriented summary from the list of identified AI-related components and descriptions. The summary can be tailored to different audiences through minor prompt modifications, including compliance officers and end users (Liu et al., 24 Nov 2025).

3. Semantic detection methodology

The core methodological distinction of LLMAID is that it does not treat AI detection as pure keyword matching. Instead, it uses static analysis to recover candidate artifacts and then relies on semantic interpretation to decide whether those artifacts implement AI functionality. This is operationalized through the two-stage LLM process of capability analysis followed by component determination (Liu et al., 24 Nov 2025).

The paper provides a component-level decision rule for a candidate component $c$:

$$\hat y_c \;=\; \begin{cases} 1, & \text{if LLM\_determine}(c,\text{desc}_c) = \text{“Yes”},\ 0, & \text{otherwise.} \end{cases}$$

Here, $\text{desc}_c$ is the short functional description returned by the capability-analysis prompt. The determination stage therefore depends not only on the raw component string but also on a semantically enriched intermediate representation. A plausible implication is that the framework is designed to handle components whose names are not themselves transparently descriptive, provided the LLM can infer their role from context (Liu et al., 24 Nov 2025).

The paper also describes an alternative ordering in which LLMAID first classifies and then analyzes. This reduces runtime but decreases detection coverage: Analysis→Detection takes 638.8 s/app, whereas Detection→Analysis takes 226.9 s/app and is 65% faster, but yields approximately 10% fewer detected AI components (Liu et al., 24 Nov 2025). This establishes an explicit accuracy–efficiency trade-off within the framework.

4. Experimental design and evaluation

LLMAID is evaluated on two datasets. Dataset 1 consists of 56,682 AI apps previously identified by the AI Discriminator rule-based tool from AndroZoo and is used for cross-validation in RQ1. Dataset 2 contains 15,517 newly curated “in-the-wild” apps, defined as latest releases on Google Play with AI keywords in descriptions and then filtered by a ChatGPT semantic check; it is used for RQ2–RQ4. For RQ1, the study samples 382 apps from Dataset 1. For RQ2–RQ4, it randomly samples 30% of apps per year from 2019 to 2023 in Dataset 2, yielding 3,819 apps (Liu et al., 24 Nov 2025).

The primary baseline is AI Discriminator, described as a state-of-the-art rule-based approach performing keyword matching across package names, API calls, and on-device models. Human annotation covers 49,371 candidate components with two authors and adjudication, and the reported metrics include precision, recall, $F_1$, and Cohen’s $\kappa$ for inter-annotator agreement. The paper defines

$$F_1 = 2 \,\frac{\mathrm{Precision}\,\times\,\mathrm{Recall}} {\mathrm{Precision} + \mathrm{Recall}}.$$

For RQ1, LLMAID reports Precision = 98.05%, Recall = 93.31%, and $\kappa = 0.82$, with app-level accuracy of 99.0% (378/382 AI apps correctly identified). For RQ2, on the in-the-wild sample, LLMAID reports Precision = 97.32%, Recall = 91.01%, and $\kappa = 0.80$. At the app level, LLMAID identifies 1,366 AI apps, whereas AI Discriminator identifies 555, corresponding to +242% more AI apps for LLMAID (Liu et al., 24 Nov 2025).

The comparative app-level classification figures are concise enough to present directly:

Method	AI apps	Non-AI
AI Discriminator	555	3,264
LLMAID	1,366	2,453

These results indicate that LLMAID classifies substantially more apps as AI-enabled than the rule-based baseline. This suggests that semantic analysis recovers AI functionality that heuristic matching fails to recognize, particularly in open-world settings where SDKs, service names, and integration patterns vary (Liu et al., 24 Nov 2025).

5. AI capability taxonomy and distribution across apps

Using LLMAID’s detections, the study performs an empirical analysis of AI capabilities across the 1,366 identified AI apps. At the component level, the paper reports 11,081 total AI components. Of these, Package/API calls account for 92.8% (10,285), On-device Models for 5.8% (644), HTTPS Requests for 1.1% (120), and Others for 0.3% (32) (Liu et al., 24 Nov 2025).

The domain taxonomy contains six top-level categories: Computer Vision (54.80%), Data Analysis (26.85%), NLP (13.38%), Audio/Speech (1.90%), Augmented Reality (0.32%), and Others (2.75%). Among individual tasks, the most common are Object Detection (25.19% of all AI features), Data Processing (23.17%), Text Recognition, and Face Recognition (Liu et al., 24 Nov 2025).

At the app level, the domain assignment counts differ from the component-level distribution:

Domain	# Apps	% of 1,366 AI Apps
Data Analysis	643	47.1%
Computer Vision	634	46.4%
NLP	43	3.1%
Audio/Speech	30	2.2%
Augmented Reality	2	0.1%
Others	14	1.0%

The paper explicitly notes that, although Computer Vision components dominate overall, many apps reuse small Data Analysis pipelines, producing nearly equal app counts for the two largest domains (Liu et al., 24 Nov 2025). This distinction is methodologically important: component prevalence and app prevalence do not coincide. A plausible implication is that ecosystem-level conclusions about “dominant AI use cases” depend strongly on whether the unit of analysis is the component or the application.

6. AI service summarization and user evaluation

LLMAID does not stop at binary detection. Its fourth task produces an AI service summary from the set of AI-related components and their descriptions. The prompt is explicitly framed to yield a concise, user-oriented account of functionality and can be adjusted for different audiences, including developers, end users, and regulators (Liu et al., 24 Nov 2025).

The paper evaluates this summarization stage through a user study with 6 professional Android developers having at least 3 years of experience. The sample contains 20 AI apps randomly drawn from the positive findings of RQ2. For each app, participants compare the original Google Play description with the LLMAID-generated AI summary. The evaluation uses a 10-point Likert scale on five criteria: accuracy of AI functions described, completeness of AI tasks covered, consistency/concreteness, recommendation for end users, and recommendation for developers. Each app is judged independently by 3 participants (Liu et al., 24 Nov 2025).

The reported ratings are as follows:

Summary Type	Accuracy	Completeness	Consistency
LLMAID Summary	8.67 ± 0.14 / 9	8.63 ± 0.13 / 9	8.55 ± 0.13 / 9
App Description	6.35 ± 0.45 / 8	6.17 ± 0.45 / 8	6.02 ± 0.45 / 8

For recommendation, the paper reports 7.50 ± 0.31 / 8 for end users and 8.57 ± 0.15 / 9 for developers for LLMAID summaries, versus 6.87 ± 0.46 / 8 and 6.47 ± 0.45 / 8 respectively for app descriptions (Liu et al., 24 Nov 2025). The stated result is that LLMAID summaries are judged significantly more accurate, complete, and consistent, and more recommendable, especially to developers.

This result clarifies an important conceptual point. LLMAID is not only a detector of AI artifacts but also a reporting system for AI capability disclosure. That reporting function is especially relevant in contexts where app-store descriptions under-specify embedded AI behavior.

7. Significance, limitations, and interpretive issues

LLMAID is positioned against existing approaches that rely on manual inspection and rule-based heuristics. Its main contribution is the integration of precise static analysis with LLM-based semantic reasoning at each stage, from candidate filtering to capability interpretation to human-readable summarization (Liu et al., 24 Nov 2025). Within the reported experiments, this combination yields over 90% precision and recall on both curated and in-the-wild apps, while discovering substantially more AI-enabled apps than the rule-based baseline.

At the same time, the motivating context of the work includes the hallucination and reliability issues of AI. This is not presented as a resolved concern; rather, it is one of the reasons why transparent identification and summarization of AI functionality is considered necessary. LLMAID therefore addresses AI opacity in mobile software, but it also depends on an LLM in its own inference pipeline (Liu et al., 24 Nov 2025).

A common misconception would be to interpret AI detection in APKs as a straightforward signature-matching problem. The reported results argue against that view: the comparison with AI Discriminator shows that heuristic methods can miss many apps in open-world conditions. Another possible misconception is that the presence of on-device models alone captures AI usage. LLMAID’s component breakdown indicates otherwise, as Package/API calls account for 92.8% of detected AI components, far exceeding on-device models and HTTPS requests (Liu et al., 24 Nov 2025).

The broader significance of LLMAID lies in making AI capability auditing operational at scale for Android software. This suggests applicability to regulatory inspection, large-scale ecosystem measurement, and developer-facing software analysis, provided the assumptions of static extraction and LLM-based semantic labeling hold for the target app population.