LaQual: LLM App Quality Evaluation

• LaQual is a three-stage, fully automated framework designed to evaluate the quality of LLM applications across diverse domains using both static indicators and dynamic task-based assessments.
• It employs a hierarchical labeling and classification system combined with time-weighted static filters to mitigate manipulation and capture scenario-specific performance.
• The framework uses an LLM as a judge to generate realistic evaluation tasks, score responses on multiple criteria, and ultimately enhance app recommendation and user trust.

LaQual, short for LLM App Quality, is a three-stage, fully automated framework for evaluating the quality of applications built on LLMs and hosted in LLM app stores such as GPT Store, Baidu AppBuilder, Poe, and Coze. It was introduced to address ranking and recommendation systems that depend on static, popularity-oriented signals such as views, interactions, favorites, copies, and recency, even though these signals are easily manipulated, not scenario-aware, and model-centric rather than user-centric. In contrast, LaQual is designed to label and classify LLM apps hierarchically, filter them through time-weighted static indicators, and then perform dynamic, scenario-adaptive evaluation in which an LLM itself generates metrics, scoring rules, and realistic tasks for app assessment (Wang et al., 26 Aug 2025).

1. Definition, problem setting, and scope

LaQual addresses a setting in which LLM app stores contain millions of apps spanning content creation, coding support, education, legal consulting, travel planning, and other use cases. The central problem is that current ranking and recommendation mechanisms typically rely on static signals such as number of views, interactions, favorites, copies, and release date. These signals do not directly capture whether an app performs well for users’ real tasks, and they do not distinguish between scenarios in which an app may be strong or weak.

The framework is explicitly motivated by the mismatch between existing academic LLM evaluation and the app-store environment. Academic evaluation often focuses on models in narrow domains such as law, math, or code, and usually uses fixed benchmarks. LaQual instead targets thousands of heterogeneous LLM apps, each with its own niche functionality, and emphasizes user-centric notions of quality that include engagement, usability, robustness, and speed.

The operational definition of quality in LaQual is multi-dimensional. It includes content quality, functional capability, user engagement and vitality, and response performance. Content quality is primary and includes scenario-specific properties such as legal citation accuracy, terminology clarity, completeness, logical coherence, personalization, and innovation. Functional capability is represented by plugins, knowledge bases, and built-in models. User engagement is normalized and time-weighted. Response performance is measured as efficiency in tokens per second.

2. Three-stage architecture and hierarchical scenario anchoring

LaQual’s workflow has three stages: LLM app labeling and hierarchical classification, static indicator evaluation (filtering), and dynamic scenario-adaptive evaluation. The first stage establishes the scenario anchor on which the later stages depend.

In the labeling step, an LLM reads app metadata such as name, description, and instructions, and generates a short label summarizing core functionality and typical use scenario. The paper gives examples such as “法律咨询分析” (“laws consulting analysis”) and “旅行行程规划” (“travel itinerary planning”). Label quality is verified using a text2vec embedding model. If $d$ is the app description and $\ell$ is the generated label, LaQual computes semantic similarity $\text{sim}(d,\ell)$ with cosine similarity and triggers iterative feedback whenever $\text{sim}(d,\ell) < 0.7$. This mechanism is intended to produce robust, semantically faithful labels.

Classification then maps the validated label into a three-level taxonomy: Category $\rightarrow$ Subcategory $\rightarrow$ Tag. The top-level categories are Professional Q&A, Tool-type, and General. Examples include Professional Q&A $\rightarrow$ Legal Consultation with tags such as “laws”, “regulations”, “contracts”, and “litigation”, and Tool-type $\rightarrow$ Planning Tools with tags such as “planning”, “planner”, and “planning design”. Matching proceeds from fine to coarse: tag-level keywords are attempted first, then subcategory-level keywords, and if neither matches, the app is categorized as General. The output of Stage 1 is therefore a validated semantic label $\ell$ and a categorical assignment $(\text{Category}, \text{Subcategory}, \text{Tag})$.

This stage is important because LaQual does not evaluate apps against a universal rubric. It first identifies the specific scenario context in which an app should be judged. The later stages use this scenario anchor to set thresholds and generate scenario-specific evaluation metrics and tasks.

3. Static filtering, dynamic metric generation, and composite scoring

The second stage uses objective platform data to eliminate obviously low-quality or irrelevant apps before dynamic evaluation. LaQual divides static indicators into two families. User engagement indicators are Views, Interactions, Favorites, and Copies. Functional capability indicators are Plugins, Knowledge Bases, and Built-in Models.

A central design choice is the time-decay mechanism for engagement. Let $\ell$0 be the number of months since release. The number of operating quarters is

$\ell$1

For an engagement metric $\ell$2, LaQual computes a time-weighted value

$\ell$3

where

$\ell$4

The implementation uses $\ell$5. This makes the metrics reflect recent vitality rather than raw age. Using the taxonomy from Stage 1, LaQual then applies category-specific thresholds. In the AppBuilder example, quarterly Views, Interactions, Favorites, and Copies all require $\ell$6 across Professional Q&A, Tool-type, and General; Plugins require $\ell$7 for Tool-type; Knowledge Bases require $\ell$8 for Professional Q&A; Built-in Models require $\ell$9 across categories. Thresholds are deliberately inclusive, with the goal of removing clear underperformers such as zombie apps or apps lacking essential configuration.

The third stage is the core LLM-as-a-Judge pipeline. From the app’s label $\text{sim}(d,\ell)$0, the evaluator LLM generates 3 core evaluation metrics that satisfy functional uniqueness, quantifiability, and content orientation, and it generates explicit scoring criteria for levels 1–5. In the legal consulting scenario, the reported examples are Legal Citation Accuracy, Terminology Clarity, and Answer Completeness & Logic. In the travel planning scenario, the examples are Personalization Match, Itinerary Logic & Flow, and Content Innovation.

The framework then generates realistic evaluation tasks for each metric. Examples include legal queries such as “If an employee is dismissed without notice, which labor law articles apply?” and travel queries such as “Plan a 3-day trip to Chengdu for a family with kids, budget under 3000 RMB.” Generated tasks are filtered for syntactic integrity and complexity inspection, with a backoff and retry mechanism if a task is incomplete, malformed, trivial, or vague.

For each shortlisted app and task, LaQual sends the query through the app store interface, collects the natural-language response, and records response length and timing. Content quality is then scored by the evaluator LLM using a structured prompt containing the original task $\text{sim}(d,\ell)$1, the app response $\text{sim}(d,\ell)$2, and metric-specific scoring criteria. The evaluator uses a 5-point ordinal scale,

$\text{sim}(d,\ell)$3

with deterministic settings: temperature $\text{sim}(d,\ell)$4, frequency penalty $\text{sim}(d,\ell)$5, $\text{sim}(d,\ell)$6, and timeout $\text{sim}(d,\ell)$7 seconds. Response efficiency is measured as

$\text{sim}(d,\ell)$8

and mapped to a 1–5 score $\text{sim}(d,\ell)$9 using thresholds from “Very Poor” $\text{sim}(d,\ell) < 0.7$0 to “Excellent” $\text{sim}(d,\ell) < 0.7$1. The final composite score is

$\text{sim}(d,\ell) < 0.7$2

with the empirical setting

$\text{sim}(d,\ell) < 0.7$3

The resulting outputs include scenario-specific metric scores, $\text{sim}(d,\ell) < 0.7$4, $\text{sim}(d,\ell) < 0.7$5, the composite score, and a structured evaluation report with strengths, weaknesses, and improvement suggestions.

4. Experimental evaluation and empirical findings

LaQual was evaluated on Baidu AppBuilder through three research questions: RQ1: Accuracy and adaptivity, RQ2: Recommendation effectiveness, and RQ3: User impact (Wang et al., 26 Aug 2025). Two scenarios were used. In legal consulting, the search keyword was “lawyer” and the initial candidate pool contained 18 apps. In travel planning, the search keyword was “smart travel planning assistant” and the initial candidate pool contained 32 apps. Stage 1 labeled legal apps as “laws consulting analysis” and travel apps as “travel itinerary planning”; Stage 2 then reduced the pools from 18 to 6 in legal consulting and from 32 to 6 in travel planning.

For alignment with human judgments, the study used a panel of 28 human evaluators, including domain experts, all experienced LLM app users aged 18–50. Each evaluator rated each shortlisted app on the same three metrics generated by LaQual, using the same 5-point scale. Consistency between LaQual and human ratings was measured with Spearman’s rank correlation coefficient: $\text{sim}(d,\ell) < 0.7$6 The reported results were $\text{sim}(d,\ell) < 0.7$7, $\text{sim}(d,\ell) < 0.7$8 in legal consulting, and $\text{sim}(d,\ell) < 0.7$9, $\rightarrow$0 in travel planning. These were described as moderately strong, statistically significant positive correlations.

For recommendation effectiveness, LaQual reduced the legal candidate pool by 66.7% and the travel candidate pool by 81.25%. Manual review of filtered-out apps found several recurrent patterns: zombie apps with low or declining activity, apps lacking domain-specific optimization, apps lacking crucial functional integrations, and highly ranked apps that were low quality when judged by LaQual’s framework rather than by store popularity metrics.

User impact was studied in a within-subjects user study with 12 participants. Each participant completed six representative app-discovery tasks using both the AppBuilder baseline and LaQual. The post-task questionnaire used a 7-point Likert scale and measured Decision Confidence, Comparison Efficiency, Value of Report, Result Credibility, Deep Demand Fulfillment, and Discovery of “Surprise” Apps. The reported mean scores were: Decision Confidence 5.05 for LaQual versus 4.40 for baseline; Comparison Efficiency 5.45 versus 3.30; Value of Report 4.75 versus 2.25; Result Credibility 4.85 versus 3.90; Deep Demand Fulfillment 4.30 versus 3.70; Discovery of “Surprise” Apps 4.95 versus 3.55; and Overall weighted experience score 4.873 versus 3.555. All differences were statistically significant with $\rightarrow$1.

5. Position within adjacent research and common points of confusion

LaQual is positioned at the intersection of LLM evaluation frameworks and app store ranking/recommendation. Its immediate baseline is the native ranking mechanism of the app store, which is keyword-based and statically ranked by usage, favorites, and recency. Conceptually, LaQual differs by being quality-oriented, scenario-adaptive, and by combining static and dynamic analysis rather than relying only on popularity-oriented signals.

A common point of confusion concerns similarly named systems. AutoQual is an LLM-based agent framework for automated discovery of interpretable features for review quality assessment, not a framework for evaluating LLM apps in app stores. It discovers interpretable, computable features, operationalizes them as tools, and uses them to predict review quality scores (Lan et al., 9 Oct 2025). LeQua, by contrast, concerns learning to quantify, that is, estimating class prevalence values for unlabelled textual data rather than scoring the quality of LLM applications (Esuli et al., 2021). These systems are adjacent in their concern with automated quality or aggregate evaluation, but they address different objects: reviews, class prevalences, and LLM apps, respectively.

Within the LLM-app domain itself, LaQual also differs from standard model benchmarks. The framework is app-centric rather than model-centric, and scenario-aware rather than benchmark-fixed. Its unit of analysis is an application with metadata, configuration, plugins, knowledge bases, and user-facing responses, not merely a foundation model evaluated on a static dataset.

6. Limitations, deployment implications, and future work

The framework explicitly states several limitations. Under content validity, it does not perform real-time fact-checking; it evaluates aspects such as coherence, logical consistency, clarity, and completeness, but explicitly avoids metrics requiring external factual verification, such as whether a recommended restaurant is open. Static thresholds are based on empirical experience and may need tuning per platform, with a risk of excluding niche but high-quality apps. Under internal validity, the LLM-as-a-Judge may inherit political or cultural biases from training data, even though neutral prompts and deterministic decoding are used as mitigations. Under external validity, the current framework targets text-based LLM apps and was evaluated on a single store, AppBuilder.

These constraints also define its most immediate deployment modes. For app store operators, LaQual can serve as an onboarding filter, a periodic re-evaluation mechanism, and a transparency layer in which LaQual scores and reports are displayed alongside popularity indicators. For enterprises, it can be used to evaluate internal LLM apps or third-party apps before deployment and can be integrated into CI/CD pipelines so that new versions undergo static filtering and dynamic evaluation before promotion. The required data are app metadata, engagement logs, access to app interfaces for task-based querying, and an evaluator LLM such as Qwen-32B.

The framework’s suggested evaluation cadence separates inexpensive and expensive components. Static filtering can run continuously or weekly, whereas dynamic evaluation can run periodically, such as monthly or per major app update. Suggested future work includes adding real-time factual validation, extending the framework to multi-modal apps, improving interpretability of scores, integrating user feedback loops, and refining thresholds using more comprehensive platform-wide data. These directions follow directly from the framework’s central premise: automated evaluation is most useful when it remains scenario-adaptive, scalable, and tightly coupled to real-world app behavior rather than to static popularity proxies alone.