ToolRec: Calibrated Preference Alignment for Query Recommendation in On-Device Assistants

Abstract: LLMs have significantly advanced generative query recommendation. However, existing alignment methods primarily focus on standard chatbot scenarios, falling short in on-device intelligent assistants where users predominantly expect the rapid invocation of system-level tools. Moreover, directly aligning LLMs with real-world click logs introduces severe noise due to varying user activity levels and the failure to emphasize execution-oriented queries. To address these challenges, we propose ToolRec, a calibrated preference alignment framework tailored for on-device query recommendation. To ground query recommendation with executable actions, we first construct SysToolKit, a comprehensive repository of 708 system tools, paired with a context-aware tool retrieval mechanism to ensure recommendation relevance. We then propose a dual-level calibration mechanism to refine raw click data, effectively mitigating user behavioral noise by calibrating signals based on user activity levels, while simultaneously up-weighting click signals on system-level tool-invoking queries. Guided by these refined preference signals, we then align the model using a sample-level weighted Kahneman-Tversky Optimization (KTO). Extensive online A/B tests on our mobile assistant platform OPPO Xiaobu, which has over 150 million monthly active users, demonstrate that ToolRec can significantly improve Click-Through Rate (CTR) and total clicks volume over strong baselines while maintaining high query relevance.

• The paper introduces ToolRec, which uses a closed-loop calibration mechanism to align LLM-based recommendations with executable system tools.
• It employs dual-level calibration of user and system preference signals, harnessing prospect theory-based weighted KTO for robust online performance.
• Empirical results show significant uplifts in click numbers and CTR, validating ToolRec’s effectiveness in real-world on-device assistant applications.

ToolRec: Calibrated Preference Alignment for Query Recommendation in On-Device Assistants

Motivation and Problem Definition

On-device intelligent assistants increasingly serve as central interfaces for user-device interaction, with user expectations shifting towards actionable recommendations that can immediately trigger system-level operations rather than generic conversational responses. Conventional LLM-based query recommendation frameworks, optimized via click behavior, exhibit deficiencies for on-device scenarios: they neglect tool-invocation intent and suffer from noisy click logs due to heterogeneities in user activity and differences between general queries and tool-invoking queries.

ToolRec targets these challenges by designing a calibrated preference alignment mechanism that grounds recommendations in executable actions, refines preference signals from large-scale online logs, and aligns LLM outputs to the practical demands of on-device assistant users.

Figure 1: Examples of query recommendations as seen in both AI assistant and search engine contexts.

Empirical Insight: Tool-Invocation Dominates User Preference

Analysis on OPPO Xiaobu, with over 150 million MAUs, reveals that queries invoking system tools result in significantly higher click-through rates (CTR) and total clicks compared to general queries. This empirical discrepancy underscores the necessity for actionable, execution-oriented query recommendations in real-world assistant deployments.

Figure 2: Statistical comparison of CTR and click counts for tool-invoking versus general queries in OPPO Xiaobu logs.

Framework Overview

ToolRec operates as a closed-loop pipeline that starts with online deployment, collects click/non-click logs, calibrates these preference signals, and then optimizes the underlying LLM via weighted KTO. Key architectural components:

1. SysToolkit Construction: An exhaustive repository of 708 system tools covering all practical device functions.
2. Context-aware Tool Retrieval: Using embedding models and rerankers, ToolRec dynamically conditions the LLM on relevant tool descriptions based on the current dialogue context, ensuring recommendation relevance and maximizing actionable suggestions.
3. Dual-level Calibration: Preference data are calibrated both on the user side—handling variations in click behavior by activity strata—and on the system side—preferentially up-weighting clicks for tool-invoking queries and suppressing penalties for unclicked tool queries.
4. Sample-level Weighted KTO: Optimization proceeds using prospect theoretic loss, modulated by calibrated sample weights, allowing for unpaired data alignment and asymmetric loss aversion.

   Figure 3: Architecture summary of ToolRec’s closed-loop system, integrating tool retrieval, preference calibration, and weighted KTO alignment.

SysToolkit Repository and Retrieval Mechanism

SysToolkit is curated to encompass all major functional domains relevant to device operations: Display, Media, Communications, Networks, Health, and Miscellaneous. Given the impracticality of supplying all tools to the LLM context due to length constraints, ToolRec encodes tool descriptions using Qwen-3-embedding and ranks them for per-dialogue relevance with Qwen-3-reranker, selecting the top-N subset for each recommendation session.

Figure 4: Distribution of tool categories within SysToolkit, demonstrating comprehensive coverage.

Dual-level Preference Calibration

User-side Calibration

Click signals are weighted dynamically according to user activity profiles; very active users’ non-clicks are upweighted as strong negative feedback, while inactive users’ rare clicks are upweighted as positive exemplars. Weighting is implemented via a formula involving $uctr$, employing hyperparameters $\alpha$, $\mu$, and $s$.

System-side Calibration

Clicks on tool-invoking queries are upweighted commensurately with the frequency of tool usage, using exponentiated weights. Unclicked tool-invoking queries are masked (weight zero), preventing the model from being penalized for actionable queries that could be missed due to UI or user lapses.

Preference Alignment via Weighted KTO

The loss adopts the KTO framework, which is compatible with unpaired preference data and prospect theory. Sample weights are determined as:

• For positive samples: $w(Q_r) = \max(w_u, w_s)$
• For negative samples: $w(Q_r) = \min(w_u, w_s)$

This balances alignment aggressiveness (rewarding strong-positive samples) and conservativeness (penalizing strong negatives only when both calibration signals agree).

Experimental Validation

Main Online Results

ToolRec outperforms SFT and vanilla KTO with a 4.74% increase in click number and 3.32% increase in CTR under rigorous A/B testing with 5% user traffic slices. The relevance of its recommendations remains high, only minimally impacted despite the optimization toward tool-invocation. Notably, ToolRec achieves a 1.44% uplift in the proportion of tool-invoking queries, translating to substantial absolute gains given the platform’s scale.

Ablation and Hyperparameter Studies

Both user-side and system-side calibrations provide statistically significant gains when applied individually, but their combination is additive. Dynamic frequency-based system-side calibration further amplifies performance.

Optimal hyperparameters $\alpha=0.25$, $\gamma=1.25$ yield peak tradeoff between CTR and relevance in online experiments.

Robustness and Granular Performance

ToolRec’s performance is temporally stable across daywise traffic fluctuations and stratified by user activity cohorts, consistently attaining top-two rankings in CTR and total clicks. The model’s impact persists regardless of user engagement levels, which is attributed to the individualized calibration strategy.

Case Investigation

Qualitative case studies highlight ToolRec’s capacity to proactively surface tool-invoking queries that precisely anticipate user intent—a critical advance for real-world assistant applications.

Practical and Theoretical Implications

ToolRec demonstrates that genuine alignment for on-device assistants requires nuanced calibration of implicit preference signals. By prioritizing execution-oriented interactions, ToolRec aligns LLM recommendations with actual user utility, establishing a paradigm for practical RLHF with noisy, non-paired logs.

The closed-loop deployment-optimization pipeline enables continual feedback-driven refinement, promising practical scalability in dynamic environments with heterogeneous users and evolving system toolkits.

On the theoretical front, sample-level weighted KTO serves as a robust alignment framework, accommodating prospect theory-based loss aversion and tailoring optimization to multiple axes of preference calibration (user behavioral heterogeneity and system invocation frequency). This has implications for future alignment protocols in broader RLHF and generative recommender tasks.

Conclusion

ToolRec constitutes a calibrated preference alignment framework tailored to actionable query recommendation in on-device assistants. The foundational advances—including SysToolkit, context-aware retrieval, dual-level calibration, and weighted KTO—jointly enable robust, noise-resilient alignment to authentic user intent. Empirical evaluation at scale confirms significant uplifts in CTR and actionable recommendation, with strong generalization across user strata and system domains. The practical and theoretical innovations embedded in ToolRec are readily extensible to future work on LLM-based recommendation, RLHF with noisy implicit signals, and agent alignment in interactive environments.