Consultation Value Assessment Framework

• Consultation Value Assessment Framework is a systematic approach that evaluates historical user logs based on in-domain scope, posterior action, and time decay.
• It utilizes the VAPS model to filter and align high-value consultations with user behaviors, ensuring adaptive and precise search personalization.
• Empirical findings demonstrate that leveraging consultation value boosts ranking accuracy, retrieval effectiveness, and conversion rates in e-commerce platforms.

A Consultation Value Assessment Framework in the context of personalized e-commerce search is a principled methodology for empirically determining which historical user-AI consultation logs are most useful for search personalization and recommendation. The framework advances beyond the prevalent practice of using textual or semantic similarity between consultation histories and current queries, by introducing explicit and multi-dimensional measures of consultation value. This approach enables platforms to align search systems with actual user needs, behaviors, and satisfaction, resulting in more precise, adaptive, and conversion-effective search experiences.

1. Three-Dimensional Evaluation of Consultation Value

The core of the framework is a triadic assessment of each historical consultation’s value, founded on the following perspectives:

1. Scenario Scope Value: Determines whether a consultation pertains to the platform’s serviced domain. A consultation is considered in-scope if its terminology overlaps with the platform’s inverted index of product-related terms, effectively filtering out irrelevant or out-of-domain dialogues.
2. Posterior Action Value: Quantifies the behavioral utility of a consultation based on its subsequent user actions, such as search, click, or purchase. This dimension acts as a proxy for true consultation impact, assigning greater value to consultations that lead to rare or high-effort user behaviors.
3. Time Decay Value: Models the temporal relevance of consultations by applying an exponentially decaying weight with increased time gap between the consultation and present activity. Recent consultations are thus emphasized as more pertinent than outdated ones.

The aggregation of these three value components for user $u$, session $s$, and consultation $c$ is formalized by: $$\mathcal{O}_{u}^{s,c} = (1-\lambda_{1})\mathcal{O}_{\mathrm{time}}^{s,c} + \lambda_{1}\left(\lambda_2\mathcal{O}_{\mathrm{scope}}^{c} + (1-\lambda_2)\mathcal{O}_{\mathrm{action}}^{c}\right)$$ where $\lambda_1, \lambda_2$ are hyperparameters controlling the contribution of each factor. The posterior action value is further detailed as: $$\mathcal{O}_{\mathrm{action}}^{c} = \sum_{a}\gamma_{a} R^{T_a}_{c}$$ where $\gamma_{a}$ reflects action rarity and $R^{T_a}_{c}$ encodes whether a given action type follows consultation $c$.

2. The VAPS Model: Value-Aware Personalized Search

To operationalize the framework, the paper introduces the Value-Aware Consultation-Enhanced Personalized Search (VAPS) model. The VAPS model incorporates only those historical consultations exhibiting high aggregate value and aligns them directly with subsequent user actions:

• Filtering: Only consultations surpassing a minimum value threshold from the tri-perspective scoring are retained for downstream personalization.
• Consultation-Action Interaction (CAI) Module: Employs a cross-attention architecture to explicitly connect filtered consultations with actual user actions (e.g., searches, clicks, buys). This module integrates action type and time embeddings, permitting the model to learn fine-grained, temporally and categorically aware attention distributions.
• Explicit Value-Alignment Loss: A dedicated $\mathcal{L}_\text{VA}$ loss term enforces alignment between consultation and action-type representations, ensuring that the derived consultation embeddings are maximally predictive of user intent and conversion, not merely semantically similar.

3. Methodology and Datasets

VAPS and the associated framework are validated on both a commercial-scale e-commerce corpus and a public Amazon Reviews dataset:

Dataset	#Users	#Items	#Interactions (Search/Consult)
Commercial	2,096	2,691	24,662 / 18,774
Amazon	967	35,772	7,263 / 40,567

Experiments measure ranking metrics (Hit Ratio, NDCG) and retrieval metrics (MRR). Strong baselines span personalized search (e.g., MAPS), multi-scenario (e.g., UnifiedSSR), and standard retrieval (BM25, BGE-M3) models. Ground-truth items are paired with 99 negatives for evaluation.

4. Empirical Findings and Component Analysis

VAPS achieves consistent and substantial improvements across both datasets:

• Ranking accuracy: On the Amazon dataset, HR@10 is 0.8019 for VAPS versus 0.7835 for MAPS; NDCG@10 is 0.5213 vs. 0.4717.
• Retrieval effectiveness: VAPS attains MRR@10 of 0.4749, surpassing dense and conversational retrieval baselines (e.g., CHIQ at 0.3691, MAPS at 0.4324).

Ablation studies confirm that all three value dimensions are essential; performance degrades notably when any are removed. Posterior action value and time decay are particularly influential.

Case analyses demonstrate that high-value consultations—those timely, in-scope, and leading to rare/high-effort actions—produce the most significant gains in recommendation precision. Consultations lacking subsequent user action or relevance, even if semantically similar, are deprioritized, reducing noise in personalization.

5. Practical Impact and Applications

The framework’s core innovations support several practical benefits:

• Enhanced Personalization: Incorporating only empirically valuable consultations, rather than all semantically similar ones, makes recommendations more relevant and conversion-oriented.
• Noise Reduction: Out-of-domain or outdated consultations are algorithmically filtered, improving the signal-to-noise ratio in personalization engines.
• Behavioral Alignment: By leveraging verified behavioral responses, models achieve better alignment with actual user interests and purchase intent.
• Temporal Adaptivity: The explicit modeling of recency ensures that recommendations reflect the user’s current needs rather than historical, potentially obsolete, interests.

Applications extend to AI assistant ranking, personalized e-commerce search, and potentially other domains where consultation logs and behavioral feedback can inform personalization logic.

6. Theoretical and Methodological Contributions

By decoupling consultation value from mere semantic closeness, the framework bridges the gap between natural language alignment and actual user satisfaction or conversion. The explicit modeling of posterior user behavior as value, and the learning of value-aligned consultation-action mappings via dedicated objectives and attention layers, provide a methodological template extensible to other interactive, user-feedback-rich systems.

This approach formally distinguishes between “similarity” and “value,” challenging a widespread assumption in current research and recommender system deployments. It sets a new standard for empirical consultation value assessment in user-AI interaction logs, operationalizing this via reproducible metrics and actionable model components.

7. Future Directions

The framework’s authors identify research opportunities in optimizing model efficiency for real-time applications, automated hyperparameter selection, cold-start user/item handling, and the integration of external knowledge sources such as knowledge graphs or LLMs. Further work on dynamic, user- or domain-specific value functions is anticipated to further adapt the framework to diverse personalization scenarios.

---

Dimension	Definition	Operational Mechanism
Scenario Scope Value	Consultation is in-platform-domain	Inverted index term overlap
Posterior Action Value	Consultation followed by high-impact user behavior	Weighted sum over action types post-consultation
Time Decay Value	Recency of consultation	Exponential decay based on consultation-to-action time delta

---

This Consultation Value Assessment Framework, exemplified by the VAPS model, marks a substantive step toward empirically grounded, user-behavior-aligned personalization in interactive search and recommendation systems.