---
title: 'HMPPS: Multimodal Personalized Product Search'
url: https://www.emergentmind.com/topics/hmpps
type: topic
---

# HMPPS: Multimodal Personalized Product Search

Searching arXiv for the specified paper and the similarly named HMPSBM paper to ground the article and address potential ambiguity.
arXiv search query: 2509.18682
HMPPS, short for **Harnessing Multimodal large language models for Personalized Product Search**, is a framework for personalized product search (PPS) that recasts reranking as a multimodal instruction-following problem and augments that formulation with two query-aware refinement mechanisms: perspective-guided description summarization and two-stage user-history selection [2509.18682]. In the formulation studied, PPS seeks to retrieve products relevant to a query while accounting for user preferences expressed in purchase history, and HMPPS is used as a reranker on top of an efficient ID-based PPS model rather than as a standalone retriever [2509.18682].

## 1. Problem formulation and scope

In HMPPS, the PPS problem is defined over a set of users $\mathcal{U}$, a set of products $\mathcal{P}$, and a set of queries $\mathcal{Q}$ [2509.18682]. For a user $u \in \mathcal{U}$ with chronologically ordered purchase history
$$
H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},
$$
given a query $q \in \mathcal{Q}$ and a candidate product $p_i \in \mathcal{P}$, the objective is to predict the purchase probability
$$
y_{u,q,p_i} = \mathcal{F}_\theta(H_u, q, p_i).
$$
The standard PPS output is then a ranking of products by $y_{u,q,p}$ [2509.18682].

The framework is explicitly positioned as a reranker. An existing ID-based model $M_{ID}$ first retrieves a candidate set
$$
\mathcal{P}' = \{ p'_1, p'_2, \ldots, p'_{K_p} \}, \quad K_p \ll N_P,
$$
and HMPPS computes scores only for $p_i \in \mathcal{P}'$ [2509.18682]. This division of labor combines efficient coarse retrieval from the ID-based model with content-rich, multimodal, fine-grained reranking from HMPPS.

The multimodal input representation is central. For each product $p$, HMPPS uses textual fields—product ID in textual form, title, brand, category, and description—together with a product display image $v_p$ [2509.18682]. The textual concatenation is denoted $t_p$, the query is a textual string $t_q$, and user history is represented as a sequence of historical products, each with textual and visual content. These inputs are fed into a multimodal large language model (MLLM), which functions both as a generative reranker and as an encoder for multimodal representations [2509.18682].

## 2. Architectural design

HMPPS consists of three components: an MLLM-based PPS backbone, a perspective-guided description summarization module, and a two-stage training paradigm for user-history refinement [2509.18682]. The motivation for this architecture is twofold. First, prior LLM-based PPS methods considered only textual content and ignored images, even though product search often depends on visual cues such as brands, styles, and colors. Second, raw PPS inputs are long, noisy, and redundant, which can mislead MLLMs and increase computation cost [2509.18682].

The MLLM backbone converts PPS into a multimodal instruction-following task. For each tuple $(u,q,p_i)$, the prompt aggregates instruction text $Inst$, user history $H_u$, the query $q$, and the candidate product $p_i$ into
$$
[Inst; H_u; q; p_i].
$$
The model is trained to generate a decision text constrained to **“yes”** or **“no”**, indicating whether the user would purchase the candidate product [2509.18682]. This is a point-wise reranking formulation rather than a listwise or pairwise generation scheme.

Training uses the standard language generation loss
$$
\mathcal{L} = - \sum_{l=1}^{L} \log \Pr\left(d_l \mid d_{<l}, [Inst; H_u; q; p_i]\right),
$$
where $d_l$ is the $l$-th token in the decision text [2509.18682]. Supervision is contrastive in the sense that each search log $\langle H_u, q \rangle$ contributes one positive candidate—the actually purchased product with target decision “yes”—and several negatives: $K_s^n$ simple negatives sampled uniformly from $\mathcal{P}$ and $K_h^n$ hard negatives drawn from the retrieved set $\mathcal{P}'$ with high scores under $M_{ID}$ [2509.18682].

At inference time, HMPPS extracts the logits for “yes” and “no” and defines a continuous purchase probability through a bidimensional softmax,
$$
y_{u,q,p_i} = \frac{\exp(s_{\text{yes}})}{\exp(s_{\text{yes}})+\exp(s_{\text{no}})}.
$$
This probabilistic interpretation converts the generative output into a ranking score [2509.18682].

The default MLLM backbone is InternVL2-1B, and the study also evaluates larger InternVL2 variants and Qwen2-VL to examine scalability and architectural generality [2509.18682]. This suggests that HMPPS is designed as a framework-level construction rather than as a method tied to one specific multimodal foundation model.

## 3. Query-aware refinement mechanisms

A defining characteristic of HMPPS is its attempt to reduce redundancy and noise before reranking. One refinement path targets product descriptions; the other targets user history [2509.18682].

The **perspective-guided description summarization** module uses the textual LLM Qwen2.5-14B and operates in two steps [2509.18682]. First, given a product description and a user query, the model extracts information perspectives—attributes or aspects relevant to the search intent—using a prompt with an explicit task instruction and a one-shot demonstration. The extracted perspective lists are aggregated across training samples, and the top-$K_d$ frequent perspectives are retained as core perspectives for each dataset or domain; the paper gives $K_d = 20$ as an example [2509.18682]. Second, the LLM generates concise descriptions centered on those core perspectives. The prompt explicitly requests focus on the core perspectives, includes a chain-of-thought requirement, and provides a one-shot reasoning-plus-summary demonstration [2509.18682].

The empirical effect on the Office dataset is concrete. Original descriptions have per-product word count $\text{WC}_p$ with average 106 and maximum 3538, whereas perspective-guided summaries reduce these to average 41 and maximum 513 [2509.18682]. The same refinement improves NDCG@4 from 0.401 to 0.403 and Recall@4 from 0.524 to 0.529. Alternative prompt designs—direct summarization and reasoning-only summarization—are reported as inferior, particularly in stability and performance [2509.18682]. This suggests that the gain comes not merely from shortening descriptions, but from shortening them in a query-aligned manner.

The **two-stage training paradigm** addresses the observation that only a subset of historical purchases expresses user preference for a given query [2509.18682]. In stage 1, the MLLM is trained using randomly sampled history items, with $K_{s_1}=5$ or 7 depending on dataset. After this stage, the trained model is reused as a multimodal encoder to extract product and query embeddings:
$$
f_p = \text{Avg}\left( \mathcal{F}_{MLLM}([t_p; v_p]; \theta^{s_1}) \right), \qquad
f_q = \text{Avg}\left( \mathcal{F}_{MLLM}(t_q; \theta^{s_1}) \right).
$$
Cosine similarity is then computed between a historical product $p^u_j$ and either the query or the candidate product,
$$
r_{p^u_j x} = \cos(f_{p^u_j}, f_x), \quad x \in \{q, p_i\},
$$
to support history selection [2509.18682].

Selection proceeds in two steps. From the full history $H_u$, the method first chooses $2K_{s_2}$ products with highest product–query relevance, producing $H'_u$, and then selects $K_{s_2}$ products from $H'_u$ with highest product–candidate relevance, producing the final selected history $H''_u$ [2509.18682]. In stage 2, the MLLM is retrained using $H''_u$ instead of randomly sampled history. The stated benefits are improved ranking accuracy through better focus and reduced inference cost through shorter prompts [2509.18682].

## 4. Representation learning, training regime, and inference pipeline

The representation-learning role of the stage-1 MLLM is not auxiliary but integral to the overall design. The stage-1 embeddings are task-adapted because they are learned from PPS logs, and the paper characterizes them as encoding search-specific semantic and multimodal relationships [2509.18682]. This makes history selection dependent on a representation space shaped by downstream PPS supervision rather than by generic multimodal pretraining alone.

The experimental datasets are the 5-core Amazon product search data with four categories: Office Products, Cell Phones & Accessories, Beauty, and Sports & Outdoors [2509.18682]. Their reported sizes are: Office with 4,905 users, 2,420 products, 290 queries, and 53,258 interactions; Cell with 27,879 users, 10,429 products, 165 queries, and 194,439 interactions; Beauty with 22,363 users, 12,101 products, 249 queries, and 198,502 interactions; and Sports with 35,598 users, 18,357 products, 1,543 queries, and 296,337 interactions [2509.18682]. The train/test split is 70/30, following prior PPS work.

Optimization uses InternVL2-1B with LoRA applied to the vision module, language module, and MLP layers [2509.18682]. The reported settings are notable for their parsimony: training on only 10% of each dataset unless otherwise specified, for 1 epoch, with batch size 1, learning rate 0.0001, and the AdamW optimizer. Negative sampling uses $K_s^n = 2$ simple negatives and $K_h^n = 3$ hard negatives [2509.18682]. The comparison against larger or more data-intensive baselines is therefore framed within a deliberately data-efficient regime.

The inference pipeline is stepwise. An existing PPS model such as UniSAR retrieves the top $K_p = 10$ candidates [2509.18682]. Stored embeddings are then used to compute $r_{p^u_j q}$ and $r_{p^u_j p_i}$, which determine the selected history $H''_u$ for each candidate. The second-stage MLLM receives the multimodal prompt $[Inst; H''_u; q; p_i]$, produces yes/no logits, and yields $y_{u,q,p_i}$ through the softmax defined above. Final reranking is performed by sorting candidates by this probability [2509.18682].

## 5. Empirical results

Evaluation uses MRR@8, NDCG@4, Recall@4, and Recall@1, all standard ranking metrics in information retrieval [2509.18682]. Across six ID-based PPS baselines—HEM, ZAM, DREM, DREM-HGN, CAMI, and UniSAR—HMPPS is applied as a reranker over the top-10 candidates and shows consistent gains [2509.18682].

The paper highlights several representative improvements. For UniSAR on Office, the base model attains MRR@8 = 0.368, NDCG@4 = 0.370, Recall@4 = 0.475, and Recall@1 = 0.232, while UniSAR + HMPPS reaches 0.398, 0.414, 0.532, and 0.256, respectively, with relative improvements up to about 12% on Recall@4 [2509.18682]. On Cell, UniSAR’s MRR@8 improves from 0.155 to 0.190, described as about 22.6% improvement; on Beauty, UniSAR’s NDCG@4 improves from 0.125 to 0.188, described as about 50.4% improvement [2509.18682]. The paper characterizes the overall pattern as double-digit relative improvement in many metrics across baselines and datasets.

Against content-based rerankers, HMPPS also leads. On UniSAR candidates, the Office MRR@8 values are 0.371 for RTM, 0.382 for InstructRec, and 0.398 for HMPPS; the Cell MRR@8 values are 0.130, 0.184, and 0.190; the Beauty NDCG@4 values are 0.146, 0.180, and 0.188; and the Sports Recall@4 values are 0.239, 0.244, and 0.269 [2509.18682]. The reported comparison is especially notable because HMPPS uses a 1B MLLM and 10% of the data, whereas the reproduced InstructRec baseline uses a 3B LLM and full data [2509.18682]. This suggests that multimodal input and query-aware refinement contribute materially beyond simple model scaling.

Ablation studies support the design choices. On Office, adding vision improves MRR@8 from 0.203 to 0.220 in zero-shot evaluation and from 0.376 to 0.386 after finetuning [2509.18682]. In description summarization, the final perspective-guided method achieves NDCG@4 = 0.403 and Recall@4 = 0.529, exceeding no summarization (0.401, 0.524), direct summarization (0.401, 0.525), and reasoning-based summarization without explicit perspectives (0.400, 0.522) [2509.18682]. For history selection, even a one-stage selected-history variant with only two history items surpasses one-stage random history with five items and one-stage latest history with nine items, while the full two-stage selected variant improves further and yields a reported 4x speedup because of reduced sequence length [2509.18682].

The framework also scales with both data and model size. Performance improves monotonically as training data increase from 10% to 20%, 30%, 50%, and 100%, and larger InternVL2 variants as well as Qwen2-VL improve metrics regardless of architecture [2509.18682]. In comparison with QIN on Beauty, QIN obtains MRR@8 = 0.222, NDCG@4 = 0.231, and Recall@4 = 0.321; HMPPS trained on 10% data obtains 0.234, 0.238, and 0.315; and HMPPS* trained on 100% data reaches 0.257, 0.263, and 0.340 [2509.18682]. The qualitative case study further reports that UniSAR tends to rank high-frequency products higher, showing popularity bias, whereas HMPPS correctly identifies the target product with the lowest frequency in a hard case and uses visual brand information that is absent from text, outperforming InstructRec [2509.18682].

## 6. Deployment, limitations, and terminological disambiguation

HMPPS is reported as deployed in a large-scale online search system with **billion-level daily active users** [2509.18682]. The deployment strategy trains a large 72B-parameter MLLM offline on 20M real search logs covering 2M users and 8M items, then distills it to a smaller model of about 300M parameters for online inference [2509.18682]. The distilled HMPPS model outputs a search probability used as one factor in the final ranking fusion. Over a 14-day A/B test, replacing a conventional multimodal transformer similar to Bert4Rec for one ranking signal yields **Query-CTR** of +0.53% and **Efficient click count** of +0.77%, with p-value 1.16% [2509.18682]. Ranking 10 candidate items per query takes about 22 microseconds on the distilled model [2509.18682].

The paper also states several limitations. Even with refinement, MLLMs remain expensive and context-limited; the method assumes the availability of images and rich metadata; query-aware perspectives are mined per dataset and may not transfer across domains without re-extraction; stage-1 representations are PPS-domain specific; and performance depends strongly on the quality and size of pretrained MLLMs [2509.18682]. Additional challenges include hallucination during summarization, cold-start items and users, and calibration within complex online pipelines [2509.18682]. Suggested directions include stronger multimodal backbones, improved summarization and hallucination reduction, unified multi-task modeling, open-world and zero-shot PPS, and more efficient training and distillation [2509.18682].

A common naming confusion deserves explicit correction. **HMPPS** in [2509.18682] refers to a multimodal large-language-model framework for personalized product search. By contrast, **HMPSBM**, described in “Simultaneous global and local clustering in multiplex networks with covariate information” [2505.03441], is a hierarchical Bayesian model for multiplex networks that is sometimes informally mis-typed as “HMPPS” [2505.03441]. The two are unrelated in problem domain, methodology, and application: the former addresses multimodal reranking in e-commerce search, whereas the latter addresses simultaneous global and local clustering in multiplex network analysis.

Source: https://www.emergentmind.com/topics/hmpps