HMPPS: Multimodal Personalized Product Search

• HMPPS is a multimodal framework for personalized product search that leverages both textual and visual data to rerank candidate products based on user history.
• It employs query-aware perspective-guided description summarization to distill product details and reduce input redundancy, leading to improved ranking metrics.
• A two-stage training paradigm refines user-history selection by combining efficient coarse retrieval with fine-grained multimodal analysis, achieving consistent gains over ID-based baselines.

Searching arXiv for the specified paper and the similarly named HMPSBM paper to ground the article and address potential ambiguity. arXiv search query: (Zhang et al., 23 Sep 2025) HMPPS, short for Harnessing Multimodal LLMs for Personalized Product Search, is a framework for personalized product search (PPS) that recasts reranking as a multimodal instruction-following problem and augments that formulation with two query-aware refinement mechanisms: perspective-guided description summarization and two-stage user-history selection (Zhang et al., 23 Sep 2025). In the formulation studied, PPS seeks to retrieve products relevant to a query while accounting for user preferences expressed in purchase history, and HMPPS is used as a reranker on top of an efficient ID-based PPS model rather than as a standalone retriever (Zhang et al., 23 Sep 2025).

1. Problem formulation and scope

In HMPPS, the PPS problem is defined over a set of users $\mathcal{U}$, a set of products $\mathcal{P}$, and a set of queries $\mathcal{Q}$ (Zhang et al., 23 Sep 2025). For a user $u \in \mathcal{U}$ with chronologically ordered purchase history

$H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$

given a query $q \in \mathcal{Q}$ and a candidate product $p_i \in \mathcal{P}$, the objective is to predict the purchase probability

$y_{u,q,p_i} = \mathcal{F}_\theta(H_u, q, p_i).$

The standard PPS output is then a ranking of products by $y_{u,q,p}$ (Zhang et al., 23 Sep 2025).

The framework is explicitly positioned as a reranker. An existing ID-based model $M_{ID}$ first retrieves a candidate set

$\mathcal{P}$0

and HMPPS computes scores only for $\mathcal{P}$1 (Zhang et al., 23 Sep 2025). This division of labor combines efficient coarse retrieval from the ID-based model with content-rich, multimodal, fine-grained reranking from HMPPS.

The multimodal input representation is central. For each product $\mathcal{P}$2, HMPPS uses textual fields—product ID in textual form, title, brand, category, and description—together with a product display image $\mathcal{P}$3 (Zhang et al., 23 Sep 2025). The textual concatenation is denoted $\mathcal{P}$4, the query is a textual string $\mathcal{P}$5, and user history is represented as a sequence of historical products, each with textual and visual content. These inputs are fed into a multimodal LLM (MLLM), which functions both as a generative reranker and as an encoder for multimodal representations (Zhang et al., 23 Sep 2025).

2. Architectural design

HMPPS consists of three components: an MLLM-based PPS backbone, a perspective-guided description summarization module, and a two-stage training paradigm for user-history refinement (Zhang et al., 23 Sep 2025). The motivation for this architecture is twofold. First, prior LLM-based PPS methods considered only textual content and ignored images, even though product search often depends on visual cues such as brands, styles, and colors. Second, raw PPS inputs are long, noisy, and redundant, which can mislead MLLMs and increase computation cost (Zhang et al., 23 Sep 2025).

The MLLM backbone converts PPS into a multimodal instruction-following task. For each tuple $\mathcal{P}$6, the prompt aggregates instruction text $\mathcal{P}$7, user history $\mathcal{P}$8, the query $\mathcal{P}$9, and the candidate product $\mathcal{Q}$0 into

$\mathcal{Q}$1

The model is trained to generate a decision text constrained to “yes” or “no”, indicating whether the user would purchase the candidate product (Zhang et al., 23 Sep 2025). This is a point-wise reranking formulation rather than a listwise or pairwise generation scheme.

Training uses the standard language generation loss

$\mathcal{Q}$2

where $\mathcal{Q}$3 is the $\mathcal{Q}$4-th token in the decision text (Zhang et al., 23 Sep 2025). Supervision is contrastive in the sense that each search log $\mathcal{Q}$5 contributes one positive candidate—the actually purchased product with target decision “yes”—and several negatives: $\mathcal{Q}$6 simple negatives sampled uniformly from $\mathcal{Q}$7 and $\mathcal{Q}$8 hard negatives drawn from the retrieved set $\mathcal{Q}$9 with high scores under $u \in \mathcal{U}$0 (Zhang et al., 23 Sep 2025).

At inference time, HMPPS extracts the logits for “yes” and “no” and defines a continuous purchase probability through a bidimensional softmax,

$u \in \mathcal{U}$1

This probabilistic interpretation converts the generative output into a ranking score (Zhang et al., 23 Sep 2025).

The default MLLM backbone is InternVL2-1B, and the study also evaluates larger InternVL2 variants and Qwen2-VL to examine scalability and architectural generality (Zhang et al., 23 Sep 2025). This suggests that HMPPS is designed as a framework-level construction rather than as a method tied to one specific multimodal foundation model.

3. Query-aware refinement mechanisms

A defining characteristic of HMPPS is its attempt to reduce redundancy and noise before reranking. One refinement path targets product descriptions; the other targets user history (Zhang et al., 23 Sep 2025).

The perspective-guided description summarization module uses the textual LLM Qwen2.5-14B and operates in two steps (Zhang et al., 23 Sep 2025). First, given a product description and a user query, the model extracts information perspectives—attributes or aspects relevant to the search intent—using a prompt with an explicit task instruction and a one-shot demonstration. The extracted perspective lists are aggregated across training samples, and the top-$u \in \mathcal{U}$2 frequent perspectives are retained as core perspectives for each dataset or domain; the paper gives $u \in \mathcal{U}$3 as an example (Zhang et al., 23 Sep 2025). Second, the LLM generates concise descriptions centered on those core perspectives. The prompt explicitly requests focus on the core perspectives, includes a chain-of-thought requirement, and provides a one-shot reasoning-plus-summary demonstration (Zhang et al., 23 Sep 2025).

The empirical effect on the Office dataset is concrete. Original descriptions have per-product word count $u \in \mathcal{U}$4 with average 106 and maximum 3538, whereas perspective-guided summaries reduce these to average 41 and maximum 513 (Zhang et al., 23 Sep 2025). The same refinement improves NDCG@4 from 0.401 to 0.403 and Recall@4 from 0.524 to 0.529. Alternative prompt designs—direct summarization and reasoning-only summarization—are reported as inferior, particularly in stability and performance (Zhang et al., 23 Sep 2025). This suggests that the gain comes not merely from shortening descriptions, but from shortening them in a query-aligned manner.

The two-stage training paradigm addresses the observation that only a subset of historical purchases expresses user preference for a given query (Zhang et al., 23 Sep 2025). In stage 1, the MLLM is trained using randomly sampled history items, with $u \in \mathcal{U}$5 or 7 depending on dataset. After this stage, the trained model is reused as a multimodal encoder to extract product and query embeddings:

$u \in \mathcal{U}$6

Cosine similarity is then computed between a historical product $u \in \mathcal{U}$7 and either the query or the candidate product,

$u \in \mathcal{U}$8

to support history selection (Zhang et al., 23 Sep 2025).

Selection proceeds in two steps. From the full history $u \in \mathcal{U}$9, the method first chooses $H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$0 products with highest product–query relevance, producing $H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$1, and then selects $H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$2 products from $H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$3 with highest product–candidate relevance, producing the final selected history $H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$4 (Zhang et al., 23 Sep 2025). In stage 2, the MLLM is retrained using $H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$5 instead of randomly sampled history. The stated benefits are improved ranking accuracy through better focus and reduced inference cost through shorter prompts (Zhang et al., 23 Sep 2025).

4. Representation learning, training regime, and inference pipeline

The representation-learning role of the stage-1 MLLM is not auxiliary but integral to the overall design. The stage-1 embeddings are task-adapted because they are learned from PPS logs, and the paper characterizes them as encoding search-specific semantic and multimodal relationships (Zhang et al., 23 Sep 2025). This makes history selection dependent on a representation space shaped by downstream PPS supervision rather than by generic multimodal pretraining alone.

The experimental datasets are the 5-core Amazon product search data with four categories: Office Products, Cell Phones & Accessories, Beauty, and Sports & Outdoors (Zhang et al., 23 Sep 2025). Their reported sizes are: Office with 4,905 users, 2,420 products, 290 queries, and 53,258 interactions; Cell with 27,879 users, 10,429 products, 165 queries, and 194,439 interactions; Beauty with 22,363 users, 12,101 products, 249 queries, and 198,502 interactions; and Sports with 35,598 users, 18,357 products, 1,543 queries, and 296,337 interactions (Zhang et al., 23 Sep 2025). The train/test split is 70/30, following prior PPS work.

Optimization uses InternVL2-1B with LoRA applied to the vision module, language module, and MLP layers (Zhang et al., 23 Sep 2025). The reported settings are notable for their parsimony: training on only 10% of each dataset unless otherwise specified, for 1 epoch, with batch size 1, learning rate 0.0001, and the AdamW optimizer. Negative sampling uses $H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$6 simple negatives and $H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$7 hard negatives (Zhang et al., 23 Sep 2025). The comparison against larger or more data-intensive baselines is therefore framed within a deliberately data-efficient regime.

The inference pipeline is stepwise. An existing PPS model such as UniSAR retrieves the top $H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$8 candidates (Zhang et al., 23 Sep 2025). Stored embeddings are then used to compute $H_u = \{ p^u_1, p^u_2, \ldots, p^u_{N_u} \},$9 and $q \in \mathcal{Q}$0, which determine the selected history $q \in \mathcal{Q}$1 for each candidate. The second-stage MLLM receives the multimodal prompt $q \in \mathcal{Q}$2, produces yes/no logits, and yields $q \in \mathcal{Q}$3 through the softmax defined above. Final reranking is performed by sorting candidates by this probability (Zhang et al., 23 Sep 2025).

5. Empirical results

Evaluation uses MRR@8, NDCG@4, Recall@4, and Recall@1, all standard ranking metrics in information retrieval (Zhang et al., 23 Sep 2025). Across six ID-based PPS baselines—HEM, ZAM, DREM, DREM-HGN, CAMI, and UniSAR—HMPPS is applied as a reranker over the top-10 candidates and shows consistent gains (Zhang et al., 23 Sep 2025).

The paper highlights several representative improvements. For UniSAR on Office, the base model attains MRR@8 = 0.368, NDCG@4 = 0.370, Recall@4 = 0.475, and Recall@1 = 0.232, while UniSAR + HMPPS reaches 0.398, 0.414, 0.532, and 0.256, respectively, with relative improvements up to about 12% on Recall@4 (Zhang et al., 23 Sep 2025). On Cell, UniSAR’s MRR@8 improves from 0.155 to 0.190, described as about 22.6% improvement; on Beauty, UniSAR’s NDCG@4 improves from 0.125 to 0.188, described as about 50.4% improvement (Zhang et al., 23 Sep 2025). The paper characterizes the overall pattern as double-digit relative improvement in many metrics across baselines and datasets.

Against content-based rerankers, HMPPS also leads. On UniSAR candidates, the Office MRR@8 values are 0.371 for RTM, 0.382 for InstructRec, and 0.398 for HMPPS; the Cell MRR@8 values are 0.130, 0.184, and 0.190; the Beauty NDCG@4 values are 0.146, 0.180, and 0.188; and the Sports Recall@4 values are 0.239, 0.244, and 0.269 (Zhang et al., 23 Sep 2025). The reported comparison is especially notable because HMPPS uses a 1B MLLM and 10% of the data, whereas the reproduced InstructRec baseline uses a 3B LLM and full data (Zhang et al., 23 Sep 2025). This suggests that multimodal input and query-aware refinement contribute materially beyond simple model scaling.

Ablation studies support the design choices. On Office, adding vision improves MRR@8 from 0.203 to 0.220 in zero-shot evaluation and from 0.376 to 0.386 after finetuning (Zhang et al., 23 Sep 2025). In description summarization, the final perspective-guided method achieves NDCG@4 = 0.403 and Recall@4 = 0.529, exceeding no summarization (0.401, 0.524), direct summarization (0.401, 0.525), and reasoning-based summarization without explicit perspectives (0.400, 0.522) (Zhang et al., 23 Sep 2025). For history selection, even a one-stage selected-history variant with only two history items surpasses one-stage random history with five items and one-stage latest history with nine items, while the full two-stage selected variant improves further and yields a reported 4x speedup because of reduced sequence length (Zhang et al., 23 Sep 2025).

The framework also scales with both data and model size. Performance improves monotonically as training data increase from 10% to 20%, 30%, 50%, and 100%, and larger InternVL2 variants as well as Qwen2-VL improve metrics regardless of architecture (Zhang et al., 23 Sep 2025). In comparison with QIN on Beauty, QIN obtains MRR@8 = 0.222, NDCG@4 = 0.231, and Recall@4 = 0.321; HMPPS trained on 10% data obtains 0.234, 0.238, and 0.315; and HMPPS* trained on 100% data reaches 0.257, 0.263, and 0.340 (Zhang et al., 23 Sep 2025). The qualitative case study further reports that UniSAR tends to rank high-frequency products higher, showing popularity bias, whereas HMPPS correctly identifies the target product with the lowest frequency in a hard case and uses visual brand information that is absent from text, outperforming InstructRec (Zhang et al., 23 Sep 2025).

6. Deployment, limitations, and terminological disambiguation

HMPPS is reported as deployed in a large-scale online search system with billion-level daily active users (Zhang et al., 23 Sep 2025). The deployment strategy trains a large 72B-parameter MLLM offline on 20M real search logs covering 2M users and 8M items, then distills it to a smaller model of about 300M parameters for online inference (Zhang et al., 23 Sep 2025). The distilled HMPPS model outputs a search probability used as one factor in the final ranking fusion. Over a 14-day A/B test, replacing a conventional multimodal transformer similar to Bert4Rec for one ranking signal yields Query-CTR of +0.53% and Efficient click count of +0.77%, with p-value 1.16% (Zhang et al., 23 Sep 2025). Ranking 10 candidate items per query takes about 22 microseconds on the distilled model (Zhang et al., 23 Sep 2025).

The paper also states several limitations. Even with refinement, MLLMs remain expensive and context-limited; the method assumes the availability of images and rich metadata; query-aware perspectives are mined per dataset and may not transfer across domains without re-extraction; stage-1 representations are PPS-domain specific; and performance depends strongly on the quality and size of pretrained MLLMs (Zhang et al., 23 Sep 2025). Additional challenges include hallucination during summarization, cold-start items and users, and calibration within complex online pipelines (Zhang et al., 23 Sep 2025). Suggested directions include stronger multimodal backbones, improved summarization and hallucination reduction, unified multi-task modeling, open-world and zero-shot PPS, and more efficient training and distillation (Zhang et al., 23 Sep 2025).

A common naming confusion deserves explicit correction. HMPPS in (Zhang et al., 23 Sep 2025) refers to a multimodal large-language-model framework for personalized product search. By contrast, HMPSBM, described in “Simultaneous global and local clustering in multiplex networks with covariate information” (Corneck et al., 6 May 2025), is a hierarchical Bayesian model for multiplex networks that is sometimes informally mis-typed as “HMPPS” (Corneck et al., 6 May 2025). The two are unrelated in problem domain, methodology, and application: the former addresses multimodal reranking in e-commerce search, whereas the latter addresses simultaneous global and local clustering in multiplex network analysis.