AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce

Abstract: Multimodal representation is crucial for E-commerce tasks such as identical product retrieval. Large representation models (e.g., VLM2Vec) demonstrate strong multimodal understanding capabilities, yet they struggle with fine-grained semantic comprehension, which is essential for distinguishing highly similar items. To address this, we propose Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning (AFMRL), which defines product fine-grained understanding as an attribute generation task. It leverages the generative power of Multimodal LLMs (MLLMs) to extract key attributes from product images and text, and enhances representation learning through a two-stage training framework: 1) Attribute-Guided Contrastive Learning (AGCL), where the key attributes generated by the MLLM are used in the image-text contrastive learning training process to identify hard samples and filter out noisy false negatives. 2) Retrieval-aware Attribute Reinforcement (RAR), where the improved retrieval performance of the representation model post-attribute integration serves as a reward signal to enhance MLLM's attribute generation during multimodal fine-tuning. Extensive experiments on large-scale E-commerce datasets demonstrate that our method achieves state-of-the-art performance on multiple downstream retrieval tasks, validating the effectiveness of harnessing generative models to advance fine-grained representation learning.

• The paper introduces AFMRL, which combines attribute-guided contrastive learning with retrieval-aware reinforcement learning to improve decoder-based multimodal representations for fine-grained product matching.
• AFMRL achieves 54.28% Recall@1 and 65.69 NDCG@10 on fine-grained retrieval, outperforming VLM2Vec at 48.05% Recall@1 while also improving results across large M5Product and EIPM datasets.
• The method uses generated attributes to mine hard negatives and remove false negatives, but its retrieval-focused reinforcement learning creates a modest generalization trade-off and depends on costly teacher-model distillation.

Motivation and problem statement

Fine-grained product retrieval in E-commerce requires distinguishing items that are visually and semantically near-duplicates — for example, separating "a white t-shirt with a blue logo" from "a blue t-shirt with a white logo." The authors observe that while MLLM-based representation models such as VLM2Vec outperform dual-encoder architectures like CLIP on broad semantic matching, they underperform on fine-grained tasks. Two structural reasons are identified. First, causal-attention decoders produce embeddings via last-token hidden states or global pooling, which is incompatible with region-level (RoI) alignment techniques used by fine-grained models such as FG-CLIP (Xie et al., 8 May 2025). Second, standard InfoNCE training penalizes the model for false negatives — semantically identical products that happen to appear as in-batch negatives — a particularly acute problem in E-commerce, where many distinct listings describe the same physical item.

The paper's central thesis is that an MLLM's generative capability can substitute for architectural access to local features: if a model can generate key attributes ("dark red," "silk," "V-neck") from image and text, those attributes can serve both as a supervisory signal during contrastive training and as query enrichment at inference time.

Method

AFMRL decouples responsibilities across two components: a Representation Model (VLM2Vec initialized from Qwen2-VL-2B-Instruct, LoRA-tuned) and an Attribute Generator (Qwen2.5-VL-3B-Instruct, distilled from Qwen2.5-VL-72B-Instruct via 10,000 CoT examples). Training proceeds in two stages.

Attribute-Guided Contrastive Learning (AGCL) modifies the InfoNCE objective in two ways. BM25 scores between generated attributes of a query and each candidate identify lexically hard negatives, which receive bounded importance weights $w_{ij} = e^{1+\tanh(B_{ij})}$. Separately, a masking rule removes any candidate whose cosine similarity to the query exceeds the positive's similarity by a margin $\delta = 0.4$, excluding false negatives from the denominator. Training uses GradCache to reach an effective batch size of 2048 despite the memory footprint of the MLLM backbone.

Retrieval-aware Attribute Reinforcement (RAR) addresses the misalignment between the distilled generator's SFT objective and downstream retrieval. The frozen AGCL-trained encoder acts as part of the reward environment: the policy generates attributes for a query, the augmented input is encoded, and the reward is Recall@k over a candidate pool, with a $-0.1$ penalty for malformed outputs. Optimization uses GRPO ($\beta = 0.01$, $\epsilon = 0.2$, 8 rollouts), chosen for its sample efficiency and absence of a separate value model. The choice of $k$ is empirically motivated: $k=50$ balances reward sparsity against saturation, whereas NDCG rewards converge best at $k=10$ and precision fails to converge at $k=100$.

Results

Experiments use M5Product (5,760,482 products after filtering, 6,000+ categories) and a proprietary dataset, EIPM (~2 million same-product groups, 10M+ items). On coarse-grained and cross-modal retrieval, AGCL consistently improves both LLM2CLIP and VLM2Vec backbones; on M5Product, VLM2Vec + AGCL raises text-to-image Recall@1 from 36.4 to 38.1. These gains hold on the larger EIPM dataset, indicating the method scales beyond academic benchmarks.

The most consequential results concern fine-grained instance retrieval, where positives must match exactly on attributes:

Model	R@1	R@5	R@10	NDCG@10
FG-CLIP	31.44	49.78	68.38	50.81
VLM2Vec	48.05	64.26	69.65	60.90
+ AGCL	51.06	68.08	73.52	62.02
+ AGCL + Distilled Gen.	52.42	71.00	76.26	64.34
AFMRL (full)	54.28	72.19	77.27	65.69

Two claims deserve emphasis. First, the authors state that without attribute augmentation, "the inability to fully exploit local features nearly erases the representational advantage brought by the strong reasoning capabilities of MLLMs" — i.e., raw MLLM embeddings do not automatically confer fine-grained superiority. Second, the RL stage contributes a further +1.86 R@1 beyond distillation alone, which the paper presents as validation of its core hypothesis that optimizing attribute generation directly against the retrieval metric is maximally effective. Case studies support this mechanistically: attribute augmentation increases the distance to hard negatives while preserving similarity to positives.

On downstream classification and clustering (849,207 items, 5,146 classes), AFMRL variants also lead: Accuracy improves from 87.80 (SFT generator) to 88.00 (RL generator), and Circular Iterative Training — reusing the RL-refined generator within AGCL on only 30% of samples — reaches 89.13 Accuracy and 75.98 Purity, suggesting a viable self-improvement loop.

Analysis of RL dynamics

A notable emergent behavior is generation conciseness: average output length decreases monotonically during RL. Unlike mathematical reasoning tasks where GRPO encourages longer chains of thought, the retrieval reward penalizes redundant attributes as noise, implicitly driving the policy toward minimal sufficient attribute sets. Qualitative results confirm that RL suppresses noisy or overly general terms extracted from verbose product titles.

Limitations

The paper candidly reports an alignment tax: although $\pi_{RL}$ wins on retrieval, its advantage over $\delta = 0.4$0 on classification and clustering is marginal (+0.20 Accuracy), because Recall@k-specific optimization specializes representations at the cost of generality. Additional constraints include dependence on a 72B oracle for cold-start distillation (acknowledged as impractical for direct deployment), sensitivity of the RL signal to the reward metric and top-$\delta = 0.4$1 hyperparameter, and evaluation confined to E-commerce data; whether AGCL's false-negative masking generalizes to domains with less duplicate-heavy corpora remains untested. The margin $\delta = 0.4$2 is fixed rather than analyzed.

Conclusion

AFMRL demonstrates that MLLM-generated attributes can compensate for the architectural inability of decoder-based embedders to exploit local features, through hard-negative mining and false-negative filtering in contrastive training, followed by reinforcement learning that aligns attribute generation with retrieval recall. State-of-the-art fine-grained instance retrieval (54.28 R@1 versus 48.05 for the VLM2Vec baseline) and consistent gains across two large-scale datasets substantiate the framework, while the observed alignment tax and oracle-dependence leave open how to obtain task-specialized yet generally useful embeddings without iterative RL loops.