Decoding Style: Efficient Fine-Tuning of LLMs for Image-Guided Outfit Recommendation with Preference (2409.12150v1)

Abstract: Personalized outfit recommendation remains a complex challenge, demanding both fashion compatibility understanding and trend awareness. This paper presents a novel framework that harnesses the expressive power of LLMs for this task, mitigating their "black box" and static nature through fine-tuning and direct feedback integration. We bridge the item visual-textual gap in items descriptions by employing image captioning with a Multimodal LLM (MLLM). This enables the LLM to extract style and color characteristics from human-curated fashion images, forming the basis for personalized recommendations. The LLM is efficiently fine-tuned on the open-source Polyvore dataset of curated fashion images, optimizing its ability to recommend stylish outfits. A direct preference mechanism using negative examples is employed to enhance the LLM's decision-making process. This creates a self-enhancing AI feedback loop that continuously refines recommendations in line with seasonal fashion trends. Our framework is evaluated on the Polyvore dataset, demonstrating its effectiveness in two key tasks: fill-in-the-blank, and complementary item retrieval. These evaluations underline the framework's ability to generate stylish, trend-aligned outfit suggestions, continuously improving through direct feedback. The evaluation results demonstrated that our proposed framework significantly outperforms the base LLM, creating more cohesive outfits. The improved performance in these tasks underscores the proposed framework's potential to enhance the shopping experience with accurate suggestions, proving its effectiveness over the vanilla LLM based outfit generation.

• The paper introduces a framework for image-guided outfit recommendation using efficient fine-tuning of LLMs combined with multimodal inputs and preference feedback.
• The methodology employs MLLMs for visual feature extraction, parameter-efficient fine-tuning (PEFT) like LoRA, and reinforcement learning from human feedback (RLHF) like DPO to refine recommendations.
• Evaluation shows significant performance gains on the Polyvore dataset, achieving an 81.03% AUC on Outfit Compatibility Prediction and 61% accuracy on the Fill-in-the-Blank task, outperforming baseline methods.

An Examination of Decoding Style: Efficient Fine-Tuning of LLMs for Image-Guided Outfit Recommendation with Preference Feedback

The paper "Decoding Style: Efficient Fine-Tuning of LLMs for Image-Guided Outfit Recommendation with Preference Feedback" introduces an innovative framework for personalized fashion recommendation systems. The research explicitly targets the synergy between LLMs and multimodal inputs to enhance the recommendation of fashion items by integrating both visual and textual cues. This essay provides a critical overview of the methodologies and findings presented in the paper.

Methodology and Framework

Central to this research is the adaptation of LLMs to understand fashion compatibility and trends through a sophisticated fine-tuning process. The authors utilize multimodal LLMs (MLLMs) to bridge the visual-textual disconnect inherent in fashion item descriptions. Through image captioning, MLLMs extract pertinent style and color characteristics from curated fashion images to generate descriptive captions. These captions serve as inputs for personalized recommendation tasks using an LLM fine-tuned on the Polyvore dataset.

To further refine the output, the framework employs parameter-efficient fine-tuning (PEFT), using methods such as Low-rank Adaptation (LoRA). This approach allows the model to adapt without the computational overhead associated with full-model fine-tuning. Additionally, the use of reinforcement learning from human feedback (RLHF), specifically direct preference optimization (DPO), enhances the system's ability to align with user preferences by incorporating direct negative feedback examples.

Evaluation and Results

The framework's efficacy is evaluated on two tasks: fill-in-the-blank (FITB) and complementary item retrieval using the Polyvore dataset. Notably, the application of PEFT and DPO leads to significant improvements in both tasks. For instance, in the Outfit Compatibility Prediction (CP) task, the proposed method achieved an AUC of 81.03%, markedly outperforming the baseline LLM's 57.9%. Similarly, in the FITB task, accuracy improved to 61%, up from the baseline's 29%.

These improvements underscore the strength of integrating multimodal inputs and preference feedback in training models for fashion recommendations. The results suggest that the proposed framework not only generates more cohesive and stylish outfit combinations but aligns them more closely with seasonal trends and user preferences.

Implications and Future Directions

The research presents both practical implications and prospects for ongoing development in AI-driven fashion recommendation systems. Practically, this approach could significantly impact the retail industry by enhancing personalized shopping experiences and catering to individual style preferences. Theoretically, it extends the capabilities of LLMs by effectively incorporating multimodal data, illustrating potential applications beyond traditional text-based tasks.

Looking forward, expanding the framework could involve integrating additional contextual information such as user location or occasion, thus refining the personalization aspect. Moreover, exploring sophisticated feedback loops that include implicit metrics like click-through rates offers an exciting avenue for further improving recommendation accuracy.

In conclusion, this paper provides a comprehensive approach to leveraging LLMs for fashion recommendations, blending multimodal capabilities with efficient fine-tuning techniques. The positive outcomes from the evaluations suggest promising advancements for AI applications in fashion, with broader applications conceivable through continued research and development.