Loom: Hybrid Retrieval-Scoring Outfit Recommendation with Semantic Material Compatibility and Occasion-Aware Embedding Priors

Abstract: We present Loom, an outfit recommendation system that combines neural embedding retrieval with structured domain scoring to generate complete, coherent outfits from fashion catalogs. Given an anchor clothing item, Loom retrieves complementary pieces via slot-constrained approximate nearest neighbor search over FashionCLIP embeddings, then scores candidate outfits using a multi-objective function that integrates six signals: embedding similarity, color harmony, formality consistency, occasion coherence, style direction, and within-outfit diversity. We introduce two techniques that address limitations of purely learned or purely rule-based approaches: (1) semantic material weight, which uses CLIP embedding geometry to infer garment heaviness for layer compatibility without hand-coded material taxonomies; and (2) vibe/anti-vibe occasion priors, which embed prose descriptions of occasion contexts as anchor vectors in CLIP space and score items by differential affinity. Ablation experiments on a catalog of 620 items show that each component contributes measurably to outfit quality: the full system achieves a mean outfit score of 0.179 with a 9.3% hard violation rate, compared to 0.054 score and 16.0% violations for a category-constrained random baseline, a 3.3x improvement in score and 42% reduction in violations. Direction reranking is the single indispensable component: removing it drops score to 0.052, essentially equal to random. The system generates three stylistically distinct outfits in under 5 seconds on commodity hardware.

• The paper introduces a hybrid system that combines neural-based retrieval with rule-based scoring to improve context-aware outfit composition.
• It employs semantic material compatibility using FashionCLIP embeddings to enforce layering constraints and assess garment heaviness.
• Occasion-aware embedding priors guide the multi-objective scoring process, leading to improved stylistic diversity and reduced violation rates.

Detailed Analysis of "Loom: Hybrid Retrieval-Scoring Outfit Recommendation with Semantic Material Compatibility and Occasion-Aware Embedding Priors" (2605.09830)

Overview

Loom is a hybrid system for automated outfit recommendation that integrates neural embedding-based retrieval and structured, domain-specific scoring. The system is designed to assemble complete, contextually-coherent outfits from fashion catalogs, addressing both the compositional nature of outfit recommendation and the need to enforce domain-specific "hard" fashion constraints often overlooked by pure embedding approaches. The architecture is underpinned by FashionCLIP multimodal embeddings and incorporates additional semantic priors to enhance compatibility and suitability for user-specified occasions and style directions.

System Architecture and Methodological Innovations

The system operates via a four-stage pipeline: vision-based attribute extraction, multimodal item embedding, slot-constrained candidate retrieval, and multi-objective combinatorial scoring. Two principal contributions address key challenges in automated outfit composition:

1. Semantic Material Weighting: Loom estimates garment "heaviness" and layering compatibility by exploiting CLIP embedding geometry. Instead of relying on a manually curated taxonomy, the system computes compatibility between clothing items (e.g., ensuring outer layers are "heavier" than inner layers) via context vectors anchored by descriptive keyword sets for heavy/light materials. This approach harnesses the geometry of the FashionCLIP embedding space, enabling flexible and extensible material reasoning.
2. Vibe/Anti-vibe Occasion Priors: Occasion-awareness is achieved through embedding rich, prose-style contextual descriptions of event "vibes" and their antitheses as anchor vectors. Candidate items are scored based on differential affinity to these vectors, with occasion-specific penalty shaping. This supports fine-grained, nuanced occasion filtering that is not tied to rigid category taxonomies and can be easily extended to new use cases.

Additionally, the system blends image and structured text embeddings (70/30 weighting), mitigating visual ambiguities and permitting attribute-level discrimination absent in image-only representations. Retrieval is conducted with ANN search using pgvector HNSW indices, augmented by coloring and category constraints.

Multi-objective Scoring and Outfitting Strategy

Loom evaluates outfit candidates through a composite scoring function that jointly considers:

• Embedding similarity to intent vectors (which can incorporate user taste profiles via EMA averaging of liked/disliked styles)
• Slot-specific importance weights (emphasizing, for example, bottoms and shoes over tops)
• Bonuses for adherence to one of three style "directions" (Classic, Trendy, Bold), each with rules for color, tags, and palette diversity
• Penalties for violations of color harmony, formality consistency, occasion coherence, and statement-piece overload

The system considers the top $k=3$ candidates per slot, yielding up to $3^5=243$ outfit combinations per direction, of which one per direction is selected after scoring. This design yields simultaneously diverse and contextually consistent outfit suggestions.

Empirical Evaluation and Findings

Ablation studies on a 620-item, real-world fashion catalog demonstrate the quantitative impact of each component. The primary findings are:

• Composite System Performance: Loom achieves a mean outfit score of 0.179 and a 9.3% hard violation rate, compared to 0.054 and 16.0% for a category-constrained random baseline—a 3.3× score improvement and 42% violation reduction.
• Component Importance: Direction reranking is critical; removing it drops performance to near-random, erasing stylistic diversity and directionality. The occasion and blended-embedding components contribute modest, positive effects, while removing material compatibility penalties paradoxically increases the composite score and reduces violation rate. This suggests over-conservative penalty parameterization, motivating forthcoming refinement.
• Efficiency: Generation of three outfits takes under 5 seconds on CPU, making the system practical for both consumer and small-scale commercial deployments.
• Diversity Metrics: Inter-direction color diversity is high (mean 7.42 distinct colors across three directions), confirming stylistic differentiation in outputs.

The evaluation leverages an internally defined composite score, highlighting a limitation in the absence of human preference ratings. The automated nature of the evaluation restricts definitive conclusions about user-perceived quality.

Implications and Future Directions

Loom's hybrid architecture demonstrates that integrating neural and rule-based elements can overcome the domain-specific brittleness of pure embedding models in complex compositional tasks like outfit recommendation. The use of semantic priors in continuous embedding spaces enables extensibility (e.g., new occasions or materials), and the modular scoring function supports rapid iteration on domain logic.

However, scalability is limited by the combinatorial scoring phase, which becomes impractical for catalogs with orders of magnitude more items per slot. Addressing this will require hierarchical candidate pruning, learned scoring networks, or alternative search formulations. The absence of benchmark user studies or adaptation to standard FITB/recommendation metrics leaves open questions regarding the generalizability and subjective utility of outfits produced by the system.

Proposed future work includes training complement embeddings to model item-to-outfit relationships directly, data-driven scoring using user preference signals, and expanding to cross-catalog recommendations for wardrobe augmentation.

Conclusion

Loom establishes a blueprint for hybrid, composition-aware fashion recommendation that leverages the representational power of multimodal embeddings while respecting real-world domain constraints through structured priors and rule-based scoring. Empirical results validate the value of the hybrid approach and specifically underscore the necessity of style-directional reranking. The system is deployed in both personal and retail wardrobe settings and serves as a foundation for further development towards preference-aligned, scalable, and compositionally sophisticated outfit generation.