LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval

Abstract: Retrieval systems underpin modern AI applications -- spanning visual search, recommendation engines, and multi-modal question answering. Modern multi-stage retrieval systems require the joint optimization of highly coupled parameters, yet traditional hyperparameter optimization (HPO) methods -- including Tree-structured Parzen Estimators (TPE) and Gaussian Process Bayesian Optimization -- rely on an independence assumption that fundamentally prevents them from navigating these coupled configuration spaces. We address this limitation with a phase-aware LLM agent that conditions each proposal on its full optimization history, navigating the coupled parameter space across phase-partitioned exploration, exploitation, and fine-tuning stages. Evaluated on the HICO-DET human-object interaction retrieval benchmark using Intel VDMS (Visual Data Management System), our agent outperforms Optuna TPE by +33.3% and VDTuner by +34.2% under SIEVE (Safeguarded Index Evaluation of Vector-search Efficiency, a quality-constrained throughput metric), delivering a 15.3x throughput gain over UniIR. Validation across three benchmarks confirms that the agent's advantage grows with the degree of parameter coupling: +33.3% on HICO-DET (high coupling), methods converge within 1% on GLDv2 (moderate coupling) and within 3.6% on SIFT1M (near-independent control). Cross-system validation on Milvus confirms the optimizer ranks first on all three datasets without modification, demonstrating transferability across vector database management system (VDBMS) platforms.

• The paper introduces a phase-aware, history-conditioned LLM agent and the SIEVE quality-gated objective, achieving a mean score of 300.3 on HICO-DET—33.3% above Optuna TPE—with consistent results across seeds.
• The method targets coupled, mixed-type search spaces where ANN parameters, candidate-pool size, and reranking weights interact, reaching a strong HNSW configuration after only six of 50 evaluations.
• Experiments show the advantage depends on optimization complexity: the agent leads on HICO-DET and across Milvus validation, but converges near competing methods on less-coupled GLDv2 and SIFT1M settings.],
• follow_up_questions:[
• How does the SIEVE quality threshold change the optimizer’s preferred balance between mAP and QPS?
• Why do history conditioning and phase-aware exploration improve reliability in coupled ANN search spaces?
• How well does the proposed approach scale to million- or billion-vector retrieval systems?
• Can the phase-aware LLM agent optimize additional objectives such as latency, memory use, recall, and indexing cost simultaneously?
• Find recent papers about LLM-guided optimization of vector retrieval systems.

This paper addresses a structural deficiency in hyperparameter optimization for multi-stage vector retrieval systems: standard HPO methods (TPE, GP-BO, VDTuner) model parameters as independent marginals, which fails when ANN index parameters and reranking parameters are coupled across stages. The authors propose a phase-aware LLM agent that conditions each proposal on its full optimization history, introduce SIEVE — a quality-gated throughput objective — and repurpose HICO-DET as the first text-query human-object interaction (HOI) retrieval benchmark. The work is evaluated on Intel VDMS with cross-system validation on Milvus.

Problem setting and motivation

The target system is a two-stage retrieval pipeline: Stage 1 performs CLIP ViT-L/14-based approximate nearest neighbor (ANN) search over image embeddings stored in VDMS; Stage 2 reranks the candidate pool using DINOv2 ViT-L/14-reg4 features fused via a convex combination controlled by weight $\alpha$. The joint configuration space $\Theta$ spans seven parameters — engine (FaissFlat, FaissHNSWFlat, FaissIVFFlat), engine-conditional index parameters ($M$/$\mathit{nlist}$, $\mathit{efSearch}$/$\mathit{nprobe}$), candidate pool size $k$, fusion weight $\alpha$, reference count $n_r$, and reference strategy — totaling $|\Theta| = 102{,}144$ configurations, of which only $\Theta$0 evaluations (0.05%) are affordable given an oracle cost of roughly 8–9 minutes per call (index rebuild plus 600 batch queries).

Two couplings invalidate independence assumptions. First, pool size $\Theta$1 determines the recall ceiling available to Stage 2: at $\Theta$2, SIEVE Score collapses to zero unless $\Theta$3 is simultaneously raised to compensate; the optimal $\Theta$4 operating point is reachable only through joint search. Second, engine selection gates which continuous parameters exist at all, making the space engine-conditional and mixed-type. The paper's central empirical claim is that these couplings are precisely where LLM-guided history-conditioned reasoning pays off.

The SIEVE objective

SIEVE (Safeguarded Index Evaluation of Vector-search Efficiency) scores a configuration by its QPS only if mAP meets a quality floor $\Theta$5, otherwise zero:

$\Theta$6

The threshold is calibrated between UniIR's failing mAP (0.095) and the unoptimized CLIP baseline (0.208), with rankings stable for $\Theta$7. A notable claim here is that configurations ranked best by conventional smooth metrics routinely fall below this quality floor, meaning smooth surrogates actively mislead optimization toward infeasible configurations. This discontinuous feasibility cliff is the mechanism behind GP-BO's repeated failures across all three datasets.

The phase-aware LLM agent

The agent maintains no surrogate model; its entire state is the ordered history trajectory $\Theta$8. Each iteration constructs a prompt from four components: fixed system context encoding parameter physics, a diagnostic class $\Theta$9 mapping the latest outcome to one of five directives (excellent, low_mAP, low_QPS, below_best, moderate), the full history, and dynamically computed untried-value hints. The budget is partitioned into Exploration (40%), Exploitation (35%), and Fine-tuning (25%) phases, encoding a structural-before-continuous principle: engine choice must be resolved before $M$0 and $M$1 refinement. Anti-collapse rules embedded in the prompt force structural diversity after two consecutive $M$2-only proposals and trigger stagnation escape after five non-improving iterations. Duplicate proposals are retried up to four times with random fallback. The full 50-iteration run costs under $0.20 in LLM API tokens, negligible against the 6.5–7.5 hour oracle cost.

The paper argues that plain OPRO-style history conditioning is insufficient for three reasons: the conditional search space violates OPRO's fixed-candidate assumption, unphased proposals average over the feasibility cliff rather than committing to its feasible side, and independent marginals cannot reach the joint $M$3 optimum. The ablation study substantiates this bracketing.

Benchmark construction

HICO-DET is repurposed into a text-to-image retrieval benchmark: all 47,776 images form the corpus, one canonical template query ("a person {verb} a {object}") per HOI category yields 600 queries encoded by the same CLIP text encoder, and the 90,641 annotated positive pairs provide ground truth without new labeling. Three properties make it configuration-sensitive: compositional fine-grained discrimination (categories sharing objects but differing in verb), long-tailed category sizes (5 to over 1,000 positives), and a dual-encoder modality gap requiring joint tuning of $M$4 with Stage-1 parameters.

A centroid-displacement argument formally justifies text queries: since CLIP's contrastive training aligns text embeddings near class visual centroids, the expected squared distance from a text query to class members approximates the intra-class variance $M$5, strictly below that of any single reference image by its centroid displacement. Empirically, text queries outperform single-image references by +2.32 pp mAP, consistent with the theory.

Results on HICO-DET

The headline numbers are strong. The LLM agent achieves a mean SIEVE Score of 300.3 versus Optuna TPE at 225.2 (+33.3%), VDTuner at 223.8 (+34.2%), GP-BO at 199.3 (+50.7%), Grid Search at 183.0, and Random Search at 111.4. Relative to system baselines, this is a 36.6× gain over the unoptimized CLIP FaissFlat reference (Score 8.21) and 15.3× over UniIR, whose fine-tuned CLIP-SF variants fail SIEVE entirely (mAP 0.095/0.089 < τ). UniIR's 2.5× mAP drop relative to zero-shot CLIP on compositional HOI queries is itself a noteworthy negative result about multi-task fine-tuning disrupting compositional alignment.

Three findings deserve emphasis. First, the optimal region is boundary-adjacent and structurally non-obvious: HNSW with $M$6, $M$7, $M$8, $M$9 centroid achieves QPS ≈ 300 with mAP 0.160 just above τ; across 188 logged $\mathit{nlist}$0 trials, feasibility jumps from 46.2% at $\mathit{nlist}$1 to 84.2% at $\mathit{nlist}$2. Second, reliability differs sharply: the LLM agent reaches Score ≥ 299 on 3/3 seeds (<0.3% spread) while GP-BO spans 110–299 and Optuna TPE is bimodal (two seeds ≈188, one ≈301); a Bartlett test confirms the variance gap ($\mathit{nlist}$3). Third, convergence is fast: the agent exceeds every baseline's final score by iteration 6. Notably, the agent's final mAP (0.161) is below the unoptimized CLIP baseline (0.208) — by design, since SIEVE trades excess quality margin for throughput within the feasible region.

An optional Query Difficulty Score (QDS) variant upweights hard queries via gallery entropy minus centroid proximity, reaching 1,362 QPS with IVFFlat. The authors candidly flag this as a corpus-scale artifact: at 47K vectors, VDMS scans all Voronoi cells regardless of $\mathit{nlist}$4, making IVFFlat effectively exact; the same configuration collapses to Score ≈ 35 at million-vector scale (verified on SIFT1M).

Complexity-proportional advantage

The paper's most disciplined claim is that the LLM advantage scales with coupling complexity. On GLDv2 (762K gallery, image-to-image, moderate coupling), the top adaptive methods converge within 1% (VDTuner leads at 273.91 vs. the LLM agent's 271.45, within seed variance); GP-BO again collapses to 56.6 QPS due to the cliff effect, locking onto $\mathit{nlist}$5 for 40 of 50 iterations. On SIFT1M (single-stage HNSW, no cross-stage coupling by construction), all methods except GP-BO converge within 3.6%, with the LLM agent marginally first at 1184.5 but slower to converge than Grid Search (iteration 21 vs. 5) — sample-efficiency rankings invert exactly where architectural reasoning confers no advantage. This gradient (33.3% → <1% → <3.6%) supports the thesis that LLM-guided optimization is warranted specifically when parameters interact across stages.

Cross-system validation on Milvus, with zero code changes, shows the LLM agent ranking first on all three datasets across all nine seeds (+7.0% over VDTuner on HICO-DET, +13.3% on GLDv2, +5.8% on SIFT1M), indicating the improvement is structural rather than VDMS-specific.

Ablations

The mechanistic ablations isolate the source of the advantage. Removing history conditioning causes complete stagnation after iteration 5 (+0.1 points over 45 remaining iterations) and 0/3 seeds reaching Score ≥ 299, versus 3/3 for the full agent — the primary effect is a binary reliability switch, not a marginal gain. Removing phase structure costs only −0.3% with 3/3 seeds still crossing threshold, confirming phases accelerate rather than enable convergence. The non-monotonic interaction (removing both outperforms removing history alone) arises because history without phase boundaries anchors the agent to its plateau. Backbone portability holds: GPT-4o-mini and Llama-3.3-70B fall within 0.95% and 2.0% of the MiniMax-M2.1 mean, and results are insensitive to phase-boundary shifts (±0.53%).

Limitations and open questions

Several limitations are conceded explicitly. The QDS result depends on a corpus-size artifact specific to 47K vectors and does not generalize to million-vector scale. The diagnostic thresholds in $\mathit{nlist}$6 were empirically calibrated on HICO-DET before comparisons, raising transfer questions for other domains. The evaluation budget of $\mathit{nlist}$7 covers only 0.05% of the space, and token consumption grows linearly with history length, with top-$\mathit{nlist}$8 retention proposed but not evaluated. The SIEVE threshold $\mathit{nlist}$9 is dataset-specific (0.15 for retrieval benchmarks, 0.90 Recall@10 for SIFT1M), and the paper notes that below 0.15 spurious feasible regions emerge. Scaling to billion-vector corpora and extending the framework to genuinely multi-objective settings remain open questions stated by the authors.

Conclusion

The paper makes a well-controlled case that coupled, discontinuous configuration spaces — not flat knob spaces — are where LLM-guided optimization provides a decisive advantage over TPE, GP-BO, and VDTuner, delivering +33.3% over the strongest baseline on HICO-DET with perfect seed reproducibility while converging to parity on simpler datasets. The combination of the SIEVE metric, the HICO-DET retrieval benchmark, and the complexity-matched three-dataset protocol gives the claim unusual evidential discipline, though the calibrated guidance thresholds and small-scale QDS artifact temper generalization beyond the tested regimes.