RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation

Abstract: Recommender systems increasingly face a choice among heterogeneous agents -- collaborative filters, sequential models, content-based retrievers, and LLM-based rerankers -- yet no single agent is uniformly best. We study this choice as task-aware agent ranking under cost constraints using RouteRec, a framework that compares request-level hard selection with item-level learned aggregation over four traditional recommender agents and one LLM reranker agent. On MovieLens-1M, the full quality oracle has substantial headroom (HR@10 = 0.584), confirming that useful cross-agent signal exists. Under a leakage-free 5-fold out-of-fold protocol, however, hard selection remains below BM25 (0.223 vs. 0.254), and selective LLM escalation does not improve it. The same protocol yields a different outcome for learned aggregation: its cheap-only variant matches BM25 in HR and has a higher NDCG point estimate (0.123 vs. 0.114), while gated all-agent aggregation reaches HR@10 = 0.295 with 70.2\% LLM calls. The resulting lesson is not that routing is solved, but that request-level selection of one complete agent list is too coarse for this sparse fixed-candidate setting; item-level aggregation is the more promising action space.

• The paper introduces a novel utility maximization framework for selecting and aggregating heterogeneous recommender agents subject to latency and cost constraints.
• It proposes two methods—RouteRec-Select using cheap-probe disagreement and RouteRec-Stack leveraging item-level features—to guide decision-making.
• Empirical findings show that while hard selection underperforms fixed baselines, learned item-level aggregation significantly improves HR and NDCG metrics.

RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation

Problem Overview and Motivation

Heterogeneous recommender agents—including collaborative filtering, sequential models, content-based retrievers, and LLM-based rerankers—have proliferated, but no individual agent dominates across all request contexts or user segments. RouteRec addresses the task-aware selection and aggregation of such agents under explicit cost-quality tradeoffs. The central problem shifts from traditional model selection to instance-level policy optimization: determining which agent or combination of agents should serve a given recommendation request, subject to latency and computational resource constraints.

The paper formalizes this via utility maximization per request, where utility is a weighted sum of retrieval quality (NDCG, HR), penalized by latency and token usage. The core evaluation is conducted on MovieLens-1M with a pool of four lightweight recommenders and a Qwen-2.5-7B-Instruct LLM reranker, using strict 5-fold out-of-fold (OOF) splits to avoid overfitting and information leakage.

RouteRec Method: Architecture and Protocol

RouteRec is introduced in two main forms:

1. RouteRec-Select: A two-stage per-request router that:
   • Stage 1: Selects the most promising cheap agent using a learned compatibility function over request features, agent fingerprints, and cheap-probe disagreement statistics.
   • Stage 2: Optionally escalates to the LLM reranker via a learned gating mechanism, based on predicted marginal utility gain from escalation.

     Figure 1: RouteRec-Select architecture, demonstrating the two-stage selection pipeline driven by request encoding and cheap-probe disagreement for hierarchical agent selection and escalation.

2. RouteRec-Stack: An aggregation-based relaxation that, instead of choosing one agent, reranks the merged candidates from agent top-$k$ lists using item-level features and lightweight models (logistic regression, shallow boosting). This enables finer granularity exploitation of cross-agent signal.

The policy-level utility is explicitly defined as:

$$U(x,\pi,S) = \alpha \cdot \text{NDCG}(x,\pi) + \beta \cdot \text{HR}(x,\pi) - \lambda \cdot \text{lat}(S) - \mu \cdot \text{tok}(S)$$

with calibrations ($\alpha=1$, $\beta=0.5$, $\lambda=\mu=0.01$).

A key component is the introduction of "cheap-probe disagreement"—statistical and rank-based metrics quantifying divergence between traditional agent outputs—which informs both routing and escalation decisions.

Strong and Contradictory Empirical Findings

The empirical analysis confirms that substantial oracle headroom exists: a per-request oracle that selects the best among all agents (including the LLM) achieves HR@10 of 0.584, while the best cheap-only oracle reaches 0.508, indicating widespread complementarity.

Figure 2: Distribution of the quality-best agent across users evidencing agent diversity and headroom for routing.

Despite this, the hard selection policies underperform relative to the best single fixed agent (BM25). RouteRec-Select achieves HR@10 of $0.223 \pm 0.008$, which is lower than BM25's 0.254, and LLM escalation via the gating mechanism does not yield improvements—escalation leads to an even lower HR of $0.215 \pm 0.016$ at 10.4% LLM usage.

Figure 3: RouteRec-Select cheap-only routing traffic, revealing strong preference for BM25 and SASRec agents.

The main contradictory finding is that, while the oracle analysis suggests significant benefits from routing, in practice, the learnable hard-selection policies with available features and sparse feedback fail to reliably improve over fixed baselines. Instead, learned item-level aggregation via RouteRec-Stack is markedly superior:

• StackCheap matches BM25 in HR and slightly exceeds it in NDCG.
• StackGate (gate-controlled aggregation-switching) and StackAll both offer strict improvements as the allowed LLM fraction increases, with StackAll reaching HR@10 of $0.299 \pm 0.015$ at 100% LLM aggregation.

Analysis of Subgroups, Ablations, and Cost/Quality Frontier

Subgroup analysis shows that hard selection does not yield meaningful improvements for cold users, short histories, or instruction-rich segments, except for a minor gain on a small long-tail subset—insufficient for general efficacy claims.

Figure 4: HR@10 across subpopulations, highlighting Stack methods outperforming strict selection methods only on long-tail subgroups.

Ablation analysis demonstrates that no single feature family (user, candidate, task, probe, agent capabilities) is dominant or critical under the observed sample size and sparsity. The performance variation from removing any group is within the baseline variance.

Figure 5: HR@10 variation across feature ablation variants—no significant effect relative to full RouteRec-Select.

Further, escalation gate evaluation shows low F1, confirming difficulty in isolating LLM-worthy cases in this sample. Experiments with a stronger LLM backend (Grok 4.1 Fast) improve LLM-alone HR slightly (0.162 to 0.176), but still trail BM25.

Cost-quality analysis reveals that hard selection does not exhibit a beneficial tradeoff—more LLM calls only degrade overall utility. In contrast, learning-based stacking policies admit a controlled, monotonic improvement in HR and NDCG as the LLM call fraction increases, providing a useful operational frontier.

Figure 6: Cost-quality curves for hard selection vs. learned aggregation strategies, illustrating smooth stacking-based quality gains with LLM-investment.

Comparison with quality oracles emphasizes that item-level aggregation recovers a substantial portion of the theoretical gap, whereas hard selection is consistently suboptimal.

Figure 7: HR@10 and candidate coverage comparison between selection and aggregation policies, highlighting the headroom only partially recoverable by learning item-level stacking.

Implications and Future Research

Theoretical implications are substantial: oracle complementarity is not synonymous with high policy learnability, especially in regimes with sparse feedback, high agent candidate overlap, and limited supervision. Aggregating at the granularity of items—using deployable rank and score features from agent lists—proves to be a strictly more effective use of heterogeneous signal than committing to a single agent per request.

From a practical perspective, the findings advise caution when deploying request-level agent selection pipelines: even with sophisticated features and architecture, the empirical benefit may be limited—unless request-level supervision and candidate diversity are substantially increased. Instead, focus should shift to learning item-level aggregation models, potentially augmented with conditional fusion or calibrated confidence thresholds to mediate LLM usage and cost.

Limitations cited include the single-dataset/single-pool scope, the constraints of fixed candidate pools, and the lack of free-form input. The findings may not generalize to interactive, multi-turn, or open-ended retrieval settings.

Future research should address scaling these approaches to larger, more diverse agent pools, online or active learning setups, multi-domain and real-time systems, and data regimes with denser and more granular feedback. Additionally, candidate generation strategies that allow agents (especially LLM-based agents) to control or influence the candidate set directly could potentially unlock more of the observed oracle headroom.

Conclusion

RouteRec provides a rigorous evaluation of agent selection versus aggregation in heterogeneous recommender settings. The key finding is the empirical inadequacy of hard agent selection under plausible offline supervision, despite strong oracle signals. In contrast, learned aggregation over item-level evidence from multiple agents consistently recovers more cross-agent information and traces a usable cost-quality tradeoff frontier. Routing and aggregation system design for hybrid and agentic recommendation must therefore prioritize actionable aggregation spaces and policy learnability, not just apparent agent complementarity.

(2607.09908)