AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models alongside LLM-based AI Agents with Cloud Computing Access

Abstract: Recent advances in AI agents for software engineering and scientific discovery have demonstrated remarkable capabilities, yet their application to developing novel ranking models in commercial search engines remains unexplored. In this paper, we present an AI Co-Scientist framework that automates the full search ranking research pipeline: from idea generation to code implementation and GPU training job scheduling with expert in the loop. Our approach strategically employs single-LLM agents for routine tasks while leveraging multi-LLM consensus agents (GPT 5.2, Gemini Pro 3, and Claude Opus 4.5) for challenging phases such as results analysis and idea generation. To our knowledge, this is the first study in the ranking community to utilize an AI Co-Scientist framework for algorithmic research. We demonstrate that this framework discovered a novel technique for handling sequence features, with all model enhancements produced automatically, yielding substantial offline performance improvements. Our findings suggest that AI systems can discover ranking architectures comparable to those developed by human experts while significantly reducing routine research workloads.

• The paper introduces an expert-supervised AI Co-Scientist that generates ranking hypotheses, implements code, schedules GPU experiments, and analyzes results through multi-LLM consensus.
• The system discovered a unified long-sequence ranking architecture and four-phase learning-rate schedule, improving the evaluation metric by 0.083% over the human-designed V2 baseline and 0.201% over V1.
• The findings show that autonomous agents can produce commercially meaningful ranking improvements, but silent bugs, GPU waste, myopic optimization, and limited safety controls still require expert oversight.

Overview

This paper presents an AI Co-Scientist framework that automates the full research pipeline for commercial search ranking model development, from hypothesis generation through code implementation to GPU experiment scheduling and results analysis (2603.22376). The system is deployed in a production search ranking context at Trip.com Group, where the authors report that a 0.1% gain on their evaluation metric is statistically significant and has historically translated into approximately 0.1% conversion rate lift—worth millions of dollars in online experiments. The central claim is that LLM-based agents with cloud computing access can discover ranking architectures comparable to those designed by human experts; the authors state this is the first application of an AI Co-Scientist framework to algorithmic research within the ranking community.

System architecture

The framework consists of five modules operating in a closed loop with expert-in-the-loop oversight. A memory module uses a two-layer design: a first layer of index files (JOURNEY.md, EXPERIMENTS.md, FLOWS.md) recording research history, active experiments, and scheduled training flows, and a second layer of per-version implementation files (e.g., V{x}.{y}_IMPLEMENTATION.md) consulted only when needed—an organization pattern similar to Devin's memory practices.

The idea generation module is deliberately seeded with a human-provided hypothesis: that gains from prior transformer-based ranking models stem primarily from scaling the transformer module rather than generative next-item losses, motivated by recent evidence favoring discriminative models in high-noise, high-data, low-latency regimes. Multiple frontier LLMs (GPT-5.2, Gemini Pro 3, Claude Opus 4.5) must reach consensus before proceeding. Code implementation is delegated to single-LLM coding agents, with bugs typically resolved on a second attempt after reading debug logs. The experimentation module commits code to new git branches and schedules GPU training jobs with tunable hyperparameters. Finally, the results analysis module evaluates changes against an aggregated metric $\mathcal{M}$—the average of six AUC metrics spanning three behavior types (click, conversion, grouped conversion) and two search scenes over seven days of unseen data—and decides via multi-LLM consensus whether to continue iterating or revert, governed by a persistence threshold parameter.

Discovered architecture: V3

Starting from V1 (mean-pooling + DCN + MOE) and the human-designed V2 baseline (separate sequence features stacked into a transformer with dense-feature embeddings added at all positions), the AI Co-Scientist autonomously produced all improvements from V2 through V3.5:

Version	Key change	Seq len	LR	$\mathcal{M}$ vs V2 (%)
V2	Transformer + separate sequences (human-designed)	40	5e-5	0.000
V3.0	+ positional encoding, attention pooling	40	5e-5	−0.047
V3.1	Unified long sequence	200	5e-5	−0.050
V3.2	V3.1 + reduced LR	200	1e-5	+0.041
V3.3	+ slot type embeddings	200	1e-5	+0.061
V3.4	+ temporal embeddings	200	1e-5	+0.063
V3.5	+ four-phase LR schedule	200	adaptive	+0.083

Two findings stand out. First, the agent's initial attempts (V3.0, V3.1) failed, but it correctly diagnosed the V3.1 degradation as a learning-rate problem rather than an architectural one—reducing the LR to one-fifth recovered a +0.041% lift, validating the unified-sequence design at length 200 versus 40. Second, the agent independently designed a four-phase learning rate schedule (standard LR for 16 days, reduction, plateau escape via re-increase, then final fine-tuning) addressing the well-documented convergence instability of multi-task ranking models. Cumulatively, V3.5 improves over V2 by +0.083%, over V1 by +0.201%, and over V3.1 by +0.133%. Given the stated economics of the metric, these are commercially meaningful gains produced without manual architecture engineering.

Relation to prior work

The paper positions itself against three lines of work. End-to-end scientific discovery agents such as The AI Scientist (Lu et al., 2024) and AI Co-Scientist (Gottweis et al., 26 Feb 2025) target paper generation and biomedical hypothesis validation respectively; this work adopts the latter's expert-in-the-loop paradigm but extends agent autonomy into code implementation, GPU scheduling, and metric-driven iteration. AlphaEvolve (Novikov et al., 16 Jun 2025) shares the iterative code-modification-with-evaluator-feedback structure but targets general algorithmic improvement. Within ranking architecture research—Wukong, RankMixer, Climber, OneTrans—all prior designs are human-authored, which supports the authors' novelty claim, though the comparison remains qualitative rather than benchmarked head-to-head.

Limitations and open questions

The paper is candid about several failure modes. Coding agents occasionally introduce silent bugs—including unintentional model-size increases—that can cause buggy implementations to "win" on metrics alone, meaning the verifiable reward signal is not fully reliable without expert supervision. Idea generation remains inefficient, wasting substantial GPU resources and requiring human intervention to prevent catastrophic failures. The optimization loop exhibits myopic behavior, reinforcing short-term metric gains over directions a human would judge more promising; the authors note that human "research taste" still materially shapes outcomes. Two structural limitations are also acknowledged: the system lacks integration with scientific databases and real-time literature search, and production deployment lacks automated safety guardrails beyond human oversight. Additionally, the pilot study feeds only five of more than 200 available feature-sequence slots into the transformer, so the scalability of the discovered approach to the full feature set is untested. Whether multi-LLM consensus genuinely outperforms a single strong model is asserted by design but not ablated.

Conclusion

This paper demonstrates that an LLM-based Co-Scientist with cloud compute access can autonomously carry a production ranking model from V2 to V3.5, yielding a +0.083% offline metric improvement through a novel unified-sequence architecture and an automatically discovered four-phase learning rate schedule. The result challenges the assumption that AI cannot manage hypothesis formulation and experimentation in computational disciplines, while the documented failure modes—silent bugs, myopic optimization, GPU waste—indicate that reliable autonomous discovery in ranking research still depends on expert-in-the-loop verification.