Cycle-Consistent Search: Question Reconstructability as a Proxy Reward for Search Agent Training

Abstract: Reinforcement Learning (RL) has shown strong potential for optimizing search agents in complex information retrieval tasks. However, existing approaches predominantly rely on gold supervision, such as ground-truth answers, which is difficult to scale. To address this limitation, we propose Cycle-Consistent Search (CCS), a gold-supervision-free framework for training search agents, inspired by cycle-consistency techniques from unsupervised machine translation and image-to-image translation. Our key hypothesis is that an optimal search trajectory, unlike insufficient or irrelevant ones, serves as a lossless encoding of the question's intent. Consequently, a high-quality trajectory should preserve the information required to accurately reconstruct the original question, thereby inducing a reward signal for policy optimization. However, naive cycle-consistency objectives are vulnerable to information leakage, as reconstruction may rely on superficial lexical cues rather than the underlying search process. To reduce this effect, we apply information bottlenecks, including exclusion of the final response and named entity recognition (NER) masking of search queries. These constraints force reconstruction to rely on retrieved observations together with the structural scaffold, ensuring that the resulting reward signal reflects informational adequacy rather than linguistic redundancy. Experiments on question-answering benchmarks show that CCS achieves performance comparable to supervised baselines while outperforming prior methods that do not rely on gold supervision. These results suggest that CCS provides a scalable training paradigm for training search agents in settings where gold supervision is unavailable.

• The paper introduces a novel cycle-consistency framework that uses question reconstructability as a proxy reward, eliminating the need for gold supervision in RL training.
• It implements dual bottlenecks—final response exclusion and named entity masking—to prevent shortcut learning and ensure genuine information acquisition.
• Empirical results show CCS outperforming both gold-free and supervised methods by up to 9.8% across multiple benchmarks, highlighting its robustness in diverse domains.

Cycle-Consistent Search: Training Search Agents Without Gold Supervision

Motivation and Problem Statement

Modern search agents leveraging LLMs are increasingly expected to perform iterative, multi-step query generation and information synthesis to answer complex user queries. RL-based frameworks have become the norm for optimizing such agents, but they heavily rely on gold supervision (ground-truth answers) to define reward functions, which is costly and infeasible in specialized or rapidly-evolving domains. The paper proposes the Cycle-Consistent Search (CCS) paradigm, establishing a scalable, gold-supervision-free RL objective for search-agent training by leveraging cycle-consistency—the degree to which a search trajectory encodes enough information to reconstruct the originating query.

Figure 1: Illustrates how high-quality search trajectories serve as lossless encodings of the question's intent, whereas insufficient or irrelevant trajectories fail to preserve reconstructability.

CCS Framework and Information Bottlenecks

CCS reframes trajectory optimization as maximizing the mutual information between the input question and the information-bottlenecked search trajectory. The core hypothesis is that, for an optimal policy, the question should be recoverable from the agent's trajectory—provided the trajectory contains all observations and intermediate search steps necessary to resolve the original information need. The method leverages a reconstructor model to map the trajectory $\tau$ back to $\hat{q}$. However, naive cycle-consistency objectives are vulnerable to information leakage: superficial lexical overlap between actions and the original question enables trivial reconstruction without incentivizing genuine information acquisition.

To mitigate shortcut exploitation, CCS applies two information bottlenecks:

• Final Response Exclusion: Removes the final answer from the reconstruction input since it often paraphrases the initial question.
• Named Entity Masking: Replaces named entities in search queries with generic tags (e.g., [LOC]), reducing direct lexical copying. Observations remain unmasked.

This bottlenecked trajectory $\tilde{\tau}$ ensures that reconstruction quality reflects agentic information gain rather than linguistic redundancy.

Figure 2: Overview of CCS: the policy generates a search trajectory, which is passed through bottlenecks before the reconstructor attempts question recovery, enforcing an evidence-driven reward signal.

RL Objective and Optimization

The CCS learning objective maximizes the expected log-likelihood that the reconstructor recovers the original question, given the processed trajectory. Direct modeling is intractable, so a frozen reconstructor approximates the inverse mapping. The reward function is the semantic similarity between embeddings of $q$ and $\hat{q}$. Optimization uses Group Relative Policy Optimization (GRPO), which computes group-level normalized advantages to guide policy updates, improving learning stability and avoiding the need for a value-function critic.

Empirical Evaluation

CCS was evaluated across seven benchmark QA datasets and a deep research benchmark. Policy models were sampled from the Qwen family, with evaluations performed using Gemini 2.5 Flash as a judge. CCS consistently achieved the best performance among gold-free training methods, surpassing RLIF, CJ, and TTRL by up to 9.8\% average score improvements. Remarkably, CCS also matches or outperforms supervised RL baselines (e.g., Search-R1) on several datasets, reversing the typical performance gap between gold-free and gold-supervised approaches.

Figure 3: Exemplifies how CCS rewards only those trajectories that preserve both the search structure and supporting observations for faithful question reconstruction, penalizing information voids and shallow search depth.

Ablation and Analysis

Ablation studies demonstrate that both bottlenecks are essential. Including final responses or using unmasked actions yields lower scores due to shortcut signals. Masked actions contribute more than simply suppressing leakage—they retain structural intent necessary for credible question reconstruction. Observations alone are insufficient, confirming the need for structural preservation.

Qualitative analysis reveals that CCS effectively distinguishes between trajectories with information voids, incomplete scaffolds, and those that are genuinely informative and reconstructable.

Open-Ended Deep Research Benchmark

CCS was further assessed on the ResearchRubrics deep research benchmark, which expects evidence-backed, long-form responses across ten diverse domains. CCS not only outperformed all gold-free baselines but also demonstrated superior robustness compared to Search-R1 (a supervised RL method), with up to 20\% relative improvement in certain domains.

Figure 4: CCS achieves the highest scores across ten domains in the ResearchRubrics benchmark, demonstrating superior evidence integration and reasoning capabilities for deep research tasks.

Practical and Theoretical Implications

CCS establishes reconstructability as a viable, scalable proxy reward for search-agent optimization, reducing dependence on expensive annotation pipelines. The dual bottleneck architecture ensures that rewards genuinely reflect information acquisition and structure preservation, making CCS robust for knowledge-intensive and open-ended tasks. The framework generalizes cycle-consistency to agentic search and can theoretically extend to broader agent-based systems, where the internal trajectory must encode all information necessary for task recovery.

Practically, CCS enables RL-based training for search agents in settings devoid of gold answers, promoting rapid domain adaptation and minimizing annotation overhead. The concept of reconstructability as a reward can be generalized to other sequential decision-making environments with latent structure and incomplete external supervision.

Future Directions

Future research could focus on improving the efficiency of the reconstructor, integrating it more tightly with actor-critic architectures, and extending CCS to domains beyond QA—potentially encompassing complex planning, scientific exploration, or interactive discovery agents. The theoretical link between trajectory information conservation and reward signal quality invites deeper investigation into mutual information objectives for agentic RL.

Conclusion

Cycle-Consistent Search (CCS) provides a principled, scalable approach for gold-supervision-free search agent training, leveraging reconstructability of the original question from information-bottlenecked trajectories as a proxy reward. CCS narrows—and in some benchmarks, eliminates—the gap between gold-free and supervised RL agents, demonstrating robustness in both closed and open-ended tasks. The approach lays foundational groundwork for RL-based agentic systems in data-sparse environments and opens new directions for unsupervised agent optimization (2604.12967).