BrowseComp Harness Engineering Task

• BrowseComp Harness Engineering is defined as the design, optimization, and automated evolution of infrastructure mediating between agentic models and web browsing environments.
• It integrates modular components such as planning, retrieval, vision, and memory logging to ensure reproducibility and facilitate detailed process evaluation.
• The approach employs subgoal-driven metrics and adversarial filtering to diagnose bottlenecks, particularly in visual reasoning, and improve system reliability.

A harness-engineering task in the context of the BrowseComp suite refers to the design, optimization, diagnosis, and automated evolution of the infrastructure that mediates between an agentic model (LLM-based or multimodal) and the web browsing environment. The harness encompasses task decomposition, execution orchestration, tool access patterns, memory management, context provision, interface protocols, verification, logging, and governance. Engineering the harness, rather than solely scaling model weights or hand-tuning prompts, determines the operational stability, compositional capabilities, and reproducibility of agentic systems evaluated on complex benchmarking suites such as BrowseComp-V³, MM-BrowseComp, and their extensions (Zhang et al., 13 Feb 2026, Li et al., 14 Aug 2025).

1. Benchmark Definition and Task Stratification

BrowseComp-V³ is structured as a vertically deep, process-oriented multimodal web-agent benchmark. It spans 300 hand-crafted questions across 24 sub-domains, partitioned into five broad domains (Science, Technology, Society, Culture, Life; 60 examples each). Crucially, BrowseComp-V³ stratifies tasks along two orthogonal axes:

• Search Depth (hops $h$): Each question is tagged as requiring “shallow” ($h=2$), “medium” ($h=3$), or “deep” ($h\geq4$) sequential retrieval steps.
• Cross-modal reasoning level $L\in\{1,2,3\}$: Level 1 (intra-region alignment), Level 2 (inter-region integration), Level 3 (inter-image reasoning).

Adversarial filtering ensures only instances unsolved by the strongest available models (e.g., GPT-5.2-Thinking, Gemini-3-Pro) survive. The harness must expose and control this stratified complexity for reliable evaluation (Zhang et al., 13 Feb 2026).

2. Harness Interface: Execution, Tools, and Reproducibility

The canonical agentic harness underlying BrowseComp admits a modular architecture:

• Planning Module: LLM-based, plans next action using accumulated context $C_t$.
• Retrieval Tools:
  • TextSearch (Serper.dev or equivalent)
  • WebVisit (HTML parsing via Jina)
• Vision Tools:
  • ImageSearch (Google Images)
  • ImageCrop
  • ReverseImageSearch
• Memory & Logger: Logs full search/action trajectories, tool calls, and context transitions.

The harness enforces stepwise logging (query string, top-$k$ result URLs) for evidence traceability and replays, constrains tool access order, and restrictions per-task (e.g., search call limits, web visit step budgets) (Zhang et al., 13 Feb 2026, Wei et al., 16 Apr 2025). The harness also mediates context normalization (deduplication, Unicode normalization, canary tokens) and ensures answer objectivity by requiring answers from a finite set or as numeric values.

Example: Harness Execution Flow (BrowseComp-V³)

$h=2$6 (Zhang et al., 13 Feb 2026)

3. Process Evaluation and Subgoal-Driven Metrics

BrowseComp harnesses depart from pure final-answer metrics by integrating subgoal-driven process evaluation. For each benchmark instance $q$:

• Expert-validated subgoals $G_q = \{g_1, ..., g_{n_q}\}$ trace the canonical evidence or reasoning steps that must be acquired or integrated.
• Model-completed subgoals $\hat G_q \subseteq G_q$ are recorded; the Process Score is $h=2$0.
• The aggregate process score over the benchmark is $h=2$1.

This setup enables fine-grained diagnosis of where harness design—versus agent limitations—bottlenecks performance (Zhang et al., 13 Feb 2026).

Metric	Formula	Notes
Success Rate	$h=2$2	Final answer match to gold answer.
Process Score	$h=2$3	Independent of final answer.

4. Bottleneck Analysis and Error Taxonomy

Empirical results show that even state-of-the-art tool-augmented agents achieve at most 36–39% success rate and 57–66% process score, well below human baselines (68% SR, 83% PS). The primary bottleneck is visual grounding and perception, accounting for 50–60% of errors. As the cross-modal reasoning level increases ($h=2$4), there is a 20–30 percentage point drop in both accuracy and process score, indicating that harnesses require not only API flexibility but robust system integration for visual evidence extraction and reasoning coordination (Zhang et al., 13 Feb 2026, Li et al., 14 Aug 2025).

Further analysis highlights the following:

• Failure types: Visual grounding/misalignment, incomplete tool trajectories, mis-integration of cross-modal evidence.
• Harness-induced variance: Larger interaction budgets and best-of-$h=2$5 sampling in the harness—without changing the core model—raise accuracy by up to 15 percentage points, confirming harness configuration as a key variable (Zhang et al., 13 Feb 2026).

5. Harness Engineering Methodologies and Best Practices

Harness engineering for BrowseComp emphasizes:

• Reproducibility: All evidence must be publicly indexed and retriable; corpus slices, API versions, and random seeds are version-locked.
• Adversarial Filtering: Only retain instances the best agents cannot solve, enforcing robust evaluation.
• Transparent Logging: Orchestrate and checkpoint all tool calls, stepwise trajectories, and cross-modal transitions.
• Process Granularity: Use subgoal-driven decomposition for both evaluation and error diagnosis.

For automation and tooling:

• OmniSeeker (BrowseComp-V³): Modular harness integrating multimodal planning and tool management (Zhang et al., 13 Feb 2026).
• Reflective Frameworks (MM-BrowseComp): Maintain unified, iterative thought–action–observation cycles; synchronize tool outputs with reasoning path checklists (Li et al., 14 Aug 2025).
• Scalable Harnesses: Headless browser environments (Playwright, Selenium), robust HTML/vision parsing, and parallel evaluation pipelines (Wei et al., 16 Apr 2025).

Harness Feature	Implementation/Principle
Evidence traceability	Per-step query/result logging, index/version lock
Public searchability	All documents must exist in public search index
Reproducible trajectories	Each {TextSearch, WebVisit, ...} logged per query
Process evaluation	Expert-validated subgoals, fine-grained progress
Cross-modal integration	Coherent image/text retrieval, region-level tools
Adversarial curation	Filter tasks using strongest agent available

6. Impact and Future Research Directions

The BrowseComp harness-engineering paradigm standardizes process-oriented, multimodal agent evaluation and exposes capability gaps that are not accessible by final-answer metrics alone. Results demonstrate:

• There remains a substantial gap between current state-of-the-art MLLMs and human process proficiency, especially in visual reasoning.
• Harnesses that are process-aware and checklist-aligned enable systematic error diagnosis, facilitate reinforcement learning by dense rewards, and decompose development effort into modular, auditable, and reproducible components.
• The continual optimization and automatic search of harness configurations (as in HARBOR and Meta-Harness) are emerging as scalable methodologies, outpacing manual ablation for complex, flag-rich configurations (Sengupta et al., 22 Apr 2026, Lee et al., 30 Mar 2026).
• The increasing complexity of harness layers (context managers, process recorders, evidence verifiers) renders them critical contributors to system performance, and future research is likely to focus on automated harness evolution and hybrid learned–scripted harnesses.

BrowseComp and its derivatives anchor the scientific study of harness engineering as a first-class challenge: reproducible, process-evaluable, agent-centric, and diagnostic for agent-environment-system interactions (Zhang et al., 13 Feb 2026, Li et al., 14 Aug 2025, Wei et al., 16 Apr 2025, Chen et al., 8 Aug 2025).