Infogent: Modular Web Aggregation Framework

• Infogent is a modular, agent-based framework for web information aggregation that addresses complex, multi-faceted queries.
• It splits tasks among Navigator, Extractor, and Aggregator modules which coordinate through an iterative, feedback-driven loop.
• Empirical evaluations show notable accuracy improvements over SOTA benchmarks, although performance depends on the underlying LLM capabilities.

Infogent denotes a modular, agent-based framework for web information aggregation that is designed to solve complex information-seeking tasks beyond the traditional linear web navigation paradigm. Unlike conventional web agents that optimize for task completion through a sequence of actions leading to a single end state, Infogent targets open-ended scenarios where the agent must navigate, extract, and aggregate information from diverse web sources to address multi-faceted user queries. The architecture is flexible across access modalities, supporting both direct API-driven (text-based) and interactive visual (browser-based) interactions. Infogent demonstrates measurable improvements over previous state-of-the-art (SOTA) web search and task completion agents in benchmark settings (Reddy et al., 2024).

1. System Architecture

The Infogent system is structured around three collaborating black-box modules—Navigator, Extractor, and Aggregator—coordinated in an iterative feedback-driven loop. This modular decomposition underpins flexible system composition and principled delegation of sub-tasks.

• Navigator ($\mathcal{NG}$): Receives the user’s task $\mathcal{T}$, current webpage context $\mathcal{W}_t$ (either as URL+snippet or screenshot+DOM), feedback message $\mathcal{F}$ from Aggregator, and partial action history. It outputs an action $a_t$ from the action space $\mathcal{A}$ (including SEARCH, CLICK, AGGREGATE, GO BACK, TERMINATE). The Navigator explores the web, initiates state transitions, and decides when to hand control to the Extractor or Aggregator.
• Extractor ($\mathcal{ET}$): Given the current webpage, the task, and feedback, outputs a set of relevant text passages $P = \{p_1, ..., p_k\}$, typically by prompting an LLM to select up to $k$ pertinent paragraphs.
• Aggregator ($\mathcal{AG}$): Given newly extracted passages and the current “stack” $\mathcal{T}$0 of aggregated items, as well as the original task, decides (a) which passages to add or replace in $\mathcal{T}$1 (avoiding redundancy and promoting topical diversity), (b) generates updated feedback $\mathcal{T}$2 for the Navigator, and (c) internally determines whether information sufficiency has been met (TERMINATE).

Coordination pseudocode (see Algorithm 1 in (Reddy et al., 2024)):

$\mathcal{T}$7 Each module is implemented via purpose-built LLM or (for the visual modality) LMM prompt chains, without gradient-based training.

2. Formal Task Definitions

While no explicit RL objectives or loss functions are introduced, Infogent formalizes three module mappings:

• Navigator policy:

$\mathcal{T}$3

• Extractor mapping:

$\mathcal{T}$4

• Aggregator update:

$\mathcal{T}$5

There is no inherent scoring function; aggregation quality is evaluated at the macro level via task-level metrics.

3. Information Access Modalities

Infogent targets two modes of web interaction, supporting both textual and visual web agent paradigms:

Direct API-Driven Access:

• The Navigator (ReACT-based) issues SEARCH and AGGREGATE API calls—SEARCH returns top-5 (URL, snippet) pairs via the Google Search API; AGGREGATE invokes Extractor and Aggregator on the selected URL.
• The Extractor processes HTML content to extract relevant paragraphs.
• The Aggregator operates as previously described.

Interactive Visual Access:

• The Navigator (multimodal agent, SeeAct + Playwright) consumes sequential browser screenshots and DOM element candidates, issuing actions such as CLICK, TYPE, PRESS ENTER, SELECT, AGGREGATE, or TERMINATE.
• The Extractor uses a vision-capable LLM (e.g., GPT-4V), acquiring multiple screenshots and extracting text.
• The Aggregator’s mechanism is unchanged.

The stack-based aggregation and feedback mechanism are consistent across both paradigms.

Access Setting Comparison Table

Setting	Navigator Modality	Extraction Method
API-Driven Access	LLM + Search APIs	HTML parsing, LLM prompt
Interactive Visual Access	LMM + Screenshots	Vision-LMM prompt

4. Training Paradigm and Module Implementation

Infogent is instantiated entirely via off-the-shelf OpenAI APIs (GPT-4-Turbo, GPT-4o; ablations include GPT-4o-mini, GPT-3.5 for extraction), without any supervised module fine-tuning or novel dataset annotation. All agent logic is encoded in engineered prompt templates (cf. Tables A.3–A.5 in (Reddy et al., 2024)). There is no learned policy or gradient update at any stage, making the system trivially deployable on new information-seeking tasks but highly dependent on the capabilities of the underlying LLM/LMM backbone, especially for the Navigator.

5. Empirical Evaluation

Infogent is systematically benchmarked under both access settings against multiple SOTA and strong baseline agents.

• Benchmarks:
  • FRAMES (531 items): reasoning over web tables, temporal relationships, post-processing, and constraint satisfaction.
  • FanOutQA (310 multi-hop Wikipedia questions).
  • AssistantBench (214 browser automation tasks; both dev and test splits).
  • Human-curated FanOutQA subset (31 queries failing closed-book models).
• Metrics:
  • FRAMES: Overall LLM-judge accuracy.
  • FanOutQA: Exact-match accuracy; ROUGE-1/2/L.
  • AssistantBench: Final-answer accuracy.
• Representative Results:
  • FRAMES (Direct API):
  • Closed-Book: 23.5%
  • MindSearch: 46.3%
  • Infogent: 53.3% (+7.0 over MindSearch)
  • FanOutQA (dev, API):
  • Closed-Book: 46.6%
  • MindSearch: 47.3%
  • Infogent: 51.1% (+3.8)
  • AssistantBench (test, Visual):
  • RALM-Inst: 11.7%
  • SPA: 11.0%
  • Infogent (GPT-4o): 15.3% (+4.3 over SPA)

Ablations indicate Navigator is the most bottlenecked module: downgrading to GPT-4o-mini for navigation drops accuracy to 0% on AssistantBench, while less powerful Extractor/Aggregator models incur only minor reductions (~2–3%).

6. Analysis, Insights, and Limitations

Architectural Insights:

• Modularity enables specialization and composability: delegating extraction and aggregation relieves the Navigator, increasing coverage and reducing error rate (Reddy et al., 2024).
• The natural language feedback loop, where the Aggregator returns contextually adaptive guidance $\mathcal{T}$6, repeatedly diagnoses coverage gaps and refocuses the Navigator, yielding higher-quality aggregation.

Limitations:

• Performance is heavily gated by the LLM backbone, especially for navigation.
• The framework lacks explicit objective metrics for “information sufficiency”—termination is governed solely by Aggregator-issued feedback.
• No gradient-based learning, adaptation to domain shifts, or listener-specific priors; all prompt-driven.
• Fails on rich interactive UIs (geo-navigation), pop-ups, and elements that require persistent local state or real-time interactions.

Future Directions:

• Introducing learned components, such as retrieval modules or RL policies, to reduce dependency on generalist LLMs.
• Development of automatic and nuanced measures of information coverage, diversity, and aggregation sufficiency.
• Extension to GUI-based information aggregation beyond the web (e.g., Slack, mobile apps), leveraging the visual interaction paradigm.
• Construction of new benchmarks for subjective and open-ended aggregation tasks beyond multi-hop QA.

7. Connections and Extensions

While Infogent is focused on web-scale information aggregation, its architectural principles—modular decomposition, feedback-driven control, and aggregation stack memory—resonate with broader developments in agent-based information systems. The approach is compatible with entropy-based information selection paradigms, as in the informativity-maximizing label selection exposed by (Bracha et al., 2018). A plausible implication is that integrating entropy-based selection at the Aggregator level could yield more principled aggregation in domains where coverage and diversity are quantifiable. Nonetheless, Infogent itself does not implement Shannon-entropy reduction or listener-model adaptation as formalized in (Bracha et al., 2018). This suggests a fertile intersection for future hybrid agents incorporating both modular search/aggregation and entropy-theoretic label or passage selection mechanisms.