Heterogeneous Agent Discussion Framework

• The HAD framework is a multi-agent system architecture where diverse AI agents with distinct specializations engage in structured discussions to improve prediction accuracy.
• It utilizes weighted synthesis of agent outputs through explicit debate protocols and aggregation methods like majority voting and graph-based pooling.
• Empirical results in domains like financial sentiment analysis and visual geo-localization demonstrate measurable accuracy gains and underscore the importance of conflict modeling.

Heterogeneous Agent Discussion (HAD) Framework designates a family of multi-agent system architectures in which a collection of heterogeneous AI agents—often instantiated as LLMs or multimodal models—carry out parallelized or structured dialogue on a given task, with the aim of synthesizing more accurate or robust predictions than homogeneous or single-agent baselines. Each agent’s heterogeneity derives from differing model weights, expert specializations, or reasoning heuristics. Aggregation occurs via explicit discussion protocols and weighted synthesis. Recent applications span complex domains such as financial sentiment analysis and visual geo-localization, with theoretical underpinnings anchored in cognitive science and argumentation theory (Xing, 2024, Zheng et al., 2 Nov 2025).

1. Theoretical Foundations

The HAD framework draws on the perspective that cognition and emergent intelligence can be modeled as coalitions of diverse resource agents. This paradigm is articulated in Marvin Minsky's "Society of Mind" and "Emotion Machine" theories, where individual “resources” (i.e. simple cognitive modules) are selectively activated according to the demands of a perceptual or reasoning task. In HAD, these “resources” are operationalized as LLM- or LVLM-based agents, each rigorously focused on one aspect of a complex decision problem through domain-specific prompts or architectural constraints (Xing, 2024). This design realizes Minsky’s hypothesis that robust outcomes emerge via aggregation (“discussion”) across disparate, sometimes conflicting, analytic lines.

2. Generalized Framework Architecture

HAD implementations share a high-level structure in which a user query or data instance is broadcast to a set of $K$ specialized agents. Each agent $A_i$ is initialized with a distinct prompt, model, or modality, eliciting an “opinion” $O_i$—a proposed label, prediction, or analytic narrative. These intermediate results are then aggregated via a higher-level agent or consensus protocol, producing a final output $\hat{y}$.

Component Roles

Component Type	Role	Instantiation Example
Specialized Agent	Analyzes a particular error or facet	“Mood” or “Rhetoric” agent in FSA
Aggregator Agent	Synthesizes all agent outputs	LLM meta-prompt or graph-based pooling
Agent Discussion	Structured flow of information or debate	Prompt-based text collation, GNN fusion

Each agent operates independently or through explicit communication channels determined by the framework, with the aggregator leveraging rule-based logic, weighted voting, or learned graph pooling over the agent-generated messages (Xing, 2024, Zheng et al., 2 Nov 2025).

3. Methodologies and Protocols

Two dominant instantiations of the HAD paradigm prevail: prompt-based agent collectives and graph-based multi-agent debates.

3.1. Prompt-based Heterogeneous LLM Agents

In financial sentiment analysis, the workflow is as follows (Xing, 2024):

1. All $K$ agents simultaneously receive the same input message.
2. Each agent $A_i$ is prompted to focus on a designated error type (e.g., irrealis moods, rhetoric, dependency, aspect, reference).
3. Agents return independent opinion texts and a proposed sentiment label.
4. The aggregator receives the input message and all $O_1,\dots,O_K$ and returns a consensus label $\hat{y}$ via a meta-prompt.

The agents operate without inter-agent awareness. Aggregation may be a simple majority, weighted voting, or a learned combination based on historical agent performance:

$$\text{Score}(c) = \sum_{i=1}^K w_i \cdot \mathbb{1}(y_i = c)$$

$\hat{y} = \arg\max_c \text{Score}(c)$

where $A_i$0 are agent weights, by default uniform, but adjustable via ablation (Xing, 2024).

3.2. Graph-based Multi-Agent Debate (GraphGeo)

GraphGeo extends HAD to visual geo-localization, casting each agent as a node in a heterogeneous graph $A_i$1 (Zheng et al., 2 Nov 2025):

• Nodes represent LVLM agents with unique identity embeddings.
• Edges are typed: agreement, conflict, and transfer (from confident to less confident agents), constructed based on spatial prediction proximity and confidence scores.
• Node- and edge-level debate mechanisms support both collaborative (agreement), adversarial (conflict), and instructional (transfer) information passing.
• Topology evolves dynamically as edge weights are pruned, added, or reinforced based on ongoing information flow and prediction accuracy.

GraphGeo’s dual-level debate combines node-level message aggregation and edge-level GRU-based state tracking. After $A_i$2 debate rounds, the learned topology influences final prediction consensus and agent embedding updates.

4. Application Domains

HAD frameworks have demonstrated impact across natural language and vision tasks characterized by multiple confounded error sources or information modalities.

• Financial Sentiment Analysis: Heterogeneous LLM agents specializing in mood, rhetoric, dependency, aspect, and reference errors yield substantial gains over naive and tuned baselines, closing a significant portion of the performance gap between zero-shot and supervised approaches (Xing, 2024).
• Visual Geo-localization: Heterogeneous agents instantiated as LVLMs form a debate network that outperforms state-of-the-art and ablated homogeneous baselines, supporting the conjecture that explicit modeling of agent disagreement (conflict edges) is critical for error correction and overall accuracy (Zheng et al., 2 Nov 2025).

The general approach is extensible to any domain where (a) expertise can be decomposed, and (b) robust aggregation of divergent perspectives improves judgement.

5. Empirical Results and Ablation Insights

Empirical evaluation in both principal studies demonstrates that HAD confers measurable gains relative to single-agent or uniform-agent systems.

• Financial Sentiment Analysis: Absolute accuracy gains of 2–10% and macro-F$A_i$3 improvements up to 14% are reported on finance-specific benchmark datasets, with ablation studies confirming the importance of particular specialist agents (mood, rhetoric, aspect) and revealing that some agents (e.g., dependency, reference) may have neutral or occasionally negative impact when included indiscriminately (Xing, 2024).
• GraphGeo: Street-level localization gains range from 0.64% to 2.47% over visual geo-localization baselines, with the removal of conflict, agreement, or transfer edges causing notable accuracy degradation (up to 6.8% loss without conflict modeling), and elimination of the multi-round debate mechanism resulting in a 10.9% performance drop (Zheng et al., 2 Nov 2025).

These results substantiate that heterogeneity and structured, typed agent interactions—especially those reflecting genuine cognitive conflict—are essential for capturing elusive dependencies in complex tasks.

6. Design Considerations and Generalization

The effective deployment of HAD requires strategic agent specialization, informed by empirical error analysis and domain expertise. Specialist prompt design, agent weight adjustment, type-specific communication protocols, and adaptive aggregation methods are all critical factors.

A plausible implication is that the HAD framework offers a template for developing robust, zero-shot systems in other domains where labeled data is scarce but error taxonomies are available. The unifying tenet is that multiplicity and contestation, when properly orchestrated, drive emergent accuracy and robustness far beyond what can be achieved by monolithic models alone (Xing, 2024, Zheng et al., 2 Nov 2025).