FedAgentBench: Agent-Driven FL

Updated 5 October 2025

FedAgentBench is a modular framework that automates federated learning by mapping task specifications to a multi-agent pipeline handling client selection, data preprocessing, label harmonization, and algorithm selection.
It simulates realistic healthcare imaging environments with 201 curated datasets across six clinical workflows, addressing complex multi-step planning and operational challenges.
The benchmark evaluates 24 LLM agents and 40 FL algorithms under both fine-grained and high-level guidance to assess autonomous decision-making and adaptation.

FedAgentBench evaluates and advances agent-driven automation in real-world federated learning (FL), with particular focus on complex healthcare imaging workflows. The framework assigns specialized LLM agents to critical FL pipeline stages—client selection, server–client coordination, data preprocessing, label harmonization, and algorithm selection—moving beyond conventional FL setups that demand intensive human orchestration. FedAgentBench is designed to rigorously assess agentic capacity for autonomous decision-making and adaptation across operationally challenging, heterogeneous, and privacy-preserving medical environments.

1. Modular Agent-Driven FL Framework

FedAgentBench introduces a modular architecture for real-world FL orchestration. User-provided task specifications (𝒯) are mapped to a workspace (𝒲), containing a comprehensive data card system, FL algorithm registry, and code templates. A multi-agent system 𝒜—comprising server agents (S₁ … S₄) and client agents (C₁ … Cₙ)—collaborates to automate FL pipeline phases, iteratively refining outputs. Formally, the process unfolds as

$\{D_i, R_i\} = \mathcal{A}(D_{i-1}, R_{i-1}, \mathcal{T} \mid \mathcal{W}), \quad D_0 = R_0 = \varnothing$

where $D_i$ is the decision or code at iteration $i$ , and $R_i$ is the execution feedback. The agents interact to select clients matching dataset constraints, preprocess and normalize medical image data, harmonize heterogeneous label taxonomies, and select or adapt FL algorithms according to dynamic user and client requirements.

2. Benchmark Structure and Evaluation Protocols

FedAgentBench is a benchmark for evaluating LLM agents’ ability to automate all stages of federated medical image analysis with minimal human involvement. The benchmark simulates realistic healthcare environments, providing diverse task structures that stress the end-to-end orchestration problem, including:

Coordination between server and client agents.
Multi-step planning and execution.
Robust adaptation to noisy, uncurated datasets. Two forms of guidance—explicit stepwise instructions versus high-level objective prompts—are tested, allowing systematic analysis of agent reasoning and planning. Key evaluation metrics comprise stagewise success rate, token and time efficiency, and comparative analysis under both guidance modes.

3. Integrated Federated Learning Algorithms

FedAgentBench incorporates 40 federated learning algorithms spanning the breadth of FL paradigms:

Classical FL aggregation: FedAvg, FedProx, Scaffold for baseline update aggregation and drift mitigation.
Personalized FL: Per-FedAvg, pFedMe, FedRep to account for client-unique data distributions.
Regularization–based FL: Ditto for balancing global and personalized updates via regularization.
Knowledge Distillation: FedDF employs logits transfer for model-agnostic aggregation.
Domain Generalization: FedSR, FedDG, FedIRM for extracting invariant features in non-IID settings.
Optimization Variants: FedNova for improved convergence and stability.

Each algorithm is engineered as a plug-and-play module triggered by the agent system, adaptively deployed based on user instructions and detected data heterogeneity across federated sites.

4. Dataset Curation and Task Complexity

The benchmark simulates six real-world healthcare imaging environments (Dermatoscopy, Ultrasound, Fundus, Histopathology, MRI, X-Ray), assembling 201 carefully curated, public datasets as client-specific local data. Clients are instantiated with distinct and often noisy datasets, featuring:

Structured perturbations (resolution, file format, intensity modification).
Noisy injections (non-image files, duplicates, annotation errors).
Variable label schemas, emulating non-standard taxonomy. Tasks include disease classification, anatomical segmentation, object detection, and regression, designed to stress semantic and operational challenges endemic to multi-institutional clinical workflows.

5. Agentic Performance: LLM Agents

FedAgentBench evaluates 24 LLM agents (14 open-source, 10 proprietary) across multiple model scales. Performance analysis is conducted for each pipeline phase, using both fine-grained and goal-oriented guidance. Principal findings:

Proprietary systems such as GPT-4.1 and DeepSeek V3 achieve near-perfect task automation on simple subtasks (client selection, model training) but falter on complex, semantic, multi-step operations (notably label harmonization).
Open-source LLMs (DeepSeek-V3, Qwen QwQ 32B, LLaMA-4 variants) demonstrate competitive but less robust performance, particularly sensitive to prompt explicitness.
Fine-grained guidance generally boosts success rates for weaker and medium-scale models, especially for tasks demanding multi-step planning and semantic integration.

6. Results, Limitations, and Implications

FedAgentBench demonstrates that agent-driven FL pipelines can automate core stages of medical federated learning, achieving privacy preservation and reducing human workload in practical deployments. However, tasks involving semantic integration and cross-client label harmonization remain challenging for even the strongest LLM agents. The modular plug-and-play algorithm registry and detailed evaluation enable nuanced profiling of agent strengths and weaknesses. These results underscore the promise and the current limits of LLM-driven automation in federated healthcare AI, motivating further development in agent planning, domain-specific reasoning, and regulatory compliance mechanisms.

A plausible implication is that future FL systems will require hybrid approaches integrating agentic reasoning with domain-specific heuristics or explicit knowledge modules to tackle the semantic and operational intricacies of real-world healthcare federated workflows.

FedAgentBench extends agentic evaluation principles established in ATR-Bench (Ashraf et al., 22 May 2025) and connects to best practices for agentic benchmark validation outlined in the Agentic Benchmark Checklist (ABC) (Zhu et al., 3 Jul 2025). The use of a diverse set of FL algorithms and careful simulation of operational challenges draws from the federation realism principles of FLBench (Liang et al., 2020), while benchmarking LLM agent orchestration in noisy, multi-modal settings aligns with broader trends in FDABench (Wang et al., 2 Sep 2025). This situates FedAgentBench as a central resource for research on autonomous, privacy–preserving, and robust agent-driven federated learning for complex medical tasks.