2025 AI Agent Index

• 2025 AI Agent Index is a comprehensive framework that defines and benchmarks agentic AI systems based on autonomy, goal decomposition, and multi-domain performance.
• The index employs rigorous empirical methods, including technical annotations across 45 fields and standardized quantitative metrics, to ensure actionable insights.
• Evaluations reveal key trends in agent performance, transparency gaps, and economic impact, guiding improvements in research, regulation, and deployment.

An AI Agent Index is an empirical, multi-dimensional framework for tracking, documenting, and benchmarking the capabilities, transparency, technical scaffolding, economic impacts, and safety practices of deployed, goal-directed AI systems. The 2025 AI Agent Index encompasses statistical surveys of agentic AI deployments, detailed technical annotations, and rigorous outcome-based benchmarks spanning professional, scientific, creative, and economic domains. Its construction synthesizes methodologies from expert system registries, agent-oriented automation measurement, and outcome-agnostic agent evaluation—serving both as a public registry and as a core tool for economic, regulatory, and technical analysis (Staufer et al., 19 Feb 2026, Casper et al., 3 Feb 2025, &&&2&&&, AlShikh et al., 11 Nov 2025).

1. Scope, Definition, and Inclusion Criteria

The 2025 AI Agent Index adopts a formal definition of “agentic AI system” based on autonomy, goal decomposition, environmental interaction, and multi-domain generality. Systems must achieve at least:

• Autonomy Level 2 (“plans and executes most tasks with limited user input”)
• The ability to decompose instructions into ≥3 tool calls
• Programmatic access to APIs, files, or browsers
• Coverage of under-specified goals in multiple domains

The index excludes standalone foundation models, narrow single-task bots, internal-use-only systems, and agents released after the fixed census date (e.g., December 31, 2025). For inclusion, impact or market significance and general-purpose deployability are mandatory: at least 10,000 average monthly searches/GitHub stars, developer market cap ≥ $1B, public availability, and non-trivial user deployment (Staufer et al., 19 Feb 2026).

2. Documentation Structure and Methodology

Each indexed agent is annotated across forty-five fields grouped into product overview, company and governance, technical capabilities and architecture, autonomy and control, ecosystem interaction, and safety/evaluation/impact. Annotation relies on public technical literature, developer documentation, regulatory filings, and direct developer correspondence. Transparency is enforced through cross-checking by expert annotators and confirmation workflows with system developers. Quantitative fields—such as model family, MCP support, autonomy level, tool traceability, safety-card existence, incident reporting, and compliance certification—are standardized for direct comparison.

Agents are categorized by primary interaction model:

Paradigm	# of Systems
Chat-based	12
Browser-based	5
Enterprise workflow	13

The annotation process yields both an index for researcher inspection and a statistical snapshot for quantitative analysis (Staufer et al., 19 Feb 2026).

3. Key Metrics and Evaluation Frameworks

The 2025 AI Agent Index incorporates both descriptive and outcome-based benchmarking. Documented metrics include:

• Technical transparency (fraction of fields with released documentation, code, planning traces)
• Safety transparency (existence of published safety evaluations, guardrail disclosures, red-team findings)
• Model backbone concentration (fraction of systems dependent on top-three LLMs)
• Ecosystem protocol support (MCP compliance, agent interop protocols)

For agent efficacy, leading evaluation frameworks are integrated:

• Composite outcome metrics (e.g., AIAI: weighted sum or geometric mean over normalized Goal Completion Rate, Autonomy Index, Tool Dexterity, Multi-Step Task Resilience, Business Impact Efficiency, and others) (AlShikh et al., 11 Nov 2025)
• Automation Rate and Elo scores on sectoral human-work benchmarks (notably, the Remote Labor Index) (Mazeika et al., 30 Oct 2025)
• Sectoral and domain-relevant composite indices, enabling calculation of an economy-weighted Agent Index:

$$\mathrm{Index}_{2025} = \sum_{i=1}^D w_i \times \mathrm{auto\_rate}_i, \quad \sum_i w_i = 1$$

All fields and metrics are normalized to facilitate ranking and cross-domain comparison.

4. Empirical Findings: Landscape and Capability Trends

The 2025 Agent Index catalogs 30–67 state-of-the-art agents, covering a concentrated developer landscape:

• 70% of developers are US-incorporated, 17% China, remainder in other jurisdictions
• Model family reliance is highly centralized: 27/30 build atop GPT-4/4o, Claude Opus, or Gemini-2.5
• 85% feature API or browser tool integrations; 60% support code execution or multi-agent orchestration (Casper et al., 3 Feb 2025, Staufer et al., 19 Feb 2026)

Transparency gaps are marked: 70% publish documentation, but only 19% publish safety policies and <10% disclose red-team/audit results. Chat-based agents show lower transparency deficits than browser agents or workflow builders, with “transparency gap” fractions of 43%, 64%, and 63%, respectively. Less than one-fourth of systems are open-source (Staufer et al., 19 Feb 2026).

Benchmarking results from 2025 indicate that most agents achieve <25% Pass@1 performance on long-horizon, professional-grade work environments (e.g., 24.0% for Gemini 3 Flash on the APEX-Agents benchmark), and <3% automation on remote-work projects (Remote Labor Index). Systematic leadership in specialized settings is demonstrated by physics and chemistry olympiad agent systems (Physics Supernova and ChemLabs), which match or exceed top human thresholds via principled agentic tool integration and structured multimodal reasoning (Vidgen et al., 20 Jan 2026, Qiu et al., 1 Sep 2025, Qiang et al., 20 Nov 2025).

5. Benchmarks, Case Studies, and Innovations

Prominent technical benchmarks and results shaping the index include:

• APEX-Agents: Pass@1 for the best agent is 24.0%, with performance stratified across banking, consulting, and law workflows. Multi-tool orchestration and long-horizon planning are principal bottlenecks (Vidgen et al., 20 Jan 2026).
• Physics Supernova: CodeAgent-based architecture, modular tool integration, and answer review pipeline yield a 23.5 ± 0.8/30.0 score on IPhO, ranking 14/406 and surpassing the gold-medalist median (Qiu et al., 1 Sep 2025).
• ChemLabs on ChemO: Multi-agent hierarchy with Assessment-Equivalent Reformulation (AER) and Structured Visual Enhancement (SVE) achieves 93.6/100 on IChO, outperforming estimated human gold cutoff (Qiang et al., 20 Nov 2025).
• Remote Labor Index: Highest agent automation rate is 2.5%, highlighting the large gap to human freelance capabilities under manual adjudication (Mazeika et al., 30 Oct 2025).
• Cybersecurity AI (CAI): Alias1 architecture with dynamic entropy-driven multi-model orchestration achieves near-perfect flag capture and 98% cost reduction in CTFs, but reveals the obsolescence of static Jeopardy-style CTFs as a differentiator (Mayoral-Vilches et al., 2 Dec 2025).

Aggregate results indicate that, while agentic scaffolding enables competitive or superhuman performance in narrowly-benchmarked, highly-structured domains, open-ended, cross-application, or creative work remains largely outside the fully autonomous performance envelope.

6. Limitations, Gaps, and Future Directions

Gaps in the 2025 Index architecture and agentic evaluation include:

• Deep under-reporting of safety practices and risk-management controls; only 4/30 systems document deployment-level safety, and only 9/30 report any capability benchmarks (Staufer et al., 19 Feb 2026).
• Opaque accountability; diffusion of responsibility between model providers, orchestrators, and deployers.
• Predominance of closed-source, non-transparent systems and limited coverage of non-Western/Anglophone agents.
• Limited inclusion of highly-specialized or internal R&D systems.

Recommended extensions include machine-readable agent cards as regulatory requirements, end-to-end certification linking agentic orchestration to underlying model compliance, standardized agent identification on the web, and systematic adversarial evaluation of agentic autonomy, planning, and tool-use. Weighting of index metrics should be periodically revised to reflect shifting priorities. Robust longitudinal datasets and periodic benchmark refreshes are mandated for empirical tracking (AlShikh et al., 11 Nov 2025, Casper et al., 3 Feb 2025, Staufer et al., 19 Feb 2026).

7. Significance and Outlook

The 2025 AI Agent Index establishes a principled scaffold for technical and economic measurement of agentic AI deployments, grounded in empirical evidence, structured annotation, and sectoral benchmarking. Its multidimensional tabulation and rigorously-defined inclusion criteria enable actionable visibility into a rapidly evolving, high-impact technology layer. The principal challenges ahead lie in closing transparency and accountability gaps, developing scalable evaluation for open-ended tasks, and constructing benchmarks that reward resilience, adaptability, and robust, real-world autonomy. Only with such a foundation can the index become an instrument for responsible governance, research, and deployment in the global agentic AI ecosystem (Staufer et al., 19 Feb 2026, Casper et al., 3 Feb 2025, AlShikh et al., 11 Nov 2025, Mazeika et al., 30 Oct 2025, Vidgen et al., 20 Jan 2026, Qiu et al., 1 Sep 2025).