Measuring Agents in Production (2512.04123v1)

Abstract: AI agents are actively running in production across diverse industries, yet little is publicly known about which technical approaches enable successful real-world deployments. We present the first large-scale systematic study of AI agents in production, surveying 306 practitioners and conducting 20 in-depth case studies via interviews across 26 domains. We investigate why organizations build agents, how they build them, how they evaluate them, and what the top development challenges are. We find that production agents are typically built using simple, controllable approaches: 68% execute at most 10 steps before requiring human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation. Reliability remains the top development challenge, driven by difficulties in ensuring and evaluating agent correctness. Despite these challenges, simple yet effective methods already enable agents to deliver impact across diverse industries. Our study documents the current state of practice and bridges the gap between research and deployment by providing researchers visibility into production challenges while offering practitioners proven patterns from successful deployments.

• The paper quantifies real-world agentic AI deployment by combining large-scale surveys and detailed interviews across diverse industries.
• It shows that 73% of practitioners prioritize productivity gains, with most relying on off-the-shelf models and manual prompt engineering for controllability.
• It highlights persistent reliability challenges and the urgent need for standardized evaluation benchmarks in autonomous, multi-step AI workflows.

Measuring Agents in Production: A Systematic Characterization of Real-World Agentic AI Deployment

Introduction

The "Measuring Agents in Production" study (2512.04123) systematically investigates the deployment, architecture, evaluation, and ongoing challenges of AI agents functioning in real-world settings. AI agents, in this context, are defined as systems combining foundation models with tools, memory, and reasoning capabilities to autonomously execute multi-step tasks. Despite the proliferation of agentic AI across domains—finance, healthcare, scientific research, and others—quantitative and qualitative data regarding actual practices remains sparse. This work fills that gap via a large-scale practitioner survey and in-depth interviews across diverse organizations, providing rigorous empirical insights into the technical strategies, architectures, evaluation paradigms, and reliability constraints typical of deployed agentic systems.

Agent Adoption Motivations and Domains

The study reveals that productivity gains are the paramount driver of AI agent adoption. 73% of practitioners cite automation of routine tasks and reduction of human labor hours as their primary motivations (Figure 1).

Figure 1: Survey responses identify productivity/efficiency gains as the most compelling benefit for deploying agentic AI systems ($N=66$).

While operational stability and risk mitigation trail as lower-priority objectives, the consistent preference for measurable efficiency is hardly surprising. The survey exposes that agents are established in production across at least 26 domains (Figure 2), with highest concentrations in finance, technology, and enterprise operations.

Figure 2: Agentic AI deployment spans a wide range of application domains with significant numbers in finance, technology, and corporate services ($N=69$).

Notably, agents overwhelmingly serve human users—either internal organizational actors (52%) or external customers (40%)—and only a small fraction deploy exclusively for machine or agent-to-agent interaction. The latency tolerance in these settings is pronounced; a majority of production deployments allow response times of minutes, prioritizing task quality over real-time responsiveness. This reflects the pragmatic substitution of agents for lengthy, manual processes, making even multi-minute agent executions competitive versus human baselines.

Engineering Patterns: Models, Prompting, and Architectures

Production agentic systems exhibit a strong bias for controllability and simplicity. 70% leverage off-the-shelf proprietary models (primarily OpenAI GPT and Anthropic Claude variants) without fine-tuning, using manually engineered instructions to elicit desired behavior. Only a minority employ open-source models, typically driven by cost constraints or regulatory requirements.

Case studies further reveal a multi-model pattern: while 41% of systems use a single model, more than half employ two or more, mixing models for optimization (latency, cost) or for modality specialization (e.g., integrating text-to-speech or domain-specific models).

Figure 3: Case-study systems predominantly rely on closed-source models and minimal post-training; use of open-source or fine-tuned weights is infrequent ($N=20$).

Prompt construction remains an overwhelmingly manual process. 79% of systems report either fully manual or human-guided prompt strategies; fully automated or optimizer-driven approaches are rare and mainly observed in experimentation rather than high-impact deployment (Figure 4).

Figure 4: Manual and human-augmented prompt construction dominate production agentic AI, underscoring controllability priorities ($N=53$).

Prompt architectures vary in complexity, with a significant tail exceeding 10,000 tokens—indicative of systems encoding rich context, domain rules, or user-story flows for robust performance.

In deployment, agentic workflows are tightly bounded with limited autonomous steps and model invocations before requiring human intervention. 68% of agents execute at most ten steps autonomously, and nearly half restrict execution to less than five steps.

Figure 5: Task, subtask, and step taxonomy clarifies agentic workflow boundaries and controlled execution sequences.

Architecturally, predefined static workflows are strongly preferred over open-ended planning or unconstrained autonomy, largely as a response to reliability pressures.

Evaluation and Verification in Production

Agent evaluation practices reveal the centrality of human verification. In production, 74% of deployed agents rely primarily on human-in-the-loop (HIL) checks rather than standardized benchmarks or rule-based automated measures. Model-based strategies (LLM-as-a-judge) are frequently adopted as a complementary approach (52%), but exclusively automated modalities are almost never used in isolation.

The absence of standardized baselines and established benchmarks is notable. Practitioners frequently build custom ground-truth datasets with domain experts or resort to online A/B testing and user feedback pipelines for performance assurance. These practices are dictated by contextual requirements: many agentic applications are highly bespoke and not amenable to public datasets or generic correctness criteria.

Reliability, Latency, and Security Challenges

Reliability is an unsolved engineering constraint for agentic AI deployment, cited as the top development challenge by practitioners. The difficulties inherently stem from lack of reliable verification mechanisms, infeasibility of benchmarking in bespoke contexts, and increased non-determinism in multi-step, autonomous LLM-driven workflows. Traditional CI/CD and regression testing paradigms are insufficient due to these idiosyncrasies.

Latency emerges as a challenge in specific real-time or interactive applications but remains a secondary, mostly manageable concern for the majority of systems (Figure 6). Most production workflows are asynchronous or batch-oriented, rendering latency less critical than output quality and correctness.

Figure 6: Latency is reported as suboptimal but very rarely as a deployment blocker for practitioners ($N=27$).

Security and privacy considerations, though important for a subset of deployments (healthcare, finance), are primarily handled via architectural constraints such as read-only workflows, sandboxing, tool abstraction layers, and RBAC, with production agents heavily reliant on internal infrastructure for data handling (Figure 7).

Figure 7: Internal databases and real-time user input comprise the bulk of data sources for production agentic AI systems ($N=29$).

Future expansion is anticipated toward multimodal input support, with additional investment planned for image, video, and structured scientific data integration (Figure 8).

Figure 8: Agentic AIs are expanding toward broader modality support beyond text and code ($N=29$).

Practical and Theoretical Implications

The empirical findings reveal several critical implications:

• Agentic engineering in production is not a triumph of autonomy but a discipline of restriction. Reliability pressures force architects to favor bounded environments, short execution traces, and extensive human oversight.
• Current foundation models, without extensive post-training, suffice for a broad range of non-real-time operational tasks, enabling rapid deployment and measurable productivity gains across domains.
• Evaluation methodologies and benchmarking for agents are emergent and highly contextual. There is urgent need for standardized evaluation pipelines, scalable CI/CD for agents, and agent observability tooling.
• Expanding agent autonomy and multimodal capability is technically feasible, but contingent on advances in verification and reliability engineering. Sample-efficient, robust post-training paradigms, failure-mode identification (cf. [whydomultiagentllmsystemsfail]), and inference-time verification-search architectures will define the next generation.

Conclusion

"Measuring Agents in Production" (2512.04123) rigorously documents the technical realities and practices of agentic AI deployment. Production systems are predominantly engineered for controllability, with simple off-the-shelf models, bounded autonomy, manual prompting, and HIL evaluation defining the state-of-the-art. Reliability is the paramount challenge, with evaluation and verification paradigms lagging behind architectural sophistication. These findings delineate the contours of the emerging discipline of agent engineering, showing both the practical success of current approaches and the theoretical need for new directions in agent reliability, benchmarking, and evaluation—essential for scaling autonomy and impact in future agentic AI systems.