Redpanda Agentic Data Plane
- Redpanda ADP is an architecture for governing autonomous agents using out-of-band metadata channels that enforce security-critical policies externally.
- It separates data and control planes by utilizing deterministic, agent-inaccessible channels to propagate access policies, behavioral constraints, and audit trails.
- Its multi-layer design integrates access control, isolated compute sandboxes, and tamper-resistant audit logs to prevent agent manipulation and ensure secure operation.
Searching arXiv for the cited Redpanda ADP paper and a few related works mentioned in the provided source so the article can ground its claims in papers. Tool call: arxiv_search {"query":"arXiv (Akidau et al., 27 May 2026) The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane", "max_results": 5} Redpanda Agentic Data Plane (ADP) is an architecture for governing autonomous agents through out-of-band metadata channels: infrastructure pathways that carry security context, policy signals, and audit information deterministically and invisibly to the agent, rather than embedding those controls in the agent’s prompt, memory, or tool parameters. The architecture is presented in "The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane" as a response to a specific systems-security claim: safe autonomous agents cannot be trusted to carry or enforce their own security metadata, including access policies, data classifications, behavioral constraints, and audit trails (Akidau et al., 27 May 2026).
1. Problem formulation and motivating claim
The motivating problem is defined by a dual characterization of contemporary AI agents. They are described as less predictable than humans because they hallucinate, misinterpret instructions, and are vulnerable to prompt injection and jailbreaks, and simultaneously more capable than humans because they can act at machine speed through direct API and tool access. In the paper’s formulation, that combination makes it unsafe to rely on agents to faithfully interpret or propagate security-critical metadata. The architecture therefore rejects the assumption that metadata embedded in the agent-visible read/write path can be trusted for enforcement (Akidau et al., 27 May 2026).
The paper identifies four classes of in-band metadata that should not be trusted when carried through the agent’s own execution path: access policies, data classifications, behavioral constraints, and audit trails. The reasons are explicit. Agents may ignore or misunderstand prompt-level policies; they may actively bypass policy if authorization is expressed in agent-constructed tool parameters; and they may tamper with self-reported logs by omitting, altering, or fabricating their own transcript. The architectural consequence is a sharp trust-boundary decision: the agent, its runtime, and its tools are treated as untrusted, while gateways, brokers, identity providers, and transcript stores form the trusted platform boundary.
A recurrent misconception addressed by this framing is that prompt engineering or tool-schema design is sufficient for governance. The paper argues that such mechanisms remain in-band and therefore depend on agent compliance. ADP is defined precisely to remove that dependency.
2. Out-of-band metadata as an architectural abstraction
An out-of-band metadata channel is defined by three properties. First, it is agent-inaccessible: the agent cannot read or write the channel, even though infrastructure components it interacts with do enforce the associated policy. Second, it is deterministic: enforcement is performed by configuration and runtime policy, not by asking the agent to interpret an instruction. Third, it is interoperable: metadata must propagate across heterogeneous systems including APIs, databases, message brokers, object stores, MCP servers, and SaaS services, rather than assuming a monolithic stack (Akidau et al., 27 May 2026).
The core architectural move is summarized by the proposition that security metadata travels beside the data, not inside the agent’s prompt or tool payloads. The paper gives a simple example of determinism: a limit such as “10 trades per hour” should be enforced by a gateway counter rather than by an instruction telling the model to obey a rate limit. That formulation relocates governance from the agent’s internal reasoning to surrounding infrastructure.
This abstraction matters because it treats metadata propagation itself as part of the security mechanism. The paper’s argument is not merely that policy should exist outside the model, but that the entire lifecycle of metadata—authorization context, routing decisions, approval state, and trace provenance—must remain external to the agent-visible data path. A plausible implication is that ADP is less a prompt-governance technique than a control-plane/data-plane separation for agent systems.
3. Runtime and control-plane structure
The paper describes ADP as a runtime and control plane organized into four infrastructure layers (Akidau et al., 27 May 2026).
| Layer | Stated components and role |
|---|---|
| Access control layer | AI Gateway, MCP Gateway, enterprise identity-provider integration, and input/output guardrails |
| Data connectivity layer | Connections to managed and external MCP servers, REST APIs, databases, object stores, and streaming platforms |
| Agentic compute layer | Sandboxes with isolated network access, limited resources, and identity-scoped access |
| Accountability layer | Structured transcripts recorded automatically by infrastructure and stored as tamper-resistant, access-controlled records |
The access control layer contains an AI Gateway for LLM routing, token budgeting, failover, and model governance, together with an MCP Gateway for tool-level policy enforcement. It also integrates enterprise identity providers and input/output guardrails such as content filtering, prompt injection detection, and response validation. The salient point is that identity and authorization context are resolved by platform infrastructure rather than by the agent, and the agent never sees provider keys or policy internals.
The data connectivity layer is responsible for carrying out-of-band metadata across heterogeneous systems. The paper states that adapters propagate metadata across managed and external MCP servers, REST APIs, databases, object stores, and streaming platforms. The agent receives a uniform tool interface while policy is enforced silently by the platform.
The agentic compute layer provides sandboxes for persistent state and code execution. Within those sandboxes, network access is isolated, resources are limited, and access is identity-scoped. MCP tools are projected into the sandbox as executable actions, but the sandbox remains inside ADP’s governance boundary. The paper states that this supports both ADP-managed and external agents.
The accountability layer records every interaction into structured transcripts collected by infrastructure rather than by the agent. Those transcripts are described as tamper-resistant and access-controlled, extending out-of-band governance from authorization and execution into auditability.
4. Lifecycle governance: ingress, execution, and egress
ADP’s governance model is explicitly lifecycle-based. At ingress, when an agent requests data, identity context flows out-of-band from authentication to the data source; the data source applies row-level or resource-level filtering; the agent receives only authorized data; and the agent never sees either the filtering logic or the hidden data. The security objective is to make “asking for too much” non-fatal because unauthorized data is never exposed in the first place (Akidau et al., 27 May 2026).
At execution, when an agent proposes an action, policy constraints are carried out-of-band to the execution layer. Infrastructure then enforces thresholds, approval rules, and rate limits, and the agent cannot override the routing decision. This preserves a distinction between recommendation and execution: the agent may produce an action proposal, but the platform determines whether and how that proposal is allowed to proceed.
At egress, every interaction is captured independently by infrastructure. The paper enumerates tool calls, model calls, message exchanges, approval steps, and execution confirmations as audit events. Because collection is out-of-band, the agent cannot omit or rewrite its own history. In the paper’s trust-boundary summary, the agent-visible path contains only data and tool outputs, while the agent-invisible path carries authorization, filtering, approval, and audit.
The security analysis embedded in the paper is structured as a contrast between failure modes of in-band metadata and the intended properties of out-of-band enforcement. Prompt-based policies can be ignored, tool parameters can be forged, classification labels can be misread or dropped, and self-reported logs can be fabricated. By contrast, out-of-band enforcement does not rely on agent compliance, prevents the agent from seeing or modifying the policy channel, and preserves audit independence from self-reporting.
5. Demonstration in a multi-agent portfolio rebalancing system
The paper demonstrates ADP with a multi-agent portfolio rebalancing system in which autonomous agents monitor markets, make trade decisions, and execute orders across isolated client accounts. The system diagram includes an AI Gateway for LLM calls, an MCP Gateway for tool calls, a Signal Agent, a Decision Agent, an Approval App, an Execution Agent, and MCP tools including Perplexity MCP, MongoDB MCP, Alpaca MCP (read), and Alpaca Orders MCP (write). The figure caption emphasizes three properties: agents communicate only through asynchronous messaging channels, out-of-band metadata propagates transparently, and the autonomy threshold is an infrastructure routing decision that the agents cannot observe or affect (Akidau et al., 27 May 2026).
The workflow is staged. For each client, the Signal Agent continuously monitors market developments, queries external research APIs, and publishes structured discoveries to per-client message channels. The Decision Agent consumes those signals, inspects portfolio positions, buying power, and price history, decides whether to buy, sell, or hold, and outputs a recommendation with an estimated amount and rationale. The platform then applies policy-based routing: if the estimated order value is below a configured threshold, the order routes to autonomous execution; if above threshold, it routes to human approval. The Approval App presents above-threshold orders to a human, who approves or denies them; approved orders go to execution and denied orders are discarded with audit records. The Execution Agent consumes from the execution channels, submits orders to the brokerage API, and records execution results.
The paper gives concrete examples of how out-of-band metadata operates in this demonstration. All access to research APIs, brokerage accounts, and price feeds passes through MCP servers governed by the MCP Gateway. Each agent authenticates with an infrastructure-issued credential; the gateway maps that credential to a client scope; and the scope is injected by the gateway rather than passed by the agent. The example given is that when the Decision Agent calls get-positions, it does not construct a client_id parameter. The gateway inserts the correct one. Similarly, all LLM requests pass through the AI Gateway, where agents authenticate with OIDC credentials, identity is mapped to policy, and the gateway forwards to the model provider. The gateway thereby supports token budgets per agent, rate limits, prompt injection detection, response validation, and dynamic routing across model providers, while the agent never receives the provider API key.
Messaging is also scope-constrained. Each agent has its own messaging credential, and policies restrict which channels it may use. The paper’s examples are explicit: the Signal Agent can write to signal channels but not order channels, and the Execution Agent can consume execution channels but cannot publish signals or approve orders. Client-scoped topics shown in the figure include signals.{client_id}, orders.execute.{client_id}, and orders.pending_approval.{client_id}. The threshold policy expression shown in the figure is value < $1k?, with auto execution if true and review if false. The paper stresses that this threshold is not in the prompt; it is infrastructure logic after the agent has already produced its recommendation. The system’s trace context propagates through W3C Trace Context headers injected at each hop, supporting a tamper-proof trace of market research requests, LLM calls, tool calls, message passes, human approval, and brokerage confirmation.
6. Trust model, comparisons, limitations, and terminological disambiguation
The ADP trust model distinguishes trusted from untrusted components with unusual explicitness. Trusted components are the gateways, message brokers, identity provider, transcript store, and policy configuration. Untrusted components are the LLM, the agent runtime, tool invocations originating from the agent, and any external context the agent ingests. The paper also narrows the threat model: it is concerned with defending against the agent, not against a malicious administrator or a bad policy author. In multi-agent pipelines, the paper identifies a classic confused deputy problem, arguing that identity context can be forged, lost, or misrepresented at inter-agent boundaries, and presenting out-of-band channels as the remedy (Akidau et al., 27 May 2026).
Within related work, the paper positions ADP near several neighboring lines of research. ToolEmu and AgentDojo are cited as showing that agents can take unsafe actions or be hijacked by prompt injection; Cartagena and Teixeira are cited for the claim that alignment in text behavior does not guarantee safe tool use; authenticated workflows, mandatory access control, least-privilege tool-call systems, and service mesh or sidecar proxies are described as adjacent ideas that externalize policy or enforce integrity at boundaries; MCP security work is relevant because MCP standardizes tool exposure without itself solving governance; and AgentTrace, VET, and verifiable execution traces are presented as related on observability and verifiability. The paper’s own claim is that ADP unifies access control, action constraints, and audit under one architectural abstraction.
The paper does not provide a formal benchmark or quantitative evaluation of latency or cost overhead. It is characterized as an architecture-plus-live-demo paper, and its stated limitation is essentially future work: quantifying latency and cost overhead of gateway-mediated enforcement in production workloads. The demonstration is therefore a feasibility argument rather than a large empirical security evaluation.
A separate source of confusion is acronym overlap. ADP is also the acronym for Agent Data Protocol, a different system introduced as a lightweight representation language or “interlingua” for heterogeneous agent datasets and downstream training pipelines. That ADP unifies raw datasets into a common trajectory schema and converts them into training-ready formats for multiple agent frameworks, reducing conversion effort from to ; however, it is explicitly not a Redpanda-style runtime data plane or infrastructure governance layer (Song et al., 28 Oct 2025). For terminological precision, Redpanda Agentic Data Plane concerns runtime enforcement of security-critical metadata, whereas Agent Data Protocol concerns standardized representation of agent training data.
Taken together, these distinctions clarify the specific contribution claimed for Redpanda Agentic Data Plane: governance is achieved not by instructing an agent to behave safely, but by ensuring that authorization, filtering, approval, and audit are enforced by infrastructure channels the agent can neither see nor bypass.
