The LLMbda Calculus: AI Agents, Conversations, and Information Flow

Abstract: A conversation with a LLM is a sequence of prompts and responses, with each response generated from the preceding conversation. AI agents build such conversations automatically: given an initial human prompt, a planner loop interleaves LLM calls with tool invocations and code execution. This tight coupling creates a new and poorly understood attack surface. A malicious prompt injected into a conversation can compromise later reasoning, trigger dangerous tool calls, or distort final outputs. Despite the centrality of such systems, we currently lack a principled semantic foundation for reasoning about their behaviour and safety. We address this gap by introducing an untyped call-by-value lambda calculus enriched with dynamic information-flow control and a small number of primitives for constructing prompt-response conversations. Our language includes a primitive that invokes an LLM: it serializes a value, sends it to the model as a prompt, and parses the response as a new term. This calculus faithfully represents planner loops and their vulnerabilities, including the mechanisms by which prompt injection alters subsequent computation. The semantics explicitly captures conversations, and so supports reasoning about defenses such as quarantined sub-conversations, isolation of generated code, and information-flow restrictions on what may influence an LLM call. A termination-insensitive noninterference theorem establishes integrity and confidentiality guarantees, demonstrating that a formal calculus can provide rigorous foundations for safe agentic programming.

• The paper introduces LLMbda, an untyped call-by-value lambda calculus with generation, conversation branching, clearing, labels, and information-flow tracking for modeling agentic LLM programs.
• The paper proves termination-insensitive noninterference for restricted sublanguages—without label tests, with a single non-bottom security level, or using assertions and strong tests—while demonstrating why unrestricted tests can leak information.
• The paper’s interpreter and case studies show how CaMeL-style quarantining and trusted-data assertions can block prompt injection and prevent untrusted content from changing protected tool-call parameters, although deterministic models and limited external-tool coverage remain important limitations.

Motivation and problem

Agentic systems built around LLMs interleave LLM calls with tool invocations and code execution inside a single prompt-response conversation. This coupling creates an attack surface that is now well documented empirically—prompt injection attacks, known since late 2022, allow untrusted content to hijack later reasoning or trigger dangerous tool calls—but for which no principled semantic foundation existed. The closest prior work, CaMeL, describes a dual-LLM architecture with dynamic information-flow tracking and demonstrates practical effectiveness on the AgentDojo benchmark, yet explicitly provides no formal reasoning or proof, flagging formalization as a "crucial direction" for future work. FIDES offers formal noninterference statements but only for fixed planner program schemas, not for general higher-order programs with dynamically generated code.

The paper under review addresses this gap by introducing LLMbda, an untyped call-by-value lambda calculus extended with dynamic information-flow control and three primitives for manipulating prompt-response conversations: a generation operator @e (serialize a value, send it as a prompt, parse the response), a fork construct for temporary conversation branching, and a clear construct for quarantining history. The stated aim is a kernel expressive enough to encode both agent planners and the code they generate, together with sound security guarantees.

The calculus

The core language is Plotkin-style call-by-value lambda calculus augmented with @e, fork e, and clear. The semantics is deliberately abstract: conversations are sequences of prompt-response pairs over token sequences, serialize/parse functions appear only in the metatheory, and the only assumption on the underlying model $\mathbb{M}$ is determinism—the response is a pure function of the conversation history. This determinism assumption is what makes "what-if" experiments (varying untrusted inputs while holding everything else fixed) well-defined, and it matches earlier formal treatments such as FIDES; it is nonetheless a real idealization, since deployed LLMs are stochastic.

JSON-style data types (booleans, numbers, strings, records, arrays) are treated as Church encodings within the core calculus, so the kernel remains minimal while the interpreter implements them directly.

Information flow follows Denning's lattice model, instantiated in examples as the powerset lattice over $\{U \text{(untrusted)}, S \text{(secret)}\}$. Two primitives are added: labelled expressions $l:e$ (returning $l \sqcup l' : v$) and label tests $l?e$ returning whether the operand's label may flow to $l$. A distinctive design decision is that the result of a label test carries the pc label, not the labels being inspected—the rationale being that policy decisions must themselves be made from trusted data. As discussed below, this choice is exactly where noninterference breaks down in general.

Big-step semantics and conversation state

Judgements have the form $pc \vdash C, e \Downarrow C', V$, carrying a program-counter label following Denning and Denning, and treating the labelled conversation history $C$ as mutable state read and written by the Prompt and Clear rules. Two aspects deserve emphasis:

• No-high-upgrade discipline: because the conversation is state, illicit flows can occur along paths that do not modify it (a secret conditional deciding whether to write to a low-labelled conversation). The Prompt and Clear rules therefore require $pc$ to flow to the current history label, adopting Zdancewic's no-high-upgrade constraint as generalized by Austin and Flanagan. The authors acknowledge this is conservative: some executions are blocked unnecessarily, and blocking itself leaks through termination—permitted by the termination-insensitive guarantee.
• Erasure: an erasure function removes subterms whose labels do not flow below a threshold before serialization, justified by a central property that values differing only above threshold erase identically.

A small-step semantics with explicit forked-state tracking ($\mathsf{forked}\ C\ e$) and a label-equivalence congruence is also developed, with a correspondence theorem showing it coincides with the big-step relation.

Noninterference: results and their limits

The main theorem is termination-insensitive noninterference (TINI): if two expressions agree up to level $\{U \text{(untrusted)}, S \text{(secret)}\}$0, then their evaluations yield indistinguishable values and histories at level $\{U \text{(untrusted)}, S \text{(secret)}\}$1. Crucially, TINI does not hold for the full language. The authors give a concrete counterexample on a three-level lattice: two $\{U \text{(untrusted)}, S \text{(secret)}\}$2-indistinguishable terms $\{U \text{(untrusted)}, S \text{(secret)}\}$3 and $\{U \text{(untrusted)}, S \text{(secret)}\}$4 become distinguishable after a label test, because tests return their result at the pc level by design. This is a genuine tension: returning test outcomes at high labels would make them useless for policy enforcement.

The theorem therefore identifies three restricted sublanguages satisfying TINI:

1. expressions built without label tests;
2. expressions using only one non-bottom label (which includes any two-point lattice—noting that most prior formal noninterference results are, in fact, proved only for two levels);
3. expressions in which tests occur only as assertions (assert, which diverges via $\{U \text{(untrusted)}, S \text{(secret)}\}$5 if the check fails) or strong tests (which return the boolean at a sufficiently high label).

Two claims here stand out. First, the result that assertions can safely return a low result on an arbitrary lattice appears to be new relative to prior dynamic IFC work (Austin–Flanagan handled only two levels; Bichhawat et al. omit testing; Vassena et al.'s first-class labels correspond roughly to strong tests). Second, the authors concede that assertions leak information within what TINI accepts—in Kozyri et al.'s terminology, assert is not block-safe—and leave open whether this is inevitable or an artifact of their design.

Implementation and case studies

To ground the abstraction, the authors implemented the big-step semantics directly as a Python interpreter (~4000 lines, developed with Claude Code, Lark parser, OpenAI Responses API backing @). All examples run in it, with reported wall-clock times of a few seconds per run.

The security section reproduces the canonical CaMeL scenario: an email-reading agent asked to send a meeting reminder. A monadic tool-calling agent, sharing one conversation between untrusted emails and tool commands, correctly sends a reminder without attack but sends a cancellation when a ---SYSTEM UPDATE--- injection is embedded in the stored email. Notably, the injected output still carries the {U} taint label—the vulnerability is structural (shared channel), not a failure of tracking. The CaMeL-style defense—a privileged P-LLM generating code that calls a quarantined Q-LLM on untrusted data, with send_email enforcing via assertion that subject and body are trusted—defeats the same injection. Here the theory pays off concretely: TINI implies that varying the untrusted email content across all runs cannot alter the trusted subject and body fields, upgrading a single observed execution into a universal guarantee. A further confidentiality example shows an allowlist policy blocking exfiltration of a secret document despite an injection redirecting it to an attacker's address.

Limitations and open questions

The paper is candid about several constraints. The determinism assumption on the LLM excludes probabilistic behaviour; the authors sketch extending to weighted big-step semantics à la Borgström et al. to obtain probabilistic noninterference variants, but this is unrealized. Data sources are assumed to live in the program state; external tool calls and data sources would expand the calculus and complicate the theory. The defense case studies center on variants of a single example rather than a full model of CaMeL, and there is no benchmark evaluation against AgentDojo or SWE-Bench—an omission the authors explicitly flag as future work. Finally, the question of whether assert's non-block-safety is inherent remains open, as does a possible connection between conversations and session types.

Conclusion

This paper supplies the first lambda-calculus-based formal treatment of agentic LLM programming, combining conversation-manipulating primitives, dynamic information-flow tracking, and a termination-insensitive noninterference theorem whose restrictions (no raw tests, single-level lattices, or assert/strong-test discipline) are precisely characterized rather than glossed over. Its principal contribution is not a deployable system—the interpreter is explicitly minimal—but a rigorous semantic account in which CaMeL-style capability checks acquire proof-backed meaning, and in which the tension between expressive label testing and noninterference is made exact. It establishes a foundation on which fully practical, formally verified agentic defenses can subsequently be built and benchmarked.