Stateful Session Server Overview

• Stateful session servers are architectures that preserve per-session state across interactions, enabling consistent, context-aware operations.
• They span various implementations from FaaS with mutable caches to transformer inference and formal session-typed systems, each using distinct state management techniques.
• Key challenges include optimizing data locality, managing session-local versus shared state, and ensuring fault tolerance and security in multi-tenant environments.

A stateful session server is a server architecture in which behavior across requests is conditioned by persistent per-session state rather than by treating each request as independent. In the literature represented here, the term spans several strata: application servers that maintain conversational or per-user context; serverless platforms that treat a workflow execution as a session; inference systems that preserve a persistent KV cache across HTTP or WebSocket requests; MCP servers that keep per-session tool state; and formal session-typed servers whose legal communication states are specified by protocol and enforced mechanically (Sreekanti et al., 2020, Norgren, 13 May 2026, Rodrigues et al., 29 Jun 2026, Qian et al., 2020). Across these variants, the defining property is that state survives beyond a single request and is used to determine routing, consistency, admissible operations, or latency-critical execution.

1. Conceptual scope and defining properties

A stateful session server arises when later operations depend on state established earlier, such as an open file, an authenticated user, an in-progress transaction, a browser context, a per-call dialogue state, or a persistent model KV cache (Rodrigues et al., 29 Jun 2026, Norgren, 13 May 2026). In conventional stateless request-response systems, the client must resend sufficient context on every request, or the system must reconstruct it from external storage. The stateful alternative keeps that context server-side and reuses it across calls.

The notion of a “session” is domain-dependent. In Cloudburst, the scope of consistency is a DAG execution instance, which the paper explicitly identifies as a “session” in the sense of Terry et al. (Sreekanti et al., 2020). In stateful transformer inference, a session is a long-lived logical connection between a client and a model, backed by a persistent KV cache that lives across HTTP or WebSocket requests (Norgren, 13 May 2026). In AdvancedShelLM, each SSH connection is a session with command history, working directory, and recent conversational context, while the filesystem is global shared state visible across sessions (Sladić et al., 26 Jun 2026). In the MCP taxonomy, a Stateful Session Server is characterized by maintaining and exploiting per-session state across multiple tool calls, typically keyed by a session identifier (Rodrigues et al., 29 Jun 2026).

A useful distinction in the literature is between session-local state and shared state. Session-local state includes items such as a per-user history, a current file under edit, or a persistent transformer sequence ID (Norgren, 13 May 2026, Rodrigues et al., 29 Jun 2026). Shared state includes globally visible files in AdvancedShelLM’s JSON filesystem, replicated key-value state in Cloudburst’s Anna store, or globally shared entity state compiled into partitioned stateful operators in Stateful Entities (Sladić et al., 26 Jun 2026, Sreekanti et al., 2020, Psarakis et al., 2021). A stateful session server may therefore preserve isolated per-session context while simultaneously exposing a shared substrate.

The literature also distinguishes stateful session servers from weaker forms of affinity. At Layer 4, Prism guarantees per connection consistency (PCC): all packets of a TCP connection are forwarded to the same backend while that backend remains alive, even during server-pool changes (Cohen et al., 2020). This is sufficient for connection-level stickiness, but not for higher-level multi-connection user sessions. Application-level stateful session servers therefore either bind multiple requests to a persistent logical session or externalize session state so that multiple connections can converge on a coherent server-side object (Cohen et al., 2020, Rodrigues et al., 29 Jun 2026).

2. Architectural forms

The architectural realization of a stateful session server varies substantially by layer and workload.

Cloudburst presents a stateful Function-as-a-Service design in which function executors access low-latency mutable state through per-VM mutable caches backed by Anna, an autoscaling key-value store (Sreekanti et al., 2020). Its main components are function executors, mutable caches, stateless schedulers, monitoring and resource management, and Anna itself. Session state is stored persistently in Anna under keys such as session:<id> and cached close to executors; the scheduler uses data-locality-aware routing so that invocations referencing those keys tend to execute on a VM whose cache already holds them (Sreekanti et al., 2020).

Stateful transformer servers use a different decomposition. In “Attention Once Is All You Need,” the server architecture includes a Session Manager, a unified GPU KV cache partitioned into per-session sequences, a data plane for ingestion, a query plane for request handling, a GPU worker with a priority scheduler, a Flash Query engine, and a transient sequence pool for stateless traffic (Norgren, 13 May 2026). Each session owns one long-lived sequence in the unified KV cache, partitioned into Region 0 (frozen system prompt), Region 1 (sliding streamed data), and Region 2 (ephemeral query and response tokens). New data advances the persistent KV cache in the background; subsequent queries operate against that preserved state and do not reprocess the full history (Norgren, 13 May 2026).

Pensieve implements a related but distinct stateful serving architecture for multi-turn conversation LLMs. It treats the per-layer KV cache of all past tokens as the true session state, retains it across requests, and manages it with a two-tier GPU–CPU cache. Tokens evicted from GPU may remain in CPU KV cache; if evicted from both tiers, only raw text is kept, and dropped suffixes are recomputed on a later request (Yu et al., 2023). Unlike vLLM’s PagedAttention, Pensieve requires a multi-query attention kernel over non-contiguous KV cache so that multi-token prefills can operate on cached conversation state that is no longer physically contiguous (Yu et al., 2023).

Application-layer session servers often appear as explicit session managers over tool calls or interactive protocols. The MCP pattern catalogue describes a Stateful Session Server as an MCP server that stores SessionContext objects keyed by session ID, often in an in-memory map or Redis, and uses ordinary tool calls such as open_file and edit_file to read and mutate that state (Rodrigues et al., 29 Jun 2026). AdvancedShelLM uses a Main Script as the core session logic, a Worker LLM for shell-response generation, a Manager LLM for verification and filesystem patch generation, and a shared JSON filesystem that persists outside either model’s conversational memory (Sladić et al., 26 Jun 2026).

Formal treatments show the same pattern at the type-theoretic level. In “Client-Server Sessions in Linear Logic,” a server of type $\exc A$ repeatedly serves sessions of type $A$ while threading an internal state of type $B$ through initialization, per-client interaction, and finalization (Qian et al., 2020). In the PureScript MPST web framework, the server role is compiled from a Scribble protocol to an endpoint finite-state machine, and the server program is indexed by the current protocol state so that only legal transitions type-check (King et al., 2019).

Form	Persistent state	Session unit
Cloudburst FaaS	Anna KVS plus mutable caches	DAG execution instance
Stateful LLM serving	GPU/CPU KV cache	Long-lived sequence or conversation
MCP server	SessionContext map or Redis	Tool-call session ID
SSH honeypot	Per-session shell context plus shared JSON FS	SSH connection
MPST server	Protocol state plus application state	Session role instance

These forms differ operationally, but all externalize the same idea: later behavior is computed relative to an accumulated state rather than a self-contained request.

3. State representation and consistency semantics

A central design question is how the server represents state and what guarantees it gives about reads and writes across a session.

Cloudburst couples mutable caches with lattice-encapsulated state to support coordination-free convergence. Anna values are represented as lattices $(L,\leq,\sqcup,\bot)$, with associative, commutative, and idempotent merge operations (Sreekanti et al., 2020). The default representation is a Last-Writer-Wins lattice using a globally comparable timestamp; for causal consistency, Cloudburst uses causal lattices with vector clocks, dependency sets, and possibly multiple concurrent values (Sreekanti et al., 2020). On top of this storage substrate, Cloudburst defines distributed session repeatable read and distributed session causal consistency. For linear DAGs, the repeatable-read invariant requires that once a key $k$ is first read as version $k^{v_0}$, later reads in the same session must either observe later writes in the same session or continue to observe $k^{v_0}$ (Sreekanti et al., 2020). For causal consistency, the paper uses Lamport’s happens-before relation and requires future reads to be concurrent with or newer than versions implied by the session’s dependency set (Sreekanti et al., 2020).

PSL provides a different but related model for trusted execution environments. Its KVS stores values as pairs $V=(v,ts)$ with Lamport timestamps, and defines a partial order

$$V_1 \leq_e V_2 \iff (ts_1 < ts_2) \;\lor\; (ts_1 = ts_2 \wedge H(v_1) \leq H(v_2)).$$

The paper formalizes monotonicity, eventual progress, and validity, and states that monotonicity plus eventual progress imply linearizability (Thomas et al., 2024). Release-consistent locking through PSL-DB provides a stronger guarantee for critical sections: all writes by the previous lock holder are visible to the next lock holder before it enters its critical section (Thomas et al., 2024). For stateful session servers handling sensitive state, this yields a design in which per-session keys can often use the default KVS semantics, while high-contention invariants can be protected by explicit locks (Thomas et al., 2024).

Stateful transformer servers represent state as KV tensors rather than user-defined objects. In “Attention Once Is All You Need,” a session maintains per-layer key and value tensors for the current context, together with region offsets, position counters, sliding-window metadata, and Flash Query registrations (Norgren, 13 May 2026). Region 0 is frozen, Region 1 is appended with FIFO eviction, and Region 2 is cleared between queries (Norgren, 13 May 2026). Pensieve extends this with explicit mappings from logical token indices to physical slots in GPU or CPU KV cache, plus raw token histories for recomputing dropped suffixes (Yu et al., 2023).

Application-layer session servers may use explicit domain state stores. AdvancedShelLM represents each filesystem entry as a JSON object with fields such as "p" for absolute path, "k" for type, "mime", "size", "ctime", "mtime", "x" for content, and optionally "owner" and "perm" (Sladić et al., 26 Jun 2026). The Manager produces ADD and REMOVE patch operations, and the new global filesystem state is obtained by applying those patches to the persisted JSON store (Sladić et al., 26 Jun 2026). The MCP pattern paper uses a simpler SessionContext, for example containing filePath, content, and edits, keyed by _sessionId in tool arguments (Rodrigues et al., 29 Jun 2026).

Formal session-typed systems make the protocol state itself explicit. In Classical Linear Logic with coexponentials, the strong server rule

$\inferH{Server}{ \vdash \Gamma, B\ \vdash \negg{B}, \Delta \ \vdash \negg{B}, A, B }{ \vdash \Gamma, \Delta, \exc A }$

encodes a server that initializes internal state $A$0, serves one client of behavior $A$1 while producing a new $A$2, and eventually finalizes that state (Qian et al., 2020). Here the “session state” is not merely data but the legal continuation of the communication protocol.

4. Routing, locality, scheduling, and execution

Because state must be found and reused, stateful session servers are inseparable from routing and scheduling policy.

Cloudburst’s schedulers are explicitly data-locality-aware. If a function invocation carries KVS references as arguments, the scheduler chooses an executor whose VM cache is likely to contain those keys (Sreekanti et al., 2020). This yields a behavior analogous to sticky sessions, but without permanently binding a session to a single process: state remains logically disaggregated in Anna and can be served elsewhere when load requires (Sreekanti et al., 2020). The caches publish their key sets; Anna maintains an index from keys to caches and can selectively push updates to those caches when a key is modified (Sreekanti et al., 2020).

Prism handles the same problem at the network layer. It guarantees PCC without maintaining per-connection hardware state for all active connections by storing only migrated connections in a hardware Migrated Connection Table and keeping the full Software Connection Table in DRAM (Cohen et al., 2020). On a pool update, only signatures in bins whose server assignment changes are copied into MCT before the ECMP bin is updated (Cohen et al., 2020). This allows a stateful application server behind Prism to treat a TCP connection as a stable affinity key throughout its lifetime, even while the backend pool changes (Cohen et al., 2020). A plausible implication is that Prism supplies the connection-level substrate on which higher-level stateful session semantics can be built, while leaving user-level or multi-connection session logic to the application.

Stateful transformer servers rely on more aggressive scheduler designs because both state and accelerator time are scarce. “Attention Once Is All You Need” uses a continuous-batching GPU scheduler, cell-budget admission, adaptive chunked prefill, prefix-aware grouped prefill, and speculative decoding with a concurrency cap (Norgren, 13 May 2026). A cell is one token-layer unit of KV state, and the scheduler enforces

$A$3

so that long-context sessions cannot monopolize memory (Norgren, 13 May 2026). Requests are prioritized across classes [FLASH](https://www.emergentmind.com/topics/flash) > SESSION > POOL > STREAM, allowing interactive queries and Flash Queries to preempt ingestion at batch boundaries (Norgren, 13 May 2026).

Pensieve likewise uses unified iteration-level batching, but its distinctive requirement is support for requests whose contexts are distributed across GPU KV, CPU KV, and recomputed suffixes (Yu et al., 2023). On each turn, the scheduler determines which tokens are already on GPU, which must be swapped in from CPU, and which dropped tokens must be prepended to the prompt to reconstruct missing KV. The worker overlaps per-layer CPU→GPU transfers with computation so that later layers’ KV movement proceeds while earlier layers are already running (Yu et al., 2023).

Speculative Pre-Positioning pushes this stateful scheduling logic further. Each session retains a decision point

$A$4

after background pre-positioning of a short entry region, together with a cached next-token distribution (Norgren, 28 Jun 2026). If a confidence gate

$A$5

fires, the server serves the first token from the cached distribution with no new decode on the critical path (Norgren, 28 Jun 2026). The paper reports about 1.01 ms P50 first-token latency on the fast path, versus about 53.1 ms P50 for the cold stateful baseline and about 39 ms for a prefix-cache path that still pays a decode (Norgren, 28 Jun 2026). This shows that in stateful session servers, preserved state can be used not only to avoid recomputation but also to shift work off the request path entirely.

Stateful Entities compile the routing problem into the dataflow itself. Every entity class defines a key() function, and ingress routing partitions events on that key so that the operator subtask responsible for the corresponding entity instance reconstructs the object from local state and executes the requested method (Psarakis et al., 2021). This turns “session routing” into ordinary keyed dataflow partitioning.

5. Reliability, fault tolerance, and security

Persisting cross-request state improves expressiveness but exposes new failure modes: stale state, session leaks, replay hazards, inconsistent transitions, and cross-tenant interference.

Cloudburst addresses failures by re-executing an entire DAG after a timeout if a machine fails during a session, while Anna’s replication and lattice merge semantics allow caches to repopulate and converge after storage failures (Sreekanti et al., 2020). The paper explicitly notes that side effects outside Anna must be idempotent or handled explicitly by the application (Sreekanti et al., 2020). PSL assumes minority failures in an asynchronous setting with replicated storage servers, uses encrypted and signed blocks, and allows workers to continue through multicast even if PSL-DB is temporarily unavailable, though lock-based operations may block until PSL-DB recovers (Thomas et al., 2024).

AdvancedShelLM surfaces a different reliability problem: coherence across many simultaneous interactive users. Its explicit JSON filesystem exists precisely because earlier single-LLM designs kept state implicitly in conversation history and therefore failed to maintain consistent filesystem behavior and time-dependent outputs (Sladić et al., 26 Jun 2026). Nevertheless, the paper reports residual issues such as occasional state inconsistencies, thought leakage, and latency that humans still noticed during deception studies (Sladić et al., 26 Jun 2026). This suggests that externalizing state is necessary but not sufficient; transition validation and prompt-layer hygiene matter as well.

Security properties are especially salient in multi-tenant stateful servers. PSL’s entire design is motivated by confidentiality and integrity under malicious cloud providers, storage servers, and networks, with state encrypted outside enclaves and access guarded by attestation and key distribution (Thomas et al., 2024). AdvancedShelLM isolates attackers from the host OS by simulating all shell behavior through LLMs and a JSON filesystem rather than a real shell and filesystem (Sladić et al., 26 Jun 2026).

At the network layer, “Invisible Adversaries” shows how a stateful session server can fail when shared state is not properly partitioned. VPN connection-tracking frameworks maintain shared flow tables and NAT mappings for all clients on the server, and the paper shows that shared resource misuse and insufficient validation of session-state transitions can violate a non-interference property (Yang et al., 5 Apr 2026). Concrete attacks include port-exhaustion denial of service, TCP hijacking, and forged DNS responses, all enabled by cross-user coupling in a global state table (Yang et al., 5 Apr 2026). The paper’s design lessons generalize widely: include tenant identity in state keys, randomize externally visible identifiers when possible, impose per-tenant quotas, and constrain destructive transitions such as RST handling or loose ESTABLISHED creation (Yang et al., 5 Apr 2026). A plausible implication is that any multi-tenant stateful session server should treat state-key design as a security boundary, not just a routing convenience.

MCP servers expose a related operational risk: sessions that are not reaped cause memory leaks, and horizontal scaling requires a distributed session store such as Redis (Rodrigues et al., 29 Jun 2026). They also rely on reliable propagation of session identifiers, which the paper notes “is not guaranteed” if left entirely to the LLM (Rodrigues et al., 29 Jun 2026). Here, session identity is simultaneously a correctness, scaling, and security concern.

6. Formal models, applications, and historical significance

The idea of a stateful session server is not confined to implementation papers; it has a substantial formal and programming-language lineage.

“Client-Server Sessions in Linear Logic” introduces coexponentials $A$6 and $A$7 as modalities suited to modeling a server receiving requests from an arbitrary number of clients on a single channel (Qian et al., 2020). The server rule above directly encodes internal mutable state threaded across unbounded client interactions. The paper’s compare-and-set example shows a server with Boolean state that serves multiple clients nondeterministically while maintaining atomic state evolution (Qian et al., 2020). This provides a logical account of what later systems engineer operationally: a persistent server whose observable behavior depends on internal state accumulated across interactions.

“Using session types as an effect system” turns stateful effects such as get and put into session-typed protocols over a store process (Orchard et al., 2016). Effects become session types on an effect channel; the store itself is a session server that repeatedly offers get, put, and stop branches while preserving a mutable cell (Orchard et al., 2016). The significance for stateful session servers is methodological: state access protocols can be specified and type-checked as communication protocols, not merely documented informally.

The PureScript MPST web framework and the Erlang MPST actor framework bring that formalism into server implementations. In PureScript, a Scribble protocol is projected to endpoint finite-state machines, and generated type-class constraints ensure that only code respecting the protocol compiles (King et al., 2019). In Erlang, session actors combine OTP gen_server logic with runtime monitors generated from Scribble projections, so that protocol violations cause the process to fail in line with Erlang’s “let it fail” philosophy (Fowler, 2016). These systems make the server’s conversational state explicit and mechanically enforceable.

Beyond formal communication, the stateful session server pattern appears in empirical software architecture. The MCP taxonomy explicitly names Stateful Session Server as one of five recurring MCP server patterns, with lineage in Session and Memento patterns and the crucial LLM-specific delta that state is implicit, not in the prompt (Rodrigues et al., 29 Jun 2026). The pattern is observed in browser automation, memory servers, Git servers, and voice-dialogue infrastructure (Rodrigues et al., 29 Jun 2026). This places the concept within contemporary systems practice rather than treating it as a niche property of particular runtimes.

Application domains are correspondingly broad. Cloudburst targets serverless scientific and data-processing workflows (Sreekanti et al., 2020). Stateful transformer servers target multi-turn chat, streaming analytics, and agentic tool use (Norgren, 13 May 2026, Yu et al., 2023, Norgren, 28 Jun 2026). AdvancedShelLM uses persistent session and filesystem state to emulate an SSH environment (Sladić et al., 26 Jun 2026). Stateful Entities targets cloud applications written as object-oriented transactional programs compiled to distributed dataflows (Psarakis et al., 2021). In each case, the same abstract pattern recurs: state survives interaction boundaries, and server behavior is a function of that preserved state.

A recurrent misconception is that “stateful” simply means “sticky.” The surveyed literature contradicts that simplification. Prism provides stickiness without application state (Cohen et al., 2020); Cloudburst provides disaggregated state with opportunistic locality rather than fixed affinity (Sreekanti et al., 2020); transformer servers preserve model-internal state even when requests arrive over ordinary stateless HTTP (Norgren, 13 May 2026); MCP servers may carry state across tool calls even though MCP itself defines stateless JSON-RPC exchanges (Rodrigues et al., 29 Jun 2026). Stateful session servers are therefore best understood not as a single product category but as a general architectural pattern in which a server preserves, indexes, and exploits state across a sequence of interactions.