---
title: 'DisCEdge: Distributed Context for LLM'
url: https://www.emergentmind.com/topics/discedge
type: topic
---

# DisCEdge: Distributed Context for LLM

DisCEdge is a distributed context management system designed for efficiently serving Large Language Model (LLM) workloads at the edge. It addresses the challenges of maintaining conversational or session context for latency-sensitive and privacy-aware applications running on geo-distributed edge nodes, where standard LLM statelessness and naive client-side context management introduce excessive network latency and bandwidth overhead. DisCEdge implements tokenized, versioned storage and replication of user context, providing consistent, low-latency, and bandwidth-efficient LLM sessions across commodity hardware edge deployments [2511.22599].

## 1. System Architecture and Goals

DisCEdge is architected around a set of geo-distributed edge nodes, each comprising a Context Manager, a token-aware LLM inference engine (e.g., a modified `llama.cpp`), and a local replica of a geo-distributed key-value (KV) store such as FReD. Clients (mobile devices or edge compute devices) connect to their nearest edge node using geo-DNS or a registry, issuing completion requests that include `user_id`, `session_id`, and a monotonically increasing turn counter.

### High-Level Goals

- **Low-latency inference:** Minimize client-perceived round-trip time for sessional LLM inference.
- **Strong session consistency:** Guarantee strict ordering and continuity of user context across roaming sessions.
- **Bandwidth efficiency:** Reduce network load on both client-to-edge and edge-to-edge communication.
- **Transparent interface:** Abstract distribution, presenting a single centralized LLM service facade to clients.
- **Commodity deployment:** Operate efficiently on edge hardware without specialized accelerators (demonstrated on Nvidia Jetson TX2 and Apple M2).

### Component Structure

Each edge node consists of:
- **Context Manager:** Assigns and validates session parameters; maintains an in-memory, tokenized session state; enforces consistency semantics driven by the client turn counter.
- **LLM Service:** Receives concatenated token context and raw prompt; performs inference with minimal re-tokenization.
- **KV Store Replica:** Holds tokenized session context, supports peer-to-peer asynchronous replication, and enforces TTL-based eviction to reclaim resources.

## 2. Context Tokenization and Storage

DisCEdge transforms and stores user session context as integer token sequences rather than raw text, leading to efficiency gains in storage, transmission, and computation.

### Tokenization Pipeline

1. On session initiation (first turn), the Context Manager tokenizes the raw user input using the LLM’s tokenizer, storing the result as initial context $C_0$.
2. On subsequent turns, only the new prompt $P_k$ is tokenized ($p_k$), and the cumulative context is updated via simple concatenation: $C_k = C_{k-1} \parallel p_k$.

### Data Structures

- **KV Store**: Keyed by $(user\_id \| session\_id)$ and holding values `{ version \in \mathbb{N}, tokens = [int] }`—a version counter and a sequence of tokens.
- **In-memory**: The Context Manager caches “tokens” as a dynamic array keyed by session identifier.

### Efficiency over Raw Text

- **Compactness**: Token sequences require approximately half the bytes of raw text, reducing inter-node synchronization traffic by up to 15%.
- **Compute Reduction**: Avoids $O(\|C\|)$ repeated re-tokenization per turn, decreasing inference response latency by up to 14.46%.
- **Append Efficiency**: Context expansion per prompt is $O(1)$ amortized, since only new tokens are appended.

## 3. Distributed Context Replication Protocol

DisCEdge implements a distributed, client-driven replication protocol that guarantees session consistency semantics with minimal coordination.

### Replication Overview

- **Writes**: Updates (new turns) are written to the local KV-replica and then asynchronously propagated to peer nodes across the same model’s keygroup.
- **Reads**: Upon request, the Context Manager ensures the local context version is at least as recent as the client’s expectation (turn counter). If stale, it backs off and retries, with a maximum retry limit.
- **Consistency**: Maintains read-your-writes and monotonic reads (via client-incremented turn counters).

### Pseudocode and State Machine

The protocol at the Context Manager is as follows:

```python
OnRequest(user, session, client_turn, raw_prompt):
    retries ← 0
    while local_version < client_turn and retries < R_max:
        sleep(backoff_ms)
        (v’, T’) ← KV.read((user,session))
        local_version ← v’
        T ← T’
        retries++
    if local_version < client_turn:
        return error("consistency failure")
    p ← tokenize(raw_prompt)
    tok_context ← T ∥ p
    response ← LLM.generate(tok_context)
    spawn AsyncUpdate(user, session, local_version+1, p)
    return response

AsyncUpdate(user,session,new_ver,p):
    new_tokens ← read_local_context(user,session).T ∥ p
    KV.write((user,session) → (new_ver, new_tokens))
```

State transitions per session proceed from initialization ($S_{\mathrm{INIT}}$) to serving ($S_{\mathrm{SERVING}}$), with eventual eviction ($S_{\mathrm{EVICTED}}$) on TTL expiry.

### Complexity Analysis

Let $L = |T|$ (context length), $\Delta = |p|$ (new tokens), $R$ (number of replicas):

- Write serialization: $O(L+\Delta)$.
- Network bandwidth per update: $B_{\text{sync}} = (R-1) \times s_{\text{token}} \times \Delta$.
- Replication latency: $T_{\text{sync}}(L,\Delta,R) \approx \alpha + \beta \cdot s_{\text{token}} \cdot (L+\Delta) \cdot \log R$ (with $\alpha$ = network RTT, $\beta$ = per-byte factor).

Amortized per-turn costs grow only with number of new tokens, not total context length.

## 4. Context Operations: API, Caching, and Eviction

DisCEdge’s programming interface and memory management are optimized for interactive, large-scale, multi-session LLM use cases.

### API Semantics

- **StartSession**: Optionally assigns and returns new `user_id` and `session_id`.
- **Query (turn $k$)**: Requires identifiers, prior `client_turn = k-1`, and a new prompt. Operation blocks until the local context is current.
- **Session Update**: Upon response, the client increments its turn.

### Caching and Session Eviction

- **In-Memory Cache**: Context Manager maintains a per-session hash-map of token sequences.
- **KV Store TTL**: Enforces expiration and eviction of stale session state.
- **Anticipated Extensions**: LRU cache policies and session idle timeouts are proposed for future versions, improving multi-tenant resource efficiency.

## 5. Quantitative Evaluation

DisCEdge was evaluated using an open-source prototype on heterogeneous, resource-constrained edge devices and clients.

### Experimental Setup

- **Edge Nodes**: Nvidia Jetson TX2 (ARM), Apple M2 (ARM), both deploying llama.cpp.
- **Client Device**: Raspberry Pi 4 (ARM).
- **Network**: Local LAN, sub-1 ms RTT.
- **Scenario**: 9-turn dialogue using 4-bit quantized Qwen1.5-0.5B-Chat model.
- **Consistency Configuration**: `max_retries = 3`, `backoff = 10` ms.

### Performance Metrics

- **Median end-to-end response time**
- **LLM tokens-per-second throughput**
- **Inter-node synchronization traffic per turn**
- **Client-to-server request size**

#### Key Results

| Metric                      | TX2 (raw) | TX2 (tok) | M2 (raw) | M2 (tok) | Δ tok vs raw              |
|-----------------------------|-----------|-----------|----------|----------|---------------------------|
| Median latency (ms)         | 254.2     | 217.6     | 112.3    | 102.5    | –14.46 % / –8.75 %        |
| TPS (tokens/sec)            | 176       | 181       | 412      | 418      | +2.85 % / +1.41 %         |
| Sync bandwidth (KB/turn)    | 15.0      | 12.8      | 9.2      | 8.0      | –15 % / –13.3 %           |

| Metric (client side)          | Client-side | DisCEdge | Δ       |
|------------------------------|-------------|----------|---------|
| Median latency (ms)          | 232.4       | 219.0    | –5.93 % |
| Client→server request size (KB)| ~45        | ≈4       | –90 %   |

Across all workloads and hardware, tokenized storage consistently improves both latency and bandwidth consumption relative to raw-text or client-side context management.

## 6. Limitations and Future Work

### Scalability and Multi-Tenancy

- **Scaling limitations**: KV layer contention emerges with high multi-tenant concurrency. LLM inference throughput sets the upper bound.
- **Replica set size**: Synchronization overhead grows as $O(\log R)$ (gossip) or $O(R)$ (primary-fanout).

### Fault Tolerance

- **KV partitioning**: Context Manager may either block (strong consistency) or proceed with potentially stale state (weak consistency, configurable).
- **Future direction**: Integration of quorum-based consistency for tunable CP/CA trade-offs.

### Security and Privacy

- **Tokenization**: While integer tokens eliminate direct raw PII, token streams still encode user data.
- **Future work**: End-to-end token stream encryption and differential privacy protections are under consideration.

### Enhancement Directions

- **Summarization/snapshotting**: Periodically condense long context histories prior to token storage to control token sequence growth.
- **KV-cache integration**: Direct manipulation of the LLM’s internal attention KV cache is proposed to bypass external token sequence maintenance.
- **Predictive handovers**: Proactive context replication to likely next edge nodes, informed by user mobility.
- **Eviction policies**: Exploration of hybrid TTL and LRU for efficient memory management.

DisCEdge exemplifies an efficient approach to distributed LLM context management at the edge, leveraging tokenized storage, client-driven consistency, and decentralized KV replication to prioritize low latency, bandwidth efficiency, and seamless user experience [2511.22599].

Source: https://www.emergentmind.com/topics/discedge