Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving

Abstract: Real-time interaction models -- Moshi, MiniCPM-o, Qwen-Omni -- turn serving into a periodic real-time task: on every frame a session ingests streaming audio and must respond by a recurring wall-clock deadline, while its KV cache grows monotonically and stays pinned for the whole conversation. This regime hides a dangerous failure mode. On a real full-duplex stack, sustained load does not degrade serving gracefully: it falls off a cliff, jumping in one step from milliseconds per frame to a stalled engine when accumulated session state exhausts the KV pool. The collapse is metastable -- identical five-minute runs collapse or survive on run-to-run variance -- and silent: latency and deadline-miss metrics read healthy throughout. We show one move restores both stability and observability: bound each session's resident state, and latency starts telling the truth. Metronome's in-engine KV window eliminates the collapse (0/20 vs. 14/20 runs across two batches) and turns per-frame latency into a monotone load signal, on which an online admission controller discovers the schedulable concurrency; without the window, the identical controller over-admits into the wall. A first-order model predicts the collapse time within a few percent on the headline model, and a quality probe validates the bound's design by ablation: the window alone is quality-free in turn-based decoding, and its few pinned attention-sink tokens are what keep free-running generation healthy. Everything is measured end-to-end on real audio, across four interaction models on one GPU.

• The paper introduces a method to bound per-session KV cache, preventing catastrophic GPU memory exhaustion in real-time LLM serving.
• It presents an analytical model that predicts collapse timing and demonstrates how a windowed KV strategy sustains stable latency under heavy load.
• The study validates that using a fixed window with persistent sink tokens maintains output quality while enabling dynamic, latency-based admission control.

Metronome: Bounded KV Serving for Real-Time LLM Interaction

Introduction and Motivation

The proliferation of real-time interaction models, such as Moshi, MiniCPM-o, and Qwen-Omni, has shifted LLM serving workloads from ephemeral, turn-based processing to persistent, periodic real-time sessions. These models continuously ingest streaming audio and produce responses with stringent wall-clock deadlines, fundamentally departing from the traditional chatbot paradigm where the context is fleeting and the KV-cache is frequently reclaimed. Instead, each session persists for minutes, growing its context—and thus, its resident KV-cache—monotonically without release during the session's entire lifetime.

This serving regime, characterized by periodic deadlines and unbounded per-session state, exposes a substantial and under-recognized failure mode in current inference stacks: under sustained load, the system does not degrade gracefully but instead abruptly "falls off a cliff" when the cumulative KV state exhausts the available GPU memory, stalling all ongoing sessions. Critically, standard metrics such as per-frame latency and deadline-miss counters remain misleadingly healthy even as this catastrophic collapse occurs.

Failure Mode in Real-Time Periodic Interaction

In contrast to turn-based LLM serving—where request lifetimes are short and idle gaps enable context swapping or recomputation—real-time interaction sessions must retain their growing KV state entirely in GPU memory due to the absence of processing lulls. Modern engines, such as vLLM and SGLang, support this by keeping session KV live across frames in persistent GPU storage. However, they do not place any bound on the resident state, letting it grow toward the maximum model length.

Empirical analysis demonstrates that with unbounded KV, latency remains low until, at a critical point, the memory pool saturates and per-frame latency instantly jumps to a hard ceiling (typically just under the frame budget cutoff), at which point sessions become unschedulable and the engine stops emitting tokens. Remarkably, this collapse is:

1. Memory-Dominated: The trigger is memory pool exhaustion, not compute overload. Compute cost per frame remains low, but the session's accumulated context consumes all GPU memory.
2. Metastable: Whether a run collapses depends on a race between the pool fill rate and session length. Run-to-run randomness in generation patterns leads to unpredictable collapse, especially when system capacity is marginal.
3. Silent: Standard serving metrics remain within tolerance as the collapse unfolds. The engine returns empty frames on time, and latency/deadline-miss counters report healthy values even as user sessions permanently stall.

Figure 2: Short-burst (90s) concurrent serving maintains low per-frame latency, but under unbounded KV, five-minute runs unpredictably hit a sudden “cliff” where latency spikes and serving collapses.

Figure 4: The instant of KV pool saturation: On unbounded KV, resident memory rises monotonically until the block pool is full, after which all sessions move to the waiting queue and the system stalls.

Analytical Model: Predictability and Memory Sizing

A first-order model accurately predicts the time to collapse: the KV pool occupancy grows linearly as each session appends KV at a steady per-session rate $r$, so the time to fill is $t_{\mathrm{sat}} = \frac{1 - \rho_0}{Nr}$, where $N$ is the number of sessions. Thus, runs where this $t_{\mathrm{sat}}$ is less than the session duration collapse deterministically, while those with room to spare survive. Variance in $r$ across runs explains the observed metastability when system parameters are near the threshold.

When resident KV is bounded per session, pool occupancy instead plateaus—a direct, observable signal for allocating memory budgets and controlling concurrency.

Figure 1: Pool occupancy under unbounded (linear increase) and windowed (plateau) KV. A linear fit on the early unbounded trace predicts collapse timing with high accuracy; windowed context supports many more sessions before memory binds.

Metronome: Serving With a Bounded Resident KV

The principal solution advanced in this work is conceptually simple but highly effective: explicitly bound each session's resident state. Metronome implements this with an in-engine windowed KV mechanism that retains only the $W$ most recent tokens of context plus a fixed set of “sink” tokens (e.g., the initial system prompt and structural tokens), preventing quality regression in free-running decode settings.

This windowed context bounding yields several key properties:

• Per-frame compute and memory usage are strictly limited by window size and unaffected by session age.
• Per-frame latency increases monotonically with load, offering a reliable, actionable signal for online admission control.
• The risk of silent, catastrophic collapse is eliminated; serving degrades gracefully under high load.

Admission Control Using Latency Feedback

With resident state bounded and per-frame metrics faithful, an online AIMD (Additive-Increase/Multiplicative-Decrease) admission controller—acting on real-time measured latency—can dynamically adjust the session cap without prior configuration. The system admits new sessions until measured latency approaches the deadline threshold, and then backs off, converging swiftly on the true schedulable concurrency.

This design is falsifiable: with unbounded KV, the identical controller is misled by persistently flat latency, over-admits, and drives the system into collapse. Only under bounded state does the controller converge correctly.

Figure 3: With windowed KV, online admission control discovers and maintains the true schedulable concurrency, with latency holding far under the deadline. Unbounded KV produces a misleading signal and triggers collapse.

Quality Analysis and Ablation Studies

Degrading a model's persistent memory footprint can, in principle, harm answer quality. The paper subjects Metronome's bounding strategy to rigorous ablation:

• Turn-based decoding: With only the $W$-token window, system accuracy and answer quality remain statistically indistinguishable from the unbounded baseline. Most interaction turns fit comfortably within standard window sizes.
• Free-running decoding: Attention “sink” tokens are critical; omitting them yields rapid quality decay as the model loses orientation in the conversation. Properly pinned sinks preserve high-quality outputs even outside the window.

Figure 5: In continuous decode, removing attention sinks causes model output to degrade as the window slides, whereas the full bound (window plus sinks) preserves stable quality.

Model Generality and Practical Constraints

The pathological collapse induced by unbounded KV is not specific to a single architecture or audio model; empirical results confirm its presence across multiple state-of-the-art LLMs. Short-term session capacity is determined by model architecture (e.g., audio encoder weights, MoE sparsity, frame budget), but the long-duration memory cliff is a universal property of the serving path.

Figure 6: Streaming capacity for various models over fresh, short sessions is determined by model specifics; the latent memory collapse is consistent across architectures over longer durations.

Best Practices and Implementation Recommendations

Metronome's design choice—an internal, engine-level sliding window anchored by persistent sink tokens—outperforms application-level strategies (such as recycling resumable requests at the window boundary), which incur periodic costly re-encoding and unnecessarily complex state management.

Figure 7: In-engine windowing shows flat latency and superior efficiency compared to application-level recycling, which suffers from growing decode costs.

Additionally, window sizes substantially larger than typical conversational context (e.g., $W=2048$) are “free” in performance terms, offering ample headroom to balance quality and memory usage without tuning.

Figure 8: Even generous window sizes incur negligible latency penalty, confirming that architectural margins are wide and safe for deployment.

Broader Implications and Future Directions

While motivated by audio and real-time interaction serving, this failure mode and its solution generalize to any persistent real-time workload with unbounded per-session state—agent scratchpads, streaming multimodal contexts, or long-running RAG caches. The universal lesson is clear: unbounded, pinned session state is a latent, catastrophic risk wherever periodic real-time control is required.

A practical implication for future engine designs is to expose session-state bounding as a first-class, explicit serving parameter (e.g., “retain the first $S$ and last $W$ tokens per session”), instead of relying on indirect model length caps or application-level home-brewed solutions. This will enable robust overload control and trustworthy autoscaling in modern inference stacks.

Conclusion

Metronome introduces a single, principled change—explicitly bounding resident KV per session—that resolves a metastable, silent failure mode endemic to real-time LLM serving. This transforms per-frame latency into an actionable signal, enabling robust, online admission control and graceful degradation under overload. Both serving stability and quality are maintained, with empirical validation across multiple state-of-the-art interaction models. The methodology and architectural lessons outlined are broadly extensible to future AI systems exposed to persistent, real-time workloads, and point to per-session state bounding as a foundational control primitive for inference system design.