---
title: 'Artic: AI-RTC for MLLM Assistants'
url: https://www.emergentmind.com/topics/artic
type: topic
---

# Artic: AI-RTC for MLLM Assistants

The term **Artic** refers to a specialized AI-oriented real-time communication (RTC) framework that re-engineers the video streaming pipeline for cloud-based Multimodal Large Language Model (MLLM) video assistants. Distinct from traditional RTC systems that optimize for human perceptual quality, Artic shifts network resource allocation and protocol logic to maximize MLLM response accuracy under stringent end-to-end latency constraints, focusing on ultra-low uplink bandwidth, adaptive content-aware encoding, and automatic accuracy benchmarking. Recent implementations of Artic systems have demonstrated substantial improvements in both model response accuracy and responsiveness over standard RTC stacks, and have established new paradigms for network design in AI-driven interactive video [2602.12641][2507.10510].

## 1. Problem Definition and Quality of Experience Shift

Conventional RTC frameworks—such as WebRTC, adaptive bitrate streaming (ABR), and error correction protocols—are designed around optimizing quality of experience (QoE) for humans, targeting metrics like PSNR, SSIM, stalling rates, and resolution fluency. In AI Video Assistant scenarios, however, the “consumer” is an MLLM (e.g., Qwen2.5-Omni, Gemini Live), shifting the QoE objective to **MLLM response accuracy** plus low end-to-end latency (typically <300 ms) [2602.12641][2507.10510].

This paradigm introduces three fundamental architectural shifts:
- **Uplink-Centric Bottleneck:** Video must be streamed at high fidelity towards the cloud, while downlink traffic (text/audio) is negligible.
- **MLLM Bandwidth Efficiency:** Model input requirements (e.g., 2 FPS sampling, <1 MPix per frame) are far below the output rates of mobile cameras, motivating aggressive bitrate reduction.
- **Accuracy–Latency Trade-off:** Raising video bitrate improves human perceptual quality but—beyond a threshold—yields negligible gains in MLLM performance, instead consuming network headroom and introducing queuing delays.

Prototype experiments confirm that conventional RTC stacks cause unsustainable latency spikes and rapid accuracy degradation under uplink fluctuations, especially during rapid bandwidth collapse events (e.g., elevator rides, cellular handoffs) [2602.12641].

## 2. Response Capability-Aware Adaptive Bitrate (ReCapABR)

Artic introduces the **Response Capability-aware Adaptive Bitrate** module, which explicitly models the relationship between video bitrate ($b$) and MLLM accuracy ($A(b)$). The accuracy curve is monotonically increasing and concave, saturating at $A_\text{sat}$ for moderate $b$. The key strategy is to:
- **Proactively cap bitrate** at the minimum $b^*$ such that $A(b^*) \geq A_\text{sat} - \epsilon$ for a small tolerance $\epsilon$, preserving bandwidth headroom ($\Delta b = \hat{B} - b^*$, $\hat{B}$: instantaneous bandwidth estimate).
- **Dynamically adjust bitrate** based on feedback from server-side MLLM response confidence $C_t$, using a nonlinear control law:
  $$
  R_{t+1} = \min(\hat{B}_t,\; R_t + w_t \cdot (\hat{B}_t - R_t)), \quad w_t = \delta_t |\delta_t|^{\gamma-1},\; \delta_t=(\tau-C_t)/\tau
  $$
  where $\tau$ is the target confidence threshold [2602.12641].

This allocation suppresses latency spikes by reserving headroom for transient network drops and avoids wasteful bandwidth utilization beyond accuracy saturation. Experimental results show up to **135 ms lower average transmission latency** and **+15% MLLM accuracy** relative to WebRTC+GCC [2602.12641].

## 3. Zero-Overhead Context-Aware Streaming (ZeCoStream)

Artic’s **Zero-overhead Context-aware Streaming (ZeCoStream)** offloads the task of determining which image regions are essential for understanding to the cloud-side MLLM. The MLLM is prompted to return bounding boxes $\mathcal{K}_t$ that are critical for the current response, effectively generating a region-importance mask with no additional client-side inference [2602.12641][2507.10510].

Bitrate allocation is then performed spatially:
- Each frame is divided into $N$ patches; for patch $i$ at distance $d_i$ to the nearest box boundary, importance is set as $\rho_i = \max(0,1 - d_i/(\mu \cdot \mathrm{diag}))$, with empirically-tuned $\mu$ [2602.12641].
- Per-patch quantization parameter $QP_i$ is set via a quadratic rule:
  $$
  QP_i = Q_{\min} + (Q_{\max}-Q_{\min})(1-\rho_i)^2
  $$
Thus, high-importance regions are preserved at high fidelity, while unimportant background is heavily quantized. Under ultra-low bitrates (e.g., $\sim$290 kbps), context-aware streaming retains >50% higher MLLM QA accuracy than uniform encoding [2602.12641].

Prediction horizon compensation is accomplished by having the MLLM predict bounding boxes 1.5 s into the future for moving scenes, synchronizing encoding decisions with model input time.

## 4. Loss-Resilient Adaptive Frame Rate and Redundancy Control

Since most MLLMs operate at modest frame rates ($R_M \approx 2$ FPS), but networks can support much higher transport rates $R_t$, Artic optimizes frame redundancy to mitigate packet loss without resorting to ARQ retransmissions. For measured packet loss $p_t$ and $\kappa_t$ packets per frame, the probability a frame arrives in time is $P_f = (1-p_t)^{\kappa_t}$, and the receiver samples $K = \lceil R_t / R_M \rceil$ candidates per model step [2507.10510].

Redundancy is increased only to the level required to ensure $P_s = 1-(1-P_f)^K \geq 1-\epsilon$, eliminating ARQ-induced latency and reducing average stalling to $1.2$ ms [2507.10510].

## 5. Degraded Video Understanding Benchmark (DeViBench)

DeViBench is the first large-scale benchmark dedicated to evaluating the sensitivity of MLLM responses to task-irrelevant video degradation. The construction pipeline comprises:
- Pre-encoding videos with aggressive quantization;
- Generating QA pairs by prompting MLLMs to focus on details visible only in the high-quality half of paired video frames;
- Filtering for truly “quality-sensitive” QAs by cross-verification and answer comparison between MLLMs [2602.12641][2507.10510].

DeViBench comprises 1,968 QA samples, 88,680 s total video, and enables systematic assessment of how encoding and channel strategies impact application-level AI accuracy; roughly 8% of QAs require inter-frame temporal context.

## 6. Quantitative Evaluation and Performance Impact

Combined, Artic’s optimizations yield:
- 45–70% **uplink bandwidth reduction** vs. standard RTC stacks, with **MLLM QA accuracy gains of 5–15 percentage points** across all tested conditions [2602.12641].
- **Latency reduction**: Mean transmission latencies decrease by 95–135 ms, and the frequency of frames arriving within 200 ms increases from ~66% to >82% [2602.12641].
- **Energy and cost trade-offs**: Minimal server-side inference overhead for context extraction; monetary API costs rise by ~27% due to increased MLLM feedback traffic, offset by reduced uplink data volumes [2602.12641].

A summary of core Artic components and their functions is provided below.

| Module             | Key Technique               | Impact                                 |
|--------------------|----------------------------|----------------------------------------|
| ReCapABR           | Accuracy-based bitrate cap  | Stabilizes latency, prevents wastage   |
| ZeCoStream         | MLLM-driven region encoding | Preserves accuracy at low bitrates     |
| Redundancy Control | Adaptive frame duplication  | Minimizes retransmission latency       |
| DeViBench          | Degradation-sensitive QA    | Enables application-level optimization |

## 7. Limitations, Extensions, and Future Directions

Current Artic deployments address only quantization-based degradation, with planned extensions to include channel loss, frame-dropping, and advanced forward error correction [2602.12641]. Scaling to rapidly dynamic scenes or adversarial prompt scenarios may erode context alignment, motivating joint model–codec co-design.

Future work will explore:
- Joint spatial-temporal saliency prediction at the client, semi-autonomous redundancy scheduling,
- Model-internal pruning of vision tokens to further reduce inference latency,
- Cooperative audio–video control where AI assistants rely on multimodal input (e.g., lip-reading),
- Integration with next-generation codecs (VVC/H.266), neural codecs, and model-intensive AR endpoints.

Artic demonstrates that a full-stack rethinking of networked video—for the needs and constraints of AI consumers, rather than humans—enables significant resource and latency efficiencies without MLLM output loss, offering a blueprint for broader MLLM-driven RTC systems [2602.12641][2507.10510].

Source: https://www.emergentmind.com/topics/artic