---
title: 'UnifiedServe: GPU Orchestration for MLLMs'
url: https://www.emergentmind.com/topics/unifiedserve
type: topic
---

# UnifiedServe: GPU Orchestration for MLLMs

UnifiedServe is a GPU-internal orchestration framework designed for high-efficiency, low-latency serving of Multi-Modal Large Language Models (MLLMs), with a core focus on the vision encoding and LLM inference stages in multi-stage pipelines. It achieves this by logically decoupling task scheduling across stages while physically multiplexing shared GPU resources, enabling order-of-magnitude improvements in throughput and latency compared to conventional monolithic or split serving approaches [2512.17574].

## 1. System Context and Motivation

UnifiedServe operates within the canonical three-stage MLLM inference pipeline: (1) multimodal preprocessing (e.g., video/image decompression), (2) vision encoding (e.g., ViT-based embedding of visual inputs), and (3) LLM inference (prefill and autoregressive decode). In typical deployments, these stages either run monolithically on the same GPU—leading to severe intra-stage stalls due to resource contention, or split across separate GPUs—fragmenting memory and compute, thereby increasing time-to-first-token (TTFT), under-utilizing hardware, and decreasing peak throughput. UnifiedServe aims to eliminate inter-stage blocking without GPU partitioning, guaranteeing tight service-level objectives (SLOs) for both time-between-tokens (TBT) and TTFT, while simultaneously maximizing overall GPU utilization [2512.17574].

## 2. Architectural Design

UnifiedServe implements a three-worker, asynchronous, logically decoupled scheduling abstraction:

- **vision_process worker**: Executes multimodal preprocessing (with joint integration of FlashCodec for NVDEC-powered video decode), outputting patch-token embeddings into a local IPC buffer.
- **encode-prefill worker**: Alternates between vision encoding (chunked over patch pages) and LLM prefill on the encoded visual inputs, with both sub-stages running in hybrid or tensor parallel to modulate per-iteration GPU demand.
- **decode worker**: Launches autoregressive sequencing for token generation, interfacing with the shared IPC KV-cache, and batch-processing output requests to clients.

All three operate on the same physical GPU cluster using NVIDIA Multi-Process Service (MPS) to submit kernels concurrently, ensuring hardware parallelism at the Streaming Multiprocessor (SM), memory, and cache levels. Data exchanges leverage multi-page shared memory buffers, scatter/gather via NCCL, and CUDA IPC handle transfer, enabling simultaneous producer-consumer coupling across the stages [2512.17574].

| Component             | Role                        | Resource Scope           |
|-----------------------|-----------------------------|-------------------------|
| vision_process        | Video decode, patch emit    | All GPUs (NVDEC, SMs)   |
| encode-prefill        | Vision encode + LLM prefill | SM, memory (shared)     |
| decode                | Autoregressive decode       | Prioritized SM, memory  |

## 3. Scheduling Algorithms and Resource Allocation

UnifiedServe employs distinct, fine-grained GPU schedulers:

**A. Stall-Free Video Decoding (via FlashCodec):**
- Inputs are partitioned into video GOPs across NVDEC engines on all GPUs.
- Threadpool-based GOP dispatch eliminates decoder straggler effects; frame buffers are dynamically allocated, sharply bounding decode-stage memory growth.
- Achieves up to 9× speedup for long videos on 4×A100 systems [2512.17574].

**B. Encode-Prefill Orchestration:**
- Let $\tau$ denote the prefill token budget, $\alpha$ the encode token budget—values are selected to satisfy the decode-stage TBT SLO.
- The algorithm maintains in-flight sets for encode and prefill, greedily accumulating chunks to fill batch budgets, and pipelining multi-microbatch execution for encode and single-batch for prefill.
- Mathematical resource allocation per epoch is governed by:
  
  $$
  \max \sum_{i} R_i \qquad
  \text{s.t.}\quad \sum_{i}\alpha_i G_i \leq G_\text{total}\ ,\quad
  \text{Latency}_i(\alpha_i) \leq \text{SLO}_i\ \forall i
  $$
  where $G_i$ is per-token GPU consumption for each stage $i\in\{\text{encode, prefill, decode}\}$, $R_i$ is tokens served per second, and $\alpha_i$ is the fractional resource allocation. By dynamic tuning of $\alpha$, latency-critical decode is prioritized, while encode/prefill opportunistically borrow residual compute and memory headroom [2512.17574].

**Decode prioritization** ensures that TBT SLOs are maintained under bursty, heterogeneous traffic, while aggressively exploiting otherwise idle SM cycles for vision encode and prefill bands.

## 4. Integration with FlashCodec and End-to-End Pipeline

UnifiedServe is designed as the second component of a unified MLLM inference stack, working in concert with FlashCodec, the first-stage collaborative GPU video decode engine:

- FlashCodec apportions incoming video across all GPUs’ NVDEC engines using a fine-grained stall-free scheduler and emits patch-token sequences.
- Patch-token pages are exchanged to vision encoder via a shared IPC buffer organized with collective write queues and page-table indirection.
- UnifiedServe orchestrates encode, prefill, and decode as detailed above, reading from/writing to distributed shared memory spaces, and avoiding bottlenecks by logical decoupling.
- All compute and memory resources on each GPU are accessible to any stage as needed, subject to scheduler-imposed constraints on SLO and batch formation [2512.17574].

This integration yields a non-blocking pipeline from frame decode through token generation, with each process advancing independently yet maintaining full physical multiplexing across GPUs.

## 5. Quantitative Performance and Comparative Analysis

Experimental deployments were conducted on Qwen2.5-VL-32B and InternVL3-38B MLLMs across the MLVU (long video), EgoSchema (short video), and VisionArena (image) datasets. The testbed comprised multi-A100 GPU clusters.

Key findings for UnifiedServe paired with FlashCodec [2512.17574]:

| Metric                                    | UnifiedServe + FlashCodec | Monolithic   | Split        |
|--------------------------------------------|--------------------------|--------------|--------------|
| Requests per unit time (Sustained)         | up to $3.0\times$        | $1\times$    | $1\times$    |
| Tightest SLO met                          | $1.5\times$ stricter     | Baseline     | Baseline     |
| Peak throughput (tokens/sec)               | $4.4\times$ monolithic   | $1\times$    | $1.9\times$  |
| Average TTFT (Qwen2.5-VL-32B, MLVU)        | $1.2$ s                  | $1.1$ s      | $6.0$ s      |
| P99 TBT                                   | $70$ ms                  | $450$ ms     | $25$ ms      |

Throughput and latency improvements are attributed to (1) minimization of inter-stage blocking and (2) elimination of device idleness via fine-grained intra-GPU resource sharing and scheduling. *This suggests UnifiedServe's architectural strategy is near Pareto-optimal for both tail-latency and throughput across heterogeneous multimodal queries.*

## 6. Implementation Details

UnifiedServe is implemented on top of Sarathi-Serve, with the following salient features:

- **Shared Memory IPC**: Patch tokens and KV-Cache buffers are realized using CUDA IPC handles and virtual paging (arrays of pv_indptr, pv_page_indices, etc.) to enable efficient, lazy materialization and transfer.
- **Collective Communication**: NCCL-based scatter/gather enables efficient movement of patches and embedding states across distributed processes.
- **Kernel Concurrency**: MPS permits overlapping kernel launches from vision_process, encode-prefill, and decode, preventing context-switch latency spikes.
- **Full async video decode**: Via FlashCodec’s threadpool/NVDEC engine multiplexing, the preprocessing step does not stall subsequent pipeline stages [2512.17574].

UnifiedServe and FlashCodec together constitute a high-throughput, scalably nonblocking architecture for multi-GPU, multi-stage MLLM inference, directly addressing prior throughput/latency bottlenecks endemic to both monolithic and split designs.

## 7. Implications and Future Directions

UnifiedServe's design underscores a shift towards *processor-centric*, rather than *stage-centric*, MLLM inference scheduling. Physically shared, logically decoupled resource orchestration enables stricter SLO attainment and higher utilization, providing a blueprint for future MLLM and LLM inference frameworks in cloud and edge data centers. 

A plausible implication is that similar logical/physical decoupling with aggressive resource sharing could be further generalized beyond vision–language models; for instance, multi-modal pipelines involving audio, graph-based inputs, or other non-text signals could benefit from this architecture. Additionally, continued innovations around IPC buffer management and token-level scheduling may yield further improvements in both resource utilization and tail latency under mixed-modality, high-variance traffic situations. 

UnifiedServe represents a demonstrably effective methodology for maximizing both latency and throughput metrics in high-concurrency, multi-GPU MLLM serving environments, with direct applicability for next-generation, low-SLO, high-throughput inferencing stacks [2512.17574].

Source: https://www.emergentmind.com/topics/unifiedserve