MoLink Distributed LLM Serving Framework

• MoLink is a distributed serving framework for large language models that leverages dynamic micro-batch scheduling and chunk transmission to optimize inference performance on heterogeneous nodes.
• MoLink achieves up to 458% higher throughput than comparable systems while reducing latency and increasing cost-profit margins through adaptive workload management.
• MoLink integrates Linux, Windows, and containerized VMs via a dual-node architecture that combines Kubernetes, WSL Node Manager, and containerized VM management for seamless orchestration.

Searching arXiv for the MoLink paper and closely related entries to ground the article in current arXiv records. {"cmd":"python - << 'PY'\nimport urllib.parse, urllib.request\nqueries = [\n 'ti:\"MoLink\"',\n 'all:\"Distributed and Efficient Serving Framework for Large Models\"',\n 'all:\"Heuristic-Informed Mixture of Experts for Link Prediction in Multilayer Networks\"',\n 'all:\"CoMoL\"',\n]\nfor q in queries:\n url = 'http://export.arxiv.org/api/query?search_query=' + urllib.parse.quote(q) + '&start=0&max_results=3'\n print('\nQUERY:', q)\n try:\n data = urllib.request.urlopen(url, timeout=20).read().decode('utf-8', errors='ignore')\n print(data[:2500])\n except Exception as e:\n print('ERROR', e)\nPY"} สามสิบเอ็ดassistant to=commentary code 大发快三开奖结果.json {"cmd":"python - << 'PY'\nimport urllib.parse, urllib.request\nqueries = [\n 'ti:\"MoLink\"',\n 'all:\"Distributed and Efficient Serving Framework for Large Models\"',\n 'all:\"Heuristic-Informed Mixture of Experts for Link Prediction in Multilayer Networks\"',\n 'all:\"CoMoL\"',\n]\nfor q in queries:\n url = 'https://export.arxiv.org/api/query?search_query=' + urllib.parse.quote(q) + '&start=0&max_results=3'\n print('\nQUERY:', q)\n try:\n data = urllib.request.urlopen(url, timeout=20).read().decode('utf-8', errors='ignore')\n print(data[:2500])\n except Exception as e:\n print('ERROR', e)\nPY"} 全民彩票assistant to=commentary code ,一本道.json {"cmd":"python - << 'PY'\nimport urllib.parse, urllib.request, re\nq='all:\"MoLink: Distributed and Efficient Serving Framework for Large Models\"'\nurl='https://export.arxiv.org/api/query?search_query='+urllib.parse.quote(q)+'&start=0&max_results=5'\nprint(url)\ntry:\n data=urllib.request.urlopen(url, timeout=20).read().decode('utf-8', errors='ignore')\n print(data[:4000])\nexcept Exception as e:\n print('ERROR', e)\nPY"} тәшкиassistant to=analysis code 微信天天彩票քը The shell output wasn't shown? maybe network blocked? Let's inspect. {"cmd":"python - << 'PY'\nimport urllib.parse, urllib.request\nq='all:\"MoLink: Distributed and Efficient Serving Framework for Large Models\"'\nurl='https://export.arxiv.org/api/query?search_query='+urllib.parse.quote(q)+'&start=0&max_results=5'\nprint('URL', url)\ntry:\n with urllib.request.urlopen(url, timeout=20) as r:\n data=r.read().decode('utf-8', errors='ignore')\n print(data[:3000])\nexcept Exception as e:\n print('ERROR', repr(e))\nPY"}_久久爱assistant to=analysis code าคาร่ Still no output? Possibly commentary tool returns but not in transcript. Could use python requests with print to file and cat? Hmm. Let's do shell maybe directly. MoLink is a distributed serving framework for LLMs that is designed to make large-model inference affordable and practical on heterogeneous, weakly connected consumer-grade GPUs rather than datacenter accelerators. It targets deployments built from devices such as RTX 4090 PCs, Windows machines, Linux servers, and containerized cloud VMs, and it addresses two stated bottlenecks of such environments: limited network bandwidth with high latency, and host heterogeneity across operating systems and hardware profiles. Its core technical mechanisms are dynamic micro-batch scheduling and chunk transmission, layered on top of vLLM’s continuous batching; the reported implementation supports 18 mainstream architectures of open-source LLMs and achieves throughput improvements of up to 458% and cost-profit margin improvements of up to 151% relative to the compared systems (Jin et al., 7 Jul 2025).

1. Problem setting and design objectives

MoLink is framed around the cost of LLM serving and the observation that consumer-grade GPUs can provide a more affordable serving substrate than high-end datacenter devices. The system is explicitly intended for settings in which nodes may be connected through 100 Mbps Ethernet or public networks and may run substantially different host configurations, including Windows, Linux, or containerized VMs. The paper identifies two principal constraints: weak or constrained network conditions, and host heterogeneity (Jin et al., 7 Jul 2025).

Under weak network conditions, pipeline parallelism remains preferable to communication-heavy model-parallel schemes, but the residual activation transfer cost can still create pipeline bubbles and contention. Under host heterogeneity, uniform placement is suboptimal because nodes can differ in GPU type, GPU count, CPU performance, operating system, and link quality. Existing orchestration systems are described as inadequate for seamless management of Windows hosts and containerized VMs. MoLink is therefore organized as an engineering framework whose goal is not merely distributed inference, but distributed inference that remains usable when network and platform assumptions are substantially weaker than those of conventional datacenter-oriented serving systems.

2. System architecture

MoLink adopts a dual-node architecture consisting of a master node and multiple worker nodes. The master node is responsible for cluster management, metrics collection, and deployment scheduling, while worker nodes execute model inference. The master node runs on Linux and combines a standard Kubernetes control plane for Linux nodes with a WSL Node Manager for Windows hosts and a Containerized VM Manager for cloud VM-style instances. Metrics collection is likewise platform-aware: Linux nodes are monitored through Prometheus with exporters such as Node Exporter and DCGM, whereas Windows and VM nodes use dedicated collectors and servers (Jin et al., 7 Jul 2025).

At the serving layer, MoLink uses distributed inference engines split into a head engine and one or more downstream engines. The head engine is the service entry point and pipeline controller. It exposes the inference API, receives user queries, places them into a request queue, profiles compute and network characteristics, and drives scheduling decisions. A central component of the head engine is the Micro Batch Controller, which determines how many micro-batches to use and what batch size or token count each iteration should process. Downstream engines receive intermediate activations from the preceding stage, deserialize them, place them into a batch queue, and execute them with the same vLLM runtime. This organization preserves the continuous-batching execution model of vLLM while inserting MoLink-specific scheduling and communication control.

3. Scheduling and communication mechanisms

The paper attributes MoLink’s efficiency primarily to dynamic micro-batch scheduling and chunk transmission. Dynamic micro-batch scheduling is motivated by the claim that prior systems often set the number of micro-batches equal to the pipeline degree, implicitly assuming that communication overhead is small. MoLink instead adapts the micro-batch count to workload and network conditions so that communication-induced idle gaps can be hidden by additional useful computation (Jin et al., 7 Jul 2025).

The scheduling procedure uses a two-phase profiling strategy. During startup profiling, each engine measures computation latency over different batch sizes and sequence lengths. During runtime monitoring, engines periodically measure downstream network latency and available bandwidth. Using these profiles, the Micro Batch Controller first constrains the maximum batch size or token count for load balancing, then computes an ideal micro-batch count $N$ using incremental search starting from $N=1$, and stops increasing $N$ when the overhead of adding another micro-batch can be completely hidden by the remaining bubble. The stated objective is to select the smallest micro-batch count that fills idle communication gaps without over-fragmenting work.

Chunk transmission addresses competition between prefill and decode transmissions. The paper states that prefill can transmit up to 1000× more data than decode, so large prefill transfers can delay decode outputs under constrained bandwidth. MoLink therefore transmits prefill activations in chunks rather than as a single large transfer. Its communication process maintains a queue of pending transmission volumes split by phase type and prioritizes decode traffic over prefill traffic; a decode request is transmitted even if prefill transmissions are waiting. Prefill volumes are divided using a fixed chunk size, allowing decode traffic to proceed without being blocked by a large prefill transfer. Runtime communication uses gRPC.

The end-to-end inference path begins when a request enters the head engine’s API server and is placed in the Request Queue. The Batch Scheduler, built on top of vLLM continuous batching, organizes requests into micro-batches. Each engine runs the vLLM runtime for token generation. Intermediate results are passed to the Communication Process for adaptive chunked transmission, then received by downstream engines into an Intermediate Data Buffer, deserialized through a Data Processor, and pushed into the Batch Queue for scheduling. The resulting pipeline combines request queueing, dynamic micro-batch control, vLLM continuous batching, gRPC-based chunked transfer, and heterogeneous node orchestration.

4. Node selection, model partitioning, and heterogeneous integration

MoLink’s Task Scheduler is responsible for node selection and model partitioning. When a service is deployed through DeployLLMService(service_name, model_name, resource_specification, inference_parameters), the scheduler first selects nodes satisfying the requested GPU type and quantity. Two heuristics govern selection: preference for multi-GPU nodes when tensor parallelism is possible, and, when pipeline parallelism across nodes is needed, prioritization of nodes with lower inter-node latency and higher link bandwidth. After selection, model layers are partitioned uniformly if the nodes are homogeneous and non-uniformly if they are heterogeneous; stronger nodes receive more layers. The head node is chosen carefully because it must host the inference API and manage scheduling and control, so nodes with better CPU performance and network connectivity are preferred (Jin et al., 7 Jul 2025).

MoLink explicitly supports Linux nodes, Windows nodes, and containerized VMs.

Worker type	Integration path	Operational notes
Linux nodes	Kubernetes + pods	Kubelet receives directives; metrics via Prometheus exporters
Windows nodes	WSL2 Ubuntu 22.04 + MoLink Daemon	Engines launched inside WSL; metrics via psutil, GPUtil, and pyCUDA
Containerized VMs	Daemon + SSH tunnel port forwarding	Designed for instances that may allow only outbound connections

Linux nodes can be physical Linux servers or cloud ECS instances. Inference engines are launched in pods using container images, and node state is synchronized back to the control plane. Windows nodes are handled through WSL2 Ubuntu 22.04 virtual machines because Kubernetes and vLLM are Linux-centric; a MoLink Daemon receives commands from the WSL Node Manager and launches inference engines directly inside the WSL environment. Containerized VMs are aimed at providers such as AutoDL or vast.ai, where inbound initiation may be blocked; MoLink uses SSH tunnel port forwarding to enable cross-instance communication. The paper presents this platform abstraction as a practical mechanism for unifying resources that are otherwise difficult to manage within a single serving cluster.

5. Interfaces, supported models, and operational workflow

MoLink exposes both service-level and node-level APIs. The service APIs include DeployLLMService(...) for deployment, GetAPIKey(service_name) for authentication, CheckServiceStatus(service_name) for uptime, request counts, throughput, and utilization, and DeleteLLMService(service_name) to release resources. Node APIs include NodeAccess(node_info) to register a worker, CheckNodeStatus(node_name) for metadata and utilization, and NodeExit(node_name) for graceful decommissioning (Jin et al., 7 Jul 2025).

The implementation is described as allowing users on Windows, Linux, and containerized VMs to integrate GPUs with just a few lines of code over Ethernet or public networks. It currently supports 18 mainstream open-source LLM architectures, including BaichuanForCausalLM, ChatGLMForCausalLM, CohereForCausalLM, DeepseekForCausalLM, DeepseekV2ForCausalLM, DeepseekV3ForCausalLM, FalconForCausalLM, GemmaForCausalLM, Gemma2ForCausalLM, GlmForCausalLM, GPT2LMHeadModel, Phi3ForCausalLM, QwenLMHeadModel, Qwen2MoeForCausalLM, Qwen2ForCausalLM, Qwen3MoeForCausalLM, and Qwen3ForCausalLM. The evaluation focuses on the Llama family, especially Llama2-7B and Llama2-70B.

For model configuration in the main experiments, Llama2-7B is run in FP16, whereas Llama2-70B uses 4-bit quantization to fit memory and reduce the required pipeline width. These details place MoLink within a serving regime that assumes practical memory constraints and seeks to make large models usable on hardware outside conventional datacenter deployments.

6. Empirical performance, economics, and limitations

The main evaluation uses two servers, each with one RTX 4090 GPU, under three network settings: 100 Mbps / 10 ms, 1 Gbps / 10 ms, and 10 Gbps / 10 ms. Request traces are drawn from Azure Conversation, with inputs longer than 256 tokens or outputs longer than 512 tokens removed, and arrival rates scaled to available GPU capacity. MoLink is compared against Petals, described as a geo-distributed serving system for consumer GPUs, and vLLM, described as a representative datacenter-oriented serving system using Ray for distributed serving (Jin et al., 7 Jul 2025).

The reported metrics are TTFT, TPOT, and throughput. Across tested network settings and request rates, MoLink achieves up to 458% higher throughput than Petals and up to 44% higher throughput than vLLM, together with average 17.6% lower TTFT than Petals, average 8.4% lower TTFT than vLLM, average 22.8% lower TPOT than Petals, and average 41.5% lower TPOT than vLLM. Two concrete throughput examples are given: at 100 Mbps / 10 ms and 1 req/s, MoLink reaches 196.142 tokens/s versus 45.343 for Petals and 135.86 for vLLM; at 10 Gbps / 10 ms and 1 req/s, MoLink reaches 221.56 tokens/s versus 41.878 for Petals and 215.41 for vLLM.

The paper also evaluates cost-profit margin for locally maintained and rented devices. For locally owned RTX 4090 systems, using a 5-year amortization plus electricity cost, MoLink achieves the highest cost-profit margin, outperforming Petals by up to 151% and vLLM by 50.7%. The local-cost table uses a Llama2-7B API price of \$0.25 per million output tokens, RTX 4090 power of 450W, electricity price of 0.538 yuan/KWh, and RTX 4090 price of 12999 yuan. For rented cloud devices, the comparison uses S1: vLLM on H20-96GB, S2: vLLM on A100-80GB, and S3: MoLink on 4× RTX 4090-24GB, using rented AutoDL instances. In that setting, distributed consumer GPUs are reported to reach a peak workload capacity of 7 req/s, and the RTX 4090-based deployment can yield up to 75% higher cost-profit margin than an A100-based deployment. At 5 req/s, MoLink on 4×4090 achieves 737.74 tokens/s.

The stated limitations are equally important to the system’s interpretation. The paper does not present a formal fault-tolerance mechanism beyond node registration, status tracking, and graceful node exit, and it does not discuss checkpointing or automatic recovery in detail. The chunk transmission strategy also uses a fixed chunk size for prefill activations. These constraints indicate that MoLink is primarily a serving and orchestration framework for heterogeneous weak-network environments, rather than a full distributed systems stack with extensive recovery semantics.

7. Terminological scope and relation to similarly named methods

In the cited literature, MoLink denotes the distributed LLM serving framework described above. A separate paper on multilayer link prediction presents a method called MoE-ML-LP and states that it is “what you are calling ‘MoLink’ in spirit,” but its actual method name is MoE-ML-LP rather than MoLink; that method addresses multilayer link prediction through a heuristic-informed mixture-of-experts design, not LLM serving (Cava et al., 29 Jan 2025).

Likewise, CoMoL, or Core Space Mixture of LoRA, is a PEFT/MoE-LoRA method that compresses LoRA experts into a shared low-rank core space and performs token-level routing and soft merging in that space. Its problem setting is parameter-efficient fine-tuning rather than distributed inference or heterogeneous serving infrastructure (Cao et al., 28 Feb 2026).

This distinction is substantive rather than nominal. MoLink belongs to distributed systems and serving infrastructure for large models; MoE-ML-LP belongs to multilayer network link prediction; and CoMoL belongs to PEFT and MoE-LoRA adaptation. Their shared use of mixture, modularity, or efficiency-oriented design does not imply a common application domain.