YALIS: Research LLM Inference Engine

• YALIS is a research platform that provides a modular, instrumentable engine for controlled multi-node LLM inference studies and communication analysis.
• It integrates modern techniques like CUDA Graphs, tensor parallelism, and speculative expert prefetching to facilitate precise systems experiments.
• Empirical studies with YALIS report up to 1.72x latency reductions and 14% token speed improvements, validating its role in experimental inference research.

YALIS is an open-source, research-oriented LLM inference engine developed as an experimental substrate for systems studies of inference-time bottlenecks and mitigations. In one source it is explicitly expanded as “Yet Another LLM Inference System” and described as a system built “as a research vehicle to study multi-node LLM inference”; in another it is described as “a light-weight research inference engine” that serves as the runtime for a Mixture-of-Experts (MoE) optimization based on speculative expert prefetching (Singhania et al., 12 Nov 2025, Madan et al., 9 Mar 2026). Across these uses, YALIS is characterized less as a production-serving framework competing on feature completeness and more as a modular, instrumentable runtime that remains sufficiently performant to support controlled experiments on distributed inference, communication collectives, CPU offloading, and decode-path scheduling.

1. Identity and role in the literature

YALIS appears in the recent literature as a systems research platform rather than as a general-purpose serving stack. Its central function is methodological: it provides a minimally confounded environment in which specific implementation changes can be isolated, measured, and attributed. In multi-node studies, YALIS is the platform used for controlled analysis of tensor-parallel strong scaling and for integrating a replacement all-reduce implementation. In MoE studies, it is the runtime that integrates CPU offloading, asynchronous expert transfers, CUDA-stream overlap, and one-layer-ahead expert-prefetch logic (Singhania et al., 12 Nov 2025, Madan et al., 9 Mar 2026).

Paper	Role of YALIS	Representative result
"LLM Inference Beyond a Single Node: From Bottlenecks to Mitigations with Fast All-Reduce Communication" (Singhania et al., 12 Nov 2025)	Research vehicle for multi-node LLM inference and communication studies	NVRAR integrated into YALIS yields up to a 1.72x reduction in end-to-end batch latency for Llama 3.1 405B in decode-heavy tensor-parallel workloads
"Speculating Experts Accelerates Inference for Mixture-of-Experts" (Madan et al., 9 Mar 2026)	Runtime for memory-constrained MoE offload and speculative expert prefetching	Expert prefetching in YALIS yields up to 14% lower time per output token

This dual role is significant. YALIS is not treated as a mere microbenchmark harness, because both papers rely on it to execute modern LLM inference paths under realistic optimizations. At the same time, it is not presented as a deployment-mature analogue of vLLM or SGLang. A plausible implication is that YALIS occupies a niche similar to a reference systems platform: representative enough for comparative performance work, but intentionally simple enough to expose the internal mechanisms under study.

2. Architectural organization and runtime model

The multi-node paper describes YALIS as having three central components: a unified model-definition layer adapted from LitGPT, an execution layer based on Torch Compile and CUDA Graphs, and tensor model parallelism implemented via AxoNN both within and across nodes (Singhania et al., 12 Nov 2025). It also supports multiple attention backends, paged KV caching, and speculative decoding. These features establish that YALIS is a full inference engine rather than a synthetic communication benchmark.

The MoE paper emphasizes a different portion of the runtime stack. There, YALIS is the system into which expert offloading and speculative prefetching are integrated. Only expert weights are offloaded; attention and router weights remain on GPU because they are relatively small. Offloaded experts are stored in pinned CPU memory for faster CPU-to-GPU DMA. Execution uses a separate CUDA copy stream for asynchronous transfers and a compute stream for model execution. Transfers are serialized with each other because they share the same copy engine, but a copy can overlap with compute. The implementation also uses double buffering, alternating expert buffers across adjacent layers, and offloads previous-layer experts after use (Madan et al., 9 Mar 2026).

Taken together, these descriptions show a consistent architectural philosophy. YALIS combines modern performance features—kernel fusion, CUDA Graph capture, optimized attention backends, model parallelism, and decode-path specialization—with a deliberately exposed execution model. This suggests that its primary design objective is not abstraction hiding, but controlled systems experimentation.

3. Multi-node tensor-parallel inference studies

In the multi-node study, YALIS is used chiefly as the tensor-parallel platform for strong-scaling experiments on GPU supercomputers. The paper evaluates YALIS alongside vLLM V1 (v0.11.0) and SGLang (v0.5.1) for pure tensor parallelism, and alongside vLLM V0 (v0.10.0) and SGLang for hybrid tensor-plus-pipeline parallelism. Although “YALIS (HP)” is mentioned briefly in later discussion, the main scaling comparisons and controlled communication experiments center on YALIS in TP mode (Singhania et al., 12 Nov 2025).

The reported environments are Perlmutter and Vista. Perlmutter provides A100 nodes, with 4 GPUs per node, fourth-generation NVLink intra-node, and HPE Slingshot-11 inter-node. Vista provides one GH200 96 GB GPU per node with InfiniBand interconnect. The models are Llama 3.1 70B Instruct and Llama 3.1 405B Instruct in bf16. Workloads are divided into prefill-heavy and decode-heavy regimes: prompt length 2363 with decode length 128, and prompt length 1426 with decode length 3072, each with number of prompts $\#P \in \{8, 32\}$ (Singhania et al., 12 Nov 2025).

YALIS is presented as “research-oriented but representative.” On Perlmutter, for Llama 3.1 70B Instruct, YALIS is reported to be within 5–16% of vLLM V1 under TP at 8 GPUs and beyond, especially in more memory-bound workloads. For Llama 3.1 405B Instruct, it remains within 8% across all GPU counts. The paper interprets this as evidence that YALIS reproduces scaling trends seen in production-oriented engines closely enough to justify using it as an experimental instrument (Singhania et al., 12 Nov 2025).

The principal systems finding exposed through YALIS is that decode-heavy multi-node tensor parallelism is limited by collective communication. In the decode-heavy regime, tensor parallelism does reduce matrix multiplication time, but the improvement is partially canceled by all-reduce overhead. The paper notes that communication time in YALIS TP increases by about 1.6x when moving from 8 to 16 GPUs in a decode-heavy case. The dominant all-reduce messages are small, with sizes in the 128 KB to 1 MB range; for the 70B model with batch size 8 and hidden dimension 8192, the all-reduce message size is 128 KB (Singhania et al., 12 Nov 2025). This small-message regime is where inter-node latency dominates and where standard NCCL behavior becomes the limiting factor.

4. Communication research and NVRAR integration

YALIS is also the integration point for NVRAR, an NVSHMEM-based hierarchical recursive all-reduce implementation, optimized for small-message inter-node communication (Singhania et al., 12 Nov 2025). NVRAR separates fast intra-node communication from slower inter-node communication using three phases: intra-node reduce-scatter, inter-node recursive-doubling all-reduce, and intra-node all-gather. Within a node, the implementation relies on NCCL through NVSHMEM host APIs; across nodes, it uses GPU-initiated NVSHMEM communication, including one-sided non-blocking put_nbi operations.

The motivation for NVRAR follows directly from YALIS-based bottleneck analysis. For decode-heavy tensor parallel inference, the primary collective is all-reduce over small $B \times H$ messages. The paper’s analytical comparison argues that the inter-node portion of NVRAR scales logarithmically in the number of nodes during the latency-dominated regime, whereas a flat ring incurs effectively linear inter-node latency growth in total GPU count (Singhania et al., 12 Nov 2025).

Standalone collective benchmarks under CUDA Graph capture report that NVRAR achieves up to 1.9x–3.6x lower latency than NCCL for approximately 128 KB to 2 MB message sizes on HPE Slingshot and InfiniBand; the conclusion gives more precise ranges of 1.06–1.92x on Slingshot and 1.14–3.57x on InfiniBand for 256 KB to 2 MB (Singhania et al., 12 Nov 2025). More important for YALIS itself, replacing NCCL with NVRAR in YALIS TP produces end-to-end batch-latency reductions in decode-heavy inference. On Perlmutter, the paper reports about 1.3x speedup for the 70B model with $\#P=8$ on 32 GPUs, 1.86x for the 70B model with $\#P=32$ on 32 GPUs, and 1.17x to 1.72x for the 405B model. On Vista, YALIS shows up to 1.92x acceleration for the 70B model with $\#P=32$ and 16 GPUs (Singhania et al., 12 Nov 2025).

This use of YALIS is methodologically important. The engine is held sufficiently constant that the observed end-to-end gains can be attributed to a substitution in the collective primitive rather than to unrelated scheduler or runtime changes. In that sense, YALIS functions as a controlled experimental substrate for communication-systems research.

5. Memory-constrained MoE decoding and speculative expert prefetching

In the MoE study, YALIS is the runtime realization of a decode optimization for memory-constrained inference in sparse MoE models (Madan et al., 9 Mar 2026). In this setting, the full model does not fit in GPU HBM, so the bulk of expert weights are stored in CPU RAM and only the active experts are moved to GPU during decoding. The paper emphasizes that sparse activation reduces per-token compute but not expert parameter size; decode therefore repeatedly pays CPU-to-GPU transfer costs for expert weights. For Qwen3-30B-A3B on an A6000, these transfers account for about 84%–88% of time per output token, making them the dominant latency source (Madan et al., 9 Mar 2026).

The key YALIS optimization is to speculate one layer ahead. At layer $l$, internal signals are used to predict the experts that the router will choose at layer $l+1$, so that those experts can be transferred early while the current layer is still computing. The relevant representations are the normalized residual stream $s_l$ and the quasi-hidden state $q_l$. The paper defines the default-vector aggregate as

$d_l = \sum_{e \in \mathcal{E}_l} g_{l,e} d_{l,e},$

and the quasi-hidden state as

$B \times H$0

where $B \times H$1 is the post-attention residual and $B \times H$2 is the normalization applied before routing at layer $B \times H$3 (Madan et al., 9 Mar 2026). Empirically, when $B \times H$4 is more aligned with the true next-layer router input $B \times H$5 than $B \times H$6 is, future experts are predicted more accurately.

The paper distinguishes four concepts that YALIS implements as separate runtime decisions: predicting future router-selected experts, prefetching those experts, speculatively executing them when the next layer arrives, and thereby avoiding re-fetching the true router-selected experts (Madan et al., 9 Mar 2026). That final step is the crucial systems point. Prior cache-hint approaches may still fetch the true experts later on a miss, putting transfer back onto the critical path. YALIS instead studies direct execution of the speculated experts; if task accuracy remains acceptable, the corrective transfer is eliminated and the overlap between communication and computation is preserved.

The decode pipeline has a cold start at the first layer, where there are no prefetched experts. For later layers, YALIS reuses already prefetched expert indices and gating weights for the current layer, computes the next layer’s predicted experts and weights with GetNextExperts(...), waits for any required current-layer transfer to complete, asynchronously launches the next-layer copy on the copy stream, and performs current-layer MoE compute on the compute stream. The implementation includes a wait_and_prefetch context manager and double buffering across adjacent layers (Madan et al., 9 Mar 2026).

The paper evaluates prediction quality primarily through recall@$B \times H$7, the expert-prefetch hit rate between predicted and ground-truth experts, and through expert rank alignment. Higher-ranked experts are predicted more reliably than lower-ranked ones, which helps explain why approximate routing can preserve downstream quality even with imperfect hit rate. For cases where naive speculative execution is insufficient, the paper introduces a shallow neural estimator trained to match next-layer router logits using KL divergence. Reported estimator sizes are 4M parameters for GPT-OSS-20B, 45M for GPT-OSS-120B, and 17M for Qwen3-30B-A3B (Madan et al., 9 Mar 2026).

6. Performance envelope, quality tradeoffs, and significance

The systems gains reported for YALIS’s MoE optimization are moderate but explicit. Across models and hardware, speculative expert prefetching reduces TPOT by 5%–14%; for Qwen3-30B-A3B on the A6000, the reduction is 9%–14%, with larger gains at longer context lengths because decode compute grows while copy time remains roughly constant (Madan et al., 9 Mar 2026). On stronger GPUs such as the A100 and GH200, gains are smaller, about 5%–8%, because there is less compute time available to hide transfer behind. The paper notes that the theoretical maximum speedup is $B \times H$8 when copy and compute are equal, but that in practice transfer is much slower than compute, so realized gains are bounded by the amount of computation available for overlap (Madan et al., 9 Mar 2026).

The quality results are architecture-dependent. For GPT-OSS-20B, baseline average accuracy is 0.799 and Router-PF yields 0.779. For GPT-OSS-120B, baseline average is 0.849 and Router-PF is 0.857, i.e. slightly better within evaluation noise. By contrast, Qwen3-30B-A3B is much more sensitive: baseline average accuracy is 0.817, Router-PF drops to 0.641, Est-PF recovers to 0.750, and Hybrid-PF reaches 0.766 (Madan et al., 9 Mar 2026). The degradation is especially pronounced on math-heavy tasks such as GSM8k and AIME24, consistent with the paper’s explanation that early-layer drift in Qwen makes inaccurate speculative routing more damaging.

Several limitations recur across the YALIS literature. The multi-node paper explicitly labels YALIS a prototype and a research vehicle, and notes missing data points caused by torch.compile out-of-memory failures (Singhania et al., 12 Nov 2025). The MoE paper limits the relevance of its optimization to CPU-offload regimes where expert weights do not fit on GPU, and to small-batch decode settings, especially $B \times H$9; as batch size approaches $\#P=8$0, the set of active experts tends to cover all experts and selective prediction loses value (Madan et al., 9 Mar 2026). The same paper also notes that its implementation does not support KV-cache offloading, which constrained the longest contexts studied (Madan et al., 9 Mar 2026).

Within these constraints, YALIS’s significance lies in its combination of representativeness and malleability. It is sufficiently close to mainstream inference engines to support comparative conclusions, yet sufficiently transparent to serve as an instrument for studying communication bottlenecks, collective design, offload scheduling, and speculative execution. The literature therefore treats YALIS less as an endpoint in itself than as an enabling systems platform for experimentally grounded LLM inference research (Singhania et al., 12 Nov 2025, Madan et al., 9 Mar 2026).