veScale: Eager-Mode Distributed Training

• veScale is an eager-mode system for distributed tensor programming that uses SPMD to maintain single-device consistency during large-scale training.
• It decouples standard PyTorch model code from distributed execution via a declarative parallelization plan and effective DTensor integration.
• veScale-FSDP extends the framework with flexible, structure-aware sharding and planning algorithms to optimize communication and memory efficiency.

veScale is an eager-mode training system for distributed tensor programming that fully embraces the Single Program Multiple Data (SPMD) paradigm while targeting two requirements that are often in tension in large-scale training: consistency with single-device execution and high performance at scale. In the formulation presented for LLM training, veScale decouples original single-device PyTorch model code from distributed execution through a declarative parallelization plan, integrates with DistributedTensor semantics, and introduces a distributed random number generation algorithm intended to preserve single-device-equivalent results under arbitrary sharded operators. A closely related system component, veScale-FSDP, extends the veScale ecosystem to Fully Sharded Data Parallel training by combining a flexible sharding format, RaggedShard, with a structure-aware planning algorithm for efficient communication and memory management at large scale (Li et al., 5 Sep 2025, Wang et al., 25 Feb 2026).

1. Historical and technical setting

veScale is situated within the evolution of distributed training for increasingly large and heterogeneous models, especially settings requiring 3D parallelism and more intricate tensor placement strategies. The stated motivation is that the growing size and complexity of LLMs have made distributed training stacks increasingly complicated, which in turn motivates a shift toward simpler and more debuggable programming paradigms such as SPMD. In eager execution, however, two key challenges are identified: ensuring consistency with single-device execution and achieving high performance at scale (Li et al., 5 Sep 2025).

The system is presented against a background in which existing eager-mode distributed tensor frameworks are described as suffering from inconsistent results relative to single-device execution and from CPU or communication bottlenecks. A recurring concern is that systems such as PyTorch DTensor, Megatron-LM, DeepSpeed, and TorchTitan do not guarantee that running the same model definition on a single device and on multiple devices produces identical results for operators that use randomness, including initialization and dropout. Another concern is that per-operation dispatch logic and fine-grained communication can impose substantial overhead in eager execution (Li et al., 5 Sep 2025).

Within this setting, veScale is characterized not simply as a collection of optimization techniques but as a programming model intended to “democratize distributed tensor programming.” This suggests a broader goal: to make advanced parallelization expressible without coupling model authoring to distributed intrinsics. A plausible implication is that veScale is best understood as both a runtime system and an abstraction layer over distributed tensor execution rather than only as a library of kernels or sharding recipes (Li et al., 5 Sep 2025).

2. Programming model and system architecture

The programming model centers on “zero model changes”: veScale requires no modifications to the original single-device PyTorch model code, and distributed execution is described separately through a declarative parallelization plan. This plan specifies how to shard weights, inputs, and other tensors across devices, and when to redistribute or annotate them. The summaries describe the plan interface as expressive enough to support regex-based targeting of multiple similar layers and timing specifications such as initialization-only or forward/backward-time actions (Li et al., 5 Sep 2025).

The architecture contains several named components. A Plan IR transformation converts user plans into an intermediate representation that is optimized, including merging and reordering, and then attached as PyTorch hooks that orchestrate sharding and resharding at runtime. Tensors are transparently replaced with PyTorch DistributedTensor objects supporting sharding and replication. An efficient runtime then uses hooks to enforce placements and trigger optimized communication and random number generation (Li et al., 5 Sep 2025).

A representative API pattern is given in the source material:

$0.19$3

This interface is important because it shows that veScale treats distributed execution as a separate specification layer rather than embedding parallel semantics into model definitions. The associated “Plan Zoo” provides reusable plan recipes for common models and parallel schemes. This suggests a design in which reuse and composability are considered first-class requirements, especially for multi-axis distributed configurations (Li et al., 5 Sep 2025).

The relationship between veScale and PyTorch DTensor is also central. veScale does not replace DTensor semantics outright; instead, it uses DTensor integration while redesigning surrounding dispatch, planning, and execution pathways. In this respect, veScale can be interpreted as a systems layer that attempts to preserve familiar tensor programming semantics while changing the operational characteristics of eager distributed execution (Li et al., 5 Sep 2025).

3. Consistency and distributed random number generation

A defining claim of veScale is that it preserves single-device-equivalent results in distributed eager-mode SPMD execution. The technical obstacle identified is random number generation: in prior seed-based or offset-based designs, per-device RNG streams diverge, so the distributed execution of random operators no longer corresponds to the result of a single-device run. veScale addresses this with a distributed RNG algorithm described as compatible with arbitrary sharded operators (Li et al., 5 Sep 2025).

The key idea is to virtualize the multi-device system as a single global virtual GPU. For each local tensor element to be generated, veScale maps the local index to its global tensor index, derives a corresponding virtual thread and offset, and invokes the backend RNG as if the local computation were a partition of a single global sequence. The summary describes this as working for arbitrary tensor shapes and non-contiguous or interleaved sharding, with the intended property that merging all local tensors reconstructs the single global tensor exactly as if it had been generated on one device (Li et al., 5 Sep 2025).

In algorithmic form, the source material gives the following simplified structure:

$0.19$4

The reported consequence is that veScale matches single-device tensors bitwise on tested random operators, shapes, and sharding patterns, with remaining discrepancies attributed only to floating-point sum associativity in collective operations. The evaluation summary states that previous baselines show maximal per-element differences in initialization on the order of $0.16$ to $0.19$, whereas veScale achieves differences of at most $10^{-5}$, and that its loss curves match single-device training exactly in the reported experiments (Li et al., 5 Sep 2025).

This property is notable because distributed consistency is often treated as an approximate goal rather than a strict semantic target. A common misconception is that eager-mode SPMD inconsistency is merely an unavoidable artifact of parallelism; the veScale formulation argues instead that a large part of the problem is algorithmic, specifically the definition of distributed RNG under sharded execution (Li et al., 5 Sep 2025).

4. Performance mechanisms in eager-mode SPMD

Beyond consistency, veScale is designed to reduce overheads that arise in eager distributed execution. The reported bottleneck in PyTorch DTensor is dispatch: every operation dynamically infers and validates tensor metadata, which becomes costly for models with many lightweight operators per forward pass. veScale addresses this through three main techniques: rule-based bypass for operators whose metadata inference can be skipped, a sharding propagation cache that memoizes inference results for operator-plus-input combinations, and a C++ implementation of key logic previously executed in Python (Li et al., 5 Sep 2025).

A further step is “Static Eager mode.” The underlying observation is that, for most operations, DTensor metadata does not change at runtime. veScale therefore precomputes and statically assigns placements, allowing most operators to map directly to local tensor operations while using hooks only where redistribution or placement annotation is required. In the summary, this is presented as eliminating dispatch overhead entirely for those operators (Li et al., 5 Sep 2025).

Communication optimization is treated as a second performance pillar. veScale generalizes gradient bucketing beyond classic data parallelism so that many small tensors can be coalesced into larger collectives across all parallelism axes. It also introduces N-dimensional communication fusion: rather than performing an AllReduce separately for each mesh axis, it flattens or fuses axes into a single AllReduce across the mesh. The source text expresses the contrast schematically as $T_{vanilla} \propto 2SB \cdot N$ versus $T_{fused} \propto 2SB$, where $S$ is buffer size, $B$ bandwidth, and $N$ the number of mesh dimensions (Li et al., 5 Sep 2025).

The reported quantitative results emphasize both speed and code complexity. veScale is said to deliver up to $2.2\times$ end-to-end speedup over state-of-the-art training systems such as TorchTitan, reduce DTensor overhead by about $95\%$ with a $0.19$0 improvement, and cut code complexity by $0.19$1 in adapting a single-device model to 4D distributed training. The line-count comparison in the summary reports 63–100 lines of code for TorchTitan and 29–38 for veScale, with model code unchanged in the latter case (Li et al., 5 Sep 2025).

These claims position veScale against a common trade-off in distributed systems design: that higher-level, more debuggable programming models necessarily sacrifice performance relative to specialized runtimes. The veScale results are presented as evidence that eager-mode SPMD can remain competitive when metadata handling and collective execution are redesigned (Li et al., 5 Sep 2025).

5. veScale-FSDP and structure-aware sharding

veScale-FSDP is a redesigned Fully Sharded Data Parallel system within the veScale ecosystem. Its motivation is the observation that current FSDP systems struggle with structure-aware training methods such as block-wise quantized training and with non-element-wise optimizers such as Shampoo and Muon. According to the source material, the difficulty arises because fixed element-wise or row-wise sharding formats conflict with block-structured computations, while existing implementations also fall short in communication and memory efficiency at very large scale (Wang et al., 25 Feb 2026).

The central abstraction introduced by veScale-FSDP is RaggedShard, described as a general and flexible sharding format. RaggedShard supports customizable sharding granularity, including arbitrary block shapes and sizes rather than just elements or rows, and supports arbitrary distribution, including uneven numbers of blocks per device. Its stated purpose is to align sharding boundaries perfectly with block boundaries so that block-wise quantization and matrix optimizers preserve block atomicity (Wang et al., 25 Feb 2026).

RaggedShard is also described as composable with existing formats such as Shard(0), Shard(1), Replicate, and Partial, allowing integration with tensor parallelism, expert parallelism, and related distributed strategies. This composability is important because it means that FSDP is not treated as an isolated subsystem; rather, it can participate in multidimensional parallel training recipes without requiring model or optimizer code changes (Wang et al., 25 Feb 2026).

For block-wise quantization, the summaries state that RaggedShard ensures each quantization block lies entirely within one device’s local memory, so per-block statistics can be computed locally and extra communication is eliminated. For non-element-wise optimizers such as Muon and Shampoo, RaggedShard guarantees that matrix updates can be gathered and redistributed at matrix or block granularity while preserving SPMD programming through the DTensor API (Wang et al., 25 Feb 2026).

A misconception addressed implicitly by this design is that FSDP flexibility is inherently limited to element-wise or row-wise sharding. veScale-FSDP argues that the limitation belongs to current formats and planners, not to sharded data parallelism as such (Wang et al., 25 Feb 2026).

6. Planning, distributed buffers, and large-scale evaluation

veScale-FSDP pairs RaggedShard with a structure-aware planning algorithm for grouped communication. The planning problem is formulated over a set of tensors, each with a block size and number of elements distributed across devices, and seeks a global communication buffer of minimal per-device size under three constraints: no block split, contiguous placement for each tensor, and balanced per-device workload. The source material states that the problem is NP-hard, mapping to classic partitioning problems, so exact ILP methods are impractical at scale (Wang et al., 25 Feb 2026).

The practical solution is a polynomial-time, dynamic-programming-based heuristic that exploits regularities in transformer models, especially repeating block shapes. Tensors may be permuted by block size or default order; inter-tensor padding is batched between tensors rather than inserted within blocks; and buffer layout is aligned with collective-operation requirements such as NCCL alignment. The reported outcomes are minimal padding overhead, below $0.19$2 in most use cases, zero-copy communication after collectives, and balanced, aligned buffers that improve network efficiency (Wang et al., 25 Feb 2026).

A second systems abstraction is the Distributed Buffer, or DBuffer. It manages collective communication buffers for groups of DTensors and provides zero-copy pointer remapping into fixed buffer addresses, fused pre- and post-collective operations, and batched allocations to reduce fragmentation and improve determinism. In the presented design, DBuffer abstracts over device topologies and low-level collective details while serving as the execution substrate for structure-aware communication (Wang et al., 25 Feb 2026).

The evaluation claims are substantial. veScale-FSDP is reported to achieve 5–66% higher throughput and 16–30% lower peak memory usage than existing FSDP systems. The summary attributes these gains to optimized communication overlap, zero-copy grouping through DBuffer, custom sharding granularity that avoids padding and copying, deterministic batched allocation, and reduced mixed-precision duplicate buffers. Scalability is reported as linear to over 10,000 GPUs, with model scaling to 2.4T parameters on 1K GPUs and no performance drop in the cited experiments (Wang et al., 25 Feb 2026).

The baselines named in the comparison include DeepSpeed ZeRO, PyTorch FSDP1, PyTorch FSDP2, and Megatron-FSDP. Relative inefficiencies assigned to these systems include fragmented collectives, copy-in/copy-out overheads, and fixed block sizes that inflate padding. In this comparative frame, veScale-FSDP is not simply a feature extension but a redesign of the communication and placement substrate for sharded data parallelism (Wang et al., 25 Feb 2026).

7. Significance, scope, and interpretation

Taken together, veScale and veScale-FSDP define an approach to distributed training in which semantic consistency, programmability, and systems efficiency are treated as mutually reinforcing rather than mutually exclusive. The base veScale system emphasizes eager-mode SPMD with single-device-equivalent execution semantics and minimal model-code intrusion; veScale-FSDP extends the same general philosophy to sharded data parallelism under block-structured training regimes (Li et al., 5 Sep 2025, Wang et al., 25 Feb 2026).

Two themes recur across the material. First, model-system decoupling: model authors write standard PyTorch, while distributed execution is expressed through plans and tensor placements. Second, structure awareness: both RNG and sharding are defined with respect to global tensor semantics rather than device-local convenience. This suggests that veScale’s underlying systems thesis is that distributed correctness and distributed efficiency both depend on preserving the logical structure of tensor programs under partitioning (Li et al., 5 Sep 2025, Wang et al., 25 Feb 2026).

The reported reductions in code complexity, together with the preservation of single-device-equivalent results, are especially relevant for debugging and reproducibility. The summaries explicitly frame veScale as enabling interactive debugging in eager mode with standard Python and PyTorch tooling. A plausible implication is that veScale aims to narrow the gap between research ergonomics and production-scale execution, particularly in environments where iterative experimentation on complex parallelization schemes is routine (Li et al., 5 Sep 2025).

Within the current literature snapshot, veScale therefore denotes both a specific eager-mode SPMD training system and a broader distributed programming framework whose FSDP subsystem addresses block-wise quantization, non-element-wise optimizers, and communication efficiency at tens-of-thousands-of-GPU scale. Its distinguishing claims are not limited to throughput; they also include semantic reproducibility, plan-based parallelization, flexible sharding, and systems mechanisms designed to preserve these properties under large-scale execution (Li et al., 5 Sep 2025, Wang et al., 25 Feb 2026).