Lucia Training Framework (LTF)

• Lucia Training Framework (LTF) is a unified training layer that ensures numerically stable and efficient large-scale training on early-life AI accelerators.
• LTF employs proactive numerical validations and early collapse detection to catch divergence issues and optimize parallelism during training.
• Complementing the Lucia Training Platform (LTP), LTF focuses on training-side recovery and throughput optimization, enabling state-of-the-art performance in large MoE pretraining.

Searching arXiv for the specified paper and related context. arxiv_search.query({"^^^^^^^^6search_query6 arxiv_search.query({"^^^^^^^^6search_query6 An AI-Empowered Training Stack on Early-Life Hardware\"","^^^^^^^^6start6 Lucia Training Framework (LTF) is the framework layer of the SIGMA training stack for early-life AI accelerators. In the paper’s architecture, PRESERVED_PLACEHOLDER_^^^^6search_query6^^^^: Lucia Training Platform (LTP) provides platform-layer reliability, node validation, fault isolation, recovery, and cluster operations, while LTF provides the framework-layer mechanisms required for numerically stable and efficient large-scale training on top of that platform. Built on Megatron-LM, LTF is described as a “unified and extendable training framework” for stable, efficient large-scale training on early-life hardware, with the practical division that LTP keeps the cluster alive and recoverable, whereas LTF keeps the training run correct, stable, and fast (&&&^^^^6search_query6^^^^&&&).

^^^^6id:(Qu et al., 15 Dec 2025)6^^^^. Architectural position within SIGMA

The paper presents SIGMA as a two-layer stack that “collapses the conventional multi-layer AI software stack into two cleanly separated layers.” Within that decomposition, LTF is explicitly the framework layer, not the platform layer. Its scope is training stability and efficiency; LTP remains responsible for cluster-level reliability, node repair workflows, and operational recovery (&&&^^^^6search_query6^^^^&&&).

This division is operationally important. LTF assumes that healthy resources can be allocated by LTP and then focuses on validating kernels and numerics, detecting impending model collapse, tuning parallelism, removing throughput noise, and making a large MoE run converge. The paper therefore does not present LTF as a standalone scheduler or orchestration substrate. A common misconception is to treat SIGMA’s framework and platform functions as interchangeable. The paper rejects that view by separating cluster survivability from training correctness and throughput optimization.

A plausible implication is that the framework/platform boundary is itself part of the design methodology: platform-layer fault tolerance is necessary but not sufficient for successful pretraining on early-life hardware, because model-in-the-loop failures can persist even after cluster operations become dependable.

^^^^6start6^^^^. Problem formulation and motivation

The rationale for LTF is a three-way problem associated with large-scale training on early-life AI accelerators. First, reliability at the platform level is poor: jobs fail often, failures are not always well classified, and recovery can take hours or days. Second, numerical stability at the framework and model level is fragile: vendor backends, new kernels, and MoE training can silently diverge or accumulate error until the loss collapses. Third, efficiency and parallelism are difficult to optimize because the best TP/EP/PP/MBS configuration depends on topology, model shape, and evolving hardware/software behavior, while local stragglers and noisy subsystems can produce large throughput swings (&&&^^^^6search_query6^^^^&&&).

LTP addressed the first problem. LTF was needed because, even when the platform was dependable enough to keep jobs running, the training stack still had to validate numerical correctness, catch collapse early, choose a good parallelism recipe, eliminate stragglers, and sustain high MFU on immature hardware. The paper therefore frames LTF as a response to the residual failure modes that remain after cluster availability is improved.

The specific numerical issues reported in the software stack illustrate why framework-level intervention was required. The paper identifies a flash-attention gradient computation error with a vendor-specific backend, diverged GEMM results for equal tiles in BLAS, grouped GEMM error in some ranks with expert parallelism enabled, and accumulated gradient error in attention k-layernorm. These are presented as errors that may be invisible in small tests but fatal in long pretraining runs. LTF’s response is proactive numerical validation before training rather than post hoc debugging after collapse (&&&^^^^6search_query6^^^^&&&).

^^^^6max_results6^^^^. Stability mechanisms

LTF organizes its stability work around proactive numerical validation and early training collapse detection. Before each training session, it runs a numerical test pipeline over the components used by the workload. The expected behavior is that outputs and gradients on new hardware should closely match those on established hardware, using both random and real inputs and weights, while also comparing a few optimizer steps to expose accumulation errors. This validation is performed at the module level (&&&^^^^6search_query6^^^^&&&).

The paper reports the following module-level cosine similarities between software stacks after ^^^^6id:(Qu et al., 15 Dec 2025)6search_query6^^^^ optimizer steps, with results below PRESERVED_PLACEHOLDER_^^^^6id:(Qu et al., 15 Dec 2025)6^^^^ treated as abnormal:

Module	Outputs	Parameters / Gradients
Attention	^^^^6search_query6^^^^.9998	Parameters: ^^^^6search_query6^^^^.^^^^6id:(Qu et al., 15 Dec 2025)66start66 Gradients: ^^^^6search_query6^^^^.^^^^6all:\6max_results6id:(Qu et al., 15 Dec 2025)6all:\6
Bias-Dropout-Add	^^^^6search_query6^^^^.9^^^^6max_results6search_query6	Gradients: ^^^^6search_query6^^^^.9^^^^6search_query6search_query6search_query6
Embedding	^^^^6search_query6^^^^.899^^^^6all:\6	Parameters: ^^^^6search_query6^^^^.999^^^^6max_results6 Gradients: ^^^^6search_query6^^^^.9^^^^6id:(Qu et al., 15 Dec 2025)6search_query6start6
MoE (EP=8)	^^^^6search_query6^^^^.9998	Parameters: ^^^^6search_query6^^^^.999^^^^6search_query6 Gradients: ^^^^6search_query6^^^^.9^^^^6search_query6max_results6search_query6

Using this preflight procedure, the framework uncovered a one-time error in qk-layernorm, a ^^^^6id:(Qu et al., 15 Dec 2025)6search_query6^^^^-step accumulated error in the output linear layer bias, a ^^^^6id:(Qu et al., 15 Dec 2025)6search_query6^^^^-step accumulated error in the MoE router, and RNG implementation differences in bias-drop-add and embedding. The significance of this component is methodological: the paper portrays LTF as converting “mysterious convergence failure” into a compatibility test performed before large-scale execution (&&&^^^^6search_query6^^^^&&&).

LTF’s second stability mechanism is a parameter-norm heuristic for MoE collapse detection. The key observation is that, in successful pretraining, the average norm of MoE parameters tends to increase with layer depth, whereas a collapsed model exhibits a “valley phenomenon” in the middle layers. No explicit loss function is introduced; the method is a robustness heuristic based on layerwise norm trajectories. In the reported unstable case, collapse was typically detected after PRESERVED_PLACEHOLDER_^^^^6start6^^^^ steps, whereas the LTF-enabled run detected a problem at PRESERVED_PLACEHOLDER_^^^^6max_results6^^^^ steps even though the actual collapse point was around PRESERVED_PLACEHOLDER_^^^^6search_query6^^^^ steps. This suggests that LTF treats collapse detection as an early-warning problem rather than a terminal-failure diagnosis problem.

The paper also attributes stability gains to a concrete initialization choice. It contrasts Megatron’s scaled output layer initialization with Gaussian initialization:

PRESERVED_PLACEHOLDER_^^^^6all:\6^^^^

$$\text{Gaussian Initialization: } \sigma \text{ for all parameters.}$$

According to the reported results, Gaussian initialization produced smoother training loss, fewer gradient-norm spikes, and MoE parameter norms that matched successful open-source models within ^^^^6all:\6^^^^,^^^^6search_query6search_query6search_query6^^^^ iterations. The paper presents this as an engineering choice that improved stability signals rather than as a general theorem about initialization.

^^^^6search_query6^^^^. Efficiency mechanisms and throughput control

LTF’s efficiency work combines AI-assisted noise diagnosis, parallelism-search pruning, and a sequence of fused or runtime-level optimizations. The paper states that throughput on early-life hardware is not determined only by FLOPs. Local disruptions such as page cache churn, Python GC, memory copy behavior, and topology-sensitive sharding can create stragglers and large MFU swings (&&&^^^^6search_query6^^^^&&&).

To identify such issues, LTF uses LLMs as assistants for throughput-noise analysis. Two concrete sources of noise were discovered: lazy OS page cache invalidation and Python garbage collection. The corresponding mitigations were explicit dropping of page cache before each job run and disabling automatic Python GC in favor of manual GC at fixed intervals. The paper characterizes these as low-level but important controls because they remove non-deterministic rank-to-rank stalls.

The effect on throughput stability is reported quantitatively. After fixing the two stragglers, throughput improved from

$$\text{Before: } \min/\text{peak}=29.47\%, \quad \text{avg}/\text{peak}=85.46\%, \quad \text{std}=22.80$$

to

$$\text{After: } \min/\text{peak}=55.05\%, \quad \text{avg}/\text{peak}=98.11\%, \quad \text{std}=1.98$$

The paper also describes pruned parallelism tuning rather than brute-force search. LTF performs a small-scale exhaustive search on ^^^^6id:(Qu et al., 15 Dec 2025)66^^^^ accelerators across EP, TP, PP, and MBS in order to discover topology and model constraints, and then searches a refined space at scale. Two empirical rules are given for Sigma-MoE on the target stack: $TP=1$ and PRESERVED_PLACEHOLDER_^^^^6id:(Qu et al., 15 Dec 2025)6search_query6^^^^ best matched the current topology and communication behavior; and there is a tradeoff between PRESERVED_PLACEHOLDER_^^^^6id:(Qu et al., 15 Dec 2025)6id:(Qu et al., 15 Dec 2025)6^^^^ and PRESERVED_PLACEHOLDER_^^^^6id:(Qu et al., 15 Dec 2025)6start6^^^^, where more PRESERVED_PLACEHOLDER_^^^^6id:(Qu et al., 15 Dec 2025)6max_results6^^^^ reduces memory pressure while larger PRESERVED_PLACEHOLDER_^^^^6id:(Qu et al., 15 Dec 2025)6search_query6^^^^ can improve end-to-end performance by up to PRESERVED_PLACEHOLDER_^^^^6id:(Qu et al., 15 Dec 2025)6all:\6^^^^ (&&&^^^^6search_query6^^^^&&&).

Additional throughput-oriented engineering choices named in the paper include fused attention with vendor backend, fused Bias-SwiGLU, fused gradient accumulation, fused grouped MLP GEMM with vendor backend, fused RoPE, vendor-specific runtime tuning for D^^^^6start6^^^^H/H^^^^6start6 copies, more balanced pipeline stage assignment, fused MoE token permute/unpermute, PRESERVED_PLACEHOLDER_^^^^6id:(Qu et al., 15 Dec 2025)66^^^^, selective MoE layer recomputation, and prioritization of more balanced-routing MoE layers for recompute. Taken together, these choices are presented as a framework-level program for removing communication/computation imbalance and suppressing local noise.

^^^^6all:\6^^^^. Interaction with LTP and training-side recovery

The paper is explicit that LTF does not subsume LTP. The interaction is described as a sequence: LTP validates and allocates healthy nodes; LTF runs numerical prechecks on the selected software/hardware stack; LTF selects a pruned parallelism recipe based on small-scale experimentation; training proceeds while LTF monitors for instability and throughput anomalies; and, if a collapse pattern appears, LTF can stop the bad recipe early and revise the training setup (&&&^^^^6search_query6^^^^&&&).

This operational logic clarifies the meaning of recovery in the framework layer. LTF’s recovery is mainly training-side recovery. It does not repair hardware or perform cluster orchestration; instead, it prevents wasted compute by catching software and model issues early and adjusting the recipe. A common misunderstanding is to interpret any large-scale training stack as a monolithic system that handles node failure, scheduling, numerics, and model dynamics uniformly. The SIGMA design, as described in the paper, rejects that monolith: LTP handles healthy resource provisioning and recoverability, while LTF handles correctness, stability, and efficiency once those resources are available.

A plausible implication is that this separation makes fault attribution cleaner. When a run becomes unstable, the framework can evaluate numerics, collapse signals, or parallelism choices without conflating them with node-health uncertainty already absorbed by the platform layer.

6. Sigma-MoE training results and reported significance

LTF enabled pretraining of Sigma-MoE, a ^^^^6start6search_query6search_query6^^^^B-parameter MoE model with ^^^^6start6search_query6^^^^B activated parameters per token. The model is described as having 6^^^^6max_results6^^^^ blocks, each block containing a GQA layer followed by an MoE layer; each MoE layer has 96 experts; and 8 experts are activated per token. The training run used ^^^^6start6^^^^,^^^^6search_query6search_query6 accelerators and lasted 7^^^^6all:\6^^^^ days (&&&^^^^6search_query6^^^^&&&).

The reported outcomes are central to the framework’s evaluation. During the 7^^^^6all:\6^^^^-day run there was only one stability incident, and that incident was promptly resolved by improving the MoE load-balancing loss function. Final training achieved ^^^^6start6id:(Qu et al., 15 Dec 2025)6^^^^.^^^^6search_query6 MFU, and the system reported state-of-the-art downstream accuracy. The paper further states that the model was “comparable with DeepSeek V^^^^6start6^^^^.”

The optimization trajectory is reported as a stepwise increase in MFU from 9.86% baseline to ^^^^6start6id:(Qu et al., 15 Dec 2025)6^^^^.^^^^6search_query6 final:

Optimization stage	MFU
initial baseline	9.86%
TP^^^^6id:(Qu et al., 15 Dec 2025)6^^^^-CP^^^^6id:(Qu et al., 15 Dec 2025)6	^^^^6id:(Qu et al., 15 Dec 2025)6search_query6^^^^.^^^^6max_results6search_query6
larger global batch size	^^^^6id:(Qu et al., 15 Dec 2025)6max_results6^^^^.^^^^6search_query6search_query6 then ^^^^6id:(Qu et al., 15 Dec 2025)6max_results6^^^^.^^^^6all:\6
fused ops and runtime tuning	^^^^6id:(Qu et al., 15 Dec 2025)6max_results6^^^^.8^^^^6start6
more balanced PP	^^^^6id:(Qu et al., 15 Dec 2025)66^^^^.^^^^6search_query6search_query6
further layer count / stage balancing	^^^^6id:(Qu et al., 15 Dec 2025)6^^^^8.78%
fused MoE token permute/unpermute	^^^^6start6search_query6^^^^.^^^^6search_query6id:(Qu et al., 15 Dec 2025)6
VPP=^^^^6start6^^^^ + selective recompute + routing-aware recompute	^^^^6start6id:(Qu et al., 15 Dec 2025)6^^^^.^^^^6search_query6

The downstream benchmark values reported for the pretraining run are MMLU: 8^^^^6search_query6^^^^.^^^^6all:\6 MMLU-Pro: ^^^^6all:\6^^^^9.8, BBH: 8^^^^6start6^^^^.^^^^6max_results6 GSM8K: 8^^^^6search_query6^^^^.^^^^6id:(Qu et al., 15 Dec 2025)6 MATH: ^^^^6all:\6search_query6^^^^.6, and HumanEval pass@^^^^6id:(Qu et al., 15 Dec 2025)6^^^^: ^^^^6all:\6^^^^7.9. The paper compares Sigma-MoE-Base against DeepSeek-V^^^^6start6^^^^-Base, Qwen^^^^6start6^^^^.^^^^6all:\6 and LLaMA-^^^^6max_results6^^^^.^^^^6id:(Qu et al., 15 Dec 2025)6 stating competitive results, especially on reasoning and coding tasks (&&&^^^^6search_query6^^^^&&&).

The paper’s broader interpretation is that early-life AI accelerators are not merely slower or less mature versions of standard GPUs; they constitute a different systems problem involving immature kernels and vendor backends, evolving numerical behavior, shifting parallelism optima, and noisy performance at scale. On that view, LTF matters because it contributes a stable Megatron-LM-based framework for early-life hardware, pretraining-time numerical validation, early collapse detection for MoE models, practical parallelism tuning under topology and memory constraints, throughput-noise diagnosis and mitigation, and a successful ^^^^6start6search_query6search_query6^^^^B MoE training run on ^^^^6start6^^^^,^^^^6search_query6search_query6 accelerators. This suggests that, within SIGMA, LTF is the component that converts platform reliability into training viability.