Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet

Abstract: Recently, there has been increased interest in globally distributed training, which has the promise to both reduce training costs and democratize participation in building large-scale foundation models. However, existing models trained in a globally distributed manner are relatively small in scale and have only been trained with whitelisted participants. Therefore, they do not yet realize the full promise of democratized participation. In this report, we describe Covenant-72B, an LLM produced by the largest collaborative globally distributed pre-training run (in terms of both compute and model scale), which simultaneously allowed open, permissionless participation supported by a live blockchain protocol. We utilized a state-of-the-art communication-efficient optimizer, SparseLoCo, supporting dynamic participation with peers joining and leaving freely. Our model, pre-trained on approximately 1.1T tokens, performs competitively with fully centralized models pre-trained on similar or higher compute budgets, demonstrating that fully democratized, non-whitelisted participation is not only feasible, but can be achieved at unprecedented scale for a globally distributed pre-training run.

• The paper introduces a novel permissionless protocol that pre-trains a 72B LLM using trustless, globally distributed peers with blockchain-based incentivization.
• It employs dynamic FSDP and SparseLoCo techniques with aggressive quantized pseudo-gradients to achieve 94.5% hardware utilization under commodity Internet constraints.
• Benchmark results demonstrate competitive zero-shot and chat model performance, validating decentralized training as a viable alternative to centralized approaches.

Covenant-72B: Permissionless Pre-Training of a 72B LLM Over the Internet

Introduction

The paper "Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet" (2603.08163) presents the first demonstration of a competitive large-scale LLM trained over globally distributed, permissionless, untrusted infrastructure. The work introduces and evaluates Covenant-72B, a 72-billion-parameter, dense decoder-only Transformer pre-trained collaboratively across a dynamic set of peers connected via commodity Internet, orchestrated by a blockchain-based incentivization and validation protocol and enabled by a highly communication-efficient optimization protocol.

Protocols for Trustless, Distributed LLM Pre-Training

Covenant-72B’s core contribution is a protocol stack facilitating scalable, robust, and communication-efficient pre-training among trustless, permissionless compute providers. Each peer hosts a full SparseLoCo replica, using dynamic FSDP for local intra-node model sharding. Pseudo-gradients are computed after local steps, aggressively chunk-topk-sparsified, two-bit quantized, and error-feedback compensated prior to all-gather aggregation across selected peers.

Figure 1: Each peer executes a SparseLoCo replica, sharding states across 8xB200 GPUs with dynamic offloading/swapping of optimizer and error-feedback buffers for efficient memory reuse and compressed communication.

Importantly, selection and incentivization of contributions are performed by Gauntlet, a protocol operating atop the Bittensor blockchain. Contributions are validated, scored by localized loss reduction, and ranked, with normalization to suppress extremely large updates. Assignment of training data is randomized and enforced by cross-verifying the impact of submitted pseudo-gradients on validation shards, robustly mitigating Sybil and copy attacks. The protocol's asynchrony and open validation enable dynamic peer churn and enforce real-world security and liveness constraints absent in previous curated/whitelisted collaborative efforts.

Systems Design and Scheduling

The parallelism and communication schedule combine dynamic FSDP intra-peer sharding, peer-level SparseLoCo outer-loop with fixed per-round compute windows, and object storage-based pseudo-gradient exchange. The compute–communication pipeline overlaps outer-loop state offloading and compressed pseudo-gradient communication for maximal hardware utilization, even under typical Internet bandwidth constraints (500 Mbps down, 110 Mbps up).

Figure 2: Timeline showing training round breakdowns: black = compute, red = communication/synchronization. With a 7.2× larger model, Covenant-72B sustains high hardware utilization and low idle time compared to INTELLECT-1’s previous best decentralized run.

Training Setup and Optimization Dynamics

Covenant-72B adopts a LLaMA-3-style dense architecture: 80 layers, 8192 width, GQA, rotary position embeddings (RoPE, θ=500,000), and the Gemma 3 tokenizer (262,208 tokens). Training covers ~1.1T tokens: a main webtext phase (DCLM), followed by an annealing phase on curated high-quality and replay data. Each peer employs SparseLoCo (AdamW inner optimizer, batch size 192, sequence length 2048, H=30 inner steps, error-feedback decay β=0.95, outer LR α=1, chunk-topk 64/4096, 2-bit quantization: 146× comms compression).

The inner AdamW schedule utilizes linear warmup, cosine decay, and a mid-training flatten window reflecting dynamic peer population. An annealing phase on target datasets is performed with a tailored schedule for downstream retention and SFT readiness.

Figure 3: Left: Pre-training inner learning rate schedule with warmup, extended decay, flattening for participation/adaptive length, and final anneal. Right: Two-stage SFT schedule for context extension and replay.

Benchmarking and Downstream Performance

Covenant-72B establishes that permissionless, non-whitelisted collaborative Internet-scale training can achieve competitive performance with best-in-class, centralized datacenter training runs. The model achieves strong results on zero-shot language understanding/QA benchmarks (ARC-Challenge 56.8, MMLU 67.1, WinoGrande 75.9) that are either on-par with or exceed closed-cluster models of similar scale and superior to all previous decentralized runs, including INTELLECT-1 (10B), Psyche Consilience (40B), and LLM360 K2 (65B) under comparable data and compute regimes.

Notably, the protocol maintains robust model quality in the presence of dynamic peer churn and sharply limited bandwidth, without restricting participation to a curated node set. The adaptive Gauntlet selection ensures an average of 16.9 contributing peers per round (capped at 20), with over 70 unique peers participating through the run.

Figure 4: The evolution of selected contributing peers per aggregation round, demonstrating robust dynamic participation and high liveness.

Participation is further dissected by segregating all active submitters versus the subset accepted for pseudo-gradient aggregation, giving insight into the efficacy of trustless filtering and enforcement.

Figure 5: Active (red) and selected/contributing (black) peers per round; substantial submission filtering ensures aggregation security despite open-access incentives.

Communication Efficiency and Resource Utilization

Crucially, the communication pipeline maintains ${\sim}94.5\%$ hardware utilization at 72B parameters under commodity Internet—incurring only ~70 seconds communication overhead per ~20 minute compute window. This is a marked reduction compared to INTELLECT-1 (8.3 min per round, 10B), and comparable to the most communication-efficient published local-update protocols at lower scale [diloco, sparseloco].

Supervised Fine-Tuning and Chat Model Results

Post-pre-training, Covenant-72B undergoes a two-stage SFT regime, extending context from 2k to 8k tokens, incorporating replay for catastrophic forgetting mitigation. On five- and multi-shot downstream tasks, the chat-tuned model (Covenant-72B-Chat) ranks competitively against centralized SFT baselines (LLaMA-2-70B-Chat, K2-Chat), attaining highest or near-highest results on instruction following (IFEval 64.7), math (MATH 26.3), and robust comprehension (MMLU 67.4).

Theoretical and Practical Implications

This work establishes, for the first time, that robust, highly performant, and competitively scaled LLMs can be collaboratively pre-trained over globally distributed, untrusted, and bandwidth-constrained hardware via permissionless protocols. It demonstrates that key obstacles—Sybil and copy attacks, straggler handling, peer churn, bandwidth bottlenecks, selection robustness, and optimization at scale—can be mitigated by jointly optimizing incentivization, trustless filtering, aggressive quantized sparsification, and chunk-based pseudo-gradient top-k.

These results contradict prior assumptions that such outcomes are tractable only with whitelisted or highly synchronized hardware and centralized trust. Architecturally, the work also suggests design adaptations (dynamic FSDP, state offloading/swap) for practical real-world distributed infrastructure.

Outlook and Future Directions

Future research will need to address scaling to larger, more heterogeneous and less resource-homogeneous peer sets, increasing adversarial robustness, alternative incentivization schemes outside blockchain, and federated extensions under high fault tolerance constraints. This paradigm has critical implications for resource democratization—potentially shifting large-scale LLMs from the domain of large resource-holding institutions to wider public collaboration.

Conclusion

Covenant-72B (2603.08163) provides a rigorous demonstration that globally distributed, permissionless LLM pre-training at high scale is practical, robust, and capable of yielding competitive downstream performance. The combination of communication-efficient local update protocols (SparseLoCo), blockchain-based peer validation and incentivization (Gauntlet), and careful systems scheduling extends the feasible frontiers of collaborative AI infrastructure and points toward a new standard for large model development and training.