Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

Abstract: LLM pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over $1.5M, and reproducing SmolLM3-3B needs over $700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than $6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about $4.4K, less than $5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.

• The paper demonstrates training of a 2B-parameter language model using RTX 5090 GPUs, successfully, costing approximately $5090 with comparable benchmark performance.
• Puro-2B includes an open-source, cost-effective approach to language modeling, supported by detailed sourcing data and optimized framework
• The authors of the language modelling were able to improve quality while also improving speed with their training system.

Research objective and contribution

“Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090$” (2608.27370) addresses a practical gap in open language-model research: releasing weights and code does not make a model genuinely reproducible if the compute required to rerun pretraining remains inaccessible. The paper presents a complete, open, from-scratch pretraining recipe for a roughly 2B-parameter dense decoder-only Transformer, trained on approximately 1.4T tokens using consumer-grade RTX 5090 GPUs. Its central claim is that a compact base model with competitive benchmark performance can be trained for a few thousand dollars of accelerator cost, provided that hardware, numerical precision, optimization, data selection, data ordering, and cost accounting are designed jointly.

The contribution is therefore broader than a model checkpoint. The authors release model weights, intermediate checkpoints, data manifests and materialized components, training configurations, preprocessing code, training code, and evaluation artifacts under Apache 2.0, subject to the licenses of upstream datasets. This positioning follows the open-recipe philosophy exemplified by OLMo (Grattafiori et al., 2024), but adds an explicit accessibility criterion: the full production run should be feasible on comparatively inexpensive hardware.

The paper reports two principal production points. A uniform-data variant costs approximately $4.37K and achieves an aggregate score of 55.14 across the paper’s 15-task evaluation used for cost-performance comparisons. The canonical curriculum-based variant costs approximately $6.89K and reaches 57.81. Under the paper’s accounting protocol, the $4.4K model exceeds Qwen2-1.5B, while the $6.9K model approaches Qwen2.5-1.5B. The comparison is meaningful as a reproducibility-cost analysis, although it is not an estimate of total project expenditure.

Figure 1: Model performance versus reproduction cost under the paper’s rental-equivalent accounting protocol.

End-to-end system design

Puro-2B adopts the Qwen3-1.7B architectural configuration with untied input embeddings and output projection, resulting in approximately 2B parameters. Training uses sequence length 4,096, global batch size 1,536, and micro-batch size 2. The production run is divided into two stages:

Stage	Tokens	GPUs	Parallelism	Throughput
Phase 1	438.84B	24 RTX 5090	PP 2, DP 12	238 TFLOP/s/GPU
Phase 2	959.99B	96 RTX 5090	PP 4, DP 24	192 TFLOP/s/GPU
Total	1.40T	—	—	22,514 active GPU-hours

Phase 1 establishes broad linguistic coverage and uses a power-decay schedule. Phase 2 increases the share of mathematics, code, Chinese, and instruction-formatted material and uses a linear decay schedule. A 43.9B-token transition interpolates between Phase 1 replay and Phase 2 data. The final training traces reach validation losses of 2.730 after Phase 1 and 2.488 after Phase 2.

The infrastructure design is unusually important to the paper’s argument. RTX 5090 offers substantially lower absolute throughput and memory capacity than data-center accelerators, and its 32GB memory and lack of NVLink impose constraints on parallelism and optimizer state placement. The authors enable PCIe peer-to-peer communication through modified drivers and platform configuration, increasing one-way bandwidth from 31.5 to 56 GB/s, bidirectional bandwidth from 32 to 111 GB/s, and eight-GPU AllReduce bus bandwidth from 14.75 to 27.34 GB/s. Enabling GPUDirect RDMA increases 24-GPU AllReduce bandwidth from approximately 8.87 to 19.93 GB/s.

These modifications are unsupported by NVIDIA and are hardware-topology dependent. The paper explicitly cautions that P2P can be counterproductive on congested PCIe root complexes and that the reported configuration is not a generally portable property of RTX 5090 systems. Within the tested environment, however, the resulting mixed-precision MFU is approximately 73%, demonstrating that consumer hardware can sustain nontrivial distributed pretraining when communication and memory placement are treated as first-class design variables.

The parallel strategy avoids tensor parallelism because its frequent intra-layer collectives are poorly matched to PCIe connectivity. Instead, the authors combine pipeline and data parallelism, use topology-aware rank ordering, adjust micro-batch size through kernel benchmarking, and assign fewer Transformer layers to the pipeline stage containing the computationally heavy embedding and language-model head. A memory-aware placement strategy then distributes Muon and AdamW state across devices.

Blockwise FP8 training

The production system uses blockwise FP8 from random initialization, without a BF16 warm-up or later precision switch. Linear-layer forward, backward, and weight-gradient GEMMs use E4M3 operands, while attention, numerically sensitive operations, master weights, and optimizer states remain in BF16 or FP32. Activations and activation gradients use groups of 128 values, and weights use $128 \times 128$ blocks with online scaling. This distinction matters: FP8 is used as a compute and activation-storage format rather than as the persistent representation of the complete model.

The paper reports a validation-loss penalty of only 0.0031–0.0039 relative to BF16 across five tested scales. A shared-shape fit translates this gap into 98.0% BF16-equivalent compute retention. At the approximately 1.7B scale, FP8 increases median throughput by 1.36x; after accounting for the quality penalty, the estimated net speedup is 1.34x, corresponding to 25.2% fewer GPU-hours at matched quality.

This result supports the paper’s numerical-efficiency claim, but its scope is narrower than a universal FP8 result. The throughput measurement is a 1.7B proxy for the 2B production model, and the quality-adjusted GPU-hour comparison is extrapolated from a scaling ladder rather than obtained from a full-horizon, matched BF16 production run. The evidence nevertheless indicates that blockwise FP8 can provide a favorable quality-throughput trade-off in this model and hardware regime.

MuonH and effective learning-rate control

The optimization recipe applies MuonH to selected attention and MLP matrices and AdamW to embeddings, normalization layers, the language-model head, and remaining parameters. MuonH combines Muon-style updates with a Hyperball constraint: each wrapped matrix is projected back to its initial Frobenius radius after every update. The resulting update magnitude is expressed relative to a fixed matrix scale, making the effective learning rate an explicit schedule rather than an emergent consequence of changing weight and update norms.

The paper’s diagnostic comparison is particularly informative. In 170M-parameter BF16 experiments, ordinary Muon receives the same scalar learning-rate schedule used by MuonH but induces a rapidly decaying effective learning rate. It achieves lower validation loss early but ends at 3.073. MuonH follows the prescribed effective-learning-rate trajectory and ends at 3.029. When ordinary Muon is adjusted online to match MuonH’s effective learning-rate trace, it reaches 3.030.

Figure 2: MuonH, effective-learning-rate-aligned Muon, and ordinary-learning-rate Muon exhibit similar final behavior when their induced effective learning-rate schedules are aligned.

The near-equivalence between MuonH and effective-LR-aligned Muon yields an important, somewhat counterintuitive conclusion: much of MuonH’s observed advantage in this diagnostic is attributable to effective-learning-rate control rather than solely to radial projection. The result does not establish that Hyperball projection is unnecessary in general. It shows instead that comparing optimizers through scalar learning rates can be misleading when the parameterization is approximately scale invariant.

The authors further fit a Multi-Power Law model to validation trajectories. Replacing ordinary scalar learning rate with induced effective learning rate reduces held-out RMSE from 0.0265 to 0.0210 in the reported diagnostic. This supports effective learning rate as a more predictive schedule descriptor for Muon-like optimization, though the diagnostic uses a small model and only two runs.

The production schedule reflects this analysis. Phase 1 uses a power-decay schedule whose MuonH effective learning rate decreases from approximately $5 \times 10^{-2}$ to $1.04 \times 10^{-2}$. Phase 2 continues from that value with a long linear decay toward $10^{-5}$. In WSD sweeps, larger effective peaks require longer decay ratios, and longer training horizons also shift the competitive region toward longer decay. The authors use two-anchor MPL fits as a low-compute heuristic for estimating decay preferences, while acknowledging that this is not a certificate of optimality.

Data selection and curriculum construction

The data recipe separates source selection from shard reconstruction. Candidate datasets are evaluated through controlled proxy continuation experiments using a shared Qwen3-0.6B checkpoint, a fixed continuation schedule, and a 15-benchmark evaluation suite. Large scored datasets are sampled at approximately the 0th, 25th, 50th, and 75th score quantiles; smaller or unscored datasets receive random 4B-token slices; other datasets are omitted from proxy evaluation.

Figure 3: Proxy benchmarking evaluates candidate source slices under a common checkpoint, continuation schedule, and downstream suite.

The resulting benchmark vectors are treated as capability profiles rather than as a single global data-quality score. This is a defensible design choice because source-specific quality scores are not directly comparable across datasets. The final mixture is selected according to target capability axes and token constraints. Phase 1 emphasizes English at 73.2% of its materialized tokens. Phase 2 reduces English to 59.4% and increases mathematics to 18.3%, code to 11.5%, Chinese to 9.4%, and instruction-formatted data to 1.3%.

Figure 4: Phase 1 emphasizes broad coverage, whereas Phase 2 allocates more capacity to mathematics, code, Chinese, and instruction-formatted data.

The data curriculum is source-local rather than globally ranked. Within each scored source, examples are ordered from lower to higher configured quality; unscored sources use fixed random order. Each component is partitioned into normalized within-source rank intervals, and corresponding intervals are combined into approximately 2.5B-token buckets. This preserves the intended cross-component mixture while moving each scored source toward its preferred examples later in training.

The optimization and data ordering are coupled through Curriculum Model Averaging. The authors argue that a conventional terminal learning-rate decay can waste high-quality examples if they are encountered when updates are already small. The canonical run therefore follows the scheduled Phase 2 trajectory, resumes from a late checkpoint with constant learning rate, and averages six subsequent checkpoints.

The ablation results support the joint design. Curriculum ordering improves the endpoint aggregate from 55.99 to 57.17 without averaging and from 55.57 to 57.18 when comparing averaged decay trajectories. However, averaging alone is not uniformly beneficial: it reduces the uniform endpoint by 0.42 points and changes the curriculum endpoint by only 0.01 points without constant-LR continuation. The selected constant-LR continuation from step 218,000 reaches 57.81, exceeding the corresponding step-215,000 continuation by 1.01 points and the curriculum endpoint without constant-LR continuation by 0.63 points.

These results contradict a simplistic interpretation that checkpoint averaging is independently responsible for the gain. The strongest evidence instead favors an interaction among within-source curriculum ordering, late constant-LR continuation, and checkpoint averaging. The fitted cost equivalent of the canonical recipe is approximately $16.55K on the uniform scaling curve, or 2.40x its measured $6.89K reproduction cost. Because the comparison changes several ingredients simultaneously, this is a recipe-level estimate, not an isolated causal effect of curriculum ordering.

Capability and cost results

The evaluation uses deterministic OpenCompass configurations across mathematics, code, reasoning, and knowledge tasks. Generation-based evaluation is used for GSM8K, MATH, sanitized-MBPP, HumanEval, MMLU-Pro, and BBH; the remaining tasks use perplexity-based candidate ranking. The reported aggregate is an unweighted arithmetic mean, and the cost-performance figure uses all 15 benchmarks.

Model	Reproduction cost	15-task average
Puro-2B, uniform	$4.37K	55.14
Puro-2B, canonical	$6.89K	57.81
Qwen2-1.5B	$84.30K estimated	55.14
Qwen2.5-1.5B	$216.78K estimated	60.73
SmolLM3-3B	$718.85K estimated	65.85
Yulan-Mini-2.4B	$48.46K estimated	58.80

The cost comparisons require careful interpretation. For Puro-2B, cost is based on measured active-training GPU-hours and a stated RTX 5090 rental-equivalent rate. For many comparison models, cost is inferred from token counts, estimated MFU, nominal FLOPs, and reference accelerator prices. The paper’s figures therefore compare heterogeneous evidence sources, and most comparator costs are lower-bound or idealized estimates rather than audited invoices.

On the four mathematics and code benchmarks, canonical Puro-2B obtains an average of 43.50, exceeding Qwen2-1.5B by 3.21 points and falling 4.02 points below Qwen2.5-1.5B. Its score is below SmolLM3-3B, Yulan-Mini-2.4B, and MobileLLM-R1-950M on this capability grouping, indicating that its cost advantage does not translate into dominance over all open-recipe baselines.

On the eleven reasoning and knowledge benchmarks, Puro-2B reaches 63.02, exceeding Qwen2-1.5B by 2.48 points and approaching Qwen2.5-1.5B within 2.51 points. It is 0.11 points below Instella-3B and 5.33 points below SmolLM3-3B. Thus, the strongest empirical claim is not that Puro-2B is the best compact model, but that it is competitive with substantially more expensive open-recipe systems under a transparent, reproducible cost boundary.

The paper’s Puro Cost Scaling Law fits a recipe-specific logarithmic relationship between performance and incremental Phase 2 cost while holding Phase 1 cost fixed at $1.84K. The fitted curve places Qwen2-1.5B-level aggregate performance at approximately $4.4K. This law is explicitly a fixed-architecture, fixed-recipe scale-down relationship; it should not be interpreted as a universal scaling law across model families or hardware platforms.

Post-training persistence of pretraining choices

The post-training experiments test whether differences between uniform and curriculum pretraining remain after identical SFT. This is important because similar pretraining losses do not imply equivalent representations or transfer behavior.

In focused mathematics SFT, curriculum initialization reaches 68.66% mean GSM8K accuracy versus 66.89% for uniform initialization, a gain of 1.77 percentage points. In the larger mathematics setup, the corresponding scores are 76.12% and 74.10%, a 2.02-point gain. Curriculum is ahead in all three repetitions in the scaled experiment.

Figure 5: Curriculum initialization improves GSM8K accuracy under both focused and scaled mathematics SFT.

The broad instruction experiment extends the comparison beyond mathematics. Curriculum initialization raises the 15-task macro-average from 54.99% to 56.58%, a 1.59-point improvement, and improves 13 of 15 component evaluations. Across additional IFEval, BBH, and MMLU-Pro evaluations, the curriculum variant improves 15 of 18 reported comparisons.

These results imply that the curriculum endpoint contains transferable differences rather than merely a transient advantage in the pretraining evaluation checkpoint. However, the experiments compare the complete curriculum/CMA recipe against the complete uniform recipe, so they do not isolate whether the downstream gain is caused by data order, late constant-LR continuation, checkpoint averaging, or their interaction.

Limitations and open questions

The headline cost excludes data acquisition, preprocessing, proxy experiments, failed runs, ablations, evaluation, post-training, storage, networking, CPU computation, ownership costs, taxes, depreciation, and research labor. It is therefore a marginal accelerator-cost estimate for rerunning the finalized production pretraining job, not the total cost of developing or independently reconstructing the project.

The comparison corpus is assembled largely from processed public datasets, and the authors do not provide a strict corpus-wide exact- and near-duplicate audit against every benchmark and proxy task. They acknowledge that contamination may contribute to the curriculum advantage, although the post-training persistence results indicate that contamination alone is unlikely to explain all observed differences. A complete contamination audit remains necessary.

The Puro Cost Scaling Law is fitted only within a fixed 2B dense architecture and a heavily overtrained regime of approximately 700 tokens per parameter. It provides no evidence for model-size scaling beyond this setting. RTX 5090 memory capacity and PCIe communication would become more restrictive at larger scales, and the reported driver modifications are unsupported and topology dependent.

Finally, the benchmark suite emphasizes mathematics, code, reasoning, and knowledge. Chinese capability is included in the corpus but is not a primary optimization target and is not reported as a headline evaluation dimension. The paper consequently leaves open how the same data-selection and curriculum procedures behave under explicitly multilingual objectives, different architectural families, or alternative post-training regimes.

Conclusion

Puro-2B demonstrates an integrated recipe for open, from-scratch language-model pretraining under a narrowly defined but reproducible accelerator budget. Its strongest contributions are the end-to-end release, the detailed consumer-GPU systems engineering, the blockwise FP8 implementation, the effective-learning-rate analysis of MuonH, the source-local data curriculum, and the explicit cost-performance accounting.

The reported $4.37K uniform run reaches Qwen2-1.5B-level aggregate performance, while the $6.89K canonical run improves the 15-task average to 57.81 and approaches Qwen2.5-1.5B. Curriculum-based pretraining also retains measurable benefits after SFT, including gains of 1.77 and 2.02 GSM8K points in focused and scaled settings and 1.59 points on broad instruction evaluation. The results support the paper’s narrower thesis: fully inspectable billion-parameter pretraining can be made substantially more accessible when the entire computational and data pipeline is co-designed around cost efficiency.