H2LooP Spark Preview for Embedded Code

• The paper demonstrates that continual pretraining on a curated embedded corpus significantly reduces in-domain and held-out perplexity while enhancing code-completion performance.
• It employs the SpecMap hierarchical datasheet-to-code mapping to process 23.5B tokens from diverse embedded domains, ensuring detailed token sequence coverage.
• The approach leverages BF16 LoRA with rank-stabilized scaling to achieve up to 70% perplexity reduction, rivaling larger systems on vendor-specific low-level code.

H2LooP Spark Preview is a domain-adapted 7B open-weight LLM and continual pretraining pipeline for low-level embedded systems code. It is presented as an adaptation of OLMo-3-7B / OLMo-3-1025-7B to a software domain characterized by hardware register manipulation, vendor-specific SDKs, real-time operating system APIs, hardware abstraction layers, linker artifacts, device trees, startup code, and assembly stubs that are underrepresented in standard pretraining corpora. The system is trained on repository–datasheet pairs processed with the SpecMap hierarchical datasheet-to-code mapping approach, and the reported outcome is substantial reduction in in-domain and held-out perplexity together with strong code-completion performance across 13 embedded domains (Singh et al., 11 Mar 2026).

1. Domain definition and motivation

H2LooP Spark Preview is defined around the claim that general-purpose code LLMs are limited in embedded firmware not because they cannot reason, but because they have not seen enough of the relevant distribution. The paper identifies the missing distribution as bare-metal register manipulation, vendor SDK conventions, startup and interrupt patterns, HAL code, RTOS APIs, linker artifacts, device trees, assembly stubs, and hardware-specific headers. It argues that broad code models tend to hallucinate register names, misuse APIs, and violate hardware-specific initialization patterns because those forms are sparse in standard web/code corpora (Singh et al., 11 Mar 2026).

The embedded setting is described as unusually fragmented and vendor-specific. A valid STM32 CAN setup sequence does not transfer to an NXP FlexCAN implementation; RTOS and HAL APIs differ across ecosystems; and many critical artifacts are not ordinary functions, but macros, enums, typedefs, constants, linker scripts, startup assembly, and device-tree declarations. The paper therefore frames continual pretraining, rather than instruction tuning or training from scratch, as the central intervention. In its formulation, the failure mode is a lack of token-sequence coverage at the domain level, so the remedy is to continue the causal language-modeling objective on a curated embedded corpus (Singh et al., 11 Mar 2026).

This positioning also shapes the model’s intended significance. The work is not presented as a new transformer architecture. Its core claim is that careful domain adaptation and carefully constructed domain data can let a relatively small open model rival larger systems on specialized technical code. A plausible implication is that the paper belongs more to the literature on domain-aligned continual pretraining than to the literature on generic code assistants.

2. Repository–datasheet corpus and embedded taxonomy

The training corpus is built from repository–datasheet pairs using the SpecMap hierarchical datasheet-to-code traceability system. The paper reports a pool of 818 repository–datasheet pairs, 117 manufacturers, 19 component categories, 61 component classes, and 76.4 GB of raw repository data. The abstract describes “100B tokens of raw embedded systems data across 117 manufacturers,” while the processed training corpus is described as a curated split of 23.5B tokens across 13 embedded domains (Singh et al., 11 Mar 2026).

SpecMap is organized into four stages. $M_1$, Folder Discovery, maps datasheet sections to relevant folders using LLM-based semantic analysis with adaptive chunking. $M_2$, File Discovery, identifies specific implementation files inside those folders through structure generation and LLM relevance scoring. $M_3$, Code Symbol Discovery, extracts symbols such as functions, macros, structs, constants, enums, and typedefs using Universal Ctags for C/C++ parsing. $M_4$, Validation & Gap Analysis, validates the complete mapping sequentially while preserving context and determines implementation status. In the paper’s framing, this hierarchical mapping is what turns fragmented embedded repositories and specifications into grounded training units (Singh et al., 11 Mar 2026).

Section-aware chunking then splits the mapped outputs with 16 parallel workers. Samples are constrained to a maximum of 7,500 characters and a minimum of 50 characters; later, tokenized sequences are truncated to 2,048 tokens with <|endoftext|> appended. Quality filtering removes repeated separator lines, excessive repeated characters, ASCII art, box-drawing characters, and empty comment blocks, and drops samples with more than 70% garbage after cleaning. Remaining samples must either contain code indicators such as #include, #define, void, int, struct, typedef, or control-flow keywords, or contain at least 20 natural-language words (Singh et al., 11 Mar 2026).

The final processed training data is reported as about 20 GB in 289 Parquet files, about 11.5M samples, about 23.5B tokens, average sample length about 1,450 tokens, and approximately 95% packing efficiency. The corpus spans 13 evaluation domains: wireless/BLE/WiFi, Linux kernel, NXP i.MX, device tree, USB stack, Zephyr RTOS, crypto, ARM Cortex assembly, STM32 HAL, general embedded code, Infineon AURIX, AMD GPU registers, and register defines. The held-out benchmark repositories are drawn from never-seen public repositories such as torvalds/linux, Infineon/AURIX_code_examples, STMicroelectronics/STM32CubeF4, nxp-mcuxpresso/mcux-sdk, ARM-software/CMSIS_5, zephyrproject-rtos/zephyr, wolfSSL/wolfssl, Mbed-TLS/mbedtls, and hathach/tinyusb (Singh et al., 11 Mar 2026).

3. Continual pretraining method and systems implementation

The paper describes the base model as OLMo-3-7B in the abstract and as OLMo-3-1025-7B in the detailed description. It is adapted with BF16 LoRA and rank-stabilized scaling (RSLoRA), targeting q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, embed_tokens, and lm_head. Standard LoRA is written as

$W' = W_0 + \frac{\alpha}{r}BA,$

while RSLoRA replaces the $1/r$ factor with $1/\sqrt{r}$: $W' = W_0 + \frac{\alpha}{\sqrt{r}}BA.$ The paper argues that this makes unusually high-rank adapters practical and reports that the full-module $r=512$ configuration corresponds to about 839M trainable LoRA parameters, about 12% of the 7B base model (Singh et al., 11 Mar 2026).

The production configuration uses LoRA rank $r=512$, LoRA alpha $M_2$0, LoRA dropout $M_2$1, BF16 precision, AdamW with dual parameter groups, cosine decay with 10% warmup, per-device batch size 4, gradient accumulation 8, 8 GPUs, maximum sequence length 2048, max gradient norm 5.0, and seed 3407. The main learning rate is $M_2$2, while the embedding/LM-head group uses $M_2$3. Tokens per optimizer step are explicitly calculated as

$M_2$4

The training loss is standard causal LM cross-entropy,

$M_2$5

and the learning-rate schedule is a warmup-plus-cosine-decay schedule (Singh et al., 11 Mar 2026).

The implementation uses a single node with 8 NVIDIA H100 80GB SXM GPUs interconnected via NVLink/NVSwitch, backed by 192 CPU cores and local NVMe storage. The software stack includes PyTorch DDP via Hugging Face Accelerate, Flash Attention 2, BF16, TF32 enabled for appropriate FP32 matmuls, enlarged DDP all-reduce buckets, gradient checkpointing with use_reentrant=False, NCCL transport optimizations, and CUDA allocator settings intended to reduce fragmentation. Rank 0 tokenizes the 289 Parquet files with 64 parallel workers, writes a memory-mapped Apache Arrow cache, and other ranks load the same Arrow cache through OS page cache sharing. File-based synchronization is used instead of dist.barrier() during long preprocessing, and sequence packing is reported to improve throughput by about 40% (Singh et al., 11 Mar 2026).

The reported hero run lasted 294.7 wall-clock hours, or 12.3 days, consumed 2,357 GPU-hours on 8 H100s, processed 8.6B tokens, and averaged 8,644 tokens/s. The paper also reports two throughput regimes: steps 0–6000 at 10,945 tok/s and 45.5 s/step, and steps 7000–16,440 at 7,078 tok/s and 70.3 s/step after a resume, with the slowdown attributed to memory pressure, optimizer state, or fragmentation after checkpoint restoration. A spike detector,

$M_2$6

is used for training monitoring (Singh et al., 11 Mar 2026).

4. Evaluation protocol and empirical results

The evaluation combines perplexity, teacher-forced completion, free-form generative code completion, and a limited forgetting check on general C++. Perplexity is defined as

$M_2$7

In-domain evaluation uses 2,270 samples from the last two Parquet files of each processed category, and held-out evaluation uses 1.36M samples from nine never-seen public repositories. Teacher-forced completion splits each sample at 75% of its natural length and evaluates the 25% suffix using teacher forcing; free-form generation uses greedy decoding with up to 512 generated tokens and reports token-level accuracy and BLEU-4 (Singh et al., 11 Mar 2026).

The headline perplexity numbers are a weighted in-domain PPL reduction from 4.06 for the base model to 1.20 at checkpoint 15,000, a 70.4% reduction, and a held-out repository PPL reduction from 3.92 to 1.33, a 66.1% reduction. Per-category in-domain reductions are largest in wireless/BLE/WiFi, Linux kernel, and NXP i.MX, and remain substantial across device tree, USB stack, Zephyr RTOS, crypto, ARM Cortex assembly, STM32 HAL, general embedded code, Infineon AURIX, AMD GPU registers, and register defines. The paper interprets the largest gains as occurring in categories least represented in the original pretraining mix (Singh et al., 11 Mar 2026).

Teacher-forced completion results are near-saturated in some categories. At checkpoint 15,000, top-1 token accuracy rises from 0.819 to 0.998 for Infineon AURIX, from 0.821 to 0.988 for register defines, from 0.739 to 0.980 for ARM Cortex ASM, from 0.724 to 0.972 for STM32 HAL, and from 0.621 to 0.919 for Linux kernel. On held-out repositories, similarly large gains are reported for register defines, general, AURIX, STM32 HAL, ARM Cortex ASM, Linux kernel, and wireless/BLE/WiFi (Singh et al., 11 Mar 2026).

The free-form generative benchmark is the basis of the paper’s comparison to larger frontier systems. H2LooP Spark Preview is reported to have the highest generative token accuracy on 8 of 13 categories: General, Register Defines, ARM Cortex ASM, Infineon AURIX, STM32 HAL, Device Tree, NXP i.MX, and AMD GPU Registers, where it ties Qwen3-Coder-30B. The categories it does not win are USB Stack, Linux Kernel, and Wireless/BLE/WiFi, which are won by Claude Opus 4.6, and Crypto and Zephyr RTOS, which are won by Qwen3-Coder-30B. The paper’s interpretation is that the model is strongest on highly structured, vendor-specific, low-level domains rather than on broader software stacks (Singh et al., 11 Mar 2026).

To examine catastrophic forgetting, the paper evaluates FullStackBench C++ with 107 samples. Compile rate drops from 44.9% for the base model to 41.1% for the adapted checkpoint, and test pass rate drops from 15.9% to 13.1%. Easy tasks remain identical at 62.5% compile and 50.0% pass, while medium and hard degrade modestly. The paper presents this as evidence of large embedded-domain gains with limited degradation on a general C++ benchmark (Singh et al., 11 Mar 2026).

5. Nomenclature and disambiguation

Despite its title, H2LooP Spark Preview is not an Apache Spark analytics paper and is not related to ALICE Hyperloop. “Hyperloop -- The ALICE analysis train system for Run 3” describes a centralized ALICE analysis train framework built for the $M_2$8 computing model, with a React.js UI, JIRA-based analysis management, automatic wagon testing, and Grid or Analysis Facility submission, and explicitly notes that the paper does not mention Apache Spark at all (Quishpe et al., 2021). By contrast, H2LooP Spark Preview is a continual pretraining pipeline for a LLM in embedded systems (Singh et al., 11 Mar 2026).

It is also distinct from papers that use Apache Spark as the execution engine for large-scale scientific data processing. “Using Big Data Technologies for HEP Analysis” evaluates Spark for CMS data reduction and reports a 1 PB to 1 TB reduction in 3.8 hours under an optimized configuration (Cremonesi et al., 2019), while “Exploiting Apache Spark platform for CMS computing analytics” studies Spark for CMS metadata analytics and ML workflows over billions of HDFS-resident records (Meoni et al., 2017). A different line of work, “Understanding the Challenges and Assisting Developers with Developing Spark Applications,” uses “preview”-like ideas to show intermediate information and hints for chained Spark pipelines (Wang, 2021). H2LooP Spark Preview belongs to none of these categories; it uses PyTorch DDP, Hugging Face Accelerate, Flash Attention 2, BF16 LoRA, and H100 GPUs rather than Spark SQL, Spark Streaming, or Spark-based cluster analytics (Singh et al., 11 Mar 2026).

This suggests that the phrase “Spark Preview” in the title functions as a project or model name rather than as a reference to the Apache Spark runtime. That interpretation is consistent with the paper’s complete focus on continual pretraining, datasheet-grounded data construction, and embedded code evaluation.

6. Limitations, documentation issues, and release

The paper explicitly describes the training as incomplete and states that more gains are expected by extending from the reported run to 100B tokens, about 4.26 epochs, particularly on weaker categories such as Zephyr RTOS and crypto. It also states that there is no embedded functional correctness benchmark: evaluation relies on perplexity, token accuracy, and BLEU-4 rather than compile/run metrics for generated firmware. Further limitations noted in the paper include use of a single base model, incomplete deduplication analysis, and unexplained throughput degradation after resume (Singh et al., 11 Mar 2026).

The documentation contains several internal inconsistencies. The abstract states “After training on 5.2 billion tokens ($M_2$9 of one epoch),” while the main experimental section describes a hero run of 8.6B tokens over 16,440 steps, and the appendix identifies the release as step 15,000 at “$M_3$0 tokens processed.” The abstract also names OLMo-3-7B, while the detailed description uses OLMo-3-1025-7B. These are not merely editorial details: they affect the exact interpretation of the released checkpoint and of the reported training extent (Singh et al., 11 Mar 2026).

The release is reported on Hugging Face at https://huggingface.co/h2loop-ai/spark-cpt-base-ckpt. According to the paper, the release includes model weights, training configuration and hyperparameters through the paper, and usage instructions for loading with PEFT/Hugging Face Transformers and vLLM as described in the model card. The LoRA adapter weights, merged model, and associated H2LooP artifacts are under a Research-Only License, while the underlying OLMo base model remains under its own Apache 2.0 license. The paper does not describe release of the full curated 23.5B-token dataset or the full training code/scripts (Singh et al., 11 Mar 2026).

Within those stated limits, the paper’s broader significance is clear. It advances the claim that specialized continual pretraining can matter more than sheer model scale when the target domain is low-level, vendor-specific, and structurally rigid. The reported results support that claim most strongly in STM32 HAL, AURIX, ARM Cortex assembly, device tree, register defines, and NXP i.MX, where the adapted 7B model is reported to outperform Claude Opus 4.6 and Qwen3-Coder-30B on token accuracy (Singh et al., 11 Mar 2026).