Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference

Abstract: Small LLMs are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it. The result keeps full attention in only 6 of its 18 blocks. The other 12 use short convolutions whose memory is two timesteps wide no matter how long the conversation gets, so two thirds of the network never re-reads a growing cache. Trained from scratch on 59.9B tokens, the model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685. To check the architecture rather than the training recipe, we trained a conventional all-attention model of the same size on the same data, and wrote down the winning condition before scoring either. The hybrid won the chosen quality metric by 0.81%, matched it on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76x faster at 2048 tokens of context, 2.08x against an external model of similar size. In every measurement the speed advantage is near zero at an empty context and grows with length, which is what the mechanism predicts and what a merely leaner model would not show. A simple bandwidth calculation predicts only 1.17x, so memory volume alone does not explain the gap. We also report what did not work: an unmitigated 4-bit quality cost, roughly half the convolution channels ending up inert and impossible to remove, and a vocabulary larger than this model size warrants.

• The paper introduces a 160.49M-parameter hybrid language model with six grouped-query attention blocks and twelve fixed-state convolution blocks designed for batch-size-one CPU decoding.
• Daedalus-150M matches its parameter-matched all-attention twin on downstream quality while achieving 1.76× faster decoding at a 2048-token context, with speed gains increasing as context grows.
• The full model scores 47.31 on a five-task benchmark, produces a 6.3% smaller 4-bit artifact, and highlights remaining challenges including dead convolution channels, quantization loss, and single-seed evaluation.

Design premise: architecture chosen for the deployment regime

Daedalus-150M (2608.20210) is a 160.49M-parameter LLM designed around an explicit deployment target rather than adapted to one after training: single-user decoding on an ordinary CPU at batch size one, 4-bit weights, 2048-token context. The paper's central argument is that this regime inverts standard efficiency priorities. With no batching over which to amortise weight loading, throughput is governed by bytes read per token; memory bandwidth binds before arithmetic throughput; and the key–value (KV) cache of an all-attention decoder is a per-token cost that grows linearly with context length and dominates long-context decoding on CPU.

The architectural response is a hybrid stack of 18 blocks at $d_{\text{model}}=768$: six full-attention blocks interleaved among twelve short-convolution blocks whose recurrent state is exactly two timesteps wide regardless of context length. Each convolution block applies RMSNorm, an input projection producing gating terms $B$, $C$ and $x$, a depthwise convolution of kernel length $L=3$ over $B \odot x$, output gating by $C$, and an output projection — a gated fixed-kernel design that maps onto existing CPU inference kernels without new operators. Attention blocks use grouped-query attention (4 KV heads for 12 query heads) with RoPE, reducing cache bytes further. Supporting choices follow MobileLLM-style findings: tied embeddings (the 49,152-entry vocabulary alone holds 37.7M parameters, 23% of the model) and a narrow feed-forward inner dimension of 2048 ($2.67\times d_{\text{model}}$) that shifts parameters away from the widest, most bandwidth-hungry tensors. Mixture-of-experts routing, positional-encoding-free attention, and distillation were all rejected for concrete engineering or budget reasons.

The paper is explicit that the contribution is not a new operator but the observation that, in the batch-size-one CPU regime, the ratio of cache-free to attention layers is the dominant deployable design variable — tested here with a controlled experiment at matched parameter count.

Pre-registered evaluation

To prevent post-hoc rationalisation, the evaluation bar was fixed before any headline number existed: a five-task mean over HellaSwag, ARC-Easy, PIQA, OpenBookQA and WinoGrande under lm-evaluation-harness conventions, with all peers re-scored on the same harness rather than quoted from their own publications. The bar was 42.20, the strongest of four beatable peers (GPT-2 124M, Pythia-160M, OPT-125M, GPT-neo-125M), each trained on three to six times more data. MobileLLM-125M was a stretch target; a 2T-token peer (51.2) was conceded in advance as a quality-for-speed trade.

The central ablation: hybrid versus dense twin

The architectural claim was tested with a pre-registered, parameter-matched comparison: a 24-layer all-attention dense twin (161.25M parameters, within 0.5% of the hybrid), trained on identical data and schedule for 5B tokens each and fully decayed. The decision rule — validation bits-per-byte over a 645M-token held-out set with a 0.5% margin floor — was fixed before either arm was scored, and a dense win beyond that floor would have changed the main run's architecture.

Quality. The hybrid won the pre-registered metric by 0.81% (0.910398 vs 0.917774 bits-per-byte), clearing the floor. On the five-task mean the dense twin was nominally ahead by 0.14 points, roughly $0.24\sigma$ against suite noise of $\approx 0.58\sigma$, with tasks swapping winners in both directions. At WinoGrande both arms sit near the 50% chance floor, so roughly two of five downstream tasks measure nothing at this budget. The honest summary the paper gives is that the hybrid matches on quality and wins decisively on decode — it is not presented as a downstream-task win.

Decode speed. Measured with 4-bit weights on 8 threads, generating 128 tokens after priming a context of stated depth, with arms alternating within a single pass so background load perturbs both equally:

Context depth	Hybrid (tok/s)	Dense twin (tok/s)	Ratio
0	1111.9 ± 25.7	922.8 ± 14.2	1.20×
512	960.3 ± 10.7	664.4 ± 6.5	1.45×
2048	739.3 ± 35.5	420.3 ± 5.5	1.76×

The shape of this table is the thesis: near-unity at empty context, growing monotonically with depth. A merely leaner model would show a constant advantage; divergence identifies the mechanism. Against an external 135M-parameter peer written by a different author, the same signature reproduces — 1.06× at depth zero rising to 2.08× at 2048 tokens — evidence the result is not an artefact of the project's own training or export code. One confound runs against the hybrid: the dense twin receives the reference runtime's better-optimised attention graph, so the margin is achieved despite inferior kernel optimisation on the convolution path.

Artefact size. Matched at half precision within 0.5%, the hybrid's shipped 4-bit file is 95.56 MiB versus the twin's 101.62 MiB — 6.3% smaller — because quantised size depends on tensor shapes, not parameter count alone.

A cost model that under-predicts

A first-order bandwidth model, $B$0 with $B$1 the cache bytes per context token, predicts only 1.17× at depth 2048 against the measured 1.76×. The paper rejects bandwidth-only explanation and attributes the residual to latency-bound softmax reduction over the cache when the working set exceeds last-level cache (versus perfect locality of a two-element conv state), plus the twin paying per-layer fixed costs across 24 layers instead of 18. The practical consequence is that the architecture's benefit exceeds naive byte accounting, and better attention kernels would narrow but not close the gap — the cache the hybrid does not keep cannot be optimised.

Data pipeline and training

Training used a ten-source English mixture of 16.93B unique tokens weighted toward educational and reasoning-dense text (FineWeb-Edu 37.5%, DCLM-baseline 22.5%, Stack-Edu Python 9%, etc.), consumed for 59.9B tokens (~3.5 epochs). Each source is capped at four epochs following data-constrained scaling findings, with freed share redistributed by water-filling; the cap is load-bearing for tiny sources like everyday-conversations (~400k tokens). Optimisation split Muon (122.68M params in 2D matrices) and AdamW (37.81M in embeddings/norms/biases), warmup-stable-decay with linear decay to zero, batch ramped 128k→512k tokens/step, bf16, on a single RTX 5090. The paper documents that the loss plateau during the stable phase is the schedule operating as designed, after an observer mistook it for a stall.

Headline results

At full scale, Daedalus-150M scores 47.31 on the five-task mean against the pre-registered bar of 42.20 — beating every peer in its class, each trained on three to six times more data, and exceeding MobileLLM-125M's published figure from a trillion tokens. Validation bits-per-byte is 0.8685, a 4.6% improvement over the 5B-token arm's 0.9104. Notably, even the 5B-token ablation arm scored 44.7, already clearing the bar — moving the open question from whether the architecture can reach it to how far twelve times more data carries past it. Quantising to Q4_0 costs about 6% perplexity (9.18 → 9.75), higher than the 2.5% measured at smaller scale.

Limitations

The paper reports its failures plainly. Mixture drift: the realised corpus mixture exceeded its pre-fixed $B$2 limit (10.42 vs 10.0 percentage points), partly because the final 4.8B tokens came from a snapshot 0.42B tokens smaller. Interrupted run: the last ~8% of training resumed without optimiser state and with a reset data cursor, so the released weights are not one continuous trajectory. No quantisation-aware training: the planned final fake-quantisation pass produced non-finite loss on its first step and was disabled without diagnosis, so the released model carries the full post-training 4-bit penalty (~6% perplexity); half-precision weights are published so this is fixable without retraining. Oversized vocabulary: 49,152 entries inherited from a dropped distillation plan, where scaling laws suggest 24–32k is optimal for this size, wasting ~13M parameters. Single seed, English only: the 0.81% ablation margin is not a confidence interval, and decode claims are bounded at the trained 2048-token context.

Negative result: dead channels cannot be reclaimed

Approximately 47.9% of short-convolution channels are inert, a stable plateau across training representing ~13.6M dead parameters (8.5% inefficiency). Structural pruning at export was tested and conclusively fails: the reference runtime shape-checks convolution tensors at fixed model width, rejecting narrowed files, with a full-width rebuild control proving the rejection is the narrowing itself. Reclaiming the channels would require patching the runtime, forfeiting stock-binary compatibility for 7.7MB. The remedy belongs to initialisation and regularisation in a successor model, not a retrofit.

Evaluation methodology

Two protocol details carry weight. Peer models are re-scored on the project's own harness, since published figures run 0.5–1.5 points above same-harness scores due to task-subset and normalisation differences. Decode measurements alternate arms within a single pass; an earlier back-to-back measurement produced a 1.29× figure that does not reproduce, and the paper explicitly instructs that two historical figures (1.29× and the 1.15–1.17× depth-zero row) should not be quoted. Prompt processing is excluded from decode measurements because it is compute-bound and would flatter the hybrid.

Conclusion

The proposition was specific and falsifiable: for single-user CPU decoding, replacing two thirds of attention layers with fixed-state short convolutions buys a large speed advantage at no quality cost. Both halves held — a pre-registered quality win of 0.81%, a downstream tie, a 6.3% smaller artefact, and 1.76× faster decoding at trained context (2.08× externally), with the advantage growing with depth in every measurement. At full scale the model clears its pre-set bar by 5.11 points. What remains open is engineering rather than research: diagnosing the quantisation-aware-training failure, discouraging dead channels during training, choosing a right-sized vocabulary, multi-seed replication of the ablation, and a retrieval-sensitive evaluation to bound how far the attention fraction can fall.