Megalodon: Diverse Advances in Modeling & Verification

• Megalodon is a polysemous research term defining three distinct systems: a long-context sequence model, a transformer-equivariant design for 3D molecule generation, and a higher-order set-theoretic proof checker.
• The sequence-model variant uses chunked streaming, complex exponential moving averages (CEMA), and causal normalization to overcome scaling bottlenecks and improve performance over traditional Transformers.
• The molecular design and formal verification systems innovate by fusing discrete and continuous domains, ensuring physical realism in 3D structures and streamlining LLM-driven autoformalization.

Megalodon is a polysemous technical term in recent research literature. It denotes a neural architecture for efficient sequence modeling with unlimited context length, a family of scalable transformer models for de novo 3D molecule generation, and a higher-order set-theoretic proof checker used in large-scale autoformalization experiments. The shared name therefore refers not to a unified framework but to separate systems in long-context language modeling, geometric generative modeling, and formal reasoning (Ma et al., 2024, Reidenbach et al., 23 May 2025, Urban, 6 Jan 2026).

1. Polysemy and research contexts

The three uses of the name occupy distinct methodological and application domains. In one line of work, Megalodon is positioned against vanilla Transformers and other sub-quadratic sequence models. In another, it is a transformer-equivariant architecture for molecular co-design. In a third, it is Chad Brown’s proof checker, instantiated in a classical higher-order set-theoretic setting and used through a command-line interface.

Usage	Characterization	Source
Sequence modeling	Neural architecture for efficient sequence modeling with unlimited context length	"Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length" (Ma et al., 2024)
3D molecule generation	Family of scalable transformer models enhanced with basic equivariant layers and trained using a joint continuous and discrete denoising co-design objective	"Applications of Modular Co-Design for De Novo 3D Molecule Generation" (Reidenbach et al., 23 May 2025)
Formal proof checking	Higher-order set-theoretic proof checker developed by Chad Brown	"130k Lines of Formal Topology in Two Weeks: Simple and Cheap Autoformalization for Everyone?" (Urban, 6 Jan 2026)

A common misconception is to treat these systems as related implementations. The cited works instead use the same name for separate projects with different objects of study, different mathematical formalisms, and different evaluation criteria.

2. Sequence-model Megalodon: architecture and computational scaling

In the sequence-modeling paper, Megalodon is introduced to address two stated limitations of vanilla Transformers: the quadratic cost of self-attention, $O(n^2\!\cdot d)$ per layer, and poor extrapolation to very long contexts (Ma et al., 2024). The architecture inherits the gated-attention plus exponential-moving-average inductive bias from Mega and combines it with chunked attention. The intended outcome is unlimited, in practice arbitrarily long, context length via streaming over fixed-size chunks, together with subquadratic runtime and memory.

The core computational strategy is explicit. The sequence is chunked into blocks of size $c$, with attention applied only within each chunk; long-range state is propagated across chunks via a moving-average mechanism; and normalization and gated-residual patterns are specialized for large-scale stability. Because chunk size is fixed, per-layer time becomes $O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$ when $c\sim d$, while memory is $O(n\!\cdot d)$. Standard Transformer self-attention is contrasted with per-layer time $O(n^2\!\cdot d)$ and memory $O(n^2+n\!\cdot d)$.

Four technical components are central. First, the real-valued damped EMA of Mega is generalized into a complex exponential moving average, or CEMA:

$$h_t^{(j)}=\alpha_j\cdot e^{i\theta_j}\odot u_t^{(j)}+(1-\alpha_j\odot\delta_j)\cdot e^{i\theta_j}\odot h_{t-1}^{(j)},\qquad y_t^{(j)}=\mathrm{Re}(\eta_j^\top h_t^{(j)}).$$

The phase vector is parameterized by

$$\theta_{j,k}=\left(\frac{2\pi k}{h}\right)\omega_j.$$

The stated rationale is that complex eigen-values allow CEMA to model oscillatory and long-range dependencies with richer dynamics than a purely real EMA.

Second, the architecture introduces a causal cumulative normalization along the auto-regressive dimension. If features are split into groups of size $d_g$, then at timestep $c$0,

$c$1

$c$2

The implementation uses efficient CUDA kernels with Welford/Kahan updates in $c$3 per token.

Third, Megalodon replaces raw dot-products by normalizing a shared representation:

$c$4

$c$5

The normalization on $c$6 removes the need for explicit scaling inside softmax.

Fourth, a pre-norm two-hop residual configuration rewires the block as

$c$7

The original input $c$8 is therefore reused as the residual in both sublayers.

3. Sequence-model Megalodon: training behavior, benchmarks, and open questions

The reported head-to-head pretraining comparison uses the scale of 7 billion parameters and 2 trillion training tokens (Ma et al., 2024). In that setting, Llama2-7B reaches a final training loss of 1.75, Llama2-13B reaches 1.67, and Megalodon-7B reaches 1.70, explicitly described as landing mid-way between the two. Training curves in negative log-likelihood versus tokens are reported to show faster convergence and fewer spikes than Llama2-7B under identical data and compute budgets.

The efficiency profile depends on context length. At 4 K context, Megalodon-7B is approximately 6% slower per GPU than Llama2-7B, with the overhead attributed to CEMA and TimestepNorm. At 32 K context, Megalodon-7B is approximately 32% faster, and chunk-parallel distributed training achieves approximately 94% utilization when scaling from 4 K to 32 K contexts. This suggests that the architectural overhead is short-context-sensitive, whereas the scaling advantage appears at long context.

On downstream evaluation, Megalodon-7B is reported to outperform Llama2-7B across the board on standard benchmarks including MMLU, BoolQ, HellaSwag, and PIQA, while approaching Llama2-13B performance. On Scrolls long-context question answering, with contexts up to 2 M tokens, perplexity declines monotonically as context grows; the model achieves state-of-the-art F1 on NarrativeQA and competitive results on Qasper and QMSum. Additional evaluations on LRA, WikiText-103, PG-19, ImageNet, and Speech Commands are described as showing robust improvements over both Transformers and prior state-space or convolutional models.

The paper also states concrete limitations and open questions. There is slight overhead at short contexts due to CEMA and TimestepNorm. Chunk boundary artifacts remain a possible source of information loss, although CEMA is said to mitigate them. Complex parameters $c$9 introduce extra memory and implementation complexity. Proposed future work includes integrating occasional full-context or sliding-window attention for cross-chunk mixing, extending to multi-modality pretraining such as video, code, and multimodal fusion, exploring alternative moving-average kernels or learned convolutional filters in the CEMA framework, and scaling to even larger parameter counts, stated as $O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$0 B, together with longer contexts in open-ended generation tasks.

4. Molecular-generation Megalodon: transformer-equivariant co-design

In molecular generative modeling, Megalodon is presented as a family of scalable transformer models for de novo 3D molecule generation (Reidenbach et al., 23 May 2025). The core architectural idea is to fuse a transformer trunk for discrete graph data with a light equivariant layer for continuous 3D coordinates. For a molecule with $O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$1 atoms, the inputs are atom coordinates $O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$2 together with discrete one-hot representations for atom types, bond types, and charges:

$O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$3

The network repeats blocks $O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$4 times, with $O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$5 given as an example. Each block contains a Fused Invariant Transformer, or FiT, and an EGNN-style equivariant structure update. The FiT block fuses node and edge embeddings plus distance features into a per-molecule invariant token $O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$6, applies multi-head attention on $O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$7, and then uses separate feed-forward layers to update atom-type and bond-type embeddings. It also uses adaptive, time-conditioned LayerNorm and query-key prenormalization.

The equivariant update modifies coordinates according to

$O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$8

where $O(n\!\cdot c\!\cdot d)\simeq O(n\!\cdot d^2)$9 and $c\sim d$0 are small MLPs on invariant features. The model outputs a continuous head projecting back to $c\sim d$1 and discrete heads projecting to atom-type, bond-type, and charge logits.

Training is organized around a joint continuous and discrete denoising co-design objective. For continuous diffusion on coordinates, the forward noising process is

$c\sim d$2

with DDPM-style denoising loss

$c\sim d$3

For discrete diffusion over atom types, bond types, and charges, the forward process is defined by a time-dependent transition matrix $c\sim d$4, with D3PM cross-entropy loss

$c\sim d$5

A distinctive co-design choice is that the model does not share a single scalar time. Instead, it samples two independent times, $c\sim d$6 and $c\sim d$7, each with its own schedule. The architecture also supports a flow-matching instantiation, including a continuous CFM objective and discrete flow matching via Multiflow cross-entropy.

The training setup is specified on GEOM-Drugs, with average 44 atoms and explicit hydrogens. Batching uses an adaptive data loader grouping similar-size molecules with batch cost approximately 200. Self-conditioning feeds an auxiliary prediction on the previous denoised output back through a residual MLP. Two model scales are given: Megalodon-quick, with approximately 19 M parameters, 10 FiT+EGNN layers, edge-feat $c\sim d$8, and node-feat $c\sim d$9; and Megalodon, with approximately 40 M parameters, edge-feat $O(n\!\cdot d)$0, and vector-dim $O(n\!\cdot d)$1. Training uses AdamW with the same learning rate and weight decay as EQGAT-diff and SemlaFlow; Megalodon-quick is trained on $O(n\!\cdot d)$2A100 for 250 epochs, and Megalodon on $O(n\!\cdot d)$3A100 for 250 epochs, stated as about 2 days.

5. Molecular-generation Megalodon: benchmarks, scaling, and physical realism

The evaluation suite includes unconditional 3D molecule generation on GEOM-Drugs, conditional conformer generation, and new structure-energetics benchmarks (Reidenbach et al., 23 May 2025). For unconditional generation, 2D and topological metrics are Atom Stability, Molecule Stability, and Connected Validity; 3D distributional metrics are weighted Wasserstein-1 distances on bond-angle and dihedral-angle distributions. Conditional conformer generation uses Precision and Recall Coverage at RMSD threshold $O(n\!\cdot d)$4 together with AMR. The energetics benchmarks run a GFN2-xTB relaxation to the nearest local minimum and then measure $O(n\!\cdot d)$5bond-length, $O(n\!\cdot d)$6bond-angle, $O(n\!\cdot d)$7dihedral, and relaxation energy

$O(n\!\cdot d)$8

On unconditional generation with 500 diffusion steps or 100 flow-matching steps, the reported GEOM-Drugs results are explicit. EQGAT-diff records Atom Stability 0.998, Molecule Stability 0.935, Validity 0.830, bond-angle $O(n\!\cdot d)$9 0.858, and dihedral $O(n^2\!\cdot d)$0 2.860. Megalodon-quick records 0.998, 0.961, 0.900, 0.689, and 2.383. Megalodon records 0.999, 0.977, 0.927, 0.461, and 1.231. SemlaFlow (FM) records 0.998, 0.979, 0.920, 1.274, and 1.934. Megalodon-flow records 0.997, 0.990, 0.948, 0.976, and 2.085.

The scaling result singled out in the paper concerns large molecules with sizes greater than 72 atoms. Megalodon at 40 M parameters is reported to generate up to 49 times more valid, stable large molecules than EQGAT-diff at 12 M parameters. Even the 19 M quick model achieves a 4–10 times boost. On the xTB-relaxation benchmark, EQGAT-diff has median $O(n^2\!\cdot d)$1 of 6.36 kcal/mol and mean 11.06; Megalodon-quick has 5.78 and 9.74; Megalodon has 3.17 and 5.71; SemlaFlow has 32.96 and 93.13; and Megalodon-flow has 20.86 and 46.86. The diffusion-based 40 M model is therefore described as achieving 2–10 times lower relaxation energies than prior state of the art and as operating within a thermally relevant window of about 2.5 kcal/mol.

The significance attributed to these results is specific. The work describes Megalodon as the first 3DMG model to jointly co-design continuous 3D coordinates and discrete graph generation in a single transformer-equivariant framework. It further argues that careful coupling of discrete and continuous denoising through dual-time sampling is critical for unconditional generation and downstream conditional conformer design. A plausible implication is that the paper treats physical realism, rather than only 2D validity or topological stability, as a primary criterion for generative quality.

6. Megalodon as a higher-order set-theoretic proof checker

A third usage of the name denotes Chad Brown’s proof checker, characterized as a relatively “unknown” yet highly capable higher-order set-theoretic system and used as the backbone of Josef Urban’s recent autoformalization of general topology (Urban, 6 Jan 2026). The project had produced 160k lines of formalized topology as of January 4, 2026, with about 130k lines completed in two weeks from December 22 to January 4, for an LLM subscription cost of about \$100. The formalized corpus includes a 3k-line proof of Urysohn’s lemma, a 2k-line proof of Urysohn’s Metrization theorem, a more than 10k-line proof of the Tietze extension theorem, and in total more than 1.5k lemmas and theorems.

The logical design is classical higher-order logic with a single universal type set. All mathematical objects—sets, relations, functions, numbers, topologies, metrics, and orders—are represented as elements of this one type, and higher-order quantification is permitted. Classical logic is built in through the Law of Excluded Middle, xm, and proof by contradiction is freely available. Functions are represented extensionally as sets of ordered pairs, while topologies, metrics, and other structures are sets satisfying suitable comprehension axioms.

The development model is deliberately minimal. A project lives in one .mg file, such as Math_Background.mg, which the source describes as eliminating module-import issues for the LLM. The trusted core is also small: ZF-style axioms including extensionality, pairing, union, power set, replacement, and choice are formalized, along with Brown’s construction of surreal numbers and the real line via Dedekind cuts or Cauchy sequences. Many initial proofs are temporarily admitted, while a small number of fully checked model proofs remain to teach proof style.

The proof language uses a simple concrete syntax. Definitions begin with declarations of the form Definition foo : forall x:set, ....; proofs use Proof ... Qed. blocks; and comments take the form (** ... **). At the command line, the canonical invocation is $O(n^2\!\cdot d)$5 and the checker prints nothing if the file type-checks, or lists errors and warnings if it fails.

The core library supports a conventional topology and analysis development in set-theoretic notation. Representative definitions include

$O(n^2\!\cdot d)$2

the continuity criterion

$O(n^2\!\cdot d)$3

and a uniform-limit continuity lemma formulated in terms of $O(n^2\!\cdot d)$4.

7. LLM-driven autoformalization pipeline and practical constraints

The proof-checking work describes a simple long-running feedback loop between an LLM and Megalodon as a command-line checker (Urban, 6 Jan 2026). A Linux bubblewrap sandbox contains only the Megalodon binary, the current .mg file, the prompt script, and minimal tooling. The coding agent is either ChatGPT 5.2 via the Codex CLI or Claude Sonnet 4.5, guided by a short boot prompt pointing to a rules file such as CLAUDE.md. The agent edits Math_Background.mg, typically appending to the topology section. An automated tmux script detects a pause of 60 seconds with no new output and resends the same prompt, allowing uninterrupted continuation.

Verification is then performed with a timeout-wrapped checker call: $O(n^2\!\cdot d)$6 Errors are fed back into the next prompt; silent execution indicates correctness. Progress tracking is augmented by mgdeps6.pl, which extracts each theorem’s admit-status and direct dependencies, together with PROGRESSXXXX and DEPSXXXX files used to manage section-level status and avoid leaving admitted lemmas in the dependency graph. The paper emphasizes that no deep integration API or plugin was needed: interaction occurred through files and CLI exit codes.

The reported performance characteristics are unusually favorable for this workflow. The initial core library compiles in under 1 second. After 170,000 lines of definitions and proofs, a full Megalodon check takes about 32 seconds on a modern laptop. The entire setup is said to fit in a few hundred lines of shell script plus a 14,000-line rules prompt. The one-file approach means the LLM does not need to reason about imports or namespaces, and the higher-order set-theoretic framework is described as expressive enough to handle metrics, orders, nets, uniform limits, and surreal numbers in a common formal setting.

The limitations are equally explicit. After sessions exceed about 250 MB of transcript, the LLM exhibits context decay and “forgets” parts of the file. Forward-reference issues were significant in the Tietze Extension proof because the relevant uniform-limit and completeness lemmas occurred later in the textbook. Admitted foundational proofs remain a risk. Although Megalodon includes a TH0 export and can invoke Vampire via its “aby” hammer, proof reconstruction is not yet implemented, which limited the benefit of hammering. The LLM subscription was the main cost, described elsewhere in the paper as around \$200 per month for ChatGPT Pro plus Codex CLI, and usage limits were reached weekly. File growth also became material: at 8 MB and 170,000 lines, future extensions were said to require file splitting or admitting completed sections to keep check times acceptable.

Taken together, these three usages of “Megalodon” mark distinct research trajectories: long-context sequence modeling with chunked streaming and CEMA, transformer-equivariant co-design for 3D molecular generation, and minimalist higher-order proof checking for LLM-driven formalization. The shared nomenclature masks substantial differences in objectives, mathematical structure, and evaluation protocol; the substantive commonality is only that each system is designed as an efficiency-oriented response to a scaling bottleneck in its own domain.