Speaking the Language of Science: Toward a General-Purpose Generative Foundation Model for the Natural Sciences

Abstract: In this report, we present LOGOS (Language Of Generative Objects in Science), a scientific generative LLM that unifies heterogeneous tasks across the natural sciences within a single autoregressive framework based on a shared scientific grammar. It encodes diverse scientific objects and their spatial interactions as token sequences over a common vocabulary. By representing spatial contact and constraint patterns as discrete tokens, the model captures complex structural interactions in a purely sequential manner, without relying on explicit coordinates or geometric neural networks. This unified representation enables a wide range of downstream tasks to be formulated consistently as next-token prediction in the same grammar space, creating strong alignment between continued multi-domain pre-training and downstream objectives. Across diverse tasks, LOGOS consistently matches or outperforms domain-specific baselines, providing preliminary evidence for the feasibility of "one model fits all" in the natural sciences. We train LOGOS models at different scales (1B, 3B, and 8B parameters) and find a consistent positive correlation between model size and performance. This suggests that the future of AI for Science (AI4S) may not lie in building an independent technical stack that is separated from LLMs. Instead, it may depend on deeply aligning scientific foundation models with LLMs through shared architectures, shared training paradigms, and shared inference infrastructure, so that LLMs can truly become a new entry point for AI4S. We release the model weights and associated resources to facilitate further research.

• The paper introduces LOGOS, a 1B–8B-parameter autoregressive foundation model that serializes proteins, molecules, reactions, materials, pockets, and complexes into a shared scientific grammar for next-token prediction.
• LOGOS matches or exceeds specialized systems across six tasks, including Vina −7.76 for pocket-conditioned ligand generation, 74.8% Top-1 retrosynthesis accuracy, and a 17.78% novel-building-block rate for MOF generation.
• The results show that full LLM backbone initialization, cross-representation translation data, and joint multi-task fine-tuning drive performance, while natural-language data and explicit 3D inputs are not required—but nucleic acids, larger-scale testing, and richer geometric modeling remain open challenges.

LOGOS (Language Of Generative Objects in Science) is a multi-domain generative foundation model that unifies heterogeneous natural-science tasks—proteins, antibodies, small molecules, chemical reactions, materials, binding pockets, and protein–ligand complexes—within a single autoregressive framework (2606.16905). Its central design decision is to abandon natural language as a cross-modal interface and instead define a shared "scientific grammar": a discrete token vocabulary with boundary and task tokens under which all scientific objects and their spatial interactions are serialized as sequences. All downstream tasks are then formulated as next-token prediction in this grammar space, aligning pre-training objectives directly with downstream generative applications. Models at 1B, 3B, and 8B parameters (built on Qwen3-8B and Llama3.2 backbones) match or exceed domain-specific baselines across six tasks without any explicit 3D geometric inputs.

The unified scientific grammar

The pre-training corpus comprises seven modalities totaling 44.87 billion tokens. Proteins derive from UniRef90; antibodies from OAS (filtered for provenance, isotype, and non-standard residues, then clustered at 70% identity); small molecules from PubChemQC as canonical SMILES; reactions from ORD and ECReact, encoded bidirectionally with directional tokens <React>/<ReverseReact>; and MOF materials decomposed via metal-oxo decomposition into metal clusters (<MetalS>...<MetalE>) nested within material boundary tokens alongside reused molecule tokens.

The most consequential grammatical innovation concerns spatial interactions. Binding pockets predicted by P2Rank on PDB structures are represented in four progressive forms: (1) in-sequence pocket annotation via <PocketS>/<PocketE>; (2) expansion of each pocket residue into its side-chain SMILES, placing the interface in chemical representation space; (3) an amino-acid-to-SMILES transformation sequence joined by a <Trans> token; and (4) a <Search>-conditioned sequence casting pocket identification itself as generation. Protein–ligand complexes from Q-BioLiP extend this by selecting residues within 5–10 Å of ligand atoms at multiple radii and appending the ligand SMILES, encoding "which residues bind which ligand" in one sequence. This discretization of 3D proximity into token records is what allows the model to learn structural interaction patterns purely sequentially.

Exploratory studies informing the design

Three ablation-driven findings shape the framework. First, on initialization: inheriting full LLM weights (embedding, backbone, LM head) yields the best Vina score ($-7.43$ vs. $-6.91$ for random init on ligand design), while inheriting only the embedding slightly degrades performance relative to random initialization. The transferable benefit resides in the Transformer backbone's sequence-modeling priors, not token representations. Second, on linguistic priors: increasing the proportion of natural-language corpus during continued pre-training consistently degrades ligand-design performance across all three scales, revealing a capacity trade-off under fixed parameter budgets. This motivates the paper's central claim—that AI4S models should inherit LLM weights but not natural-language data, using the scientific grammar rather than language as the cross-modal interface. Third, on grammar composition: removing pocket and complex data drops ligand-generation performance to near-random levels (Vina $-3.57$); adding them recovers substantial capability ($-6.25$); adding the <Trans> translation data produces the largest single gain ($-7.64$), despite containing no ligand information. This indicates that cross-representation semantic alignment between amino-acid symbols and molecular structures—not mere co-occurrence of modalities—is what enables conditional generation. Finally, multi-task joint SFT outperforms single-task SFT on all four evaluated tasks, providing evidence of genuine cross-domain knowledge transfer under the shared grammar.

Training pipeline

Models retain their original LLM vocabularies extended with grammar special tokens, initialized fully from LLM weights. Continued pre-training uses standard next-token prediction over the seven-modality corpus (AdamW, peak LR $2\times10^{-5}$, bfloat16, on 32 A800 GPUs). Post-training applies unified SFT over four representative tasks—retrosynthesis, material generation, pocket identification, and ligand design—with loss computed only on output spans, deliberately mixing domains to exploit the observed cross-task synergy.

Evaluation results

Pocket-conditioned ligand generation (PDBBind core set): LOGOS-8B achieves Vina $-7.76$, QED 0.57, SAS 0.73, outperforming all coordinate-based baselines including TargetDiff ($-7.38$) and DiffBP ($-7.28$). Notably, LOGOS-1B ($-7.64$) surpasses NatureLM (8×7B, $-6.91$0) with roughly 1/56 of the parameters—a strong parameter-efficiency result attributable to concentrating capacity on domain-native representations. Performance improves monotonically from 1B to 8B without sacrificing drug-likeness.

Binding site identification (COACH420/HOLO4K): LOGOS-8B reaches Top-$-6.91$1 of 66.5 and 58.5 respectively, second only to P2Rank—but all baselines require 3D structures while LOGOS uses sequence alone. The gap to P2Rank is partly structural: LOGOS's training annotations are themselves P2Rank predictions, which upper-bounds achievable performance. The practical implication is extending pocket identification to the far larger space of proteins without resolved structures.

Retrosynthesis (USPTO-50K): LOGOS-8B attains the best Top-1 accuracy of 74.8%, exceeding NatureLM (8×7B) at 71.9% and EditRetro at 60.8%. However, its Top-3 accuracy (75.6%) falls well below NatureLM's 87.4%, a trade-off the authors attribute to concentration of probability mass versus candidate diversity—an honest concession that the sequential native-representation paradigm favors precision over recall.

MOF generation: LOGOS-8B achieves Valid 45.19%, VNU 39.02%, and NBB 17.78%—a 76% relative improvement in novel-building-block rate over MOFFlow-2 (10.10%) and far above MOFDiff (0.00%). The NBB result is the paper's strongest evidence of generative exploration beyond the training distribution, since it requires producing chemically plausible linkers absent from the training set.

Generalization to unseen formats: On protein editing (GFP, AAV), where the task format was not pre-trained, LOGOS reaches Fitness 0.93 on GFP Hard difficulty versus 0.35 for the best baseline (GGS), with near-invariance to difficulty increases. On antibody CDR design under the inverse-folding setting—competing against methods conditioned on 3D backbone geometry—LOGOS-8B achieves the best AAR and scRMSD on CDR1/CDR2 loops (e.g., 85.18% on CDR-L1) but trails RADAb on CDR-H3 (46.95% vs. 57.02%). The authors correctly note that CDR-H3 diversity arises from V(D)J recombination and is poorly predictable from framework context alone, making this gap an inherent limitation of sequence-only modeling rather than a framework failure; competitive AntiBERTy plausibility scores suggest the model generates distributionally valid alternatives even when reference recovery is low.

Limitations and open questions

The paper concedes several constraints plainly. Nucleic-acid modalities (genomic, transcriptomic) are absent due to computational limits, so "general-purpose" coverage of the natural sciences remains incomplete. The scaling study spans only 1B–8B parameters and a subset of public data per domain, leaving the scaling behavior of scientific modalities at larger scales uncharacterized. Spatial information is captured only through discretized proxies; whether explicit geometric conditioning would improve structure-sensitive tasks such as CDR-H3 design is left open. Additionally, the pocket-identification result depends on self-distilled P2Rank annotations, and the retrosynthesis Top-3 deficit suggests the paradigm may systematically trade ranking precision for diversity—both points warranting further investigation.

Conclusion

LOGOS demonstrates that a unified discrete scientific grammar can support competitive multi-domain generative modeling in a purely sequential paradigm, with consistent positive scaling from 1B to 8B and measurable cross-domain transfer during both pre-training and fine-tuning. Its strongest empirical claims—the parameter-efficiency advantage over language-interfaced alternatives, the critical role of cross-representation translation data, and the high novel-building-block rate in MOF generation—together provide preliminary but concrete support for the thesis that scientific foundation models should align deeply with LLM infrastructure while operating in domain-native representation spaces.