Navigating heterogeneous protein landscapes through geometry-aware smoothing

Abstract: The evolutionary fitness landscape of biological molecules is extremely sparse and heterogeneous, with functional sequences forming isolated dense ``islands'' within a vast combinatorial space of largely non-functional variants. Protein sequences, in particular, exemplify this structure, yet most generative artificial intelligence models implicitly assume a homogeneous data distribution. We show that this assumption fundamentally breaks down in heterogeneous biological sequence spaces: fixed global noise levels impose a destructive trade-off, either oversmoothing dense functional clusters or fragmenting sparse regions and producing non-functional hallucinations. To address this limitation, we introduce \emph{Density-Dependent Smoothing} (DDS), a geometry-aware generative framework that adapts stochastic smoothing to the local density of the underlying sequence landscape. By inversely coupling diffusion noise to estimated sequence density, DDS enables gentle refinement in high-density functional regions while promoting controlled exploration across sparse regions. Implemented as a plug-in mechanism for discrete molecular sampling, DDS consistently outperforms state-of-the-art diffusion and autoregressive models across antibody repertoires, therapeutic antibody design, antimicrobial peptide generation and coronavirus antibody design. Together, these results show that fixed global smoothing assumptions fundamentally limit generative modeling in sparse biological sequence spaces, and that geometry-aware smoothing removes this constraint, enabling reliable exploration and design previously unattainable with fixed-noise generative models.

• The paper presents Density-Dependent Smoothing (DDS) to dynamically adapt noise based on local density, mitigating oversmoothing and fragmentation in protein sequence generation.
• It employs kernel density estimation on biophysical feature projections to condition noise, ensuring enhanced mode recovery and biological plausibility.
• Empirical benchmarks demonstrate DDS’s superior performance over fixed-noise models in reducing L1 error, preserving diversity, and maintaining structural confidence.

Geometry-Aware Smoothing for Protein Sequence Generation: A Technical Analysis

Motivation and Problem Statement

The combinatorial complexity and extreme sparsity of biological sequence spaces, especially protein sequences, pose fundamental challenges to generative modeling. Functional protein sequences form isolated, dense “islands” within an astronomically vast space, leading to severe mode fragmentation and oversmoothing when naïvely applying globally-tuned noise scales in diffusion-based or autoregressive generative models. The assumption of homogeneous data density underpinning these methods leads, as demonstrated, to destructive trade-offs: high fixed noise merges distinct modes, erasing local structural detail, while low fixed noise causes sampling trajectories to stall within sparse voids and induces memorization, undermining both novelty and biological plausibility.

Figure 1: The impact of fixed and adaptive noise scales on synthetic protein landscapes, illustrating oversmoothing, fragmentation, and the DDS mechanism.

The paper introduces Density-Dependent Smoothing (DDS)—a geometry-aware generative framework that dynamically adapts stochastic smoothing to local data density. By directly estimating local sequence density via biophysical property-based kernel density estimation, DDS inversely couples diffusion noise to density, applying gentle refinement in dense functional clusters and strong smoothing in sparse regions. As a plug-in for discrete molecular sampling models, DDS aims to resolve the fundamental geometric mismatch inherent in fixed-noise generative modeling, enabling reliable exploration and mode recovery without excessive hallucination or trivial memorization.

Methodology

The core innovation is the replacement of global noise schedules with a per-sample, data-dependent noise scale, $\sigma(x)$, proportional to $(1-\hat{p}(f(x)))$ for a given feature-based density estimate $\hat{p}(f(x))$. Sequence representations are projected into a low-dimensional biophysical feature space (e.g., hydrophobicity, molecular weight, isoelectric point, aromaticity, instability index, $\beta$-sheet content). Kernel density estimation (KDE) in this domain is accelerated via Random Fourier Features to scale to large antibody datasets (OAS, DBAASP, CoV-AbDab). During training, denoising objectives are conditioned on $\sigma(x)$; during inference, $\sigma$ is sampled from the empirical distribution, ensuring the generative trajectory reflects local sparsity.

Sampling is implemented via a $\sigma$-conditioned score network, using Langevin dynamics and jump projection onto the discrete sequence domain. The method is theoretically linked to classifier-free guidance (CFG), with geometric density serving as a continuous conditioning class rather than a discrete label.

Synthetic Benchmark: Fidelity and Exploration

Controlled experiments on a synthetic 4-dimensional sequence space validate DDS against standard fixed-noise baselines, measuring $L_1$ error, mode coverage, and spurious generation. DDS consistently achieves lowest $L_1$ error, recovers true and held-out modes without favoring false modes, and generalizes beyond training support more efficiently than any single fixed-noise baseline. This demonstrates that per-sample geometric smoothing mitigates destructive over- and under-smoothing, enabling controlled exploration.

Figure 2: $L_1$ norm error rate for DDS versus fixed-noise baselines on the synthetic multimodal landscape.

Biological Sequence Design Task Performance

Antibody Sequence Generation (OAS)

DDS was evaluated alongside dWJS, SeqVDM, DEEN, IgLM, ESM2, and GPT-3.5 on OAS antibody repertoires, using metrics including uniqueness, intra-diversity (ID), edit distance (ED), distributional consistency score (DCS), and Wasserstein distance (WD). DDS maintains robust diversity (ID, ED) while outperforming baselines on distributional consistency and biochemical fidelity (WD). Fixed-noise baselines either exhibit high diversity with low sample quality or vice versa, showing DDS’s superior balance.

CDR Mutant Generation

On the hu4D5 dataset, DDS achieves high binding probability (classifier-estimated), ID, and harmonic mean (HM), outperforming GFlowNets, gg-dWJS, and autoregressive models. DDS retains evolutionary plausibility and functional diversity in mutant discovery, bridging the previously observed trade-off between novelty and quality.

Antimicrobial Peptide Generation

AMP generation task (DBAASP) further corroborates DDS as delivering high classifier-verified AMP probability, diversity (ID, ED), and harmonic mean, suggesting that geometry-aware smoothing faithfully explores diverse, biologically plausible peptide designs. Uniform noise assignment yields inferior results, confirming the criticality of density-based $\sigma$ estimation.

Coronavirus Antibody Design

DDS generates diverse VH–VL sequences matching ground-truth immunogenicity distribution (NetMHCIIpan metrics, KS, WD) more closely than baselines, preserving novelty and intra-diversity while avoiding memorization. This demonstrates DDS’s utility in safety-critical biological design scenarios.

Structural, Evolutionary, and Homology Analysis

DDS-generated sequences exhibit high structural foldability (ESM-2 pLDDT scores), sequence naturalness (pseudo-perplexity, mean log-probability), and evolutionary novelty (BLAST homology analysis). Unlike fixed high-noise models, which achieve high LLM and structure scores via memorization, DDS balances validity and novelty, reducing extreme duplication while maintaining functional plausibility.

Practical and Theoretical Implications

DDS establishes geometry-aware smoothing as a necessary principle for generative modeling in heterogeneous sequence spaces. Fixed global noise scales induce damaging trade-offs: oversmoothing or fragmentation, diversity loss or unrealistic exploration, and trivial memorization in biological design. DDS’s per-sample density adaptation enables scalable mode recovery, exploration, and biologically faithful de novo protein design, outperforming both fixed-noise diffusion and autoregressive models across multiple evaluation metrics and tasks.

The methodological foundations—KDE-based density estimation, $\sigma$-conditioned score networks, Langevin sampling, and self-guidance in continuous density space—are tractable and extensible. Future research may formalize error bounds linking density estimation, $\sigma$ selection, and score approximation, and further automate selection of noise range bounds. Theoretical analysis may relate DDS’s performance to underlying manifold geometry or connect it with nonparametric empirical Bayes denoising guarantees.

Strong empirical results: On multiple real-world biological sequence generation tasks, DDS consistently achieves superior trade-offs between quality and diversity, as measured by classifier scores, edit distance, biophysical similarity, immunogenicity, structural confidence, and homology. The method avoids destructive memorization seen in fixed-noise models, generating genuinely novel and plausible protein sequences.

Contradictory claims: The paper robustly demonstrates that fixed-noise generative modeling is fundamentally incompatible with highly heterogeneous biological landscapes. This contradicts prevailing assumptions and provides rigorous evidence for mode fragmentation, oversmoothing, and memorization failures.

Conclusion

Density-Dependent Smoothing (DDS) offers a geometric solution to generative modeling in sparse biological sequence spaces, outperforming fixed-noise baselines in fidelity, diversity, and biological plausibility. Geometry-aware smoothing enables principled, scalable exploration and design in settings inaccessible to global noise-based generative frameworks. As protein and molecular design tasks grow in complexity, adaptive stochastic smoothing aligned with local density will likely become a foundational tenet for robust, trustworthy generative AI. Advances in density estimation and theory may further refine the methodology, extending its applicability to increasingly large and heterogeneous biological datasets.