Sequence-Axis Hybridization in Deep Modeling

• Sequence-axis hybridization is a paradigm that enables per-token, bidirectional mixing of different modalities or mechanisms in deep sequence models.
• It dynamically routes between computational strategies like attention and linear recurrence, allowing models to allocate resources based on token context.
• Empirical results in RNA design and language modeling demonstrate improved efficiency, robust representation sharing, and enhanced performance over traditional hybrid approaches.

Sequence-axis hybridization is a paradigm in deep sequence modeling and alignment that enables flexible, bidirectional mixing of distinct representational modalities or sequence-mixing mechanisms along the positional axis of an input sequence. In contrast to traditional hybrid models, which alternate mechanisms at the layer or block level, sequence-axis hybridization dynamically allocates mixing mechanism choice at the token level, allowing models to adapt computational resources or representational pathways as a function of positional or contextual token needs. This concept has emerged in both biological sequence modeling (e.g., the fusion of sequence and structure axes in RNA foundation models) and neural sequence models for language and signal processing (e.g., per-position switching between attention and linear recurrent mixers). Sequence-axis hybridization addresses the challenges inherent in fusing heterogeneous modalities or mechanisms—such as combinatorial mapping, data imbalance, and representational compatibility—by leveraging shared internal embeddings and cross-modal objectives or shared parameterizations.

1. Conceptual Foundations and Definitions

In sequence modeling, hybridization conventions have historically focused on the network depth axis—interleaving different blocks (e.g., attention and recurrence) across layers—or the local fusion of outputs within each block. Sequence-axis hybridization, by contrast, empowers the model to select, at every token position, a mechanism or modality appropriate for that segment. The token-level flexibility can, for example, allow efficient computation on uninformative positions via linear recurrences and deploy higher-capacity mechanisms, such as softmax attention, on salient or context-dependent segments (Li et al., 27 May 2026).

In biological sequence modeling, notably RNA foundation modeling, this paradigm appears as native bidirectional fusion between sequence tokens and structure-encoding axes (e.g., dot-bracket secondary structure schemes). A model embodying sequence-axis hybridization forms a unified latent space permitting free mapping between axes—for instance, predicting structural features from raw sequence (Seq2Str) or reconstructing valid sequences possessing defined structural motifs (Str2Seq) (Yang et al., 2024).

2. Sequence-Axis Hybridization in Foundation Models: Genomic Context

In RNA and genome-scale LLMs, sequence–structure alignment embodies sequence-axis hybridization by facilitating bidirectional mappings between a nucleotide sequence and its corresponding structure. The two principal subproblems are:

• Seq2Str: Sequence-to-structure prediction; for each base, predicting its pairing status ("(", ")", or ".")
• Str2Seq: Structure-to-sequence prediction; inferring the nucleotide sequence consistent with a given secondary structure label.

Key obstacles in this context are:

• Combinatorial explosion: A length-$N$ RNA sequence admits exponentially many secondary-structure foldings; a single structure maps to many compatible sequences.
• Data imbalance: Scarce experimentally-validated structure data, and ill-posedness in one-to-many Str2Seq supervision.

Foundation models such as OmniGenome directly address these challenges by introducing cross-modal, structure-contextualized objectives into pretraining rather than relying solely on masked language modeling (MLM) of nucleotide sequences. The modeling pipeline supports both sequence-only and structure-plus-sequence inputs/outputs and accomplishes hybridization by a single token-embedding vocabulary, shared Transformer encoder, and explicit bidirectional (Seq2Str, Str2Seq) training losses (Yang et al., 2024).

3. Sequence-Axis Hybridization in Neural Sequence Models

In the context of neural language modeling and sequence mixing, sequence-axis hybridization has been operationalized by per-token routing between mixers with divergent algorithmic and computational properties (e.g., quadratic attention and linear recurrence). The Oryx model exemplifies this approach:

• Each block comprises shared parameterizations (key, value, gating, and output projections) and dual-mode query projections, enabling two distinct computation pathways.
• At each timestep, the model can select either the attention mixer ($o^{\attention}_t$) or the linear recurrent mixer ($o^{\lin}_t$), driven by a binary mode selector $m(t)$.
• Both mechanisms operate on a shared internal representation, ensuring seamless state handover upon mode-switching (Li et al., 27 May 2026).

This mechanism circumvents the inefficiencies and representational discontinuities of depth- or head-fusion hybridization. It enables adaptive allocation of quadratic (attention) or linear (recurrence) computation as required, and supports training regimes where models experience mixed-mode operation over token "chunks."

4. Architectural and Training Principles

The realization of sequence-axis hybridization in both genomic and neural modeling domains is predicated on the tight coupling of shared internal representation spaces and modality-independent tokenization. For instance:

• OmniGenome’s architecture utilizes a single vocabulary embedding both nucleotide and dot-bracket structure tokens, with a Transformer encoder that feeds structure and sequence tokens through interleaved attention (Yang et al., 2024).
• The Oryx model architecture maintains >90% shared parameters between attention and linear mixers, differing only in mode-specific query projections and a minimal set of mixer-specific state parameters (Li et al., 27 May 2026).

Training strategies for Oryx employ chunked mixed-mode sampling—assigning each token chunk a mixer mode at random (e.g., 1/4 attention, 3/4 linear)—to enforce robustness and representation-sharing across modes (Li et al., 27 May 2026). For OmniGenome, cross-modal reconstruction and prediction losses are combined in pretraining to encode both axes into the shared latent space (Yang et al., 2024).

5. Empirical Results and Implications

The effectiveness of sequence-axis hybridization has been empirically validated in both biological and language modeling contexts.

• In RNA design on EternaV2, OmniGenome—when leveraging explicit structure-contextualized Str2Seq mapping—solved 74% of puzzles, while sequence-only baselines and prior FMs failed beyond 5%. Feeding structure to the model lifts Rfam F1 from 20% (ViennaRNA) to 83%, exceeding purely algorithmic folding methods (Yang et al., 2024).
• For LLMs, Oryx matches or surpasses pure-Transformer baselines in attention mode, and outperforms competitive linear recurrent models (Mamba-2, Gated DeltaNet) in linear mode, with both modes exceeding single-mixer performance by +0.7 percentage points at the 1.4B parameter scale under identical training budgets (Li et al., 27 May 2026).
• On retrieval tasks, Oryx processes only a small fraction of tokens in attention mode (e.g., <10%) yet maintains Transformer-level retrieval accuracy, and mid-sequence mode switches yield a rapid recovery in token-wise perplexity to non-switch baselines, substantiating the compatibility of shared internal representations.

These results establish that bidirectional, per-token hybridization is tractable and effective in sequence modeling, and provides advantages in resource adaptation, representational flexibility, and sample efficiency.

6. Advantages, Limitations, and Future Directions

Sequence-axis hybridization yields token-level adaptive compute, enabling focused use of computationally intensive mechanisms when and where required. The sharing of >90% of parameters across mechanisms supports unified representations and seamless mid-sequence mode transitions, addressing prior incompatibilities seen in purely stacked or fused hybrid models. Efficiency gains are realized when most tokens can be processed in linear fashion, resorting to quadratic complexity only for select segments.

Limitations include the increased implementation complexity of maintaining multiple mixer states (e.g., KV cache and linear recurrent state), reliance on currently static or random routing policies (e.g., chunk sampling rather than learned switching), and the necessity of providing both modalities or mixer-specific supervision to exploit the full bidirectional potential.

This suggests that learned, dynamic routing (e.g., via reinforcement learning or differentiable routing with Gumbel-softmax) represents a promising extension. In genomic contexts, expanding to broader structure types, more realistic in-vivo mappings, or leveraging fully unsupervised cross-modal alignments may further extend the paradigm.

7. Relation to Adjacent Hybridization Paradigms

Sequence-axis hybridization contrasts with layerwise (inter-layer) or intra-layer hybrid mechanisms, where mixing choices are applied to blocks or within heads/outputs locally. Depth-axis (inter-layer) hybrids alternate block types, which can introduce representational discontinuities and blunt adaptation to local token properties. Intra-block (within-layer) hybridization fuses outputs via weighted sums, potentially diluting the individualized expressivity of each mechanism. Sequence-axis approaches move the granularity of mixing to the token level, allowing maximal adaptability and compute efficiency.

In summary, sequence-axis hybridization constitutes both a representational and architectural innovation, enabling flexible, bidirectional, per-token modal fusion through unified internal spaces. The result is improved performance on bidirectional mappings (e.g., sequence ↔ structure in RNA models), adaptation to diverse downstream tasks, and efficient long-context inference in large-scale neural models (Yang et al., 2024, Li et al., 27 May 2026).