HydraMamba: Hybrid Mamba-Derived Architectures

• HydraMamba is a family of architectures built on selective state-space models, augmented with auxiliary pathways (e.g., self-attention, convolution) to recover bidirectionality and locality.
• It employs mechanisms such as quasiseparable mixers and dual scanning (forward and backward) to achieve efficient linear-time processing with fixed per-token memory.
• Hybrid models like Nemotron-H demonstrate HydraMamba’s advantage in long-context generation speed and scalability by strategically mixing Mamba layers with sparse attention.

to=arxiv_search.search 天天中彩票软件 大发pkाज্ঞata 天天买彩票.^^^^^^^^17query17^^^^^^^^ code: {"^^^^^^^^17query17 OR \17^^^^^^^^"Nemotron-H\" OR \17^^^^^^^^"Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers\"^^^^^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^^^^^", "^^^^^^^^17max_results17 ^^^^17HydraMamba OR \17query17^^^^, "^^^^^^^^17sort_by17 "^^^^^^^^17submittedDate17 สูตรบาคาร่า to=arxiv_search.search code ＿色ية 彩神争霸是 to=arxiv_search.search 彩票娱乐注册.^^^^^^^^17query17^^^^^^^^ code: {"^^^^^^^^17query17 "^^^^^^^^17max_results17 ^^^^17HydraMamba OR \17query17^^^^, "^^^^^^^^17sort_by17 "^^^^^^^^17submittedDate17 to=arxiv_search.search 无码不卡高清免费.^^^^^^^^17query17^^^^^^^^ code: {"^^^^^^^^17query17 OR \17 OR Jamba hybrid Transformer-Mamba17^^^^^^^^"Nemotron-H\" OR ^^^^^^^^17HydraMamba OR \17 OR Jamba hybrid Transformer-Mamba17^^^^^^^^"HydraMamba\" OR ^^^^^^^^17HydraMamba OR \17 OR Jamba hybrid Transformer-Mamba17^^^^^^^^"Hydra\" OR \17submittedDate17^^^^^^^^"Mamba\"", "^^^^^^^^17max_results17 ^^^^17HydraMamba OR \17query17^^^^, "^^^^^^^^17sort_by17 "relevance"} HydraMamba is a label attached to several recent Mamba-derived architectures that share a common premise: the selective state-space model is treated as the computational trunk, but it is supplemented by additional pathways to recover properties that a plain unidirectional SSM does not provide, such as content-based retrieval, bidirectionality, locality, or heterogeneous conditional capacity (&&&^^^^17query17^^^^&&&, &&&^^^^17HydraMamba OR \17^^^^&&&, &&&^^^^17 OR \17^^^^&&&). In current arXiv usage, the term does not identify a single canonical model. Instead, it spans hybrid Mamba–Transformer LLMs, quasiseparable bidirectional matrix mixers, multi-head latent Mamba recommenders, and point-cloud backbones with locality-aware SSM blocks. This suggests that HydraMamba is best understood as a design family organized around Mamba plus auxiliary heads, paths, or mixers.

^^^^17HydraMamba OR \17^^^^. Terminological scope

Across the cited literature, “HydraMamba” is used in several related but non-identical senses. In the Nemotron-H report, it denotes a hybrid architecture that mixes a small number of Transformer self-attention layers with many Mamba-^^^^17 OR \17^^^^ layers to obtain “the best of both worlds”: content-based reasoning and global retrieval from attention, together with linear-time, constant-memory generation from Mamba (&&&^^^^17query17^^^^&&&). In “Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers,” the central construction is a bidirectional extension of Mamba implemented as a quasiseparable matrix mixer, intended primarily for non-causal encoder-style modeling rather than autoregressive decoding (&&&^^^^17HydraMamba OR \17^^^^&&&). In point cloud learning, “HydraMamba” is the official name of an SSM-based backbone with shuffle serialization, ConvBiS^^^^17submittedDate17^^^^, and multi-head S^^^^17submittedDate17^^^^ (&&&^^^^17 OR \17^^^^&&&). In sequential recommendation, “Hydra” denotes a multi-head latent Mamba architecture rather than a Transformer–Mamba hybrid (&&&^^^^17submittedDate17^^^^&&&).

Usage	Domain	Defining mechanism
Nemotron-H “HydraMamba”	Autoregressive LMs	Sparse attention plus many Mamba-^^^^17 OR \17^^^^ layers
Hydra	Bidirectional sequence modeling	Quasiseparable matrix mixer with forward/backward scans
Hydra	Sequential recommendation	Multi-head latent Mamba with latent subspaces
HydraMamba	Point cloud learning	Shuffle serialization, ConvBiS^^^^17submittedDate17^^^^, and MHS^^^^17submittedDate17^^^^

Two nearby cases are explicitly disambiguated in their own papers. “Decision Mamba-Hybrid” is the correct name for the hybrid Mamba–Transformer in-context RL agent, and the paper states that HydraMamba is not an official term there (&&&^^^^17query17^^^^&&&). Likewise, the ^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^D human pose model is officially named HGMamba, and its paper explicitly states that there is no separate model named HydraMamba in that setting (&&&^^^^17all:HydraMamba17^^^^&&&). A recurring misconception is therefore to treat HydraMamba as a single universally defined architecture; the literature instead supports a broader family resemblance centered on Mamba augmented by multiple computational modes.

^^^^17 OR \17^^^^. Shared mathematical substrate

Despite this terminological diversity, the underlying mechanism is stable. The common starting point is the selective state-space model. In the continuous-time form used in the Nemotron-H exposition, an SSM can be written as

PRESERVED_PLACEHOLDER_^^^^17query17^^^^

Under Mamba-style discretization with data-dependent parameters,

PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17^^^^

where PRESERVED_PLACEHOLDER_^^^^17 OR \17^^^^, PRESERVED_PLACEHOLDER_^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^, and PRESERVED_PLACEHOLDER_^^^^17max_results17^^^^ depend on the input PRESERVED_PLACEHOLDER_^^^^17sort_by17^^^^ and the sequence is computed by a linear-time scan (&&&^^^^17query17^^^^&&&). The key consequence is that a Mamba layer performs recurrent selective scanning with per-token cost independent of processed context length PRESERVED_PLACEHOLDER_^^^^17submittedDate17^^^^ during generation, while maintaining a fixed recurrent state rather than an PRESERVED_PLACEHOLDER_^^^^17query17^^^^ KV cache.

The Hydra matrix-mixer framework generalizes this view by treating a sequence mixer as a linear map along the length dimension. In that formulation, self-attention is an input-dependent mixer PRESERVED_PLACEHOLDER_^^^^17all:HydraMamba17^^^^ applied to projected values, while selective SSMs such as Mamba induce lower-triangular semiseparable mixers whose structure permits linear-time application via scans (&&&^^^^17HydraMamba OR \17^^^^&&&). Hydra’s main conceptual move is to lift Mamba from the semiseparable class to the quasiseparable class, so that both lower- and upper-triangular off-diagonal structure are modeled. Operationally, Hydra implements

PRESERVED_PLACEHOLDER_^^^^17max_results17^^^^

with a forward selective scan, a backward selective scan, and a free diagonal term. The paper argues that this diagonal freedom makes quasiseparable mixers strictly more expressive than addition-based bidirectional SSM heuristics (&&&^^^^17HydraMamba OR \17^^^^&&&).

A second shared idea is that HydraMamba variants are rarely content with a single homogeneous mixer. They typically introduce a second path or a head-wise decomposition: sparse attention in Nemotron-H and Jamba, forward/backward generators in Hydra, low-dimensional head-specific Mamba blocks in recommendation Hydra, or convolutional locality branches in point clouds. This suggests that the “Hydra” aspect is not merely nominal; it usually denotes architectural multiplicity around an SSM core.

^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^. Autoregressive hybrid LLMs

The clearest large-scale language-model instantiation appears in “Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models” (&&&^^^^17query17^^^^&&&). Nemotron-H includes Nemotron-H-^^^^17all:HydraMamba17^^^^B-Base / -Instruct / -VLM, Nemotron-H-^^^^17sort_by17submittedDate17^^^^B-Base / -VLM, and Nemotron-H-^^^^17max_results17query17^^^^B-Base, the last obtained from the ^^^^17sort_by17submittedDate17^^^^B model via the MiniPuzzle pruning-and-distillation pipeline. Its HydraMamba stack uses about ^^^^17all:HydraMamba17^^^^% self-attention layers, evenly dispersed through depth, while the remainder alternate Mamba-^^^^17 OR \17^^^^ and FFN layers. The ^^^^17all:HydraMamba17^^^^B model has ^^^^17sort_by17 OR \17^^^^ total layers with ^^^^17max_results17^^^^ attention layers; the ^^^^17sort_by17submittedDate17^^^^B model has ^^^^17HydraMamba OR \17HydraMamba OR \17all:HydraMamba17^^^^ total layers with ^^^^17HydraMamba OR \17query17^^^^ attention layers. The first layer is Mamba-^^^^17 OR \17^^^^, the last layer is FFN, every self-attention layer precedes an FFN, and each of Mamba-^^^^17 OR \17^^^^, attention, and FFN has its own residual skip connection. The implementation uses RMSNorm, squared ReLU in FFNs, no dropout, no linear-layer bias, separate embedding and output weights, no positional embeddings, GQA with ^^^^17all:HydraMamba17^^^^ KV heads, and Mamba state dimension ^^^^17HydraMamba OR \17 OR \17all:HydraMamba17^^^^ for ^^^^17all:HydraMamba17^^^^B and ^^^^17 OR \17sort_by17submittedDate17^^^^ for ^^^^17sort_by17submittedDate17^^^^B (&&&^^^^17query17^^^^&&&).

The motivating asymptotics are explicit. In causal generation with KV cache, self-attention incurs per-token compute PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17query17^^^^ and KV-cache memory PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17HydraMamba OR \17^^^^, whereas Mamba keeps per-token compute PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17 OR \17^^^^ and stack-wide recurrent state memory PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17 OR Jamba hybrid Transformer-Mamba17^^^^, constant in PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17max_results17^^^^ (&&&^^^^17query17^^^^&&&). Because only about ^^^^17all:HydraMamba17^^^^% of Nemotron-H layers pay KV-cache costs, long-context throughput improves substantially. Under H^^^^17HydraMamba OR \17query17query17^^^^ benchmarking with ^^^^17submittedDate17sort_by17^^^^,^^^^17sort_by17 OR Jamba hybrid Transformer-Mamba17submittedDate17^^^^ input tokens and ^^^^17HydraMamba OR \17^^^^,^^^^17query17 OR \17max_results17^^^^ generated tokens, Nemotron-H-^^^^17sort_by17submittedDate17^^^^B-Base generates ^^^^17 OR \17^^^^.^^^^17max_results17^^^^× more tokens/sec per GPU than Qwen-^^^^17 OR \17^^^^.^^^^17sort_by17^^^^- OR \17^^^^B and Llama-^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17HydraMamba OR \17^^^^-^^^^17query17query17^^^^B; the distilled Nemotron-H-^^^^17max_results17query17^^^^B-Base reaches ^^^^17 OR \17^^^^.^^^^17max_results17^^^^×, and the ^^^^17all:HydraMamba17^^^^B model is ^^^^17HydraMamba OR \17^^^^.^^^^17all:HydraMamba17^^^^× faster than Qwen-^^^^17 OR \17^^^^.^^^^17sort_by17^^^^- and ^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^× faster than Llama-^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17HydraMamba OR \17^^^^-^^^^17all:HydraMamba17^^^^B on long contexts (&&&^^^^17query17^^^^&&&).

Accuracy does not uniformly favor either side of the hybrid. On large-model evaluation, Nemotron-H-^^^^17sort_by17submittedDate17^^^^B-Base scores ^^^^17submittedDate17query17^^^^.^^^^17sort_by17^^^^ on MMLU-Pro, ^^^^17all:HydraMamba17max_results17^^^^.^^^^17 OR \17^^^^ on MMLU, ^^^^17max_results17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17query17^^^^ on GSM^^^^17all:HydraMamba17^^^^K, ^^^^17sort_by17max_results17^^^^.^^^^17max_results17^^^^ on MATH, and ^^^^17submittedDate17query17^^^^.^^^^17max_results17^^^^ HumanEval pass@^^^^17HydraMamba OR \17^^^^; the ^^^^17max_results17query17^^^^B compressed model is near-lossless, scoring ^^^^17submittedDate17HydraMamba OR \17^^^^.^^^^17all:HydraMamba17^^^^, ^^^^17all:HydraMamba17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17submittedDate17^^^^, ^^^^17max_results17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^, ^^^^17sort_by17query17^^^^.^^^^17max_results17 and ^^^^17submittedDate17HydraMamba OR \17^^^^.^^^^17query17^^^^ respectively (&&&^^^^17query17^^^^&&&). The paper is careful to note task sensitivity: Qwen-^^^^17 OR \17^^^^.^^^^17sort_by17^^^^- OR \17^^^^B remains stronger on some tasks such as MATH at ^^^^17submittedDate17max_results17^^^^.^^^^17submittedDate17^^^^ versus ^^^^17sort_by17max_results17^^^^.^^^^17max_results17 A central design trade-off therefore remains unresolved: the hybrid ratio that optimizes inference-time scaling need not be the ratio that maximizes every reasoning benchmark.

“Jamba: A Hybrid Transformer-Mamba LLM” provides an earlier hybrid language-model template with an additional MoE axis (&&&^^^^17HydraMamba OR \17query17^^^^&&&). Its released configuration contains ^^^^17 OR Jamba hybrid Transformer-Mamba17 OR \17^^^^ layers arranged as four Jamba blocks, each with PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17sort_by17^^^^ layers and attention-to-Mamba ratio PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17submittedDate17^^^^, yielding ^^^^17max_results17^^^^ attention layers and ^^^^17 OR \17all:HydraMamba17^^^^ Mamba layers overall. MoE is inserted every PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17query17^^^^ layers, with ^^^^17HydraMamba OR \17submittedDate17^^^^ experts and top-PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17all:HydraMamba17^^^^ routing at PRESERVED_PLACEHOLDER_^^^^17HydraMamba OR \17max_results17^^^^ per token, resulting in ^^^^17HydraMamba OR \17 OR \17^^^^B active parameters and ^^^^17sort_by17 OR \17^^^^B total parameters. Jamba reports ^^^^17 OR \17sort_by17submittedDate17^^^^K context support in the released base model, a ^^^^17max_results17^^^^GB KV cache at ^^^^17 OR \17sort_by17submittedDate17^^^^K tokens versus ^^^^17 OR Jamba hybrid Transformer-Mamba17 OR \17^^^^GB for Mixtral ^^^^17all:HydraMamba17^^^^×^^^^17query17 and ^^^^17HydraMamba OR \17 OR \17all:HydraMamba17^^^^GB for Llama-^^^^17 OR \17^^^^ ^^^^17submittedDate17^^^^.^^^^17query17 and approximately ^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^× throughput over Mixtral at long context (&&&^^^^17HydraMamba OR \17query17^^^^&&&). Its ablations argue that even sparse attention is important for induction-like in-context learning and format adherence, while Mamba is the efficiency mechanism that makes long-context serving practical.

^^^^17max_results17^^^^. Bidirectional and encoder-style formulations

Hydra in the matrix-mixer literature addresses a different problem class: non-causal sequence modeling in which information must flow in both directions (&&&^^^^17HydraMamba OR \17^^^^&&&). The paper starts from a unifying view in which a sequence mixer is a linear map on the input sequence, then identifies “sequence alignment” as the key parameterization axis that explains the success of input-dependent mixers such as attention and selective SSMs. A sequence-aligned matrix ties subsets of parameters to specific sequence positions, enabling token-wise parameter generation and extendability beyond training length. Attention, linear attention, SSD/Mamba, and Hydra all satisfy this property in the paper’s taxonomy (&&&^^^^17HydraMamba OR \17^^^^&&&).

Hydra’s specific contribution is to construct a bidirectional Mamba by replacing Mamba’s lower-triangular semiseparable mixer with a full quasiseparable mixer. The lower triangle is implemented by a forward scan, the upper triangle by a backward scan, and the diagonal is learned independently. Computationally, Hydra runs two linear-time scans plus a diagonal map, preserving PRESERVED_PLACEHOLDER_^^^^17 OR \17query17^^^^ complexity and memory similar to Mamba while allowing non-causal interactions (&&&^^^^17HydraMamba OR \17^^^^&&&). The paper emphasizes that this is not merely “forward SSM plus backward SSM plus addition”: the free diagonal PRESERVED_PLACEHOLDER_^^^^17 OR \17HydraMamba OR \17^^^^ is part of the reason the quasiseparable construction is more expressive than the common addition-based bidirectional heuristic.

Empirically, Hydra is framed as a drop-in replacement for attention layers in encoder-style stacks. On GLUE, after masked-LM pretraining on C^^^^17max_results17^^^^ in the BERT-base parameter regime, Hydra reaches ^^^^17all:HydraMamba17max_results17^^^^.^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^ average versus ^^^^17all:HydraMamba17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17sort_by17^^^^ for BERT, with stronger pretraining validation accuracy on C^^^^17max_results17^^^^ as well (&&&^^^^17HydraMamba OR \17^^^^&&&). In a ViT-Base backbone on ImageNet-^^^^17HydraMamba OR \17^^^^K, replacing Transformer layers with Hydra mixers yields ^^^^17all:HydraMamba17HydraMamba OR \17^^^^.^^^^17query17^^^^% Top-^^^^17HydraMamba OR \17^^^^ and ^^^^17max_results17sort_by17^^^^.^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^% Top-^^^^17sort_by17^^^^, or ^^^^17all:HydraMamba17HydraMamba OR \17^^^^.^^^^17submittedDate17^^^^/ with EMA, outperforming ViT-B at ^^^^17query17all:HydraMamba17^^^^.^^^^17all:HydraMamba17 OR \17^^^^. In the paper’s ablations, unidirectional Mamba underperforms on non-causal settings, simple bidirectional SSM combinations improve, and Hydra’s quasiseparable mixer is best, reaching ^^^^17all:HydraMamba17HydraMamba OR \17^^^^.^^^^17query17^^^^ GLUE average and the lowest C^^^^17max_results17^^^^ validation loss among the tested bidirectional variants (&&&^^^^17HydraMamba OR \17^^^^&&&). Within the HydraMamba family, this paper is important because it formalizes a route from causal selective scans to encoder-style bidirectional mixers without reverting to attention.

^^^^17sort_by17^^^^. Domain-specific adaptations

In sequential recommendation, Hydra is not a Transformer–Mamba hybrid but a multi-head latent Mamba architecture designed to address long, noisy user histories and billion-scale item vocabularies (&&&^^^^17submittedDate17^^^^&&&). Each layer splits the representation into PRESERVED_PLACEHOLDER_^^^^17 OR \17 OR \17^^^^ latent subspaces, applies per-head Mamba-^^^^17 OR \17^^^^ blocks to the historical path, processes an item-information path with RoPE, and fuses the two by

PRESERVED_PLACEHOLDER_^^^^17 OR \17 OR Jamba hybrid Transformer-Mamba17^^^^

The paper’s complexity analysis contrasts Transformer training FLOPs PRESERVED_PLACEHOLDER_^^^^17 OR \17max_results17^^^^, single large-state Mamba-^^^^17 OR \17^^^^ training FLOPs PRESERVED_PLACEHOLDER_^^^^17 OR \17sort_by17^^^^, and Hydra multi-head latent interaction training FLOPs PRESERVED_PLACEHOLDER_^^^^17 OR \17submittedDate17^^^^, with the key condition PRESERVED_PLACEHOLDER_^^^^17 OR \17query17^^^^ (&&&^^^^17submittedDate17^^^^&&&). On Amazon Reviews ^^^^17 OR \17query17 OR \17 OR Jamba hybrid Transformer-Mamba17^^^^ domains, Hydra-^^^^17query17^^^^.^^^^17max_results17all:HydraMamba17 reaches R@^^^^17HydraMamba OR \17query17^^^^=^^^^17all:HydraMamba17^^^^. and N@^^^^17HydraMamba OR \17query17^^^^=^^^^17sort_by17^^^^. OR \17sort_by17^^^^ on Movies & TV with embedding inputs, Hydra-^^^^17HydraMamba OR \17^^^^B reaches R@^^^^17HydraMamba OR \17query17^^^^=^^^^17sort_by17^^^^. and N@^^^^17HydraMamba OR \17query17^^^^=^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17 OR \17sort_by17^^^^ on Books, and Hydra-^^^^17query17^^^^.^^^^17max_results17all:HydraMamba17 with item LLM improves to R@^^^^17HydraMamba OR \17query17^^^^=^^^^17max_results17^^^^. and N@^^^^17HydraMamba OR \17query17^^^^=^^^^17sort_by17^^^^. on Movies & TV. Training efficiency is similarly emphasized: Hydra-^^^^17query17^^^^.^^^^17 OR \17all:HydraMamba17^^^^B takes ^^^^17HydraMamba OR \17 OR \17^^^^,^^^^17max_results17all:HydraMamba17HydraMamba OR \17^^^^ seconds per epoch on Movies & TV, versus ^^^^17HydraMamba OR \17 OR \17sort_by17^^^^,^^^^17all:HydraMamba17max_results17max_results17^^^^ seconds for SASRec-^^^^17HydraMamba OR \17^^^^B and ^^^^17max_results17query17^^^^,^^^^17query17query17sort_by17^^^^ seconds for HSTU-large-^^^^17HydraMamba OR \17^^^^B (&&&^^^^17submittedDate17^^^^&&&).

In point cloud learning, “HydraMamba: Multi-Head State Space Model for Global Point Cloud Learning” gives the most literal official use of the name (&&&^^^^17 OR \17^^^^&&&). The model is built around three mechanisms. First, shuffle serialization randomizes among six axis-priority ^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^D Hilbert variants—xyz, xzy, yxz, yzx, zxy, zyx—to convert unordered point sets into locality-preserving causal sequences. Second, ConvBiS^^^^17submittedDate17^^^^ combines a bidirectional S^^^^17submittedDate17^^^^ branch for global context with a ^^^^17HydraMamba OR \17^^^^D convolution branch over the serialized sequence for explicit locality. Third, MHS^^^^17submittedDate17^^^^ extends selective SSMs to multiple heads. The paper reports ^^^^17max_results17max_results17^^^^.^^^^17query17 OA on ModelNet^^^^17max_results17query17^^^^, ^^^^17all:HydraMamba17submittedDate17^^^^.^^^^17all:HydraMamba17 instance mIoU on ShapeNet Part, ^^^^17query17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17submittedDate17^^^^% mIoU on S^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^DIS Area-^^^^17sort_by17^^^^, and ^^^^17all:HydraMamba17all:HydraMamba17^^^^.^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^% OA on ScanObjectNN PB_T^^^^17sort_by17query17^^^^_RS, while using ^^^^17sort_by17max_results17^^^^ ms and ^^^^17sort_by17^^^^.^^^^17max_results17^^^^ GB for single-scene S^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^DIS inference on an RTX ^^^^17max_results17query17max_results17query17^^^^, compared with ^^^^17max_results17max_results17^^^^ ms and ^^^^17submittedDate17^^^^.^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^ GB for Point Transformer V^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^ (&&&^^^^17 OR \17^^^^&&&). Its ablations are especially diagnostic: randomized six-variant serialization reaches ^^^^17max_results17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17max_results17submittedDate17^^^^% OA versus ^^^^17max_results17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17 OR \17 OR Jamba hybrid Transformer-Mamba17^^^^% for fixed sequential assignment, and no Hilbert serialization collapses to ^^^^17all:HydraMamba17max_results17^^^^.^^^^17 OR Jamba hybrid Transformer-Mamba17all:HydraMamba17^^^^%.

Two additional papers show the broader spread of Hydra-like Mamba hybrids, even though they explicitly reject HydraMamba as the official name. Decision Mamba-Hybrid separates roles between a Mamba module that scans long across-episodic context and emits sub-goals every PRESERVED_PLACEHOLDER_^^^^17 OR \17all:HydraMamba17^^^^ steps, and a Transformer module that predicts the next PRESERVED_PLACEHOLDER_^^^^17 OR \17max_results17^^^^ actions conditioned on those sub-goals; it reports a total D^^^^17max_results17^^^^RL average of ^^^^17all:HydraMamba17query17^^^^.^^^^17sort_by17 OR Jamba hybrid Transformer-Mamba17^^^^ ± ^^^^17query17^^^^.^^^^17sort_by17max_results17^^^^ versus ^^^^17all:HydraMamba17 OR \17^^^^.^^^^17all:HydraMamba17max_results17^^^^ ± ^^^^17query17^^^^.^^^^17query17query17^^^^ for AD-Transformer, and online testing approximately ^^^^17 OR \17all:HydraMamba17^^^^× faster than Transformer baselines on long-horizon D^^^^17max_results17^^^^RL evaluation (&&&^^^^17query17^^^^&&&). HGMamba, for ^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^D human pose lifting, couples a Hyper-GCN stream for multi-granularity local dependencies with a Shuffle-Mamba stream for global spatio-temporal scanning, reaching P^^^^17HydraMamba OR \17^^^^=^^^^17 OR Jamba hybrid Transformer-Mamba17all:HydraMamba17^^^^.^^^^17submittedDate17sort_by17^^^^ mm and P^^^^17 OR \17^^^^=^^^^17 OR Jamba hybrid Transformer-Mamba17 OR \17^^^^.^^^^17all:HydraMamba17query17^^^^ mm on Human^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17submittedDate17^^^^M estimated ^^^^17 OR \17^^^^D and P^^^^17HydraMamba OR \17^^^^=^^^^17HydraMamba OR \17max_results17^^^^.^^^^17 OR Jamba hybrid Transformer-Mamba17 OR Jamba hybrid Transformer-Mamba17^^^^ mm on MPI-INF-^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^DHP for HGMamba-B (&&&^^^^17all:HydraMamba17^^^^&&&). These cases reinforce the general pattern: Mamba is used as the long-range backbone, while another path restores structure that pure scanning misses.

^^^^17submittedDate17^^^^. Compression, training, and open research directions

One major reason HydraMamba architectures remain active research objects is that they are not only new sequence mixers but also new deployment regimes. Nemotron-H introduces MiniPuzzle, a compression pipeline that ranks layers by the MSE change in penultimate activations when a layer is removed, ranks FFN neurons by aggregated post-activation magnitude, enumerates about ^^^^17max_results17query17query17^^^^ constrained NAS candidates, shortlists ^^^^17HydraMamba OR \17 OR Jamba hybrid Transformer-Mamba17query17^^^^ using next-token accuracy and next-token parent agreement, and then applies short and long distillation to obtain Nemotron-H-^^^^17max_results17query17^^^^B-Base (&&&^^^^17query17^^^^&&&). The final compressed model retains ^^^^17sort_by17^^^^ attention layers, ^^^^17max_results17max_results17^^^^ Mamba-^^^^17 OR \17^^^^ layers, ^^^^17max_results17max_results17^^^^ FFN layers, and FFN width ^^^^17 OR Jamba hybrid Transformer-Mamba17query17query17 OR \17query17^^^^, is about ^^^^17HydraMamba OR \17^^^^.^^^^17 OR \17^^^^× faster on long contexts, and is deployable in FP^^^^17max_results17^^^^ on an RTX ^^^^17sort_by17query17max_results17query17^^^^ with ^^^^17 OR Jamba hybrid Transformer-Mamba17 OR \17^^^^ GiB. The same paper also presents an FP^^^^17all:HydraMamba17^^^^ training recipe for the ^^^^17sort_by17submittedDate17^^^^B model using E^^^^17max_results17^^^^M^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^ for weights and activations, E^^^^17sort_by17^^^^M^^^^17 OR \17^^^^ for gradients, per-tensor current scaling, and BF^^^^17HydraMamba OR \17submittedDate17^^^^ retention in the first and last four layers, reporting downstream accuracy equal or better than BF^^^^17HydraMamba OR \17submittedDate17^^^^ at the same token horizon without overtraining (&&&^^^^17query17^^^^&&&).

A more modular forward-looking extension appears in “Hydra: A ^^^^17HydraMamba OR \17^^^^.^^^^17submittedDate17^^^^B-Parameter State-Space LLM with Sparse Attention, Mixture-of-Experts, and Memory” (&&&^^^^17 OR Jamba hybrid Transformer-Mamba17 OR Jamba hybrid Transformer-Mamba17^^^^&&&). This Hydra is decoder-only and combines a Mamba-style SSM backbone with intermittent sparse global attention, chunk-level Top-^^^^17 OR \17^^^^ MoE routing, and dual memories consisting of a workspace scratchpad plus Product-Key Memory. The target design envelope is approximately ^^^^17HydraMamba OR \17^^^^.^^^^17submittedDate17query17all:HydraMamba17^^^^B trainable parameters, typical active parameters around ^^^^17query17^^^^.^^^^17all:HydraMamba17query17 native ^^^^17HydraMamba OR \17submittedDate17^^^^k context with burst to ^^^^17submittedDate17max_results17^^^^k via segmental processing, ^^^^17 OR \17max_results17^^^^ blocks organized as ^^^^17all:HydraMamba17^^^^ tri-path triples, ^^^^17all:HydraMamba17^^^^ sparse global attention layers, and ^^^^17HydraMamba OR \17 OR \17^^^^ MoE pools (&&&^^^^17 OR Jamba hybrid Transformer-Mamba17 OR Jamba hybrid Transformer-Mamba17^^^^&&&). The paper explicitly presents itself as an architectural proposal rather than a finished system, and its experiments are toy-scale; nevertheless, they show the expected long-context throughput crossover, with speedup rising to ^^^^17 OR Jamba hybrid Transformer-Mamba17^^^^.^^^^17HydraMamba OR \17query17^^^^× at ^^^^17HydraMamba OR \17submittedDate17^^^^,^^^^17 OR Jamba hybrid Transformer-Mamba17all:HydraMamba17max_results17^^^^ tokens versus a parameter-matched Transformer. The same paper also makes the risks explicit: expert collapse, memory under-utilization, specialization dynamics, variable latency from controller-driven skipping, and safety concerns around memory writes (&&&^^^^17 OR Jamba hybrid Transformer-Mamba17 OR Jamba hybrid Transformer-Mamba17^^^^&&&).

Several design rules recur across the literature. Retaining some attention remains valuable for global content-based retrieval in autoregressive LMs (&&&^^^^17query17^^^^&&&, &&&^^^^17HydraMamba OR \17query17^^^^&&&). Bidirectionality is most naturally handled by quasiseparable rather than semiseparable mixers in non-causal encoders (&&&^^^^17HydraMamba OR \17^^^^&&&). Multi-head or multi-branch decompositions help recover capacity lost by replacing dense attention with structured state updates (&&&^^^^17submittedDate17^^^^&&&, &&&^^^^17 OR \17^^^^&&&). At the same time, compression is selective: Nemotron-H reports that pruning Mamba heads or reducing Mamba head dimension degraded accuracy, so depth and FFN width were the preferred axes (&&&^^^^17query17^^^^&&&). The resulting research trajectory is toward architectures in which Mamba provides the always-on linear-time substrate, while attention, MoE, memory, or locality modules are activated sparsely and strategically. This suggests that HydraMamba is less a single invention than a durable architectural doctrine for reconciling long-context efficiency with the expressive deficits of a plain SSM.