HiSS Blocks: Hierarchical State Space Models

• Hierarchical State Space (HiSS) Blocks are advanced neural modules that embed multi-scale state-space recurrences to capture both local and global dependencies.
• They leverage hierarchical partitioning and layered SSM computations to process temporal, spatial, and irregular data efficiently.
• HiSS Blocks demonstrate superior scalability and performance in applications such as video segmentation, point-cloud analysis, and hierarchical reinforcement learning.

Hierarchical State Space (HiSS) Blocks are a class of neural building blocks characterized by the embedding of state-space recurrences within a hierarchical, multi-scale architecture. HiSS Blocks have emerged as a unifying paradigm for scalable sequence modeling, high-dimensional feature aggregation, and abstracted representation learning across diverse modalities—including time series, spatial data, irregular structures, and complex control tasks. Their central motif is the composition of state-space models (SSMs) at multiple resolutions, leveraging both local and global dependency modeling, and—distinct from flat SSMs—explicitly injecting multi-scale structure or domain-aligned scanning strategies. Recent advances deploy HiSS Blocks in vision, point-cloud analysis, temporal task segmentation, hierarchical reinforcement learning, and 3D generative modeling, demonstrating superior scalability and inductive bias compared to both “flat” state-space and attention-based models (Bhirangi et al., 2024, Zhang et al., 17 Apr 2025, Wu et al., 26 Jun 2025, Chharia et al., 24 May 2026, Steccanella et al., 2021).

1. Mathematical Foundation of State-Space Hierarchies

A HiSS Block typically extends the conventional linear state-space model: $$x_{k} = A x_{k-1} + B u_{k}, \quad y_{k} = C x_{k} + D u_{k}$$ by either stacking SSMs at multiple temporal/spatial scales, partitioning the state or input domains, or both. In canonical continuous sequence-to-sequence instantiations, the input is segmented into “chunks” (length $K$), each processed by a low-level SSM. Chunk features are then processed by a higher-level SSM operating at a lower effective rate, yielding a multi-scale abstraction: $$x^{(2)}_k = A^{(2)} x^{(2)}_{k-1} + B^{(2)} h^{(1)}_k, \quad h^{(1)}_k = \text{low-level SSM}(u_{(k-1)K+1:kK})$$ This structure generalizes to spatial domains by replacing temporal sequences with token grids, local neighborhoods, or learned partitions (Bhirangi et al., 2024, Zhang et al., 17 Apr 2025, Chharia et al., 24 May 2026).

Distinct HiSS Block families employ various strategies:

• Chunked hierarchy: Non-overlapping local windows, with layer 1 SSM extracting fine-scale features, layer 2 SSM modeling long-range or global context (Bhirangi et al., 2024).
• Spatial hierarchy: Recursive downsampling and aggregation (e.g., via farthest point sampling in point clouds) at each level, followed by SSM processing (Zhang et al., 17 Apr 2025).
• Hybrid sequence/image hierarchy: Joint use of SSM recurrences (including multi-axis scans) and self-attention in stacked “block” fashion (Chharia et al., 24 May 2026).
• Hierarchical input-dependent SSMs: Independent SSMs over locally defined segments (e.g., pseudo-phases), followed by global SSMs for dependencies across segments (Wu et al., 26 Jun 2025).

2. Core Architectural Variants and Integration Strategies

The implementation of HiSS Blocks varies across domains but universally includes hierarchical partitioning and multi-level SSM computation.

• Time Series (Sensor and Video Data): Input sequences are divided into fixed-size chunks, processed by low-level SSMs that summarize fast, local dynamics. High-level SSMs operate on these summaries to capture slow, global trends (Bhirangi et al., 2024, Wu et al., 26 Jun 2025). In surgical phase recognition, Hierarchical Input-Dependent State Space Blocks combine block-diagonal local SSMs (segment-wise) with full dense global SSM mixers (Wu et al., 26 Jun 2025).
• Spatial/Structural Data (Point Clouds): At each hierarchy level, HiSS Blocks aggregate local features using neighborhood-based SSMs (vanilla Point SSMs over k-NN groups), then combine information globally with group SSMs that scan the entire set under multiple coordinate orderings (e.g., {x, y, z, inside-out}) (Zhang et al., 17 Apr 2025). Serialization enables SSMs to model structure in irregular domains.
• 2D and 3D Grid Data (Images, Gaussian Primitives): MVCHead employs HiSS Blocks to process $H \times W$ token grids. Each block fuses global self-attention and Hierarchical Bi-directional State Scans (HiBiSS) along both spatial axes and at increasing refinement levels (Chharia et al., 24 May 2026).
• Reinforcement Learning (Abstract State Blocks): HiSS-Blocks partition state spaces into “blocks” using learned soft (then hard) abstractions. Transition policies between blocks are learned as subtask options; the manager plans over abstract transitions, while option policies solve intra-block transitions (Steccanella et al., 2021).

Domain	Hierarchy Mechanism	SSM Application
Sensor Time Series	Chunks in time	Layered SSMs (fast+slow)
Irregular Point Clouds	Farthest point sampling at M levels	KNN local + group SSMs
Grid/Images	Grid upsampling/refinement	2D multi-axis SSM + attention
RL/MDP	State abstraction partitions (“blocks”)	Options over block transitions
Video (Surgical)	Phase-based segmentation	Block-diag + global ID-SSM/Mixer

3. Mathematical Details of Block Operations

Most HiSS Block variants are grounded in discrete or continuous state-space equations. For instance, in (Wu et al., 26 Jun 2025), each input-dependent SSM block within the hierarchy involves:

$$x_t = \overline{A}_t x_{t-1} + \overline{B}_t u_t,\quad y_t = C_t x_t$$

where parameters $\overline{A}_t$, $\overline{B}_t$, $C_t$ are dynamically generated from $u_t$ (feature at time t), potentially incorporating a learnable timescale $\Delta_t$. Local SSMs are typically block-diagonal (each segment/phase processed independently), global SSMs are full lower-triangular (dense recurrence). Temporal selectivity emerges via data-adaptive state updates, e.g., large $K$0 “resets” the recurrent state at phase boundaries (Wu et al., 26 Jun 2025).

MVCHead’s spatial HiSS Block intertwines self-attention and four 1D SSM scans (“HiBiSS”) along both grid axes and directions. Each scan applies separate, learned linear operators and fuses results via concatenation and projection (Chharia et al., 24 May 2026).

4. Applications and Empirical Performance

HiSS Blocks deliver demonstrable advantages in tasks involving spatiotemporal structure, multi-scale reasoning, and/or very long input sequences:

• Continuous Sequence Modeling: Across six real-world sensor tasks, HiSS Blocks reduce mean squared error by at least 23% compared to best flat SSMs (Mamba), LSTM, and Transformers, while scaling linearly in sequence length. They exhibit enhanced sample efficiency, outperforming alternatives even in low-data regimes (Bhirangi et al., 2024).
• Medical Point Clouds: Hierarchical Point SSM Blocks outperform transformer-based and vanilla SSM baselines on shape classification, completion, and segmentation benchmarks (MedPointS), due to their local-to-global SSM pipeline and learned, domain-specific serialization (Zhang et al., 17 Apr 2025).
• Video-based Temporal Segmentation: Hierarchical Input-Dependent SSMs yield state-of-the-art accuracy in long surgical video phase recognition, attaining +2.8 to +12.9 percentage point improvement depending on the dataset (Wu et al., 26 Jun 2025). The architecture models fine “short-term” local phase dynamics and long-range temporal context without quadratic attention cost.
• 3D Generation (Gaussian Splatting): HiSS Blocks in MVCHead enable progressive, coarse-to-fine refinement of 3D Gaussian primitives from single 2D images without multi-view supervision, incorporating both axis-aligned local consistency (HiBiSS) and global self-attention. This approach achieves state-of-the-art perceptual quality and consistency in 3D head avatar synthesis (Chharia et al., 24 May 2026).
• Hierarchical RL: In gridworld domains with sparse rewards, HiSS-Blocks (state abstraction + options) yield near-universal success on tasks where flat DQN or self-imitation learning fail. Transfer learning (reusing learned blocks/options) accelerates learning in new, related environments by 30–50% (Steccanella et al., 2021).

5. Implementation and Hyperparameter Choices

Implementation details are strongly task- and modality-dependent but share these features:

• Chunk Size/Resolution: A key hyperparameter in temporal models; larger chunks decrease high-level layer compute but may underspecify fast dynamics (Bhirangi et al., 2024).
• Hierarchy Depth: Number of SSM levels; more levels grant finer scale separation at the cost of greater compute/memory (Chharia et al., 24 May 2026, Zhang et al., 17 Apr 2025).
• Local Neighborhood Sizes: In point cloud HiSS blocks, KNN radii and number of scans (e.g., 16/32 neighbors, 4 axes) control the granularity of feature aggregation (Zhang et al., 17 Apr 2025).
• Conditional Normalization: Some architectures modulate SSM computation at each level via learned latent vectors (e.g., AdaIN in MVCHead) (Chharia et al., 24 May 2026).
• Optimization: State-space parameters are trained jointly with feature extractors, usually via Adam or AdamW, with learning rates (1e-3 or task-specific) and dropout regularization (Bhirangi et al., 2024, Zhang et al., 17 Apr 2025, Wu et al., 26 Jun 2025).

6. Limitations, Ablations, and Future Directions

Empirical ablations indicate:

• Fixed chunk sizes, while practical, may restrict adaptivity. Future HiSS work may employ learnable or data-adaptive segmentation for chunking (Bhirangi et al., 2024).
• SSMs currently struggle with extremely high-dimensional targets (e.g., 60-joint kinematics) relative to LSTMs; integrating richer nonlinearities or cross-attention might increase robustness (Bhirangi et al., 2024).
• Information flow is typically bottom-up; cross-level or top-down modeling (attention, gating) is an open extension.

Across domains, HiSS Blocks consistently outperform both flat SSMs and quadratic-attention models in scaling, efficiency, and parameter efficiency. A plausible implication is that domain-aligned multi-scale recurrence in the hierarchy is a crucial inductive bias. Extensions to deeper hierarchies, multimodal fusion, and non-linear SSM updates represent primary research directions (Chharia et al., 24 May 2026, Bhirangi et al., 2024).