---
title: 'HiSS Blocks: Hierarchical State Space Models'
url: https://www.emergentmind.com/topics/hierarchical-state-space-hiss-blocks
type: topic
---

# HiSS Blocks: Hierarchical State Space Models

Hierarchical State Space (HiSS) Blocks are a class of neural building blocks characterized by the embedding of state-space recurrences within a hierarchical, multi-scale architecture. HiSS Blocks have emerged as a unifying paradigm for scalable sequence modeling, high-dimensional feature aggregation, and abstracted representation learning across diverse modalities—including time series, spatial data, irregular structures, and complex control tasks. Their central motif is the composition of state-space models (SSMs) at multiple resolutions, leveraging both local and global dependency modeling, and—distinct from flat SSMs—explicitly injecting multi-scale structure or domain-aligned scanning strategies. Recent advances deploy HiSS Blocks in vision, point-cloud analysis, temporal task segmentation, hierarchical reinforcement learning, and 3D generative modeling, demonstrating superior scalability and inductive bias compared to both “flat” state-space and attention-based models [2402.10211, 2504.13015, 2506.21330, 2605.25220, 2106.01655].

## 1. Mathematical Foundation of State-Space Hierarchies

A HiSS Block typically extends the conventional linear state-space model:
\[
x_{k} = A x_{k-1} + B u_{k}, \quad y_{k} = C x_{k} + D u_{k}
\]
by either stacking SSMs at multiple temporal/spatial scales, partitioning the state or input domains, or both. In canonical continuous sequence-to-sequence instantiations, the input is segmented into “chunks” (length $K$), each processed by a low-level SSM. Chunk features are then processed by a higher-level SSM operating at a lower effective rate, yielding a multi-scale abstraction:
\[
x^{(2)}_k = A^{(2)} x^{(2)}_{k-1} + B^{(2)} h^{(1)}_k, \quad h^{(1)}_k = \text{low-level SSM}(u_{(k-1)K+1:kK})
\]
This structure generalizes to spatial domains by replacing temporal sequences with token grids, local neighborhoods, or learned partitions [2402.10211, 2504.13015, 2605.25220].

Distinct HiSS Block families employ various strategies:

- **Chunked hierarchy:** Non-overlapping local windows, with layer 1 SSM extracting fine-scale features, layer 2 SSM modeling long-range or global context [2402.10211].
- **Spatial hierarchy:** Recursive downsampling and aggregation (e.g., via farthest point sampling in point clouds) at each level, followed by SSM processing [2504.13015].
- **Hybrid sequence/image hierarchy:** Joint use of SSM recurrences (including multi-axis scans) and self-attention in stacked “block” fashion [2605.25220].
- **Hierarchical input-dependent SSMs:** Independent SSMs over locally defined segments (e.g., pseudo-phases), followed by global SSMs for dependencies across segments [2506.21330].

## 2. Core Architectural Variants and Integration Strategies

The implementation of HiSS Blocks varies across domains but universally includes hierarchical partitioning and multi-level SSM computation.

- **Time Series (Sensor and Video Data):** Input sequences are divided into fixed-size chunks, processed by low-level SSMs that summarize fast, local dynamics. High-level SSMs operate on these summaries to capture slow, global trends [2402.10211, 2506.21330]. In surgical phase recognition, Hierarchical Input-Dependent State Space Blocks combine block-diagonal local SSMs (segment-wise) with full dense global SSM mixers [2506.21330].
  
- **Spatial/Structural Data (Point Clouds):** At each hierarchy level, HiSS Blocks aggregate local features using neighborhood-based SSMs (vanilla Point SSMs over k-NN groups), then combine information globally with group SSMs that scan the entire set under multiple coordinate orderings (e.g., {x, y, z, inside-out}) [2504.13015]. Serialization enables SSMs to model structure in irregular domains.
  
- **2D and 3D Grid Data (Images, Gaussian Primitives):** MVCHead employs HiSS Blocks to process $H \times W$ token grids. Each block fuses global self-attention and Hierarchical Bi-directional State Scans (HiBiSS) along both spatial axes and at increasing refinement levels [2605.25220].
  
- **Reinforcement Learning (Abstract State Blocks):** HiSS-Blocks partition state spaces into “blocks” using learned soft (then hard) abstractions. Transition policies between blocks are learned as subtask options; the manager plans over abstract transitions, while option policies solve intra-block transitions [2106.01655].

| Domain                        | Hierarchy Mechanism                           | SSM Application                   |
|-------------------------------|-----------------------------------------------|------------------------------------|
| Sensor Time Series            | Chunks in time                               | Layered SSMs (fast+slow)           |
| Irregular Point Clouds        | Farthest point sampling at M levels          | KNN local + group SSMs             |
| Grid/Images                   | Grid upsampling/refinement                   | 2D multi-axis SSM + attention      |
| RL/MDP                        | State abstraction partitions (“blocks”)      | Options over block transitions      |
| Video (Surgical)              | Phase-based segmentation                     | Block-diag + global ID-SSM/Mixer   |

## 3. Mathematical Details of Block Operations

Most HiSS Block variants are grounded in discrete or continuous state-space equations. For instance, in [2506.21330], each input-dependent SSM block within the hierarchy involves:

\[
x_t = \overline{A}_t x_{t-1} + \overline{B}_t u_t,\quad y_t = C_t x_t
\]
where parameters $\overline{A}_t$, $\overline{B}_t$, $C_t$ are dynamically generated from $u_t$ (feature at time t), potentially incorporating a learnable timescale $\Delta_t$. Local SSMs are typically block-diagonal (each segment/phase processed independently), global SSMs are full lower-triangular (dense recurrence). Temporal selectivity emerges via data-adaptive state updates, e.g., large $\Delta_t$ “resets” the recurrent state at phase boundaries [2506.21330].

MVCHead’s spatial HiSS Block intertwines self-attention and four 1D SSM scans (“HiBiSS”) along both grid axes and directions. Each scan applies separate, learned linear operators and fuses results via concatenation and projection [2605.25220].

## 4. Applications and Empirical Performance

HiSS Blocks deliver demonstrable advantages in tasks involving spatiotemporal structure, multi-scale reasoning, and/or very long input sequences:

- **Continuous Sequence Modeling:** Across six real-world sensor tasks, HiSS Blocks reduce mean squared error by at least 23% compared to best flat SSMs (Mamba), LSTM, and Transformers, while scaling linearly in sequence length. They exhibit enhanced sample efficiency, outperforming alternatives even in low-data regimes [2402.10211].

- **Medical Point Clouds:** Hierarchical Point SSM Blocks outperform transformer-based and vanilla SSM baselines on shape classification, completion, and segmentation benchmarks (MedPointS), due to their local-to-global SSM pipeline and learned, domain-specific serialization [2504.13015].

- **Video-based Temporal Segmentation:** Hierarchical Input-Dependent SSMs yield state-of-the-art accuracy in long surgical video phase recognition, attaining +2.8 to +12.9 percentage point improvement depending on the dataset [2506.21330]. The architecture models fine “short-term” local phase dynamics and long-range temporal context without quadratic attention cost.

- **3D Generation (Gaussian Splatting):** HiSS Blocks in MVCHead enable progressive, coarse-to-fine refinement of 3D Gaussian primitives from single 2D images without multi-view supervision, incorporating both axis-aligned local consistency (HiBiSS) and global self-attention. This approach achieves state-of-the-art perceptual quality and consistency in 3D head avatar synthesis [2605.25220].

- **Hierarchical RL:** In gridworld domains with sparse rewards, HiSS-Blocks (state abstraction + options) yield near-universal success on tasks where flat DQN or self-imitation learning fail. Transfer learning (reusing learned blocks/options) accelerates learning in new, related environments by 30–50% [2106.01655].

## 5. Implementation and Hyperparameter Choices

Implementation details are strongly task- and modality-dependent but share these features:

- **Chunk Size/Resolution:** A key hyperparameter in temporal models; larger chunks decrease high-level layer compute but may underspecify fast dynamics [2402.10211].
- **Hierarchy Depth:** Number of SSM levels; more levels grant finer scale separation at the cost of greater compute/memory [2605.25220, 2504.13015].
- **Local Neighborhood Sizes:** In point cloud HiSS blocks, KNN radii and number of scans (e.g., 16/32 neighbors, 4 axes) control the granularity of feature aggregation [2504.13015].
- **Conditional Normalization:** Some architectures modulate SSM computation at each level via learned latent vectors (e.g., AdaIN in MVCHead) [2605.25220].
- **Optimization:** State-space parameters are trained jointly with feature extractors, usually via Adam or AdamW, with learning rates (1e-3 or task-specific) and dropout regularization [2402.10211, 2504.13015, 2506.21330].

## 6. Limitations, Ablations, and Future Directions

Empirical ablations indicate:

- Fixed chunk sizes, while practical, may restrict adaptivity. Future HiSS work may employ learnable or data-adaptive segmentation for chunking [2402.10211].
- SSMs currently struggle with extremely high-dimensional targets (e.g., 60-joint kinematics) relative to LSTMs; integrating richer nonlinearities or cross-attention might increase robustness [2402.10211].
- Information flow is typically bottom-up; cross-level or top-down modeling (attention, gating) is an open extension.

Across domains, HiSS Blocks consistently outperform both flat SSMs and quadratic-attention models in scaling, efficiency, and parameter efficiency. A plausible implication is that domain-aligned multi-scale recurrence in the hierarchy is a crucial inductive bias. Extensions to deeper hierarchies, multimodal fusion, and non-linear SSM updates represent primary research directions [2605.25220, 2402.10211].

Source: https://www.emergentmind.com/topics/hierarchical-state-space-hiss-blocks