S4oP: Operator-level Pruning of Structured State Space Models for Resource-Constrained Devices

Abstract: Structured State Space Models (SSMs), including the S4 and S4D architectures, have recently emerged as powerful alternatives to attention-based models for capturing long-range dependencies in sequential data. Despite their strong empirical performance, deploying these models in time- and resource-constrained settings remains challenging due to their computational and memory demands. In this paper, we propose a novel incremental, operator-level pruning approach for S4- and S4D-based models that significantly reduces inference cost while preserving predictive performance. To the best of our knowledge, this is the first work to systematically investigate structured operator pruning for SSMs. Our method progressively prunes model operators by interleaving structured masking with fine-tuning, while jointly monitoring accuracy and inference latency. We implement this approach within a unified training and evaluation framework that enables systematic exploration of efficiency-accuracy trade-offs. Experiments across multiple benchmark datasets show that pruning up to 70% of the model operators preserves the performance of the original models in most cases, while substantially reducing inference latency. These results demonstrate that structured operator pruning is an effective and previously unexplored strategy for improving the efficiency of SSMs and facilitate their deployment in practical, resource-constrained scenarios.

• The paper demonstrates an incremental operator-level pruning framework for SSMs that preserves predictive accuracy while reducing inference latency and memory footprint.
• It employs depth-aware channel allocation with random selection and fine-tuning after each pruning step to optimize efficiency across model layers.
• Experiments on tasks including IMDb, Pathfinder, and ECG show up to 70% operator reduction with minimal accuracy loss, enabling deployment on resource-limited hardware.

Operator-Level Pruning of S4/S4D Structured State Space Models

Introduction and Motivation

Structured State Space Models (SSMs) such as S4 and S4D have recently demonstrated strong performance in sequential data modeling, effectively capturing long-range dependencies with parallel computation that overcomes the bottlenecks of traditional recurrent architectures. However, practical deployment of these models in resource-constrained environments (e.g., edge or IoT devices) remains hindered by their computational and memory demands. While pruning, especially at the parameter level, has been extensively studied for convolutional and transformer networks, channel- or operator-level pruning in SSMs has not been systematically explored.

The paper introduces S4oP, an incremental operator-level pruning methodology for S4/S4D models, designed to reduce inference latency and memory footprint while retaining predictive accuracy. Channels (i.e., independent SSM operators within a layer) are pruned in a structured manner, with each pruning step followed by fine-tuning, thereby enabling controlled exploration of the efficiency-accuracy frontier.

Figure 1: Overview of the proposed pruning approach, detailing incremental pruning interleaved with fine-tuning and evaluation.

Structured State Space Models: Architectural Insights

S4 and S4D models leverage parallel execution of channel-wise SSM operators within each layer, followed by mixing and residual connections. S4 parameterizes the state matrix to enable efficient convolutional kernel computation, while S4D further reduces complexity with a diagonal approximation. Each channel of an S4/S4D layer is essentially independent, which structurally facilitates operator-level pruning:

Figure 2: Architecture of an S4 layer with multiple parallel SSM operators, supporting straightforward channel-wise pruning.

This independence allows pruned channels to be replaced with identity mappings, ensuring tensor shapes are preserved and downstream layers remain operational without computational disruption. Unlike parameter-level pruning, this method does not interfere with FFT-based kernel computations.

Pruning Methodology

S4oP leverages a depth-aware incremental channel allocation policy, distributing pruning budgets across layers with exponentially increasing aggression toward deeper layers. This is motivated by empirical findings that early layers are more critical to model accuracy, whereas deeper layers exhibit greater redundancy and are more tolerant to operator removal.

Channel selection for pruning is performed randomly for each targeted pruning rate—rather than ranking by importance. Fine-tuning after each pruning step mitigates loss of capacity, while multiple random seeds minimize risk of suboptimal channel selection. The result is a nested progression of pruning configurations, each optimized for the current pruning budget.

Experimental Evaluation

The framework is evaluated across LRA benchmarks (ListOps, Pathfinder, IMDb) and the CODE ECG long-sequence dataset, providing diverse sequencing tasks from symbolic reasoning to temporal signal analysis.

Figure 3: IMDb benchmark results for operator-level pruning across increasing rates.

Figure 4: Pathfinder benchmark results, highlighting robustness in accuracy under aggressive pruning.

Incremental S4oP strategy consistently outperforms one-shot pruning methods in accuracy preservation, especially at moderate and high pruning rates. Pruning up to 30% of operators generally yields negligible accuracy loss across the models and datasets, while reductions of up to 70% remain viable in several cases (e.g., IMDb and Pathfinder) as shown in Figure 5.

Figure 5: IMDb benchmark demonstrating preserved accuracy even at high pruning rates.

Figure 6: Pathfinder benchmark, similarly exhibiting stability under channel pruning.

On ECG, S4D in particular attains F1-score improvements post pruning, indicating a beneficial regularization effect. ListOps emerges as the most pruning-sensitive, especially for S4, confirming the greater importance of structural redundancy in symbolic reasoning tasks.

Latency measurements on Jetson Orin Nano empirically validate proportional speedups with increased pruning rates. Application of S4oP delivers up to 60% reductions in inference latency and substantial parameter savings at moderate pruning, facilitating deployment on limited hardware.

Figure 7: Speedup and accuracy across pruning rates on an embedded device, demonstrating near-linear latency gains.

Practical and Theoretical Implications

The operator-level pruning method identifies depth-dependent redundancy: early layers should be pruned conservatively, while deeper layers admit substantial removal without compromising downstream learning. This insight aligns with established findings for transformers and convolutional networks, but the systematic application and validation in SSMs constitutes a valuable advance for sequence modeling.

Furthermore, the empirical resilience of S4D over S4 under aggressive pruning suggests that architectural simplification (e.g., diagonal state matrix) enhances robustness, which has implications for future model design targeting resource-constrained inference.

The observed benefits of random channel selection challenge the necessity of sophisticated importance-based ranking, reducing computational overhead and simplifying deployment pipelines in practical settings.

Speculation on Future Research

The methodology outlined can be extended towards dynamic or adaptive pruning regimes, possibly integrated with hardware-aware cost models. Combining pruning with quantization or low-rank approximations could further optimize SSMs for real-time inference in ultra-low-power settings. The random channel selection findings may catalyze exploration of probabilistic model surgery in other sequence architectures.

Operator-level pruning also invites theoretical examination of representational sufficiency in overparameterized SSMs, particularly for tasks with varying dependency lengths and signal modalities.

Conclusion

S4oP establishes a robust, incremental operator-level pruning framework for S4 and S4D models, enabling efficient deployment in resource-constrained environments. The approach combines depth-aware channel allocation, random channel selection, and progressive fine-tuning to preserve accuracy while achieving substantial latency and parameter reductions. Empirical results indicate up to 70% operator pruning is viable in many settings, and S4D architectures exhibit particularly strong robustness. The framework’s simplicity and effectiveness suggest broad applicability, with theoretical implications for redundancy analysis and efficient sequence model scaling.