Hybrid Stateful Ensemble (HSE) Overview

• Hybrid Stateful Ensemble is a unified framework combining deep neural networks and classical time-series models, synchronized through a joint state-space representation.
• It employs online joint optimization via particle filtering to enhance prediction accuracy and reduce errors in sequential forecasting tasks.
• The framework provides robust defense against adversarial attacks by monitoring temporal and spatial query patterns through explicit state tracking.

A Hybrid Stateful Ensemble (HSE) is a class of machine learning frameworks that integrates heterogeneous model components—often nonlinear deep learners with classical statistical models—into a joint state-space system with explicit state tracking and sequential dependency modeling. HSE frameworks are utilized both to improve predictive modeling of complex sequential data and as a defense mechanism against adversarial attacks in security-critical contexts. Distinctively, HSE methods leverage stateful coordination between their constituent models, either for jointly optimized inference or for robust detection of temporal anomalies in model querying behavior.

1. State-Space-Based Hybrid Model Structure

Hybrid Stateful Ensembles address the limitations of traditional ensemble models that treat base learners as independent black boxes, often training and updating them in isolation. Instead, HSE frameworks provide a unified state-space representation wherein nonlinear and linear components interact at each timestep. For sequential prediction, HSE typically couples a deep recurrent neural network (e.g., LSTM or GRU) and a classical time-series model (e.g., SARIMAX), synchronizing their internal states and updating them jointly through probabilistic filtering methods such as particle filtering.

The joint state vector at time $t$ is given by

$$s_t = \begin{bmatrix} s_{t,\text{LSTM}} \ s_{t,\text{SX}} \end{bmatrix}$$

where $s_{t,\text{LSTM}}$ encodes the RNN states and parameters (cell state $c_t$, hidden state $h_t$, trainable weights $\theta_t$), and $s_{t,\text{SX}}$ represents the SARIMAX block (time-varying AR/MA/seasonal coefficients, exogenous regressors). The complete state transition is governed by both nonlinear (RNN) and linear (ARMA/SARIMAX) update equations, with random walk adaptation for non-stationary or streaming data. The model’s observation at each step is a sum of the respective predictions from both blocks, i.e., $$y_t = y_{t,\text{LSTM}} + y_{t,\text{SX}} + \epsilon_t$$ (Aydın et al., 2023).

2. Joint Optimization via Particle Filtering

Unlike conventional ensemble learning where base models are trained disjointly and later combined, HSE frameworks employ a joint online learning strategy facilitated by sequential Monte Carlo (particle filtering). For each new observation, a set of $N$ particles is propagated through the augmented state-space, each representing a hypothetical trajectory for the joint model parameters and hidden states. Weights for each particle are updated with the likelihood of newly observed data given the corresponding hidden state, and resampling is performed as needed for stability.

This mechanism ensures that nonlinear feature-extraction (from LSTM/GRU) and interpretable time-series structure (from SARIMAX or similar) are optimized in concert, rather than being independently tuned. Regularization is incorporated through isotropic process and observation noise, while optional shrinkage can further stabilize the linear block. Empirically, this procedure significantly reduces mean squared error (MSE) and mean absolute percentage error (MAPE) across multiple real-world and competition datasets, outperforming both individual and disjointly trained ensemble baselines (Aydın et al., 2023).

3. Architecture and Algorithmic Details

The HSE architecture for sequential data prediction comprises:

• An LSTM (or GRU/TCN) feature extractor with stochastic parameter and state transitions:

$$c_t = \gamma(x_t, h_{t-1}, c_{t-1}) + u_t,\quad h_t = \tau(x_t, h_{t-1}, c_t) + v_t,\quad \theta_t = \theta_{t-1} + \epsilon_t$$

where $u_t, v_t, \epsilon_t$ are process noise terms.

• A SARIMAX state block including ordinary and seasonal AR/MA dynamics, exogenous variable coefficients, and a random walk transition.
• A measurement model combining the outputs of both blocks:

$$y_t = w_t^\top h_t + \tilde r_t^\top s_{t,\text{SX}} + \epsilon_t$$

where $w_t$ is the LSTM regression head and $\tilde r_t$ aggregates past values, residuals, and exogenous factors.

• Online learning via particle filtering, with per-step computational cost $O(N\,\text{dim}(s_t))$, where $N$ is the number of particles.

The design supports modular swapping of component models—for example, replacing LSTM with GRU or switching SARIMAX for ETS—by adapting the relevant state-transition and measurement mappings. Offline and online preprocessing steps include input standardization and, where required, differencing of target series (Aydın et al., 2023).

4. Applications in Model Extraction Defense

A distinct application domain for HSE, exemplified by its deployment in adversarial defense, leverages the stateful tracking component to detect non-i.i.d. querying patterns indicative of optimization-driven attacks. In the context of model extraction attacks employing semantic priors (such as those generated by latent diffusion models, e.g., DiMEx), HSE combines:

• A spatial consensus mechanism: evaluating a consensus score among predictions from a master model and a set of submodels, each trained on disjoint or randomly partitioned data. Queries lacking consensus trigger immediate flagging.

$$\mathcal{C}(x) = \frac{1}{N} \sum_{i=1}^N \mathbb{I}(\arg\max M_0(x) = \arg\max S_i(x))$$

• A temporal drift detector: monitoring the cosine similarity of step vectors in penultimate-layer feature space across sequential queries in a session, highlighting sustained directional drift characteristic of optimization trajectories.

$$\text{drift\_score} = \sum_{j=1}^{k-1} \cos(\vec v_{t-j}, \vec v_{t-j+1}), \quad \vec v_t = h_t - h_{t-1}$$

The hybrid approach captures both anomalous spatial distribution (across models) and temporal coherence (within-query sequence), effectively suppressing the attack success rate relative to purely static distributional or OOD detectors (Thesia et al., 4 Jan 2026).

5. Experimental Results and Sensitivity

Empirical evaluations on tasks such as CIFAR-10 under semantic model extraction attack (e.g., DiMEx) demonstrate that the full HSE, integrating both spatial and temporal checks, achieves an attack success rate (ASR) of 21.6%, a reduction by over 3× compared to PRADA’s 61.2%. The latency overhead remains negligible (under 15 ms per query), and the impact on benign user accuracy is limited (1.4% drop versus greater losses for alternative defenses). Ablation studies reveal that the temporal and spatial components are synergistic, with neither individually matching the combined defense’s efficacy. Parameter tuning (buffer length $k$, drift threshold $\tau_{\text{drift}}$, spatial consensus threshold $\tau_{\text{spatial}}$) provides a mechanism to balance false positive rates and sensitivity, with practical choices ($k=8$–12, $\tau_{\text{drift}} \approx 4$) yielding under 2% false-positive rates on benign traffic (Thesia et al., 4 Jan 2026).

For sequential prediction tasks, HSE outperforms both RNN-only and classical models on both online (Bank, Elevators, Kinematics, Pumadyn) and offline forecasting datasets (M4 competition Hourly, Daily, Yearly subsets), as shown in the table below:

Dataset	Naive	SARIMAX	LSTM	LSTM-SX (HSE)
Bank	0.0307	0.0183	0.0190	0.0129
Elevators	0.0714	0.0465	0.0119	0.0098
Kinematics	0.1305	0.0897	0.0884	0.0627
Pumadyn	0.0218	0.0107	0.0094	0.0013

All values are cumulative MSE at $t=T$ (Aydın et al., 2023).

6. Limitations and Extensibility

HSE frameworks assume session-wise continuity and the availability of meaningful state or feature representations across queries. Their efficacy in sustaining robustness to adversarial attacks partially depends on the adversary’s inability to mimic random-walk or benign i.i.d. query behavior. Attackers employing session resets or interleaving benign queries can degrade detection accuracy at the expense of attack efficiency. For sequential prediction, performance and computational complexity scale with particle count and state dimension.

The architectural genericity of HSE allows substituting RNN with GRU or TCN and SARIMAX with other state-space models (ETS), as well as employing alternate filtering methods (EKF, UKF) instead of particle filtering, providing flexibility across problem domains and data regimes. Code implementations and further algorithmic detail are publicly available (Aydın et al., 2023).

7. Impact and Significance

Hybrid Stateful Ensembles represent a shift toward integrating deep representation learning and interpretable statistical modeling within unified, sequentially optimized architectures. Their design is motivated by both predictive performance (in online and time-series contexts) and by the necessity of robust, stateful security in the face of emerging attacks that exploit distributional indistinguishability. HSE demonstrates superior empirical performance across a range of datasets and threat models, while maintaining low computational overhead and interpretability for at least a subset of its constituent models. As research continues to interrogate the frontiers of hybrid learning and adversarial robustness, HSE provides a generalizable, extensible template for stateful ensemble methodologies (Thesia et al., 4 Jan 2026, Aydın et al., 2023).