MCNN+LSTM: Multi-Granularity Sequence Modeling

• MCNN+LSTM is a composite neural architecture that fuses multi-scale convolutional filters with LSTM temporal modeling to improve sequence prediction.
• It employs parallel CNN pathways with varied kernel sizes to capture both short-term fluctuations and long-range patterns in time series data.
• Empirical studies show the model achieves state-of-the-art results in autonomous driving, financial forecasting, and weather prediction compared to single-scale approaches.

A Multi-granularity Convolutional Neural Network combined with Long Short-Term Memory (MCNN+LSTM, or “Multi-Scale CNN+LSTM”) is a composite neural architecture engineered to extract and fuse temporal and multi-resolution spatial features for sequence modeling and prediction tasks. In this context, “multi-granularity” or “multi-scale” refers to the parallel application of convolutional filters with distinct receptive fields, capturing local and global patterns in input sequences, while the LSTM component models longer-term temporal dependencies. MCNN models and their variants have demonstrated state-of-the-art results in domains such as autonomous driving, financial time series forecasting, and weather prediction (Zhang et al., 2021, Shen et al., 2024, Guo et al., 2021).

1. Conceptual Foundation

MCNN+LSTM architectures are motivated by the complementary strengths of convolutional and recurrent neural networks for sequential data. Convolutional neural networks (CNNs) are adept at identifying local and multi-scale spatial patterns via learnable kernels. When convolution kernels span multiple temporal windows (“scales”), the resulting architecture can simultaneously summarize short-term fluctuations and long-range periodicities. Long Short-Term Memory (LSTM) networks, conversely, maintain a persistent representation of the sequence context. The fusion of multi-scale convolutional processing and LSTM-based sequence modeling provides a hierarchical, expressive summary that enhances predictive accuracy for complex time series and behavior recognition tasks (Zhang et al., 2021, Shen et al., 2024, Guo et al., 2021).

2. Typical Architectural Components

A standard MCNN+LSTM model is a two-branch or sequential framework in which a multi-scale CNN (MSCNN or MRC) and an LSTM operate in parallel (feature fusion) or series (convolutional preprocessing before LSTM). The essential components are:

• Multi-Scale CNN Branch: Applies parallel 1D convolutions with different kernel sizes (e.g., $k \in \{2,3,4\}$ or $k \in \{2,4,8\}$), extracting pattern representations at various scales.
• LSTM Branch: Operates either in parallel with the CNN branch (as in (Zhang et al., 2021)) or sequentially after multi-scale feature extraction (as in (Guo et al., 2021)), capturing sequential context and long-term dependencies.
• Feature Fusion and Classification/Regression: The outputs of the convolutional and LSTM branches are concatenated or otherwise fused, followed by one or more fully connected layers for the final task-specific prediction.
• Auxiliary Components: May include self-attention modules (as in (Shen et al., 2024)), residual connections (as in (Guo et al., 2021)), or normalization and pooling operations for stability and expressivity.

3. Mathematical Formulation

3.1 Multi-Scale CNN Block

Given a sequence $S \in \mathbb{R}^{T \times F}$, with $T$ time steps and $F$ features per step, the multi-scale convolution module applies multiple 1D convolutions in parallel:

$$Y^{(k)} = \mathrm{ReLU}\bigg( \sum_{c=1}^F \sum_{\delta=0}^{k-1} w_{k,c}(\delta)\;S_c(t+\delta) + b_k \bigg)\,,$$

for each kernel size $k \in \mathcal{K}$. The outputs for all scales are concatenated:

$$y_\text{multi} = \mathrm{concat}\bigl(\mathrm{vec}(Y^{(k_1)}),\ldots,\mathrm{vec}(Y^{(k_m)})\bigr)$$

where $k_1,\ldots,k_m$ are the scales and $\mathrm{vec}(\cdot)$ denotes flattening. These are projected to fixed-size feature vectors by fully connected layers as in (Zhang et al., 2021, Guo et al., 2021).

3.2 LSTM Branch

For each time step $k \in \{2,4,8\}$0, the LSTM cell equations are:

$k \in \{2,4,8\}$1

with possible bidirectional and layered extensions. Outputs are temporally pooled (e.g., by averaging or final hidden state extraction) prior to fusion or further processing (Zhang et al., 2021).

3.3 Feature Fusion and Output

The pooled representations from CNN and LSTM are concatenated:

$k \in \{2,4,8\}$2

This vector is mapped to task outputs (e.g., class logits via softmax or regression scores) using a dense layer. Cross-entropy loss is used for classification (Zhang et al., 2021); mean squared error (MSE) is used for regression (Guo et al., 2021, Shen et al., 2024).

4. Benchmark Applications and Empirical Findings

MCNN+LSTM variants have been implemented in several domains, achieving top results relative to simpler baselines:

Domain	Model	Key Results	Paper
Driving behavior	MCNN	Vehicle Bal. Acc. 90.85%, F1 88.07%, Recall 90.27%	(Zhang et al., 2021)
Cryptocurrency price	MRC-LSTM	BTC RMSE 261.44, MAE 166.52, R² 93.10%	(Guo et al., 2021)
Weather forecasting	Multi-Scale CNN-LSTM-Attn	RMSE 0.8107 (vs. 1.43 for single-scale), MSE 1.9783	(Shen et al., 2024)

In all benchmarked cases, the use of multi-granularity convolution improves performance by 2%–7% over single-scale CNN+LSTM architectures and LSTM alone. The inclusion of class balancing (e.g., Random Over-Sampling) is critical for imbalanced datasets (Zhang et al., 2021).

5. Ablation Studies and Impact of Multi-Granularity Convolution

All referenced works report ablation studies quantifying the incremental gains from each architectural component.

• In (Zhang et al., 2021), adding the MSCNN branch to a Bi-LSTM (with ROS) raises balanced accuracy from 88.92% to 90.85% for vehicle behavior recognition.
• In (Guo et al., 2021), replacing the multi-scale (residual) module with a single-scale CNN increases RMSE on Bitcoin from 261.44 to 270.66—demonstrating the gain from rich multi-scale representation.
• In (Shen et al., 2024), extending from single-scale (k=2) to multi-scale (k=2,4,8) convolutions reduces RMSE on temperature prediction from ≈1.43 to 0.81, a relative error reduction of over 40%. Removal of the attention module further degrades performance, but the largest single drop is due to loss of multi-scale convolution.

A plausible implication is that multi-scale convolution exposes sequence models to higher-order short- and long-range features, facilitating downstream LSTM modules’ extraction of complex temporal dynamics.

6. Typical Training Protocols and Hyperparameter Choices

Training of MCNN+LSTM models generally employs adaptive gradient methods (Adam or Nadam), learning rates in the range [0.001, 0.005], and batch sizes from 50 to 256. Regularization methods include dropout in recurrent layers (e.g., 0.3, (Shen et al., 2024)), random over-sampling for class balancing (Zhang et al., 2021), and learning rate decay or early stopping to prevent overfitting. Number of epochs varies by domain and dataset size (e.g., up to 2,000 for BTC price forecasting (Guo et al., 2021), 60 for driving behavior (Zhang et al., 2021)).

Kernel widths for the multi-scale CNNs are selected to reflect the temporal scales relevant for each application: e.g., daily, weekly, bi-weekly for climate; short and mid-term for trajectory sequences; or multiple lags for financial data (Zhang et al., 2021, Shen et al., 2024, Guo et al., 2021).

7. Significance and Broader Implications

Multi-granularity CNN+LSTM models offer a principled and empirically validated framework for combining local, scale-adaptive feature extraction with sequence modeling. Their utility spans domains typified by complex, interleaved local and temporal dynamics, including autonomous vehicle behavior understanding, multivariate financial forecasting, and environmental time series prediction. Empirical evidence consistently supports the claim that introducing parallel, scale-diverse convolutions enhances downstream sequential modeling, yielding measurable improvements over single-scale or purely recurrent baselines (Zhang et al., 2021, Guo et al., 2021, Shen et al., 2024).