Multi-Kernel Decomposed LSTM Overview

• The paper shows that replacing standard LSTM gates with multiple kernel transformations captures multi-scale motion patterns effectively.
• It leverages both convolutional branching with flow-based attention and tensor-decomposition (KCP) to reduce parameter count and computational load.
• Empirical results on benchmarks like UCF-101 and UCF11 demonstrate enhanced accuracy and significant compression compared to traditional LSTM architectures.

Searching arXiv for the cited papers to ground the article. Multi-Kernel Decomposed LSTM denotes a class of LSTM variants in which the standard monolithic recurrent transformation is replaced by multiple kernels, tensor-decomposed factors, or both. In the literature represented here, the term spans at least two technically distinct constructions. One line augments ConvLSTM with parallel convolutional kernels of different spatial sizes, optionally followed by a $1\times1$ mixing layer and a flow-based attention mechanism, yielding a “Network-in-LSTM” organization for video modeling (Agethen et al., 2019). A second line compresses conventional LSTM weight matrices by expressing them in Kronecker–CP form as a sum of $K$ Kronecker terms with internal low CP rank, together with fast contraction algorithms for the resulting tensorized gates (Wang et al., 2020). A related theoretical line derives LSTM-like gated memory cells and their $n$-gram, multi-kernel generalizations from recurrent kernel machines, thereby placing multi-kernel sequence models within an RKHS-based framework (Liang et al., 2019).

1. Conceptual scope and terminology

The defining departure from the standard LSTM is the replacement of a single transition operator by a structured collection of operators. In the convolutional setting, each gate no longer uses a single convolution $W_x * x_t$ or $W_h * h_{t-1}$; instead, it uses a set of convolutions indexed by kernel size $k \in K=\{k_1,\dots,k_n\}$, with channel allocations $C_k$ satisfying $\sum_k C_k=C$ (Agethen et al., 2019). In the compression setting, the dense weight matrix $W \in \mathbb{R}^{M\times N}$ is reshaped into a tensor and represented by a rank-$K$ Kronecker–Tensor factorization whose factors are further expressed in small CP form, producing what the paper calls the KCP format (Wang et al., 2020).

These constructions share a common structural motif: the recurrent update is decomposed into multiple sub-transformations whose outputs are later concatenated, mixed, or summed. This suggests a useful umbrella interpretation of “multi-kernel decomposed” LSTM as a family of architectures that exchange a single recurrent operator for a structured composition of several lower-level operators.

The term kernel is not uniform across the relevant literature. In "Deep Multi-Kernel Convolutional LSTM Networks and an Attention-Based Mechanism for Videos" (Agethen et al., 2019), it denotes convolutional kernels such as $K$0 and $K$1. In "Kernel-Based Approaches for Sequence Modeling: Connections to Neural Methods" (Liang et al., 2019), it denotes RKHS kernels $K$2. The term decomposition is likewise used in different senses: tensor decomposition for weight compression in KCP-LSTM (Wang et al., 2020), and multi-branch convolutional factorization inside the gate transition in ConvLSTM-MK (Agethen et al., 2019).

2. Multi-kernel ConvLSTM gate construction

In the standard ConvLSTM recalled by Agethen and Hsu, each gate is parameterized by a single convolution: $K$3 The multi-kernel modification replaces each single-kernel convolution by a set of convolutions of different spatial sizes (Agethen et al., 2019). For each gate $K$4,

$K$5

$K$6

where $K$7 and $K$8. An optional $K$9 convolution then mixes these channels: $n$0 The gates become

$n$1

$n$2

followed by the usual cell and hidden-state updates (Agethen et al., 2019).

The architecture is explicitly described as resembling a Network-in-LSTM approach. Operationally, there are parallel branches for each kernel size $n$3, each branch having its own $n$4 and $n$5. The branch outputs may be combined either by interleaving or by a $n$6 “mix” convolution before gate-wise splitting. In practice, the paper uses $n$7 with $n$8 (Agethen et al., 2019).

A central design claim is that the single-kernel ConvLSTM imposes “an unavoidable trade-off between effectiveness and efficiency,” whereas multiple kernels permit simultaneous accommodation of fine-scale and coarse-scale motion patterns (Agethen et al., 2019). The paper’s critical discussion states this directly as multi-scale motion modeling: small kernels capture fine, slow motion; large kernels capture larger displacements.

3. Flow-based attention and branch specialization

The same work introduces an attention-based mechanism designed specifically for the multi-kernel extension. Let $n$9 denote optical-flow feature maps at time $W_x * x_t$0. For each kernel size $W_x * x_t$1, the model learns a mask convolution $W_x * x_t$2, producing

$W_x * x_t$3

This mask is applied pixelwise to every channel of the input: $W_x * x_t$4 The resulting masked input enters the kernel-specific branch through

$W_x * x_t$5

after which the outputs are concatenated across kernels (Agethen et al., 2019).

The stated goal is to let each kernel specialize on regions of the input with appropriate motion speed. During back-propagation, the masks $W_x * x_t$6 specialize so that large kernels “see” large-displacement (fast) regions and small kernels “see” slow-motion regions (Agethen et al., 2019). This is a concrete architectural claim in that paper, not merely an interpretation.

The attention mechanism modulates only the input-to-hidden transition. The paper identifies hidden-to-hidden attention as a future direction, stating that the current attention only modulates input-to-hidden and that the next step is to extend to $W_x * x_t$7 branches (Agethen et al., 2019). It also notes that optical flow must be computed or predicted, adding overhead and potential error, which makes the attention mechanism technically meaningful but not free.

A common misconception is to treat the multi-kernel attention mechanism as a generic self-attention substitute. The reported mechanism is narrower: it is a set of learned mask convolutions on optical-flow feature maps that gates the input differently for each convolutional kernel size (Agethen et al., 2019).

4. Kronecker–CP decomposition of LSTM weights

A different notion of decomposition appears in the KCP-LSTM formulation. Let $W_x * x_t$8 be one of the two-dimensional input-to-gate or hidden-to-gate weight matrices in an LSTM. Assuming

$W_x * x_t$9

the matrix is reshaped to a $W_h * h_{t-1}$0-mode tensor

$W_h * h_{t-1}$1

A rank-$W_h * h_{t-1}$2 Kronecker–Tensor factorization writes

$W_h * h_{t-1}$3

where $W_h * h_{t-1}$4 and $W_h * h_{t-1}$5, with $W_h * h_{t-1}$6 the mode-wise Kronecker product (Wang et al., 2020).

Each factor is then expressed in small CP form: $W_h * h_{t-1}$7

$W_h * h_{t-1}$8

with $W_h * h_{t-1}$9 and $k \in K=\{k_1,\dots,k_n\}$0. Theorem 3.1 then shows that $k \in K=\{k_1,\dots,k_n\}$1 admits a single “sparse-partitioned” CP: $k \in K=\{k_1,\dots,k_n\}$2 where

$k \in K=\{k_1,\dots,k_n\}$3

and each $k \in K=\{k_1,\dots,k_n\}$4 has size $k \in K=\{k_1,\dots,k_n\}$5, with

$k \in K=\{k_1,\dots,k_n\}$6

This is the KCP format (Wang et al., 2020).

In an LSTM, one typically concatenates all four gates into a single large matrix $k \in K=\{k_1,\dots,k_n\}$7, so that the same KCP factorization is shared by $k \in K=\{k_1,\dots,k_n\}$8 gates, with optional further weight-sharing across gates (Wang et al., 2020). This is the precise point at which multi-kernel and decomposed structures meet in this line of work: the model is multi-kernel in the sense of multiple Kronecker terms $k \in K=\{k_1,\dots,k_n\}$9, and decomposed in the tensor-factorization sense.

For gate computation, if $C_k$0, $C_k$1, and $C_k$2, then for each gate $C_k$3,

$C_k$4

The usual LSTM nonlinearities then apply: $C_k$5 (Wang et al., 2020).

5. Fast multiplication and complexity regimes

A naïve CP contraction for the KCP-formatted tensor is stated to have complexity

$C_k$6

which the paper characterizes as infeasible (Wang et al., 2020). Two fast multiplication algorithms are therefore introduced.

Algorithm 1, “Strict Fast Multiplication,” precomputes

$C_k$7

forms each $C_k$8 by concatenating $C_k$9, and successively reshapes and contracts the running tensor with $\sum_k C_k=C$0, alternating left- and right-multiplications to keep intermediate sizes small. The stated complexity is

$\sum_k C_k=C$1

(Theorem 4.3) (Wang et al., 2020).

Algorithm 2, “Relaxed Fast Multiplication,” does not build the full big $\sum_k C_k=C$2. Instead, at each mode $\sum_k C_k=C$3 and for each $\sum_k C_k=C$4, it applies $\sum_k C_k=C$5 and sums over $\sum_k C_k=C$6 whenever an even number of modes has been processed, as justified by Theorem 4.2. Its stated complexity is

$\sum_k C_k=C$7

which is always less than or equal to the cost of Algorithm 1 (Wang et al., 2020).

The paper also provides a theoretical comparison against TT, BT, TR, and HT. Under $\sum_k C_k=C$8, $\sum_k C_k=C$9, and $W \in \mathbb{R}^{M\times N}$0 denoting TT/BT/TR/HT rank, the summarized asymptotics are as follows.

Format	Param $W \in \mathbb{R}^{M\times N}$1	FLOPs per gate
Ori.	$W \in \mathbb{R}^{M\times N}$2	$W \in \mathbb{R}^{M\times N}$3
TT	$W \in \mathbb{R}^{M\times N}$4	$W \in \mathbb{R}^{M\times N}$5
BT	$W \in \mathbb{R}^{M\times N}$6	$W \in \mathbb{R}^{M\times N}$7
TR	$W \in \mathbb{R}^{M\times N}$8	$W \in \mathbb{R}^{M\times N}$9
HT	$K$0	$K$1
KCP	$K$2	$K$3

The paper further states that space is proportional to $K$4, hence linear in $K$5, and that the FLOPs are comparable to TT, while TT yields far poorer compression (Wang et al., 2020). It also reports that, as $K$6 grows, both the parameter count and FLOPs of KCP remain lower than TT, BT, TR, and HT for typical choices of $K$7 or $K$8.

6. Empirical results, theoretical lineage, and interpretive boundaries

The empirical profile of the multi-kernel ConvLSTM is reported on UCF-101 and Sports-1M, with additional I3D-based experiments (Agethen et al., 2019). On UCF-101 Split 1 with frozen VGG-16 conv5_3 features $K$9, the baselines are single-kernel ConvLSTM $K$00 top-1 and $K$01. A simple multi-kernel variant with $K$02 reaches $K$03, and MK plus stacked $K$04 mix reaches $K$05. On Sports-1M with 20 random sports classes, sampled at $K$06 fps with $K$07 frames per clip, the baselines are $K$08 for $K$09 and $K$10 for $K$11, while MK $K$12 reaches $K$13. In I3D-based experiments on a 17-class subset of UCF-101, the RGB-only I3D baseline is $K$14, ConvLSTM $K$15 gives $K$16, ConvLSTM $K$17 gives $K$18, and Inception-like MK gives $K$19. On full UCF-101 with end-to-end fine-tuning, ConvLSTM $K$20 gives $K$21, $K$22 gives $K$23, simple MK gives $K$24, MK with flow-attention gives $K$25, Inception MK gives $K$26, and end-to-end fine-tuning gives $K$27 (Agethen et al., 2019).

The ablation summary is correspondingly precise: mixing $K$28 and $K$29 always beats either alone by approximately $K$30–$K$31; the $K$32 mixing layer adds approximately $K$33 on UCF-101/VGG; flow-based attention adds approximately $K$34; and end-to-end fine-tuning with MK and $K$35 bottlenecks yields a large $K$36 (Agethen et al., 2019). The same source reports that MK adds approximately $K$37–$K$38 more time per iteration than the larger single-kernel baseline, and that MK $K$39 requires approximately $K$40M weights versus $K$41M for $K$42 alone and $K$43M for $K$44 alone.

For KCP-LSTM, experiments are reported on UCF11, Youtube Celebrities Face, and UCF50 (Wang et al., 2020). The comparison includes Original LSTM, TT-LSTM, BT-LSTM, TR-LSTM, HT-LSTM, and KCP-LSTM with rank settings $K$45 and $K$46 on UCF11/Youtube, and $K$47, $K$48, $K$49 on UCF50, all with and without four-gate weight-sharing. The key empirical trends are that all tensor formats achieve near-identical final Top-1 and average accuracies within $K$50–$K$51, while KCP routinely reaches higher compression ratios with fewer FLOPs. Specifically, on UCF11 without sharing, KCP reaches up to $K$52k$K$53 compression with $K$54 MFLOPs versus TT $K$55, BT $K$56, TR $K$57, and HT $K$58; with weight-sharing, KCP reaches up to $K$59k$K$60 versus BT $K$61k$K$62, TR $K$63k$K$64, and HT $K$65k$K$66; on UCF50, KCP reaches up to $K$67k$K$68 without sharing and $K$69k$K$70 with sharing (Wang et al., 2020). Figure 1 is summarized as showing nearly identical convergence across methods, and Figure 2 as showing that raising the KT-rank $K$71 is more important for expressivity than boosting $K$72 and $K$73 individually.

The KCP paper also emphasizes parallel potential. Because KCP is a sum over $K$74 of independent Kronecker blocks $K$75, Algorithm 2 can be split across $K$76 threads or processes. In a simple CPU-based timing test, inference time under a multi-process implementation decreases roughly linearly in $K$77, up to manageability overhead (Wang et al., 2020).

The theoretical lineage supplied by the kernel-machine formulation is useful for clarifying what is and is not being generalized. Starting from a Mercer kernel

$K$78

the recurrent kernel-machine derivation yields a memory-cell recursion and, after dynamic gating, an LSTM-like model

$K$79

with

$K$80

(Liang et al., 2019). Extending this to $K$81-gram filters replaces rank-1 filters by length-$K$82 convolutions: $K$83 with separate filter banks for the cell candidate and each gate. The paper states that CNNs, Gated CNNs, CIFG-LSTMs, and RANs emerge as special cases under specific gate settings (Liang et al., 2019). This suggests that multi-kernel LSTM variants can be interpreted not only as engineering modifications, but also as members of a broader family of gated convolutional sequence models.

Taken together, these results delimit two major interpretations of Multi-Kernel Decomposed LSTM. One interpretation is architectural and spatial: multiple convolutional kernels are embedded inside each recurrent gate, optionally modulated by flow-based attention (Agethen et al., 2019). The other is algebraic and compressive: the gate matrices are decomposed into multiple Kronecker factors with internal CP structure and fast contractions (Wang et al., 2020). The kernel-machine account provides a unifying theoretical backdrop in which dynamic gating and multi-kernel convolution arise from a common sequence-modeling formalism (Liang et al., 2019).