---
title: Multi-Kernel Decomposed LSTM Overview
url: https://www.emergentmind.com/topics/multi-kernel-decomposed-lstm
type: topic
---

# Multi-Kernel Decomposed LSTM Overview

Searching arXiv for the cited papers to ground the article.
Multi-Kernel Decomposed LSTM denotes a class of LSTM variants in which the standard monolithic recurrent transformation is replaced by multiple kernels, tensor-decomposed factors, or both. In the literature represented here, the term spans at least two technically distinct constructions. One line augments ConvLSTM with parallel convolutional kernels of different spatial sizes, optionally followed by a \(1\times1\) mixing layer and a flow-based attention mechanism, yielding a “Network-in-LSTM” organization for video modeling [1908.08990]. A second line compresses conventional LSTM weight matrices by expressing them in Kronecker–CP form as a sum of \(K\) Kronecker terms with internal low CP rank, together with fast contraction algorithms for the resulting tensorized gates [2008.09342]. A related theoretical line derives LSTM-like gated memory cells and their \(n\)-gram, multi-kernel generalizations from recurrent kernel machines, thereby placing multi-kernel sequence models within an RKHS-based framework [1910.04233].

## 1. Conceptual scope and terminology

The defining departure from the standard LSTM is the replacement of a single transition operator by a structured collection of operators. In the convolutional setting, each gate no longer uses a single convolution \(W_x * x_t\) or \(W_h * h_{t-1}\); instead, it uses a set of convolutions indexed by kernel size \(k \in K=\{k_1,\dots,k_n\}\), with channel allocations \(C_k\) satisfying \(\sum_k C_k=C\) [1908.08990]. In the compression setting, the dense weight matrix \(W \in \mathbb{R}^{M\times N}\) is reshaped into a tensor and represented by a rank-\(K\) Kronecker–Tensor factorization whose factors are further expressed in small CP form, producing what the paper calls the KCP format [2008.09342].

These constructions share a common structural motif: the recurrent update is decomposed into multiple sub-transformations whose outputs are later concatenated, mixed, or summed. This suggests a useful umbrella interpretation of “multi-kernel decomposed” LSTM as a family of architectures that exchange a single recurrent operator for a structured composition of several lower-level operators.

The term *kernel* is not uniform across the relevant literature. In "Deep Multi-Kernel Convolutional LSTM Networks and an Attention-Based Mechanism for Videos" [1908.08990], it denotes convolutional kernels such as \(3\times3\) and \(5\times5\). In "Kernel-Based Approaches for Sequence Modeling: Connections to Neural Methods" [1910.04233], it denotes RKHS kernels \(k_\theta(\tilde z_i,z_t)=q_\theta(\tilde z_i^\top z_t)\). The term *decomposition* is likewise used in different senses: tensor decomposition for weight compression in KCP-LSTM [2008.09342], and multi-branch convolutional factorization inside the gate transition in ConvLSTM-MK [1908.08990].

## 2. Multi-kernel ConvLSTM gate construction

In the standard ConvLSTM recalled by Agethen and Hsu, each gate is parameterized by a single convolution:
\[
\begin{aligned}
i_t &= \sigma( W_{xi} * x_t + W_{hi} * h_{t-1} + W_{ci}\odot c_{t-1} + b_i ),\\
f_t &= \sigma( W_{xf} * x_t + W_{hf} * h_{t-1} + W_{cf}\odot c_{t-1} + b_f ),\\
o_t &= \sigma( W_{xo} * x_t + W_{ho} * h_{t-1} + W_{co}\odot c_{t-1} + b_o ),\\
\tilde c_t &= \tanh( W_{xc} * x_t + W_{hc} * h_{t-1} + b_c ),\\
c_t &= f_t\odot c_{t-1} + i_t\odot \tilde c_t,\qquad h_t = o_t\odot \tanh(c_t).
\end{aligned}
\]
The multi-kernel modification replaces each single-kernel convolution by a set of convolutions of different spatial sizes [1908.08990]. For each gate \((\cdot)\in\{i,f,o,\mathcal{G}\}\),
\[
A_x^{(\cdot)}=\operatorname{concat}_{k\in K}[W_x^{(\cdot),(k)} * x_t]\in\mathbb{R}^{H\times W\times C},
\]
\[
A_h^{(\cdot)}=\operatorname{concat}_{k\in K}[W_h^{(\cdot),(k)} * h_{t-1}]\in\mathbb{R}^{H\times W\times C},
\]
where \(W_x^{(\cdot),(k)}\in\mathbb{R}^{C_k\times C_x\times k\times k}\) and \(W_h^{(\cdot),(k)}\in\mathbb{R}^{C_k\times C_h\times k\times k}\). An optional \(1\times1\) convolution then mixes these channels:
\[
B^{(\cdot)} = W^{1\times1,(\cdot)} * (A_x^{(\cdot)} + A_h^{(\cdot)}) + b^{(\cdot)}.
\]
The gates become
\[
i_t=\sigma(B^{(i)} + W_{ci}\odot c_{t-1}),\quad
f_t=\sigma(B^{(f)} + W_{cf}\odot c_{t-1}),
\]
\[
o_t=\sigma(B^{(o)} + W_{co}\odot c_{t-1}),\quad
\tilde c_t=\tanh(B^{(\tilde c)}),
\]
followed by the usual cell and hidden-state updates [1908.08990].

The architecture is explicitly described as resembling a Network-in-LSTM approach. Operationally, there are parallel branches for each kernel size \(k_1,k_2,\dots,k_n\), each branch having its own \(W_x^{(\cdot),(k)}\) and \(W_h^{(\cdot),(k)}\). The branch outputs may be combined either by interleaving or by a \(1\times1\) “mix” convolution before gate-wise splitting. In practice, the paper uses \(K=\{3\times3,5\times5\}\) with \(C_{3\times3}=C_{5\times5}=C/2\) [1908.08990].

A central design claim is that the single-kernel ConvLSTM imposes “an unavoidable trade-off between effectiveness and efficiency,” whereas multiple kernels permit simultaneous accommodation of fine-scale and coarse-scale motion patterns [1908.08990]. The paper’s critical discussion states this directly as multi-scale motion modeling: small kernels capture fine, slow motion; large kernels capture larger displacements.

## 3. Flow-based attention and branch specialization

The same work introduces an attention-based mechanism designed specifically for the multi-kernel extension. Let \(f_t\) denote optical-flow feature maps at time \(t\). For each kernel size \(k_i\), the model learns a mask convolution \(W_{att}^{(k_i)}\), producing
\[
m_t^{(k_i)}=\sigma\bigl(W_{att}^{(k_i)} * f_t + b_{att}^{(k_i)}\bigr)\in[0,1]^{H\times W\times 1}.
\]
This mask is applied pixelwise to every channel of the input:
\[
x_t^{(k_i)} = m_t^{(k_i)} \odot x_t.
\]
The resulting masked input enters the kernel-specific branch through
\[
A_x^{(\cdot),(k_i)} = W_x^{(\cdot),(k_i)} * x_t^{(k_i)},
\]
after which the outputs are concatenated across kernels [1908.08990].

The stated goal is to let each kernel specialize on regions of the input with appropriate motion speed. During back-propagation, the masks \(m_t^{(k_i)}\) specialize so that large kernels “see” large-displacement (fast) regions and small kernels “see” slow-motion regions [1908.08990]. This is a concrete architectural claim in that paper, not merely an interpretation.

The attention mechanism modulates only the input-to-hidden transition. The paper identifies hidden-to-hidden attention as a future direction, stating that the current attention only modulates input-to-hidden and that the next step is to extend to \(W_h*\) branches [1908.08990]. It also notes that optical flow must be computed or predicted, adding overhead and potential error, which makes the attention mechanism technically meaningful but not free.

A common misconception is to treat the multi-kernel attention mechanism as a generic self-attention substitute. The reported mechanism is narrower: it is a set of learned mask convolutions on optical-flow feature maps that gates the input differently for each convolutional kernel size [1908.08990].

## 4. Kronecker–CP decomposition of LSTM weights

A different notion of decomposition appears in the KCP-LSTM formulation. Let \(W\in\mathbb{R}^{M\times N}\) be one of the two-dimensional input-to-gate or hidden-to-gate weight matrices in an LSTM. Assuming
\[
M=\prod_{i=1}^d m_i,\qquad N=\prod_{i=1}^d n_i,
\]
the matrix is reshaped to a \(2d\)-mode tensor
\[
\mathcal{W}\in\mathbb{R}^{(m_1n_1)\times(m_2n_2)\times\cdots\times(m_dn_d)}.
\]
A rank-\(K\) Kronecker–Tensor factorization writes
\[
\mathcal{W}=\sum_{k=1}^K \mathcal{A}_k\otimes \mathcal{B}_k,
\]
where \(\mathcal{A}_k\in\mathbb{R}^{m_1\times m_2\times\cdots\times m_d}\) and \(\mathcal{B}_k\in\mathbb{R}^{n_1\times n_2\times\cdots\times n_d}\), with \(\otimes\) the mode-wise Kronecker product [2008.09342].

Each factor is then expressed in small CP form:
\[
\mathcal{A}_k = \mathcal{I}^{(A)}_k \times_1 A_k^{(1)} \times_2 A_k^{(2)} \cdots \times_d A_k^{(d)},
\]
\[
\mathcal{B}_k = \mathcal{I}^{(B)}_k \times_1 B_k^{(1)} \times_2 B_k^{(2)} \cdots \times_d B_k^{(d)},
\]
with \(A_k^{(i)}\in\mathbb{R}^{m_i\times C_k^{(A)}}\) and \(B_k^{(i)}\in\mathbb{R}^{n_i\times C_k^{(B)}}\). Theorem 3.1 then shows that \(\mathcal{W}\) admits a single “sparse-partitioned” CP:
\[
\mathcal{W}=\mathcal{I}^{(W)} \times_1 W^{(1)} \times_2 W^{(2)} \cdots \times_d W^{(d)},
\]
where
\[
W^{(i)} = [ A_1^{(i)}\otimes B_1^{(i)} \mid A_2^{(i)}\otimes B_2^{(i)} \mid \dots \mid A_K^{(i)}\otimes B_K^{(i)} ],
\]
and each \(W^{(i)}\) has size \((m_in_i)\times C^{(W)}\), with
\[
C^{(W)}=\sum_{k=1}^K C_k^{(A)}C_k^{(B)}.
\]
This is the KCP format [2008.09342].

In an LSTM, one typically concatenates all four gates into a single large matrix \(W\in\mathbb{R}^{4N\times(N+X)}\), so that the same KCP factorization is shared by \(f,i,z,o\) gates, with optional further weight-sharing across gates [2008.09342]. This is the precise point at which multi-kernel and decomposed structures meet in this line of work: the model is multi-kernel in the sense of multiple Kronecker terms \(k=1,\dots,K\), and decomposed in the tensor-factorization sense.

For gate computation, if \(x_t\in\mathbb{R}^X\), \(h_{t-1}\in\mathbb{R}^N\), and \([x_t;h_{t-1}]\in\mathbb{R}^M\), then for each gate \(\theta\in\{f,i,z,o\}\),
\[
u_t^\theta = KCP(W_\theta,[x_t;h_{t-1}]) + b_\theta
= \bigl((x_t;h_{t-1})\ \text{tensorized}\bigr)\times_1 W_\theta^{(1)} \times_2 \cdots \times_d W_\theta^{(d)}.
\]
The usual LSTM nonlinearities then apply:
\[
c_t=f_t\odot c_{t-1}+i_t\odot z_t,\qquad h_t=o_t\odot\tanh(c_t)
\]
[2008.09342].

## 5. Fast multiplication and complexity regimes

A naïve CP contraction for the KCP-formatted tensor is stated to have complexity
\[
O((\max(m,n))^{d+1}[C^{(A)}C^{(B)}K]^d),
\]
which the paper characterizes as infeasible [2008.09342]. Two fast multiplication algorithms are therefore introduced.

Algorithm 1, “Strict Fast Multiplication,” precomputes
\[
D_k^{(i)} = A_k^{(i)}\otimes B_k^{(i)} \in \mathbb{R}^{(m_in_i)\times(C_k^{(A)}C_k^{(B)})},
\]
forms each \(W^{(i)}\) by concatenating \(D_1^{(i)},\dots,D_K^{(i)}\), and successively reshapes and contracts the running tensor with \(W^{(i)}\), alternating left- and right-multiplications to keep intermediate sizes small. The stated complexity is
\[
O\bigl(d\cdot \max(m,n)^{d+1}\cdot [C^{(A)}C^{(B)}]\cdot K\bigr)
\]
(Theorem 4.3) [2008.09342].

Algorithm 2, “Relaxed Fast Multiplication,” does not build the full big \(W^{(i)}\). Instead, at each mode \(i\) and for each \(k=1,\dots,K\), it applies \((\dots((X\times A_k^{(i)})\otimes B_k^{(i)})\dots)\) and sums over \(k\) whenever an even number of modes has been processed, as justified by Theorem 4.2. Its stated complexity is
\[
O\Bigl(d\cdot \max(m,n)^d \cdot \bigl[\tfrac12 C^{(A)} + \tfrac12 C^{(B)} + C^{(A)}C^{(B)}\bigr]\cdot K\Bigr),
\]
which is always less than or equal to the cost of Algorithm 1 [2008.09342].

The paper also provides a theoretical comparison against TT, BT, TR, and HT. Under \(m_i\approx m\), \(n_i\approx n\), and \(r\) denoting TT/BT/TR/HT rank, the summarized asymptotics are as follows.

| Format | Param \((\Theta)\) | FLOPs per gate |
|---|---|---|
| Ori. | \(O((mn)^d)\) | \(O((mn)^d)\) |
| TT | \(O((d-2)mnr^2 + 2mnr)\) | \(O(d\cdot\max(m,n)^{d+1}\cdot r^2)\) |
| BT | \(O((dmnr + r^d)P)\) | \(O((d\cdot\max(m,n)^{d+1} + n^d)\cdot r^d\cdot P)\) |
| TR | \(O(d(m+n)r^2)\) | \(O(d(m^d + n^d)\cdot r^3)\) |
| HT | \(O((d-1)r^3 + d\cdot m\cdot n\cdot r)\) | \(O((2d-1)\cdot \max(m,n)^{d+1}\cdot r^{1+\log_2 d})\) |
| KCP | \(O(d(m+n)r\cdot K)\) | \(O(d\cdot\max(m,n)^d\cdot (r+r^2)\cdot K)\) |

The paper further states that space is proportional to \(d(m+n)rK\), hence linear in \(m,n,r,K\), and that the FLOPs are comparable to TT, while TT yields far poorer compression [2008.09342]. It also reports that, as \(r\) grows, both the parameter count and FLOPs of KCP remain lower than TT, BT, TR, and HT for typical choices of \(K=4\) or \(6\).

## 6. Empirical results, theoretical lineage, and interpretive boundaries

The empirical profile of the multi-kernel ConvLSTM is reported on UCF-101 and Sports-1M, with additional I3D-based experiments [1908.08990]. On UCF-101 Split 1 with frozen VGG-16 conv5\_3 features \((14\times14\times512)\), the baselines are single-kernel ConvLSTM \(3\times3\times512 \rightarrow 71.27\%\) top-1 and \(5\times5\times512 \rightarrow 72.20\%\). A simple multi-kernel variant with \(C_{3\times3}=C_{5\times5}=256\) reaches \(73.18\%\), and MK plus stacked \(1\times1\) mix reaches \(74.09\%\). On Sports-1M with 20 random sports classes, sampled at \(1\) fps with \(30\) frames per clip, the baselines are \(80.67\%\) for \(3\times3\times512\) and \(81.09\%\) for \(5\times5\times512\), while MK \((256+256)\) reaches \(81.34\%\). In I3D-based experiments on a 17-class subset of UCF-101, the RGB-only I3D baseline is \(96.62\%\), ConvLSTM \(3\times3\) gives \(96.46\%\), ConvLSTM \(5\times5\) gives \(96.62\%\), and Inception-like MK gives \(97.46\%\). On full UCF-101 with end-to-end fine-tuning, ConvLSTM \(3\times3\) gives \(86.33\%\), \(5\times5\) gives \(86.92\%\), simple MK gives \(87.21\%\), MK with flow-attention gives \(87.39\%\), Inception MK gives \(88.40\%\), and end-to-end fine-tuning gives \(90.09\%\) [1908.08990].

The ablation summary is correspondingly precise: mixing \(3\times3\) and \(5\times5\) always beats either alone by approximately \(1\)–\(3\%\); the \(1\times1\) mixing layer adds approximately \(1\%\) on UCF-101/VGG; flow-based attention adds approximately \(0.2\%\); and end-to-end fine-tuning with MK and \(1\times1\) bottlenecks yields a large \(+1.7\%\) [1908.08990]. The same source reports that MK adds approximately \(1\)–\(3\%\) more time per iteration than the larger single-kernel baseline, and that MK \((3\times3+5\times5)\) requires approximately \(8.9\)M weights versus \(13.1\)M for \(5\times5\) alone and \(4.7\)M for \(3\times3\) alone.

For KCP-LSTM, experiments are reported on UCF11, Youtube Celebrities Face, and UCF50 [2008.09342]. The comparison includes Original LSTM, TT-LSTM, BT-LSTM, TR-LSTM, HT-LSTM, and KCP-LSTM with rank settings \((4,4,2)\) and \((4,2,2)\) on UCF11/Youtube, and \((6,4,4)\), \((6,4,2)\), \((6,2,2)\) on UCF50, all with and without four-gate weight-sharing. The key empirical trends are that all tensor formats achieve near-identical final Top-1 and average accuracies within \(\pm1\)–\(2\%\), while KCP routinely reaches higher compression ratios with fewer FLOPs. Specifically, on UCF11 without sharing, KCP reaches up to \(22\)k\(\times\) compression with \(37.9\) MFLOPs versus TT \(77.2\), BT \(264.8\), TR \(211.7\), and HT \(125.4\); with weight-sharing, KCP reaches up to \(59\)k\(\times\) versus BT \(17\)k\(\times\), TR \(34\)k\(\times\), and HT \(36\)k\(\times\); on UCF50, KCP reaches up to \(122\)k\(\times\) without sharing and \(278\)k\(\times\) with sharing [2008.09342]. Figure 6 is summarized as showing nearly identical convergence across methods, and Figure 8 as showing that raising the KT-rank \(K\) is more important for expressivity than boosting \(C^{(A)}\) and \(C^{(B)}\) individually.

The KCP paper also emphasizes parallel potential. Because KCP is a sum over \(k=1,\dots,K\) of independent Kronecker blocks \(\mathcal{A}_k\otimes\mathcal{B}_k\), Algorithm 2 can be split across \(K\) threads or processes. In a simple CPU-based timing test, inference time under a multi-process implementation decreases roughly linearly in \(K\), up to manageability overhead [2008.09342].

The theoretical lineage supplied by the kernel-machine formulation is useful for clarifying what is and is not being generalized. Starting from a Mercer kernel
\[
k_\theta(\tilde z_i,z_t)=q_\theta(\tilde z_i^\top z_t),
\]
the recurrent kernel-machine derivation yields a memory-cell recursion and, after dynamic gating, an LSTM-like model
\[
h_t' = o_t\odot c_t,\qquad
c_t = \eta_t\odot \tilde c_t + f_t\odot c_{t-1},
\]
with
\[
\eta_t=\sigma(W_\eta z_t' + b_\eta),\quad
f_t=\sigma(W_f z_t' + b_f),\quad
o_t=\sigma(W_o z_t' + b_o),\quad
\tilde c_t=W_c z_t'
\]
[1910.04233]. Extending this to \(n\)-gram filters replaces rank-1 filters by length-\(n\) convolutions:
\[
\tilde c_t=\tilde{\mathbf X}\cdot X_t + \tilde H h_{t-1}',
\]
with separate filter banks for the cell candidate and each gate. The paper states that CNNs, Gated CNNs, CIFG-LSTMs, and RANs emerge as special cases under specific gate settings [1910.04233]. This suggests that multi-kernel LSTM variants can be interpreted not only as engineering modifications, but also as members of a broader family of gated convolutional sequence models.

Taken together, these results delimit two major interpretations of Multi-Kernel Decomposed LSTM. One interpretation is architectural and spatial: multiple convolutional kernels are embedded inside each recurrent gate, optionally modulated by flow-based attention [1908.08990]. The other is algebraic and compressive: the gate matrices are decomposed into multiple Kronecker factors with internal CP structure and fast contractions [2008.09342]. The kernel-machine account provides a unifying theoretical backdrop in which dynamic gating and multi-kernel convolution arise from a common sequence-modeling formalism [1910.04233].

Source: https://www.emergentmind.com/topics/multi-kernel-decomposed-lstm