---
title: 'ConvTran: Hybrid Conv-Transformer for Time-Series'
url: https://www.emergentmind.com/topics/convtran-model
type: topic
---

# ConvTran: Hybrid Conv-Transformer for Time-Series

Searching arXiv for papers on "ConvTran" and closely related time-series transformer architectures.
ConvTran is a convolution-enhanced Transformer model for time-series learning whose canonical formulation was introduced for multivariate time-series classification as a combination of convolutional input encoding, a Transformer encoder stack, and a classification head, with the central methodological contribution being a pair of position-encoding mechanisms tailored to time-series data: time Absolute Position Encoding (tAPE) and efficient Relative Position Encoding (eRPE) [2305.16642]. Subsequent work reused the name for a closely related hybrid “conv-to-transformer” classifier in knowledge-distillation studies on the UCR Archive, where ConvTran served as a transformer-based baseline and teacher architecture [2607.06796]. Taken together, the term now denotes a small family of convolution-plus-transformer designs for time-series classification; a plausible implication is that the name refers more to an architectural pattern than to a single immutable instantiation.

## 1. Origins and problem formulation

ConvTran was proposed in the context of multivariate time-series classification (MTSC), where Transformer models must encode temporal order effectively despite the fact that the efficacy of position encoding in time-series analysis “is not well-studied and remains controversial,” including whether absolute position encoding, relative position encoding, or a combination should be preferred [2305.16642]. The original formulation therefore addresses two coupled problems: input embedding for multivariate sequences and positional representation for temporal order.

The original model takes one multivariate series $X \in \mathbb{R}^{L \times d_x}$ of length $L$ with $d_x$ channels. It first constructs a learned representation by convolutional encoding, then applies Transformer blocks over the resulting sequence, and finally maps the sequence representation to class probabilities. In the original paper, this design is explicitly tied to MTSC, but the authors also state that the proposed absolute and relative position-encoding methods “can be easily integrated into transformer blocks and used for downstream tasks such as forecasting, extrinsic regression, and anomaly detection” [2305.16642].

A central motivation is that standard sinusoidal encodings can behave poorly when applied to time-series tasks with arbitrary $L$ and often small $d_{model}$. The paper argues that the dot-product similarity of standard encodings can lose monotonic “distance awareness” or become anisotropic under these conditions. ConvTran is therefore not only a convolution-front-ended Transformer; it is also a position-encoding redesign for temporal data.

## 2. Core architecture

In its original MTSC form, ConvTran consists of three stages: convolutional input encoding, a Transformer encoder stack, and a classification head [2305.16642].

The convolutional input encoding begins with a bank of 1D temporal convolutions. With $M$ temporal filters of length $K_t$, stride $1$, and padding $\lfloor K_t/2 \rfloor$, the model extracts short-term motifs over the multivariate input:

$$
\tilde H_{i,k}
=\sum_{c=1}^{d_x}\sum_{t=0}^{K_t-1}
X_{\,i+t,\;c}\;W^{\rm temp}_{t,c,k},
\quad
k=1,\dots,M,\;i=1,\dots,L.
$$

In the reported experiments, $M=64$ and $K_t=8$. The resulting $\tilde H \in \mathbb{R}^{L \times M}$ is then passed through a $1 \times 1$ “spatial” convolution to fuse feature maps into a $d_{model}$-dimensional embedding at each time step:

$$
E_{i,m}
=\sum_{k=1}^{M}\tilde H_{i,k}\;W^{\rm spatial}_{k,m}
+b_m,
\quad m=1,\dots,d_{model}.
$$

Thus $E \in \mathbb{R}^{L \times d_{model}}$ becomes the Transformer input.

The Transformer stage stacks $T$ encoder layers, with $T=3$–$6$ used in practice. Each layer adds tAPE, computes multi-head self-attention with eRPE, and applies a two-layer feed-forward network with GELU activations, residual connections, and layer normalization. The classification head then takes the final sequence output $H^T \in \mathbb{R}^{L \times d_{model}}$, applies both global max-pooling and global average-pooling, concatenates the two $d_{model}$-vectors, and feeds the resulting $2d_{model}$ representation into a linear layer with softmax [2305.16642].

Later knowledge-distillation work used a simpler ConvTran teacher for time-series classification: a single 1D temporal convolution produces embeddings $E \in \mathbb{R}^{T \times d}$, a learned positional embedding is added, one Transformer encoder block is applied, and global average pooling followed by a linear layer and softmax yields class probabilities [2607.06796]. The coexistence of these two specifications suggests a stable high-level template—convolutional tokenization followed by Transformer encoding—alongside variation in positional encoding, encoder depth, and head design.

## 3. Positional encoding and attention mechanisms

The defining methodological contribution of the original ConvTran is the joint use of tAPE and eRPE [2305.16642].

tAPE modifies sinusoidal absolute encoding by incorporating both the series length and embedding dimension. Starting from the standard sinusoidal form,

$$
p_i(2k) = \sin(i\,\omega_k),\qquad
p_i(2k+1) = \cos(i\,\omega_k),
$$

ConvTran redefines the frequency as

$$
\omega_k^{\rm tAPE}
= \omega_k \times \frac{d_{model}}{L}
= 10000^{-2k/d_{model}} \frac{d_{model}}{L},
$$

and then uses

$$
p_i^{\rm tAPE}(2k)   = \sin\bigl(i\,\omega_k^{\rm tAPE}\bigr),\qquad
p_i^{\rm tAPE}(2k+1) = \cos\bigl(i\,\omega_k^{\rm tAPE}\bigr).
$$

In practice, the paper sets $d_{model}=64$ and generates a table $P \in \mathbb{R}^{L \times d_{model}}$ once per input length $L$. The stated purpose is to restore clear, monotonic distance awareness while maintaining isotropy in low dimensions.

eRPE adds a learnable scalar relative-position bias after softmax. For per-head projections $Q,K,V \in \mathbb{R}^{L \times d_z}$, standard attention is defined as

$$
S_{i,j} = \frac{Q_i\cdot K_j}{\sqrt{d_z}},
\qquad
A_{i,j} = \frac{\exp(S_{i,j})}{\sum_{k=1}^L\exp(S_{i,k})}.
$$

ConvTran then introduces a learnable vector $w \in \mathbb{R}^{2L-1}$ and defines

$$
W^{\rm rel}_{i,j} = w_{\,i-j+L},
\qquad
\widetilde A_{i,j} = A_{i,j} + W^{\rm rel}_{i,j},
\qquad
Z_i = \sum_{j=1}^L \widetilde A_{i,j}\,V_j.
$$

The stated advantage is that eRPE uses only $2L-1$ scalar parameters, one per relative shift, rather than large $O(L \cdot d)$ or $O(L^2 \cdot d)$ tables. The paper further reports that post-softmax addition yields sharper relative weighting.

By contrast, the ConvTran variant used in the later distillation study injects position information by adding a learned positional embedding $P \in \mathbb{R}^{T \times d}$ to the convolutional embeddings, $Z^{(0)} = E + P$, and then applies standard multi-head self-attention and a position-wise feed-forward network [2607.06796]. Factually, this means that “ConvTran” in the literature spans both a position-encoding-centered design and a more conventional learned-position conv-to-transformer instantiation.

## 4. Training procedure and empirical performance

The original ConvTran is trained with cross-entropy loss, optimized by Adam with learning rate approximately $10^{-3}$, and uses early stopping on a held-out validation set; batch normalization is applied in the convolution blocks and layer normalization within the Transformer [2305.16642]. Evaluation is conducted on the UEA archive of $30$ fixed-length multivariate TSC datasets, with channels ranging from $2$ to $1345$, lengths from $8$ to $17984$, and $5$–$50$k training instances, plus two large-scale variable-length datasets, Ford Challenge and Actitracker.

The paper reports overall classification accuracy and average rank over datasets. Its key findings are specific. ConvTran achieves “the best average rank among six deep-learning models” with rank approximately $1.3$ versus the next best approximately $2.1$. On the $12$ largest datasets, defined as datasets with at least $100$ training samples per class, ConvTran wins $11/12$ times; one example given is HAR, where ConvTran reaches $90.98\%$ accuracy versus $88.31\%$ for TST and $87.75\%$ for InceptionTime. In a head-to-head comparison on $26$ UEA sets, ConvTran outperforms ROCKET on $19$, HIVE-COTEv2 on $13$, and CIF on $13$. On the largest dataset, InsectWingbeat with $25$k samples, ConvTran trains in approximately $1.6$k s versus ROCKET’s approximately $5.7$k s while delivering a $+30\%$ absolute accuracy improvement [2305.16642].

The ablation results are equally central to the model’s identity. The paper states that tAPE is better than vanilla or learned absolute encodings at $p<0.05$, eRPE is better than Shaw- or vector-style relative encodings, and the combination tAPE+eRPE is significantly better than any other pairing. The discussion attributes the gains to three synergistic effects: disjoint temporal and spatial convolutions for local feature extraction, tAPE for restoring distance awareness in low-dimensional embeddings when $L \neq d_{model}$, and eRPE for capturing relative order with a scalar, translation-equivariant bias.

## 5. Distillation and compact ConvTran variants

A later study on knowledge distillation for time-series classification evaluated ConvTran alongside FCN and Inception on the UCR Archive and treated ConvTran as a transformer-based architecture whose student models are obtained by shrinking attention heads and embedding dimensions [2607.06796]. The teacher ConvTran in that study has embedding dimension $d=16$, number of attention heads $h=8$, per-head dimension $d_k=d/h=2$, feed-forward hidden size $d_{ff}=4d=64$, dropout $p=0.1$, number of encoder layers $L=1$, and total parameters $33\,026$.

The student variants are defined by proportional shrinking of $h$ and $d$, while keeping the other components unchanged.

| Model | $h, d$ | Parameters |
|---|---:|---:|
| Teacher ConvTran | $8, 16$ | 33,026 |
| Model_6H | $6, 12$ | 24,340 |
| Model_4H | $4, 8$ | 15,174 |
| Model_2H | $2, 4$ | 8,290 |
| Model_1H | $1, 2$ | 4,265 |

The empirical evaluation covers $112$ UCR Archive datasets, with $128$ originally available and $16$ discarded for missing data. Performance is summarized by counting classification-accuracy “wins” against both the same student architecture trained from scratch and the full teacher. Against studentAlone, Model_2H wins on $62/112$ datasets versus studentAlone\_2H’s $42/112$, a $+20$ “win” improvement. Model_4H narrows the gap to approximately equal wins, while Model_6H and Model_1H show smaller or negative knowledge-distillation effects. Against the teacher, Model_6H wins on $57/112$ datasets despite being $26\%$ of teacher size, and Model_2H still wins on about one-third of datasets. The paper concludes that knowledge distillation provides the largest benefit at intermediate student complexity, especially $h=2, d=4$, acts as a strong regularizer by reducing overfitting on smaller datasets, and allows models with as few as approximately $8$K parameters to retain or even exceed the accuracy of a $33$K-parameter teacher on many time-series benchmarks [2607.06796].

## 6. Nomenclature, adjacent models, and limitations

The term “ConvTran” is not completely stable across the literature. In the MTSC paper, it denotes a convolution-enhanced Transformer whose signature elements are convolutional input encoding, tAPE, eRPE, and a dual-pooling classifier [2305.16642]. In the distillation paper, it denotes a hybrid conv-to-transformer classifier with a single temporal convolution, learned positional embedding, one encoder block, and a global-average-pooling classification head [2607.06796]. A plausible implication is that ConvTran should be understood as a model family centered on convolutional tokenization plus Transformer-based sequence modeling, with multiple concrete instantiations.

There is also a separate nomenclature collision in the pose-estimation literature. In “ConvFormer: Parameter Reduction in Transformer Models for 3D Human Pose Estimation by Leveraging Dynamic Multi-Headed Convolutional Attention,” the model is described as “ConvFormer (also called ‘ConvTran’ in what follows)” and is built for monocular 3D human pose estimation rather than time-series classification [2304.02147]. That architecture uses spatial and temporal ConvFormer stacks, dynamic multi-headed convolutional self-attention, a temporal joints profile, and a many-to-one output head for predicting the 3D pose of the center frame. It is therefore a distinct model despite the overlapping name.

The limitations explicitly identified for the original MTSC ConvTran include difficulty on “small-sample, very long-series datasets,” with EigenWorms given as an example, and open directions such as extending tAPE/eRPE to self-supervised pre-training, forecasting, regression, or anomaly detection, and exploring learned variants of the $L,d_{model}$ scaling in tAPE or smoothing in eRPE [2305.16642]. In the distilled ConvTran setting, the results indicate that the strongest gains do not occur at the smallest model size; instead, benefit is maximal at intermediate student complexity, while very small or relatively large students show smaller or negative distillation effects [2607.06796].

Within time-series research, ConvTran is therefore best characterized not as a generic Transformer with a convolutional front end, but as a line of models that combine convolutional local-pattern extraction with Transformer-based long-range interaction, with the original formulation distinguished by an explicit re-engineering of absolute and relative position encoding for temporal data.

Source: https://www.emergentmind.com/topics/convtran-model