ConvTran: Hybrid Conv-Transformer for Time-Series

• ConvTran is a hybrid model combining convolutional input encoding with Transformer blocks for efficient multivariate time-series classification.
• It introduces specialized position encodings (tAPE and eRPE) to capture temporal order and restore distance awareness beyond standard sinusoidal schemes.
• Empirical results demonstrate its superior performance on the UEA Archive, achieving high accuracy and reduced training times compared to other deep models.

Searching arXiv for papers on "ConvTran" and closely related time-series transformer architectures. ConvTran is a convolution-enhanced Transformer model for time-series learning whose canonical formulation was introduced for multivariate time-series classification as a combination of convolutional input encoding, a Transformer encoder stack, and a classification head, with the central methodological contribution being a pair of position-encoding mechanisms tailored to time-series data: time Absolute Position Encoding (tAPE) and efficient Relative Position Encoding (eRPE) (Foumani et al., 2023). Subsequent work reused the name for a closely related hybrid “conv-to-transformer” classifier in knowledge-distillation studies on the UCR Archive, where ConvTran served as a transformer-based baseline and teacher architecture (Abdullayev et al., 7 Jul 2026). Taken together, the term now denotes a small family of convolution-plus-transformer designs for time-series classification; a plausible implication is that the name refers more to an architectural pattern than to a single immutable instantiation.

1. Origins and problem formulation

ConvTran was proposed in the context of multivariate time-series classification (MTSC), where Transformer models must encode temporal order effectively despite the fact that the efficacy of position encoding in time-series analysis “is not well-studied and remains controversial,” including whether absolute position encoding, relative position encoding, or a combination should be preferred (Foumani et al., 2023). The original formulation therefore addresses two coupled problems: input embedding for multivariate sequences and positional representation for temporal order.

The original model takes one multivariate series $X \in \mathbb{R}^{L \times d_x}$ of length $L$ with $d_x$ channels. It first constructs a learned representation by convolutional encoding, then applies Transformer blocks over the resulting sequence, and finally maps the sequence representation to class probabilities. In the original paper, this design is explicitly tied to MTSC, but the authors also state that the proposed absolute and relative position-encoding methods “can be easily integrated into transformer blocks and used for downstream tasks such as forecasting, extrinsic regression, and anomaly detection” (Foumani et al., 2023).

A central motivation is that standard sinusoidal encodings can behave poorly when applied to time-series tasks with arbitrary $L$ and often small $d_{model}$. The paper argues that the dot-product similarity of standard encodings can lose monotonic “distance awareness” or become anisotropic under these conditions. ConvTran is therefore not only a convolution-front-ended Transformer; it is also a position-encoding redesign for temporal data.

2. Core architecture

In its original MTSC form, ConvTran consists of three stages: convolutional input encoding, a Transformer encoder stack, and a classification head (Foumani et al., 2023).

The convolutional input encoding begins with a bank of 1D temporal convolutions. With $M$ temporal filters of length $K_t$, stride $1$, and padding $\lfloor K_t/2 \rfloor$, the model extracts short-term motifs over the multivariate input:

$$\tilde H_{i,k} =\sum_{c=1}^{d_x}\sum_{t=0}^{K_t-1} X_{\,i+t,\;c}\;W^{\rm temp}_{t,c,k}, \quad k=1,\dots,M,\;i=1,\dots,L.$$

In the reported experiments, $L$0 and $L$1. The resulting $L$2 is then passed through a $L$3 “spatial” convolution to fuse feature maps into a $L$4-dimensional embedding at each time step:

$L$5

Thus $L$6 becomes the Transformer input.

The Transformer stage stacks $L$7 encoder layers, with $L$8–$L$9 used in practice. Each layer adds tAPE, computes multi-head self-attention with eRPE, and applies a two-layer feed-forward network with GELU activations, residual connections, and layer normalization. The classification head then takes the final sequence output $d_x$0, applies both global max-pooling and global average-pooling, concatenates the two $d_x$1-vectors, and feeds the resulting $d_x$2 representation into a linear layer with softmax (Foumani et al., 2023).

Later knowledge-distillation work used a simpler ConvTran teacher for time-series classification: a single 1D temporal convolution produces embeddings $d_x$3, a learned positional embedding is added, one Transformer encoder block is applied, and global average pooling followed by a linear layer and softmax yields class probabilities (Abdullayev et al., 7 Jul 2026). The coexistence of these two specifications suggests a stable high-level template—convolutional tokenization followed by Transformer encoding—alongside variation in positional encoding, encoder depth, and head design.

3. Positional encoding and attention mechanisms

The defining methodological contribution of the original ConvTran is the joint use of tAPE and eRPE (Foumani et al., 2023).

tAPE modifies sinusoidal absolute encoding by incorporating both the series length and embedding dimension. Starting from the standard sinusoidal form,

$d_x$4

ConvTran redefines the frequency as

$d_x$5

and then uses

$d_x$6

In practice, the paper sets $d_x$7 and generates a table $d_x$8 once per input length $d_x$9. The stated purpose is to restore clear, monotonic distance awareness while maintaining isotropy in low dimensions.

eRPE adds a learnable scalar relative-position bias after softmax. For per-head projections $L$0, standard attention is defined as

$L$1

ConvTran then introduces a learnable vector $L$2 and defines

$L$3

The stated advantage is that eRPE uses only $L$4 scalar parameters, one per relative shift, rather than large $L$5 or $L$6 tables. The paper further reports that post-softmax addition yields sharper relative weighting.

By contrast, the ConvTran variant used in the later distillation study injects position information by adding a learned positional embedding $L$7 to the convolutional embeddings, $L$8, and then applies standard multi-head self-attention and a position-wise feed-forward network (Abdullayev et al., 7 Jul 2026). Factually, this means that “ConvTran” in the literature spans both a position-encoding-centered design and a more conventional learned-position conv-to-transformer instantiation.

4. Training procedure and empirical performance

The original ConvTran is trained with cross-entropy loss, optimized by Adam with learning rate approximately $L$9, and uses early stopping on a held-out validation set; batch normalization is applied in the convolution blocks and layer normalization within the Transformer (Foumani et al., 2023). Evaluation is conducted on the UEA archive of $d_{model}$0 fixed-length multivariate TSC datasets, with channels ranging from $d_{model}$1 to $d_{model}$2, lengths from $d_{model}$3 to $d_{model}$4, and $d_{model}$5–$d_{model}$6k training instances, plus two large-scale variable-length datasets, Ford Challenge and Actitracker.

The paper reports overall classification accuracy and average rank over datasets. Its key findings are specific. ConvTran achieves “the best average rank among six deep-learning models” with rank approximately $d_{model}$7 versus the next best approximately $d_{model}$8. On the $d_{model}$9 largest datasets, defined as datasets with at least $M$0 training samples per class, ConvTran wins $M$1 times; one example given is HAR, where ConvTran reaches $M$2 accuracy versus $M$3 for TST and $M$4 for InceptionTime. In a head-to-head comparison on $M$5 UEA sets, ConvTran outperforms ROCKET on $M$6, HIVE-COTEv2 on $M$7, and CIF on $M$8. On the largest dataset, InsectWingbeat with $M$9k samples, ConvTran trains in approximately $K_t$0k s versus ROCKET’s approximately $K_t$1k s while delivering a $K_t$2 absolute accuracy improvement (Foumani et al., 2023).

The ablation results are equally central to the model’s identity. The paper states that tAPE is better than vanilla or learned absolute encodings at $K_t$3, eRPE is better than Shaw- or vector-style relative encodings, and the combination tAPE+eRPE is significantly better than any other pairing. The discussion attributes the gains to three synergistic effects: disjoint temporal and spatial convolutions for local feature extraction, tAPE for restoring distance awareness in low-dimensional embeddings when $K_t$4, and eRPE for capturing relative order with a scalar, translation-equivariant bias.

5. Distillation and compact ConvTran variants

A later study on knowledge distillation for time-series classification evaluated ConvTran alongside FCN and Inception on the UCR Archive and treated ConvTran as a transformer-based architecture whose student models are obtained by shrinking attention heads and embedding dimensions (Abdullayev et al., 7 Jul 2026). The teacher ConvTran in that study has embedding dimension $K_t$5, number of attention heads $K_t$6, per-head dimension $K_t$7, feed-forward hidden size $K_t$8, dropout $K_t$9, number of encoder layers $1$0, and total parameters $1$1.

The student variants are defined by proportional shrinking of $1$2 and $1$3, while keeping the other components unchanged.

Model	$1$4	Parameters
Teacher ConvTran	$1$5	33,026
Model_6H	$1$6	24,340
Model_4H	$1$7	15,174
Model_2H	$1$8	8,290
Model_1H	$1$9	4,265

The empirical evaluation covers $\lfloor K_t/2 \rfloor$0 UCR Archive datasets, with $\lfloor K_t/2 \rfloor$1 originally available and $\lfloor K_t/2 \rfloor$2 discarded for missing data. Performance is summarized by counting classification-accuracy “wins” against both the same student architecture trained from scratch and the full teacher. Against studentAlone, Model_2H wins on $\lfloor K_t/2 \rfloor$3 datasets versus studentAlone_2H’s $\lfloor K_t/2 \rfloor$4, a $\lfloor K_t/2 \rfloor$5 “win” improvement. Model_4H narrows the gap to approximately equal wins, while Model_6H and Model_1H show smaller or negative knowledge-distillation effects. Against the teacher, Model_6H wins on $\lfloor K_t/2 \rfloor$6 datasets despite being $\lfloor K_t/2 \rfloor$7 of teacher size, and Model_2H still wins on about one-third of datasets. The paper concludes that knowledge distillation provides the largest benefit at intermediate student complexity, especially $\lfloor K_t/2 \rfloor$8, acts as a strong regularizer by reducing overfitting on smaller datasets, and allows models with as few as approximately $\lfloor K_t/2 \rfloor$9K parameters to retain or even exceed the accuracy of a $$\tilde H_{i,k} =\sum_{c=1}^{d_x}\sum_{t=0}^{K_t-1} X_{\,i+t,\;c}\;W^{\rm temp}_{t,c,k}, \quad k=1,\dots,M,\;i=1,\dots,L.$$0K-parameter teacher on many time-series benchmarks (Abdullayev et al., 7 Jul 2026).

6. Nomenclature, adjacent models, and limitations

The term “ConvTran” is not completely stable across the literature. In the MTSC paper, it denotes a convolution-enhanced Transformer whose signature elements are convolutional input encoding, tAPE, eRPE, and a dual-pooling classifier (Foumani et al., 2023). In the distillation paper, it denotes a hybrid conv-to-transformer classifier with a single temporal convolution, learned positional embedding, one encoder block, and a global-average-pooling classification head (Abdullayev et al., 7 Jul 2026). A plausible implication is that ConvTran should be understood as a model family centered on convolutional tokenization plus Transformer-based sequence modeling, with multiple concrete instantiations.

There is also a separate nomenclature collision in the pose-estimation literature. In “ConvFormer: Parameter Reduction in Transformer Models for 3D Human Pose Estimation by Leveraging Dynamic Multi-Headed Convolutional Attention,” the model is described as “ConvFormer (also called ‘ConvTran’ in what follows)” and is built for monocular 3D human pose estimation rather than time-series classification (Diaz-Arias et al., 2023). That architecture uses spatial and temporal ConvFormer stacks, dynamic multi-headed convolutional self-attention, a temporal joints profile, and a many-to-one output head for predicting the 3D pose of the center frame. It is therefore a distinct model despite the overlapping name.

The limitations explicitly identified for the original MTSC ConvTran include difficulty on “small-sample, very long-series datasets,” with EigenWorms given as an example, and open directions such as extending tAPE/eRPE to self-supervised pre-training, forecasting, regression, or anomaly detection, and exploring learned variants of the $$\tilde H_{i,k} =\sum_{c=1}^{d_x}\sum_{t=0}^{K_t-1} X_{\,i+t,\;c}\;W^{\rm temp}_{t,c,k}, \quad k=1,\dots,M,\;i=1,\dots,L.$$1 scaling in tAPE or smoothing in eRPE (Foumani et al., 2023). In the distilled ConvTran setting, the results indicate that the strongest gains do not occur at the smallest model size; instead, benefit is maximal at intermediate student complexity, while very small or relatively large students show smaller or negative distillation effects (Abdullayev et al., 7 Jul 2026).

Within time-series research, ConvTran is therefore best characterized not as a generic Transformer with a convolutional front end, but as a line of models that combine convolutional local-pattern extraction with Transformer-based long-range interaction, with the original formulation distinguished by an explicit re-engineering of absolute and relative position encoding for temporal data.