Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Universal Time-Series Representation Learning: A Survey (2401.03717v3)

Published 8 Jan 2024 in cs.LG and cs.AI

Abstract: Time-series data exists in every corner of real-world systems and services, ranging from satellites in the sky to wearable devices on human bodies. Learning representations by extracting and inferring valuable information from these time series is crucial for understanding the complex dynamics of particular phenomena and enabling informed decisions. With the learned representations, we can perform numerous downstream analyses more effectively. Among several approaches, deep learning has demonstrated remarkable performance in extracting hidden patterns and features from time-series data without manual feature engineering. This survey first presents a novel taxonomy based on three fundamental elements in designing state-of-the-art universal representation learning methods for time series. According to the proposed taxonomy, we comprehensively review existing studies and discuss their intuitions and insights into how these methods enhance the quality of learned representations. Finally, as a guideline for future studies, we summarize commonly used experimental setups and datasets and discuss several promising research directions. An up-to-date corresponding resource is available at https://github.com/itouchz/awesome-deep-time-series-representations.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (215)
  1. Large-scale indexing, discovery, and ranking for the internet of things (iot). ACM CSUR, 51(2):1–53, 2018.
  2. Anomaly detection for iot time-series data: A survey. IEEE Internet of Things Journal, 7(7):6481–6494, 2019.
  3. A survey of physics-based attack detection in cyber-physical systems. ACM CSUR, 51(4):1–36, 2018.
  4. Deep learning-based anomaly detection in cyber-physical systems: Progress and opportunities. ACM CSUR, 54(5):1–36, 2021.
  5. Neural network–based financial volatility forecasting: A systematic review. ACM CSUR, 55(1):1–30, 2022.
  6. Longbing Cao. Ai in finance: Challenges, techniques, and opportunities. ACM CSUR, 55(3):1–38, 2022.
  7. Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities. ACM CSUR, 54(4):1–40, 2021a.
  8. A survey on deep learning for human activity recognition. ACM CSUR, 54(8):1–34, 2021.
  9. Time-series data mining. ACM CSUR, 45(1):1–34, 2012.
  10. Time-series forecasting with deep learning: A survey. Philos. Trans. Royal Soc., 379(2194):20200209, 2021.
  11. Deep learning for time series classification: A review. Data Min. Knowl. Discov., 33(4):917–963, 2019.
  12. Time series extrinsic regression: Predicting numeric values from time series data. Data Min. Knowl. Discov., 35:1032–1060, 2021a.
  13. Deep learning for anomaly detection in time-series data: Review, analysis, and guidelines. IEEE Access, 2021.
  14. Time Series Data Representation and Dimensionality Reduction Techniques, pages 267–284. Springer, 2020.
  15. A transformer-based framework for multivariate time series representation learning. In KDD, pages 2114–2124, 2021.
  16. TS2Vec: Towards universal representation of time series. In AAAI, volume 36, pages 8980–8987, 2022.
  17. Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion. In ICML, pages 25038–25054, 2022.
  18. Recurrent neural networks for edge intelligence: A survey. ACM CSUR, 54(4):1–38, 2021.
  19. A review of recurrent neural network-based methods in computational physiology. IEEE Trans. Neural Netw. Learn. Syst., 0(0):1–21, 2022.
  20. An empirical survey of data augmentation for time series classification with neural networks. PLOS ONE, 16(7):e0254841, 2021.
  21. Transformers in time series: A survey. In IJCAI, 2023.
  22. A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection. arXiv:2307.03759, 2023a.
  23. Latent odes for irregularly-sampled time series. In NeurIPS, volume 33, pages 5320–5330, 2019a.
  24. A review of deep learning methods for irregularly sampled medical time series data. arXiv:2010.12493, 2020.
  25. Label-efficient time series representation learning: A review. arXiv:2302.06433, 2023.
  26. Self-supervised learning for time series analysis: Taxonomy, progress, and prospects. arXiv:2306.10125, 2023a.
  27. A survey on time-series pre-trained models. arXiv:2305.10716, 2023.
  28. Unsupervised representation learning for time series: A review. arXiv:2308.01578, 2023.
  29. Self-supervised representation learning: Introduction, advances, and challenges. IEEE Signal Process. Mag., 39(3):42–62, 2022.
  30. Recursive time series data augmentation. In ICLR, 2023.
  31. TimesNet: Temporal 2d-variation modeling for general time series analysis. In ICLR, 2023.
  32. Time series data augmentation for deep learning: A survey. In IJCAI, pages 4653–4660, 2021.
  33. A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recognit. Lett., 42:11–24, 2014.
  34. Beyond just vision: A review on self-supervised representation learning on multimodal and temporal data. arXiv:2206.02353, 2022.
  35. A deep multi-task representation learning method for time series classification and retrieval. Inf. Sci., 555:17–32, 2021b.
  36. A real-time action representation with temporal encoding and deep compression. IEEE Trans. Circuits Syst. Video Technol., 31(2):647–660, 2020.
  37. Attentive neural controlled di01fferential equations for time-series classification and forecasting. In ICDM, pages 250–259, 2021.
  38. Crosspyramid: Neural ordinary differential equations architecture for partially-observed time-series. arXiv:2212.03560, 2022.
  39. EXIT: Extrapolation and interpolation-based neural controlled differential equations for time-series classification and forecasting. In WebConf, pages 3102–3112, 2022.
  40. FEAT: A general framework for feature-aware multivariate time-series representation learning. Knowl.-Based Syst., 277:110790, 2023a.
  41. Latent ordinary differential equations for irregularly-sampled time series. In NeurIPS, pages 5321–5331, 2019b.
  42. Learning disentangled representations of satellite image time series. In ECML PKDD, pages 306–321, 2019.
  43. MARINA: An mlp-attention model for multivariate time-series analysis. In CIKM, pages 2230–2239, 2022.
  44. Memory-augmented dense predictive coding for video representation learning. In ECCV, pages 312–329, 2020.
  45. Multi-task self-supervised time-series representation learning. arXiv:2303.01034, 2023.
  46. TriBERT: Human-centric audio-visual representation learning. In NeurIPS, volume 34, pages 9774–9787, 2021.
  47. Unsupervised scalable representation learning for multivariate time series. In NeurIPS, volume 32, 2019.
  48. Multilevel wavelet decomposition network for interpretable time series analysis. In KDD, pages 2437–2446, 2018.
  49. Ti-MAE: Self-supervised masked time series autoencoders. arXiv:2301.08871, 2023.
  50. UniTS: A universal time series analysis framework with self-supervised representation learning. arXiv:2303.13804, 2023a.
  51. T-Rep: Representation learning for time series using time-embeddings. arXiv:2310.04486, 2023.
  52. Delta: deep local pattern representation for time-series clustering and classification using visual perception. Knowl.-Based Syst., 212:106551, 2021.
  53. Learning robust and consistent time series representations: A dilated inception-based approach. arXiv:2306.06579, 2023.
  54. Audio word2vec: Sequence-to-sequence autoencoding for unsupervised learning of audio segmentation and representation. IEEE/ACM Trans. Audio Speech Lang. Process., 27(9):1481–1493, 2019.
  55. One Fits All: Power general time series analysis by pretrained lm. arXiv:2302.11939, 2023.
  56. One Transformer for All Time Series: Representing and training with time-dependent heterogeneous tabular data. arXiv:2302.06375, 2023.
  57. SSAN: Separable self-attention network for video representation learning. In CVPR, pages 12618–12627, 2021.
  58. Contrastive shapelet learning for unsupervised multivariate time series representation learning. arXiv:2305.18888, 2023b.
  59. Effectively modeling time series with simple discrete state spaces. In ICLR, 2023b.
  60. Learning representations of multivariate time series with missing data. Pattern Recognition, 96:106973, 2019.
  61. Modeling irregular time series with continuous recurrent units. In ICML, pages 19388–19405, 2022.
  62. Modeling temporal data as continuous functions with stochastic process diffusion. In ICML, pages 2452–2470, 2023.
  63. Neural continuous-discrete state space models for irregularly-sampled time series. In ICML, pages 926–951, 2023.
  64. Random warping series: A random features method for time-series embedding. In AISTATS, pages 793–802, 2018.
  65. TE-ESN: time encoding echo state network for prediction based on irregularly sampled time series data. In IJCAI, pages 3010–3016, 2021.
  66. Towards explainable representation of time-evolving graphs via spatial-temporal graph attention networks. In CIKM, pages 2137–2140, 2019.
  67. Hypertime: Implicit neural representations for time series. In NeurIPS Workshop on Synthetic Data for Empowering ML Research, 2022.
  68. Multivariate time series representation learning via hierarchical correlation pooling boosted graph neural network. IEEE Trans. Artif. Intell., 2023a.
  69. Towards learning disentangled representations for time series. In KDD, pages 3270–3278, 2022.
  70. Time series domain adaptation via sparse associative structure alignment. In AAAI, pages 6859–6867, 2021.
  71. Weakly paired associative learning for sound and image representations via bimodal associative memory. In CVPR, pages 10534–10543, 2022a.
  72. Contrast everything: A hierarchical contrastive framework for medical time-series. In NeurIPS, 2023b.
  73. ContiFormer: Continuous-time transformer for irregular time series modeling. In NeurIPS, 2023.
  74. TriD-MAE: A generic pre-trained model for multivariate time series with missing values. In CIKM, pages 3164–3173, 2023c.
  75. A multi-scale decomposition mlp-mixer for time series analysis. arXiv:2310.11959, 2023.
  76. FITS: Modeling time series with 10⁢k10𝑘10k10 italic_k parameters. arXiv:2307.03756, 2023.
  77. End-to-end incomplete time-series modeling from linear memory of latent variables. IEEE Trans. Cybern., 50(12):4908–4920, 2019.
  78. WHEN: A wavelet-dtw hybrid attention network for heterogeneous time series analysis. In KDD, page 2361–2373, 2023c.
  79. Sparse binary transformers for multivariate time series modeling. In KDD, page 544–556, 2023.
  80. Multi-time attention networks for irregularly sampled time series. In ICLR, 2021.
  81. Temporal aggregate representations for long-range video understanding. In ECCV, pages 154–171, 2020.
  82. TARNet: Task-aware reconstruction for time-series transformer. In KDD, pages 212–220, 2022.
  83. TCGL: Temporal contrastive graph for self-supervised video representation learning. IEEE Trans. Image Process., 31:1978–1993, 2022.
  84. Time series continuous modeling for imputation and forecasting with implicit neural representations. arXiv:2306.05880, 2023.
  85. NuTime: Numerically multi-scaled embedding for large-scale time series pretraining. arXiv:2310.07402, 2023.
  86. Representation learning via global temporal alignment and cycle-consistency. In CVPR, pages 11068–11077, 2021.
  87. Wave2vec: Deep representation learning for clinical temporal data. Neurocomputing, 324:31–42, 2019.
  88. Decoupling local and global representations of time series. In AISTATS, pages 8700–8714, 2022.
  89. Sqn2Vec: learning sequence representation via sequential patterns with a gap constraint. In ECML PKDD, pages 569–584, 2018.
  90. SimMTM: A simple pre-training framework for masked time-series modeling. arXiv:2302.00861, 2023.
  91. Cross-architecture self-supervised video representation learning. In CVPR, pages 19270–19279, 2022.
  92. Learning by aligning videos in time. In CVPR, pages 5548–5558, 2021.
  93. Self-supervised spatiotemporal representation learning by exploiting video continuity. In AAAI, pages 1564–1573, 2022.
  94. Self-supervised video representation learning by pace prediction. In ECCV, pages 504–521, 2020.
  95. Self-supervised video representation learning by uncovering spatio-temporal statistics. IEEE Trans. Pattern Anal. Mach. Intell., 44(7):3791–3806, 2021.
  96. TransRank: Self-supervised video representation learning via ranking-based transformation recognition. In CVPR, pages 3000–3010, 2022.
  97. Learning decomposed spatial relations for multi-variate time-series modeling. In AAAI, pages 7530–7538, 2023.
  98. Contrastive spatio-temporal pretext learning for self-supervised video representation. In AAAI, pages 3380–3389, 2022a.
  99. Cycle-contrast for self-supervised video representation learning. In NeurIPS, volume 33, pages 8089–8100, 2020.
  100. Dual contrastive learning for spatio-temporal representation. In MM, pages 5649–5658, 2022.
  101. Frame-wise action representations for long videos via sequence contrastive learning. In CVPR, pages 13801–13810, 2022.
  102. Hierarchically decoupled spatial-temporal contrast for self-supervised video representation learning. In WACV, pages 3235–3245, 2022.
  103. Learning from untrimmed videos: Self-supervised video representation learning with hierarchical consistency. In CVPR, pages 13821–13831, 2022.
  104. Learning representations from audio-visual spatial alignment. In NeurIPS, volume 33, pages 4733–4744, 2020.
  105. Modeling video as stochastic processes for fine-grained video representation learning. In CVPR, pages 2225–2234, 2023d.
  106. On temporal granularity in self-supervised video representation learning. In BMVC, page 541, 2022.
  107. PrimeNet: Pre-training for irregular multivariate time series. In AAAI, 2023.
  108. RSPNet: Relative speed perception for unsupervised video representation learning. In AAAI, pages 1045–1053, 2021c.
  109. Self-supervised contrastive pre-training for time series via time-frequency consistency. In NeurIPS, volume 35, pages 3988–4003, 2022b.
  110. Self-supervised time series representation learning with temporal-instance similarity distillation. In ICML Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward, 2022.
  111. Spatiotemporal contrastive video representation learning. In CVPR, pages 6964–6974, 2021.
  112. TCLR: Temporal contrastive learning for video representation. Comput. Vis. Image Underst., 219:103406, 2022.
  113. TempCLR: Temporal alignment representation with contrastive learning. In ICLR, 2023a.
  114. Time-equivariant contrastive video representation learning. In ICCV, pages 9970–9980, 2021.
  115. TimeAutoML: Autonomous representation learning for multivariate irregularly sampled time series. arXiv:2010.01596, 2020.
  116. TimeCLR: A self-supervised contrastive learning framework for univariate time series representation. Knowl.-Based Syst., 245:108606, 2022a.
  117. TS-Rep: Self-supervised time series representation learning from robot sensor data. In NeurIPS Workshop on Self-Supervised Learning - Theory and Practice, 2022.
  118. Unsupervised representation learning for time series with temporal neighborhood coding. In ICLR, 2021.
  119. Context consistency regularization for label sparsity in time series. In ICML, pages 31579–31595, 2023.
  120. FOCAL: Contrastive learning for multimodal time-series sensing signals in factorized orthogonal latent space. In NeurIPS, 2023.
  121. A co-training approach for noisy time series learning. In CIKM, pages 3308–3318, 2023e.
  122. TEST: Text prototype aligned embedding to activate llm’s ability for time series. arXiv:2308.08241, 2023.
  123. Time-series representation learning via temporal and contextual contrasting. In IJCAI, pages 2352–2359, 2021.
  124. Exploring temporally dynamic data augmentation for video recognition. In ICLR, 2023b.
  125. Time series contrastive learning with information-aware augmentations. In AAAI, pages 4534–4542, 2023.
  126. Frequency selective augmentation for video representation learning. In AAAI, pages 1124–1132, 2023c.
  127. Finding order in chaos: A novel data augmentation method for time series in contrastive learning. In NeurIPS, 2023.
  128. Contrastive learning of global and local video representations. In NeurIPS, volume 34, pages 7025–7040, 2021.
  129. Long short view feature decomposition via contrastive video representation learning. In ICCV, pages 9244–9253, 2021.
  130. Cross-modal mutual learning for audio-visual speech recognition and manipulation. In AAAI, pages 3036–3044, 2022b.
  131. Irregularly-sampled time series modeling with spline networks. arXiv:2210.10630, 2022.
  132. Multi-view integrative attention-based deep representation learning for irregular clinical time-series data. J-BHI, 26(8):4270–4280, 2022b.
  133. Direct embedding of temporal network edges via time-decayed line graphs. In ICLR, 2023.
  134. Learning video representations from large language models. In CVPR, pages 6586–6597, 2023.
  135. Long short-term memory. Neural Comput., 9(8), 1997.
  136. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv:1406.1078, 2014.
  137. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.01271, 2018.
  138. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd ICLR, 2015.
  139. Deep learning for time series classification and extrinsic regression: A current survey. arXiv:2302.02515, 2023.
  140. Attention is all you need. NeurIPS, 30, 2017.
  141. A survey on principles, models and methods for learning from irregularly sampled time series. arXiv:2012.00168, 2020.
  142. Experimental analysis of the effect of autoencoder architectures on time series representation learning. Korea Computer Congress, pages 842–844, 2020.
  143. word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method. arXiv:1402.3722, 2014.
  144. Evaluation methods for representation learning: A survey. IJCAI-ECAI, 2022.
  145. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, pages 11106–11115, 2021.
  146. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, pages 95–104, 2018.
  147. Monash time series forecasting archive. In NeurIPS, 2021.
  148. The ucr time series archive. IEEE/CAA J. Autom. Sin., 6(6):1293–1305, 2019.
  149. The uea multivariate time series classification archive, 2018. arXiv:1811.00075, 2018.
  150. A public domain dataset for human activity recognition using smartphones. In ESANN, volume 3, page 3, 2013.
  151. Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. Crit. Care Med., 48(2):210–217, 2020.
  152. Time series extrinsic regression. Data Min. Knowl. Discov., pages 1–29, 2021b.
  153. Matrix profile viii: domain agnostic online semantic segmentation at superhuman performance levels. In ICDM, pages 117–126, 2017.
  154. Clasp: parameter-free time series segmentation. Data Min. Knowl. Discov., 37(3):1262–1300, 2023.
  155. Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress. IEEE Trans. Knowl. Data Eng., 2021.
  156. Timesead: Benchmarking deep multivariate time-series anomaly detection. TMLR, 2023.
  157. Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In KDD, pages 3220–3230, 2021.
  158. TSB-UAD: an end-to-end benchmark suite for univariate time-series anomaly detection. VLDB, 15(8):1697–1711, 2022a.
  159. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In WWW, pages 187–196, 2018.
  160. DCdetector: Dual attention contrastive representation learning for time series anomaly detection. In KDD, page 3033–3045, 2023b.
  161. Context-aware deep time-series decomposition for anomaly detection in businesses. In ECML PKDD, pages 330–345, 2023.
  162. Choose wisely: An extensive evaluation of model selection for anomaly detection in time series. VLDB, 16(11):3418–3432, 2023.
  163. Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection. VLDB, 15(11):2774–2787, 2022b.
  164. Towards a rigorous evaluation of time-series anomaly detection. In AAAI, pages 7194–7201, 2022.
  165. Do you know existing accuracy metrics overrate time-series anomaly detections? In SAC, pages 403–412, 2022.
  166. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. IJCV, pages 1–23, 2022.
  167. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV, 2019.
  168. The sound of pixels. In ECCV, pages 570–586, 2018.
  169. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In ICLR, 2018.
  170. Time-series generative adversarial networks. NeurIPS, 32, 2019.
  171. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In CinC, pages 245–248, 2012.
  172. Af classification from a short single lead ecg recording: The physionet/computing in cardiology challenge 2017. In CinC, pages 1–4, 2017.
  173. PhysioToolkit PhysioBank. Physionet: components of a new research resource for complex physiologic signals. Circulation, 101(23):e215–e220, 2000.
  174. Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64(6):061907, 2001.
  175. uwave: Accelerometer-based personalized gesture recognition and its applications. PMC, 5(6):657–675, 2009.
  176. Lip reading in the wild. In ACCV, pages 87–103, 2017.
  177. Karol J Piczak. Esc: Dataset for environmental sound classification. In MM, pages 1015–1018, 2015.
  178. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv:1212.0402, 2012.
  179. Hmdb: a large video database for human motion recognition. In ICCV, pages 2556–2563, 2011.
  180. The kinetics human action video dataset. arXiv:1705.06950, 2017.
  181. Mmact: A large-scale dataset for cross modal human action understanding. In ICCV, pages 8658–8667, 2019.
  182. A dataset of neonatal eeg recordings with seizure annotations. Sci. Data, 6(1):1–8, 2019.
  183. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHME, 2016.
  184. Revisiting time series outlier detection: Definitions and benchmarks. In NeurIPS, 2021.
  185. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In KDD, pages 2828–2837, 2019.
  186. Practical approach to asynchronous multivariate time series anomaly detection and localization. In KDD, pages 2485–2494, 2021.
  187. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In KDD, pages 387–395, 2018.
  188. SWaT: A water treatment testbed for research and training on ics security. In CySWater, pages 31–36, 2016.
  189. WADI: a water distribution testbed for research in the design of secure cyber physical systems. In CySWater, pages 25–28, 2017.
  190. Real-world anomaly detection in surveillance videos. In CVPR, pages 6479–6488, 2018.
  191. Oops! predicting unintentional action in video. In CVPR, pages 919–929, 2020.
  192. The deepfake detection challenge (DFDC) dataset. arXiv:2006.07397, 2020.
  193. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation, 101(23):e215–e220, 2000.
  194. Coherence-based label propagation over time series for accelerated active learning. In ICLR, 2021.
  195. Are all frames equal? active sparse labeling for video action detection. NeurIPS, 35:14358–14373, 2022.
  196. Concept drift detection in data stream mining: A literature review. J. King Saud Univ. - Comput. Inf. Sci., 34(10):9523–9540, 2022.
  197. Recent advances in concept drift adaptation methods for deep learning. In IJCAI, pages 5654–5661, 2022.
  198. Adatime: A benchmarking suite for domain adaptation on time series data. TKDD, 17(8):1–18, 2023.
  199. Domain adaptation for time series forecasting via attention sharing. In ICML, pages 10280–10297, 2022.
  200. AutoGT: Automated graph transformer architecture search. In ICLR, 2023f.
  201. Neural architecture search: Insights from 1000 papers. arXiv:2301.08727, 2023.
  202. AutoAI-TS: Autoai for time series forecasting. In SIGMOD, 2021.
  203. Differentiable neural architecture search for high-dimensional time series forecasting. IEEE Access, 9, 2021.
  204. AutoTS: Automatic time series forecasting model design based on two-stage pruning. arXiv:2203.14169, 2022.
  205. Efficient automated deep learning for time series forecasting. In ECML PKDD, pages 664–680, 2022.
  206. LightCTS: A lightweight framework for correlated time series forecasting. In SIGMOD, 2023.
  207. Neural architecture search for time series classification. In IJCNN, 2020.
  208. RNTS: Robust neural temporal search for time series classification. In IJCNN, 2021.
  209. AutoTransformer: Automatic transformer architecture design for time series classification. In PAKDD, 2022.
  210. Large models for time series and spatio-temporal data: A survey and outlook. arXiv:2310.10196, 2023b.
  211. Recurrent neural networks for multivariate time series with missing values. Sci. Rep., 8(1):6085, 2018.
  212. Neural ordinary differential equations. In NeurIPS, volume 31, 2018.
  213. Interpolation-prediction networks for irregularly sampled time series. arXiv:1909.07782, 2019.
  214. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763, 2021.
  215. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904–4916, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Patara Trirat (4 papers)
  2. Yooju Shin (8 papers)
  3. Junhyeok Kang (2 papers)
  4. Youngeun Nam (2 papers)
  5. Jihye Na (1 paper)
  6. Minyoung Bae (1 paper)
  7. Joeun Kim (2 papers)
  8. Byunghyun Kim (2 papers)
  9. Jae-Gil Lee (25 papers)
Citations (9)
View on Semantic Scholar

Summary

A Survey on Universal Time-Series Representation Learning

The paper "Universal Time-Series Representation Learning: A Survey" offers a comprehensive review of the landscape of representation learning for time-series data. Time-series data pervades numerous real-world applications, ranging from satellite telemetry to wearable sensor data, and adequately learning representations from this data type is crucial for informed decision-making and accurate analysis. The paper addresses this challenge by presenting a novel taxonomy to systematize methods in this domain, focusing on three fundamental design elements: neural architectures, learning objectives, and training data.

Neural Architectures

The survey categorizes neural architectural approaches into two main areas: the combination of existing building blocks and innovative neural redesign. The combination strategy includes both network-level and module-level integrations of foundational models like convolutional and recurrent neural networks, graph neural networks, and Transformers. Such combinations harness diverse modeling capabilities from distinct architectures, improving overall representation quality. Meanwhile, innovative redesign efforts introduce new network structures or significantly modify existing components to capture complex temporal information and interactions between variables more effectively. Models like Attentive Neural Controlled Differential Equations (ANCDEs) and Time-Series Transformers (TSTs) exemplify the exploration of novel attention mechanisms and conformal geometry insights.

Learning Objectives

The paper distinguishes learning methodologies into supervised, unsupervised, and self-supervised learning, depending on the availability and nature of labeled data. Supervised approaches rely on task-specific loss functions, but are often limited by the lack of annotated datasets. Unsupervised methods typically employ reconstruction-based losses, including masked prediction tasks to exploit input dependencies. On the other hand, self-supervised learning, particularly contrastive learning, has surged in popularity due to its capacity to generate pseudo-labels from intrinsic data characteristics, utilizing functions designed to learn invariant representations of diverse augmentations.

Training Data

The authors highlight the vital role of data-centric approaches, aiming to enhance training datasets rather than merely modifying model architectures or loss functions. Data augmentation remains a key strategy, with methods varying from straightforward random augmentation to more sophisticated, policy-based approaches leveraging temporal and frequency-domain transformations. Furthermore, decomposition and transformation techniques (e.g., wavelet-based decompositions) effectively capture meaningful data features, and sample selection mechanisms ensure high-quality inputs into the model.

Implications and Future Directions

The paper identifies several challenges and future avenues for the field. One critical challenge is the scarcity of labeled time-series data, which hinders the expansive application of supervised learning strategies. Addressing this issue, methods focusing on efficient active learning are expected to grow. The challenges of handling distribution shifts and variability in time-series data also remain prominent, suggesting potential future research in robust representation learning adaptable to concept drift and domain shifts. Additionally, enhancing the reliability of data augmentation strategies and exploring the potential of neural architecture search (NAS) to autonomously design optimal networks promise further advancements. With the rising role of LLMs, there are emerging opportunities for better integrating text data with traditional time-series modalities to achieve a richer, multi-modal representation.

The survey concludes by providing a guideline for evaluating time-series representation models, covering commonly used datasets and performance metrics for evaluating downstream task effectiveness. This ensures researchers and practitioners have the information needed to select relevant datasets and appropriate assessment techniques for their specific objectives.

In conclusion, the paper provides an insightful overview of the contemporary landscape of universal time-series representation learning, emphasizing the ongoing challenges and potential directions to explore. As the field progresses, expanding the toolbox available for this type of data not only advances technical capabilities but also broadens the scope of practical applications, promising advancements across myriad domains.