Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Warpformer: A Multi-scale Modeling Approach for Irregular Clinical Time Series (2306.09368v1)

Published 14 Jun 2023 in cs.LG and cs.AI

Abstract: Irregularly sampled multivariate time series are ubiquitous in various fields, particularly in healthcare, and exhibit two key characteristics: intra-series irregularity and inter-series discrepancy. Intra-series irregularity refers to the fact that time-series signals are often recorded at irregular intervals, while inter-series discrepancy refers to the significant variability in sampling rates among diverse series. However, recent advances in irregular time series have primarily focused on addressing intra-series irregularity, overlooking the issue of inter-series discrepancy. To bridge this gap, we present Warpformer, a novel approach that fully considers these two characteristics. In a nutshell, Warpformer has several crucial designs, including a specific input representation that explicitly characterizes both intra-series irregularity and inter-series discrepancy, a warping module that adaptively unifies irregular time series in a given scale, and a customized attention module for representation learning. Additionally, we stack multiple warping and attention modules to learn at different scales, producing multi-scale representations that balance coarse-grained and fine-grained signals for downstream tasks. We conduct extensive experiments on widely used datasets and a new large-scale benchmark built from clinical databases. The results demonstrate the superiority of Warpformer over existing state-of-the-art approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Layer Normalization. ArXiv abs/1607.06450 (2016).
  2. Donald J Berndt and James Clifford. 1994. Using dynamic time warping to find patterns in time series.. In KDD workshop.
  3. (k, l)-Medians Clustering of Trajectories Using Continuous Dynamic Time Warping. In SIGSPATIAL.
  4. Language Models are Few-Shot Learners. In NeurIPS 2020.
  5. Recurrent neural networks for multivariate time series with missing values. Scientific reports 8, 1 (2018), 1–12.
  6. Neural Ordinary Differential Equations. In NeurIPS. 6572–6583.
  7. Marco Cuturi and Mathieu Blondel. 2017. Soft-dtw: a differentiable loss function for time-series. In International conference on machine learning. PMLR, 894–903.
  8. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 (2019).
  9. Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition. In ICASSP. IEEE, 5884–5888.
  10. Predicting intervention onset in the ICU with switching state space models. AMIA Summits on Translational Science Proceedings 2017 (2017), 82 – 91.
  11. Multitask learning and benchmarking with clinical time series data. Scientific data 6, 1 (2019), 1–18.
  12. Machine learning–based model for prediction of outcomes in acute stroke. Stroke 50, 5 (2019), 1263–1265.
  13. Set Functions for Time Series. In ICML, Vol. 119. 4353–4363.
  14. Mining electronic health records: towards better research applications and clinical care. Nature Reviews Genetics 13, 6 (2012), 395–405.
  15. MIMIC-III, a freely accessible critical care database. Scientific data 3, 1 (2016), 1–9.
  16. Learnable dynamic temporal pooling for time series classification. In AAAI.
  17. Predicting Path Failure In Time-Evolving Graphs. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
  18. TATC: predicting Alzheimer’s disease with actigraphy data. In SIGKDD. 509–518.
  19. Neural Speech Synthesis with Transformer Network. In AAAI.
  20. Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting. In ICLR.
  21. Human Mobility Modeling During the COVID-19 Pandemic via Deep Graph Diffusion Infomax. In AAAI.
  22. Temporal transformer networks: Joint learning of invariant and discriminative time warping. In CVPR. 12426–12435.
  23. Hitanet: Hierarchical time-aware attention networks for risk prediction on electronic health records. In SIGKDD. 647–656.
  24. AdaCare: Explainable Clinical Health Status Representation Learning via Scale-Adaptive Feature Extraction and Recalibration. In AAAI. 825–832.
  25. A comprehensive EHR timeseries pre-training benchmark. In CHIL. 257–278.
  26. Discrete Event, Continuous Time RNNs. CoRR abs/1710.04110 (2017). arXiv:1710.04110 http://arxiv.org/abs/1710.04110
  27. Machine Learning–Based Early Warning Systems for Clinical Deterioration: Systematic Scoping Review. Journal of medical Internet research 23, 2 (2021), e25187.
  28. Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences. In NueIPS. 3882–3890.
  29. A deep active survival analysis approach for precision treatment recommendations: Application of prostate cancer. Expert Systems with Applications 115 (2019), 16–26.
  30. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In ICLR.
  31. SyncTwin: Treatment Effect Estimation with Longitudinal Outcomes. NeurIPS 34 (2021).
  32. Latent Ordinary Differential Equations for Irregularly-Sampled Time Series. In NeurIPS. 5321–5331.
  33. Hiroaki Sakoe and Seibi Chiba. 1978. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing 26 (1978), 159–165.
  34. Differentiable Segmentation of Sequences. In ICLR.
  35. Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting. In ICLR.
  36. Self-Attention with Relative Position Representations. In ACL.
  37. Satya Narayan Shukla and Benjamin Marlin. 2021. Multi-Time Attention Networks for Irregularly Sampled Time Series. In ICLR.
  38. Satya Narayan Shukla and Benjamin M. Marlin. 2019. Interpolation-Prediction Networks for Irregularly Sampled Time Series. In ICLR.
  39. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 Computing in Cardiology. IEEE, 245–248.
  40. The evolved transformer. In ICML. PMLR, 5877–5886.
  41. Clinical intervention prediction and understanding with deep neural networks. In Machine Learning for Healthcare Conference. 322–337.
  42. Treatment Recommendations for COVID-19 Patients along with Robust Explanations. In CBMS. 207–212.
  43. SINDHU TIPIRNENI and CHANDAN K REDDY. 2022. Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series. ACM Trans. Knowl. Discov. Data 1, 1 (2022).
  44. Attention is all you need. In NeurIPS. 5998–6008.
  45. On Position Embeddings in BERT. In ICLR.
  46. Learning Deep Transformer Models for Machine Translation. In ACL.
  47. MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III. In CHIL. 222–235.
  48. Diffeomorphic Temporal Alignment Nets. In Neural Information Processing Systems.
  49. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. In NeurIPS.
  50. Understanding vasopressor intervention and weaning: risk prediction in a public heterogeneous clinical time series database. Journal of the American Medical Informatics Association 24, 3 (2017), 488–495.
  51. Raim: Recurrent attentive and intensive model of multimodal patient monitoring data. In SIGKDD. 2565–2573.
  52. MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records. In AAAI. 10532–10540.
  53. Lsan: Modeling long-term dependencies and short-term correlations with hierarchical attention for risk prediction. In CIKM. 1753–1762.
  54. Artificial intelligence in healthcare. Nature biomedical engineering 2, 10 (2018), 719–731.
  55. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186 (2022).
  56. Learning Robust Patient Representations from Multi-modal Electronic Health Records: A Supervised Deep Learning Approach. In SDM. 585–593.
  57. Graph-Guided Network For Irregularly Sampled Multivariate Time Series. In ICLR.
  58. Time adaptive optimal transport: A framework of time series similarity measure. IEEE Access 8 (2020), 149764–149774.
  59. Development of an early-warning system for high-risk patients for suicide attempt using deep learning and electronic health records. Translational psychiatry 10, 1 (2020), 1–10.
  60. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI.
  61. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML. 27268–27286.
Citations (15)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now