Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Prompting-based Temporal Domain Generalization (2310.02473v2)

Published 3 Oct 2023 in cs.LG

Abstract: Machine learning traditionally assumes that the training and testing data are distributed independently and identically. However, in many real-world settings, the data distribution can shift over time, leading to poor generalization of trained models in future time periods. This paper presents a novel prompting-based approach to temporal domain generalization that is parameter-efficient, time-efficient, and does not require access to future data during training. Our method adapts a trained model to temporal drift by learning global prompts, domain-specific prompts, and drift-aware prompts that capture underlying temporal dynamics. Experiments on classification, regression, and time series forecasting tasks demonstrate the generality of the proposed approach. The code repository will be publicly shared.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. Self-adaptive forecasting for improved deep learning on non-stationary time-series. arXiv preprint arXiv:2202.02403, 2022.
  2. Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In Empirical Methods in Natural Language Processing, 2022.
  3. Temporal domain generalization with drift-aware dynamic neural networks. In International Conference on Learning Representations, 2023.
  4. A theory of learning from different domains. Machine learning, 2010.
  5. Language models are few-shot learners. Advances in neural information processing systems, 2020.
  6. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 2016.
  7. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics, 2018.
  8. Adaptive methods for real-world domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  9. Adversarially adaptive normalization for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  10. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, 2015.
  11. Domain-adversarial training of neural networks. The journal of machine learning research, 2016.
  12. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021.
  13. Learn to expect the unexpected: Probably approximately correct domain generalization. In International Conference on Artificial Intelligence and Statistics, 2021.
  14. Geodesic flow kernel for unsupervised domain adaptation. In IEEE conference on computer vision and pattern recognition, 2012.
  15. Ppt: Pre-trained prompt tuning for few-shot learning. In Proceedings of Annual Meeting of the Association for Computational Linguistic, 2022.
  16. Bertese: Learning to speak to bert. Conference of the European Chapter of the Association for Computational Linguistics, 2021.
  17. Continuous manifold based adaptation for evolving visual domains. In Conference on Computer Vision and Pattern Recognition, 2014.
  18. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
  19. Domain and content adaptive convolution for domain generalization in medical image segmentation. IEEE Transactions on Medical Imaging, 2023.
  20. How can we know what language models know? Transactions of the Association for Computational Linguistics, 2020.
  21. Cdot: Continuous domain adaptation using optimal transport. Optimal Transport & Machine Learning Workhop (NeurIPS), 2019.
  22. Domain randomization for scene-specific car detection and pose estimation. In Winter Conference on Applications of Computer Vision, 2019.
  23. Self-balanced learning for domain generalization. In International Conference on Image Processing, 2021.
  24. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  25. Continuous domain adaptation with variational domain-agnostic feature replay. arXiv preprint arXiv:2003.04382, 2020.
  26. The power of scale for parameter-efficient prompt tuning. CoRR, 2021.
  27. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R. (eds.), Proceedings of the nnual Meeting of the Association for Computational Linguistics, 2020.
  28. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021.
  29. A unified feature disentangler for multi-domain image translation and manipulation. Advances in neural information processing systems, 2018.
  30. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 2023a.
  31. Gpt understands, too. AI Open, 2023b.
  32. Oscillation and chaos in physiological control systems. Science, 1977.
  33. Adagraph: Unifying predictive and continuous domain adaptation through graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  34. Representation learning via invariant causal mechanisms. In International Conference on Learning Representations, 2021.
  35. Reducing domain gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  36. Training for the future: A simple gradient interpolation loss to generalize along time. Advances in Neural Information Processing Systems, 2021.
  37. Domain generalization using shape representation. In European Conference on Computer Vision, 2020.
  38. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, 2018.
  39. Adapterfusion: Non-destructive task composition for transfer learning. Annual Meeting of the Association for Computational Linguistics, 2021.
  40. Structured domain randomization: Bridging the reality gap by context-aware synthetic data. In International Conference on Robotics and Automation, 2019.
  41. Unsupervised domain generalization for person re-identification: A domain-specific adaptive framework. arXiv preprint arXiv:2111.15077, 2021.
  42. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  43. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
  44. Fishr: Invariant gradient variances for out-of-distribution generalization. International Conference on Machine Learning, 2022.
  45. How many data points is a prompt worth? In Conference of the North American Chapter of the Association for Computational Linguistics, 2021.
  46. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, 2021.
  47. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. International Conference on Learning Representations, 2023.
  48. Generalizing across domains via cross-gradient training. International Conference on Learning Representations, 2018.
  49. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing, 2020.
  50. On transferability of prompt tuning for natural language processing. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
  51. Conditioned natural language generation using only unconditioned language model: An exploration. arXiv preprint arXiv:2011.07347, 2020.
  52. Recovering latent causal factor for generalization to distributional shifts. In Conference on Neural Information Processing Systems, 2021.
  53. Neuron coverage-guided domain generalization. The IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  54. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018.
  55. Learning a universal template for few-shot dataset generalization. In International conference on machine learning, 2021.
  56. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
  57. Generalizing to unseen domains via adversarial data augmentation. Conference on Neural Information Processing Systems, 2018.
  58. Spot: Better frozen model adaptation through soft prompt transfer. Association for Computational Linguistics, 2021.
  59. Meta-learning for domain generalization in semantic parsing. In North American Chapter of the Association for Computational Linguistics, 2021a.
  60. TransPrompt: Towards an automatic transferable prompting framework for few-shot text classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021b.
  61. Continuously indexed domain adaptation. International Conference on Machine Learning, 2020a.
  62. Generalizing to unseen domains: A survey on domain generalization. arXiv preprint arXiv:2103.03097, 2021c.
  63. Deep visual domain adaptation: A survey. Neurocomputing, 2018.
  64. K-adapter: Infusing knowledge into pre-trained models with adapters. Association for Computational Linguistics, 2020b.
  65. Multitask prompt tuning enables parameter-efficient transfer learning. In International Conference on Learning Representations, 2023.
  66. Collaborative optimization and aggregation for decentralized domain generalization and adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  67. Multivariate regression on the grassmannian for predicting novel domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  68. Spatial-temporal change trend analysis of second-hand house price in hefei based on spatial network. Computational Intelligence and Neuroscience, 2022.
  69. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In International Conference on Computer Vision, 2019.
  70. Learning to generalize unseen domains via memory-based multi-source meta-learning for person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  71. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of Association for the Advancement of Artificial Intelligence, 2021.
  72. Deep domain-adversarial image generation for domain generalisation. In Proceddings of Association for the Advancement of Artificial Intelligence, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now