Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DiffAnt: Diffusion Models for Action Anticipation (2311.15991v1)

Published 27 Nov 2023 in cs.CV

Abstract: Anticipating future actions is inherently uncertain. Given an observed video segment containing ongoing actions, multiple subsequent actions can plausibly follow. This uncertainty becomes even larger when predicting far into the future. However, the majority of existing action anticipation models adhere to a deterministic approach, neglecting to account for future uncertainties. In this work, we rethink action anticipation from a generative view, employing diffusion models to capture different possible future actions. In this framework, future actions are iteratively generated from standard Gaussian noise in the latent space, conditioned on the observed video, and subsequently transitioned into the action space. Extensive experiments on four benchmark datasets, i.e., Breakfast, 50Salads, EpicKitchens, and EGTEA Gaze+, are performed and the proposed method achieves superior or comparable results to state-of-the-art methods, showing the effectiveness of a generative approach for action anticipation. Our code and trained models will be published on GitHub.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Uncertainty-Aware Anticipation of Activities. In ICCV Workshop, 2019.
  2. Long-term anticipation of activities with cycle consistency. In GCPR, pages 159–173. Springer, 2020.
  3. SegDiff: Image segmentation with diffusion probabilistic models. arXiv preprint arXiv:2112.00390, 2021.
  4. Label-efficient semantic segmentation with diffusion models. ICLR, 2021.
  5. End-to-end object detection with transformers. In ECCV, 2020.
  6. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In CVPR, 2017.
  7. DiffusionDet: Diffusion model for object detection. arXiv preprint arXiv:2211.09788, 2022.
  8. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, pages 720–736, 2018.
  9. The epic-kitchens dataset: Collection, challenges and baselines. TPAMI, 43(11):4125–4141, 2020.
  10. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021.
  11. Temporal action segmentation: An analysis of modern technique. arXiv preprint arXiv:2210.10352, 2022.
  12. Ms-tcn: Multi-stage temporal convolutional network for action segmentation. In CVPR, pages 3575–3584, 2019.
  13. When will you do what? - Anticipating Temporal Occurrences of Activities. In CVPR, 2018.
  14. What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention. In ICCV, 2019.
  15. RED: Reinforced Encoder-Decoder Networks for Action Anticipation. In BMVC, 2017.
  16. Anticipative Video Transformer. In ICCV, 2021.
  17. Actionvlad: Learning spatio-temporal aggregation for action classification. In CVPR, pages 971–980, 2017.
  18. Future Transformer for Long-term Action Anticipation. In CVPR, 2022.
  19. Diffuseq: Sequence to sequence text generation with diffusion models. ICLR, 2023.
  20. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In CVPR, 2022.
  21. Vector quantized diffusion model for text-to-image synthesis. In CVPR, 2022.
  22. A-act: Action anticipation through cycle transformations. arXiv preprint arXiv:2204.00942, 2022.
  23. Denoising diffusion probabilistic models. In NeurIPS, pages 6840–6851, 2020.
  24. Diffusion models for video prediction and infilling. arXiv preprint arXiv:2206.07696, 2022.
  25. Timeception for complex action recognition. In CVPR, pages 254–263, 2019a.
  26. Videograph: Recognizing minutes-long human activities in videos. In ICCV Workshop, 2019b.
  27. Time-Conditioned Action Anticipation in One Shot. In CVPR, 2019.
  28. DiffusionClip: Text-guided image manipulation using diffusion models. In CVPR, 2022.
  29. Auto-encoding variational bayes. In ICLR, 2014.
  30. The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities. In CVPR, 2014.
  31. BDDM: Bilateral denoising diffusion models for fast and high-quality speech synthesis. arXiv preprint arXiv:2203.13508, 2022.
  32. Binauralgrad: A two-stage conditional diffusion probabilistic model for binaural audio synthesis. arXiv preprint arXiv:2205.14807, 2022.
  33. Diffusion-lm improves controllable text generation. NeurIPS, 35:4328–4343, 2022.
  34. In the eye of beholder: Joint learning of gaze and actions in first person video. In ECCV, pages 619–635, 2018.
  35. Diffusion action segmentation. arXiv preprint arXiv:2303.17959, 2023.
  36. Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970, 2022.
  37. Intention-conditioned long-term human egocentric action anticipation. In WACV, pages 6048–6057, 2023.
  38. A Variational Auto-Encoder Model for Stochastic Point Processes. In CVPR, 2019.
  39. Ego-topo: Environment affordances from egocentric video. In CVPR, pages 163–172, 2020.
  40. Rethinking learning approaches for long-term action anticipation. In ECCV, pages 558–576. Springer, 2022.
  41. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, ICML, 2021.
  42. Adversarial generative grammars for human activity prediction. In ECCV, pages 507–523. Springer, 2020.
  43. Weakly supervised action learning with rnn based fine-to-coarse modeling. In CVPR, pages 754–763, 2017.
  44. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  45. Temporal Aggregate Representations for Long-Range Video Understanding. In ECCV, 2020.
  46. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
  47. Denoising diffusion implicit models. In ICLR, 2021.
  48. Generative modeling by estimating gradients of the data distribution. NeurIPS, 2019.
  49. Improved techniques for training score-based generative models. NeurIPS, 2020.
  50. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
  51. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In UbiComp, pages 729–738, 2013.
  52. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021.
  53. Attention is All you Need. In NeurIPS, 2017.
  54. Masked conditional video diffusion for prediction, generation, and interpolation. arXiv preprint arXiv:2205.09853, 2022.
  55. Anticipating visual representations from unlabeled video. In CVPR, pages 98–106, 2016.
  56. Learning to schedule in diffusion probabilistic models. In KDD, 2023.
  57. Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203.09481, 2022.
  58. Asformer: Transformer for action segmentation. In BMVC, 2021.
  59. Latent diffusion energy-based model for interpretable text modeling. arXiv preprint arXiv:2206.05895, 2022.
  60. On diverse asynchronous activity anticipation. In ECCV, pages 781–799. Springer, 2020.
  61. A survey on deep learning techniques for action anticipation. arXiv preprint arXiv:2309.17257, 2023a.
  62. Anticipative feature fusion transformer for multi-modal action anticipation. In WACV, 2023b.
  63. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Zeyun Zhong (7 papers)
  2. Chengzhi Wu (15 papers)
  3. Manuel Martin (3 papers)
  4. Michael Voit (35 papers)
  5. Juergen Gall (121 papers)
  6. Jürgen Beyerer (40 papers)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now