Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 80 tok/s
Gemini 2.5 Pro 28 tok/s Pro
GPT-5 Medium 32 tok/s Pro
GPT-5 High 38 tok/s Pro
GPT-4o 125 tok/s Pro
Kimi K2 181 tok/s Pro
GPT OSS 120B 462 tok/s Pro
Claude Sonnet 4.5 35 tok/s Pro
2000 character limit reached

SlowFast Network for Continuous Sign Language Recognition (2309.12304v1)

Published 21 Sep 2023 in cs.CV

Abstract: The objective of this work is the effective extraction of spatial and dynamic features for Continuous Sign Language Recognition (CSLR). To accomplish this, we utilise a two-pathway SlowFast network, where each pathway operates at distinct temporal resolutions to separately capture spatial (hand shapes, facial expressions) and dynamic (movements) information. In addition, we introduce two distinct feature fusion methods, carefully designed for the characteristics of CSLR: (1) Bi-directional Feature Fusion (BFF), which facilitates the transfer of dynamic semantics into spatial semantics and vice versa; and (2) Pathway Feature Enhancement (PFE), which enriches dynamic and spatial representations through auxiliary subnetworks, while avoiding the need for extra inference time. As a result, our model further strengthens spatial and dynamic representations in parallel. We demonstrate that the proposed framework outperforms the current state-of-the-art performance on popular CSLR datasets, including PHOENIX14, PHOENIX14-T, and CSL-Daily.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. “Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent cnn-hmms,” in Proc. CVPR, 2017, pp. 4297–4305.
  2. “Stochastic fine-grained labeling of multi-state sign glosses for continuous sign language recognition,” in Proc. ECCV, 2020, pp. 172–186.
  3. “Fully convolutional networks for continuous sign language recognition,” in Proc. ECCV, 2020, pp. 697–714.
  4. “A deep neural framework for continuous sign language recognition by iterative training,” IEEE Trans. on Multimedia, vol. 21, no. 7, pp. 1880–1891, 2019.
  5. “Spatial-temporal multi-cue network for continuous sign language recognition.,” in Proc. AAAI, 2020, pp. 13009–13016.
  6. “C2slr: Consistency-enhanced continuous sign language recognition,” in Proc. CVPR, 2022, pp. 5131–5140.
  7. “Dilated convolutional network with iterative optimization for continuous sign language recognition.,” in Proc. IJCAI, 2018, vol. 3, p. 7.
  8. “Iterative alignment network for continuous sign language recognition,” in Proc. CVPR, 2019, pp. 4165–4174.
  9. “Dynamic pseudo label decoding for continuous sign language recognition,” in Proc. ICME, 2019, pp. 1282–1287.
  10. “Ksl-guide: A large-scale korean sign language dataset including interrogative sentences for guiding the deaf and hard-of-hearing,” in 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). IEEE, 2021, pp. 1–8.
  11. “Learning spatiotemporal features with 3d convolutional networks,” in Proc. ICCV, 2015, pp. 4489–4497.
  12. “Quo vadis, action recognition? a new model and the kinetics dataset,” in Proc. CVPR, 2017, pp. 6299–6308.
  13. “Continuous sign language recognition with correlation network,” in Proc. CVPR, 2023, pp. 2529–2539.
  14. “Slowfast networks for video recognition,” in Proc. ICCV, 2019, pp. 6202–6211.
  15. “Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers,” Computer Vision and Image Understanding, vol. 141, pp. 108–125, 2015.
  16. “Neural sign language translation,” in Proc. CVPR, 2018, pp. 7784–7793.
  17. “Improving sign language translation with monolingual data by sign back-translation,” in Proc. CVPR, 2021, pp. 1316–1325.
  18. “Connectionist temporal modeling for weakly supervised action labeling,” in Proc. ECCV, 2016, pp. 137–153.
  19. “Visual alignment constraint for continuous sign language recognition,” in Proc. ICCV, October 2021, pp. 11542–11551.
  20. “Sign language transformers: Joint end-to-end sign language recognition and translation,” in Proc. CVPR, 2020, pp. 10023–10033.
  21. “Local context-aware self-attention for continuous sign language recognition,” in Proc. Interspeech, 2022, pp. 4810–4814.
  22. “Boosting continuous sign language recognition via cross modality augmentation,” in Proc. ACM MM, 2020, pp. 1497–1505.
  23. “A simple multi-modality transfer learning baseline for sign language translation,” in Proc. CVPR, 2022, pp. 5120–5130.
  24. “Self-mutual distillation learning for continuous sign language recognition,” in Proc. ICCV, October 2021, pp. 11303–11312.
  25. “Self-emphasizing network for continuous sign language recognition,” in Proc. AAAI, 2023, vol. 37, pp. 854–862.
  26. “Self-sufficient framework for continuous sign language recognition,” in Proc. ICASSP. IEEE, 2023, pp. 1–5.
  27. “Temporal lift pooling for continuous sign language recognition,” in Proc. ECCV. Springer, 2022, pp. 511–527.
  28. “Signing outside the studio: Benchmarking background robustness for continuous sign language recognition,” in Proc. BMVC., 2022.
  29. “Two-stream network for sign language recognition and translation,” in Proc. NeurIPS, 2022, vol. 35, pp. 17043–17056.
  30. “A short note about kinetics-600,” arXiv preprint arXiv:1808.01340, 2018.
  31. “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015.
  32. “Video-based sign language recognition without temporal segmentation,” in Proc. AAAI, 2018, vol. 32.
  33. “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proc. ICCV, 2017, pp. 618–626.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up