Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards End-User Development for IoT: A Case Study on Semantic Parsing of Cooking Recipes for Programming Kitchen Devices (2309.14165v1)

Published 25 Sep 2023 in cs.CL

Abstract: Semantic parsing of user-generated instructional text, in the way of enabling end-users to program the Internet of Things (IoT), is an underexplored area. In this study, we provide a unique annotated corpus which aims to support the transformation of cooking recipe instructions to machine-understandable commands for IoT devices in the kitchen. Each of these commands is a tuple capturing the semantics of an instruction involving a kitchen device in terms of "What", "Where", "Why" and "How". Based on this corpus, we developed machine learning-based sequence labelling methods, namely conditional random fields (CRF) and a neural network model, in order to parse recipe instructions and extract our tuples of interest from them. Our results show that while it is feasible to train semantic parsers based on our annotations, most natural-language instructions are incomplete, and thus transforming them into formal meaning representation, is not straightforward.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Lexical event ordering with an edge-factored model. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1161–1171, Denver, Colorado. Association for Computational Linguistics.
  2. Semantic parsing as machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 47–52, Sofia, Bulgaria. Association for Computational Linguistics.
  3. Dario Bonino and Fulvio Corno. 2008. Dogont - ontology modeling for intelligent domotic environments. In The Semantic Web - ISWC 2008, pages 790–803, Berlin, Heidelberg. Springer Berlin Heidelberg.
  4. Xiang ‘Anthony’ Chen and Yang Li. 2017. Improv: An Input Framework for Improvising Cross-Device Interaction by Demonstration. ACM Trans. Comput.-Hum. Interact., 24(2):15:1–15:21.
  5. An end-user programming paradigm for pervasive computing applications. In 2006 ACS/IEEE International Conference on Pervasive Services, pages 325–328.
  6. Is smart home a necessity or a fantasy for the mainstream user? a study on users’ expectations of smart household appliances. International Journal of Design, 12(1):7–20.
  7. Joëlle Coutaz and James L. Crowley. 2016. A first-person experience with end-user development for smart homes. IEEE Pervasive Computing, 15(2):26–39.
  8. NeuroNER: an easy-to-use program for named-entity recognition based on neural networks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 97–102, Copenhagen, Denmark. Association for Computational Linguistics.
  9. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association (JAMIA).
  10. Gerhard Fischer and Elisa Giaccardi. 2006. Meta-design: A Framework for the Future of End-User Development, pages 427–457. Springer Netherlands, Dordrecht.
  11. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
  12. Jermsak Jermsurawong and Nizar Habash. 2015. Predicting the structure of cooking recipes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 781–786, Lisbon, Portugal. Association for Computational Linguistics.
  13. Nate Kushman and Regina Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 826–836, Atlanta, Georgia. Association for Computational Linguistics.
  14. InterPlay: a middleware for seamless device integration and task orchestration in a networked home. In Fourth Annual IEEE International Conference on Pervasive Computing and Communications (PERCOM’06), pages 10 pp.–307.
  15. Semiotics: Semantically enhanced iot-enabled intelligent control systems. IEEE Internet of Things Journal, 6(1):1257–1266.
  16. George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41.
  17. Mohammad Hasanzadeh Mofrad and Daniel Mosse. 2018. Speech Recognition and Voice Separation for the Internet of Things. In Proceedings of the 8th International Conference on the Internet of Things, IOT ’18, pages 8:1–8:8, New York, NY, USA. ACM.
  18. doccano: Text annotation tool for human. Software available from https://github.com/chakki-works/doccano.
  19. “kognichef”: A cognitive cooking assistant. KI - Künstliche Intelligenz, 31:273–281.
  20. Describing human-object interaction in intelligent space. In Proceedings of the 2Nd Conference on Human System Interactions, HSI’09, pages 392–396, Piscataway, NJ, USA. IEEE Press.
  21. Naoaki Okazaki. 2007. Crfsuite: a fast implementation of conditional random fields (crfs).
  22. Fabio Paternò and Carmen Santoro. 2017a. A Design Space for End User Development in the Time of the Internet of Things, pages 43–59. Springer International Publishing, Cham.
  23. Fabio Paternò and Carmen Santoro. 2017b. A Design Space for End User Development in the Time of the Internet of Things, pages 43–59. Springer International Publishing, Cham.
  24. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830.
  25. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  26. Natural Notation for the Domestic Internet of Things. In End-User Development, pages 25–41, Cham. Springer International Publishing.
  27. Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora.
  28. Mampf: An intelligent cooking agent for zoneless stoves. In 2011 Seventh International Conference on Intelligent Environments, pages 171–178.
  29. Yutaka Sasaki. 2007. The truth of the f-measure. Teach Tutor Mater.
  30. Albrecht Schmidt. 2015. Programming ubiquitous computing environments. In End-User Development, pages 3–6, Cham. Springer International Publishing.
  31. Michael Schneider. 2007. The semantic cookbook: sharing cooking experiences in the smart kitchen. In 2007 3rd IET International Conference on Intelligent Environments, pages 416–423.
  32. Sajjad Hussain Shah and Ilyas Yaqoob. 2016. A survey: Internet of things (iot) technologies, applications and challenges. In 2016 IEEE Smart Energy Grid Engineering (SEGE), pages 381–385.
  33. A crowdsourced database of event sequence descriptions for the acquisition of high-quality script knowledge. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3494–3501, Portorož, Slovenia. European Language Resources Association (ELRA).
  34. RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1358–1368, Brussels, Belgium. Association for Computational Linguistics.
  35. Kristina Yordanova. 2015. Discovering causal relations in textual instructions. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 714–720, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.
  36. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.
  37. 4w1h in mobile crowd sensing. IEEE Communications Magazine, 52(8):42–48.
  38. Automatically extracting procedural knowledge from instructional texts using natural language processing. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 520–527, Istanbul, Turkey. European Languages Resources Association (ELRA).

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now