Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Wizard of Oz Experimentation for Language Technology Applications: Challenges and Tools (2402.14563v1)

Published 22 Feb 2024 in cs.HC

Abstract: Wizard of OZ (WOZ) is a well-established method for simulating the functionality and user experience of future systems. Using a human wizard to mimic certain operations of a potential system is particularly useful in situations where extensive engineering effort would otherwise be needed to explore the design possibilities offered by such operations. The WOZ method has been widely used in connection with speech and language technologies, but advances in sensor technology and pattern recognition as well as new application areas such as human-robot interaction have made it increasingly relevant to the design of a wider range of interactive systems. In such cases achieving acceptable performance at the user interface level often hinges on resource intensive improvements such as domain tuning, which are better done once the overall design is relatively stable. While WOZ is recognised as a valuable prototyping technique, surprisingly little effort has been put into exploring it from a methodological point of view. Starting from a survey of the literature, this paper presents a systematic investigation and analysis of the design space for WOZ for language technology applications, and proposes a generic architecture for tool support that supports the integration of components for speech recognition and synthesis as well as for machine translation. This architecture is instantiated in WebWOZ - a new web-based open-source WOZ prototyping platform. The viability of generic support is explored empirically through a series of evaluations. Researchers from a variety of backgrounds were able to create experiments, independent of their previous experience with WOZ. The approach was further validated through a number of real experiments, which also helped to identify a number of possibilities for additional support, and flagged potential issues relating to consistency in Wizard performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (94)
  1. Utilizing mock-ups in the development of distributed information systems for semantic data federations. In Proceedings of CISIS, pages 307–312.
  2. Adapting paper prototyping for designing user interfaces for multiple display environments. Personal and Ubiquitous Computing, 12:269–277.
  3. Towards Automatic Evaluation of Multimodal User Interfaces. In Proceedings of IUI, pages 201–208.
  4. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies, 4(3):114–123.
  5. Translation by iterative collaboration between monolingual users. In Proceedings of GI, pages 39–46.
  6. A Wizard-of-Oz experiment for tutorial dialogues in mathematics. In Supplementary Proceedings of AIED (Vol. VIII: Advanced Technologies for Mathematics Education), pages 471–481.
  7. Olympus: An open-source framework for conversational spoken language interface research. In Proceedings of NAACL-HLT, pages 32–39.
  8. RavenClaw: Dialog management using hierarchical task decomposition and an expectation agenda. In Proceedings of Eurospeech 2003, Geneva, Switzerland.
  9. Sorry, I didn’t catch that! An investigation of non-understanding errors and recovery strategies. In Proceedings of 6th SIGdial Workshop on Discourse and Dialogue, pages 128–143.
  10. Wizard of Oz experiments and companion dialogues. In Proceedings of BCS-HCI, pages 117–123.
  11. Wizard of Oz experiments for companions. In Proceedings of BCS-HCI, pages 313–317.
  12. The role of beliefs in lexical alignment: evidence from dialogs with humans and computers. Cognition, 121(1):41–57.
  13. Syntactic alignment between computers and people: the role of belief about mental states. In Proceedings of the Twenty-fifth Annual Conference of the Cognitive Science Society, pages 186–191.
  14. Brooke, J. (1996). SUS - a quick and dirty usability scale. In Jordan, P. W., Thomas, B., Weerdmeester, B. A., and McClelland, L., editors, Usability Evaluation in Industry. Taylor and Francis.
  15. Buxton, W. (2007). Sketching user experiences: getting the design right and the right design. Morgan Kaufmann.
  16. Rapidly testing the interaction model of a pronunciation training system via Wizard-of-Oz. In Proceedings of LREC.
  17. Learning by doing with simulated intelligent help. Communications of the ACM, 31(9):1064–1079.
  18. A framework for rapid multimodal application design. In Proceedings of TSD, pages 393–403.
  19. AxsJAX: A talking translation bot using Google IM: Bringing web-2.0 applications to life. In Proceedings of W4A, pages 54–56.
  20. NEIMO, a multiworkstation usability lab for observing a multiworkstation usability and analyzing multimodal interaction. In Proceedings of CHI, pages 402–403.
  21. Investigating the impact of interlocutor voice on syntactic alignment in human-computer dialogue. In Proceedings of BCS HCI, pages 39–48.
  22. Wizard of Oz studies - why and how. In Proceedings of IUI, pages 193–200.
  23. Davis, J. (1998). Active help found beneficial in Wizard of Oz study. Information and Software Technology, 40:93–103.
  24. SketchWizard: Wizard of Oz prototyping of pen-based user interfaces. In Proceedings of UIST, pages 119–128.
  25. Visual interpretation of faces in the NEIMO multi-modal test-bed. In Proceedings of IJCAI. Workshop Looking at People: Recognition and Interpretation of Human Action.
  26. Assessing the effects of building social intelligence in a robotic interface for the home. Interacting with Computers, 17(5):522–541.
  27. Collaboration in translation: The impact of increased reach on cross-organisational work. Computer Supported Cooperative Work (CSCW).
  28. Wizard of Oz interfaces for mixed reality applications. In Proceedings of ACM CHI, pages 1339–1342.
  29. Wizard of Oz support throughout an iterative design process. IEEE Pervasive Computing, 4(4):18–26.
  30. Eliza meets the Wizard-of-Oz: Blending machine and human control of embodied characters. In Proceedings of CHI, pages 547–556.
  31. Laboratory vs. field experimentation in human factors: An evaluation of an experimental self-service airline ticket vendor. Human Factors, 13:521–531.
  32. Supporting progressive refinement of Wizard-of-Oz experiments. In Proceedings of the ITS, pages 62–69.
  33. An experimental evaluation of preferences for data entry method in automated telephone services. Behaviour & Information Technology, 17(2):82–92.
  34. Simulating speech systems. Computer Speech and Language, 5:81–99.
  35. Design of the VICO spoken dialogue system: evaluation of user expectations by Wizard-of-Oz experiments. In Proceedings of LREC.
  36. Navigational abilities in audial voice-controlled dialogue structures. Behaviour & Information Technology, 18(2):83–95.
  37. Building a user-derived interface. Communications of the ACM, 27(10):1032–1043.
  38. The 1984 Olympic Message System: a test of behavioral principles of system design. Communications of the ACM, 30(9):758–769.
  39. Composing letters with a simulated listening typewriter. Communications of the ACM, 26(4):295–308.
  40. Designing for usability: Key principles and what designers think. Communications of the ACM, 28(3):300–311.
  41. Hauptmann, A. G. (1989). Speech and gestures for graphic image manipulation. In Proceedings of CHI, pages 241–245.
  42. Justified advice: a semi-naturalistic study of advisory strategies. In Proceedings of CHI, pages 185–190.
  43. The impact of interface metaphor and context of use on the usability of a speech-based mobile city guide service. Behaviour & Information Technology, 24(1):67–78.
  44. WOZ Pro: A pen-based low fidelity prototyping environment to support Wizard of Oz studies. In Proceedings of CHI, pages 2453–2458.
  45. A Wizard-of-Oz environment to study referring expression generation in a situated spoken dialogue task. In Proceedings of ENLG, pages 94–97.
  46. Talking to a computer is not like talking to your best friend. In Proceedings of SCAI.
  47. Speech and Language Processing. Prentice Hall, second edition.
  48. Translation practice in the workplace: contextual analysis and implications for machine translation. Machine Translation, 25(1):35–52.
  49. A semi-automatic Wizard of Oz technique for Let’sFly spoken dialogue system. In Lecture Notes in Computer Science: Text, Speech and Dialogue. Springer.
  50. Kelley, J. F. (1983). An empirical methodology for writing user-friendly natural language computer applications. In Proceedings of CHI, pages 193–196.
  51. Kelley, J. F. (1984). An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems, 2(1):26–41.
  52. User interface design by sketching: A complexity analysis of widget representations. In Proceedings of EICS, pages 57–66.
  53. Creating corpora for speech-to-speech translation. In Proceedings of EUROSPEECH, pages 381–384.
  54. SUEDE: A Wizard of Oz prototyping tool for speech user interfaces. In Proceedings of UIST, pages 1–10.
  55. Krause, D. (1996). Using an interpretation system - some observations in hidden operator simulations of VERBMOBIL. In Proceedings of ECAI, pages 41–54.
  56. Lamel, L. (1998). Spoken language dialog system development and evaluation at LIMSI. In Proceedings of ISSD, pages 9–17.
  57. A Wizard of Oz study for an ar multimodal interface. In Proceedings of ICMI, pages 249–256.
  58. An interlingua based on domain actions for machine translation of task-oriented dialogues. In ICSLP, volume 98, pages 1155–1158.
  59. FrameWire: a tool for automatically extracting interaction logic from paper prototyping tests. In Proceedings of CHI, pages 503–512.
  60. Topiary: a tool for prototyping location-enhanced applications. In Proceedings of UIST, pages 217–226.
  61. BrickRoad: a light-weight tool for spontaneous design of location-enhanced applications. In Proceedings of CHI, pages 295–298.
  62. A Wizard-of-Oz interface to study information presentation strategies for spoken dialogue systems. In Proceedings of the 1st International Workshop on Spoken Dialogue Systems.
  63. Polonius: A Wizard of Oz interface for HRI experiments. In Proceedings of HRI, pages 197–198.
  64. The use of the pegs computational discourse framework as an interlingua representation. In AMTA SIG-IL First Workshop on Interlinguas (held at MT Summit VI), San Diego, California.
  65. Conducting a Wizard of Oz experiment on a ubiquitous computing system doorman. In Proceedings of IPNMD, pages 115–119.
  66. Effects of prompt style on user responses to an automated banking service using word-spotting. BT Technology Journal, 17(1):160–171.
  67. From vocal to multimodal dialogue management. In Proceedings of ICMI, pages 59–67.
  68. What if everyone could do it? a framework for easier spoken dialog system design. In Proceedings of ACM EICS.
  69. MDWOZ: A Wizard of Oz environment for dialog systems development. In Proceedings of LREC.
  70. Wizard of oz method for learning dialog agents. In Klusch, M. and Zambonelli, F., editors, Cooperative Information Agents V, volume 2182 of Lecture Notes in Computer Science, pages 20–25. Springer Berlin Heidelberg.
  71. Message oriented middleware for flexible Wizard of Oz experiments in HCI. Human-Computer Interaction. Design and Development Approaches, pages 121–130.
  72. Ozlab: a simple demonstration tool for prototyping interactivity. In Proceedings of NordiCHI, pages 295–296.
  73. Extending the Wizard of Oz methodology for language-enabled multimodal systems. In Proceedings of LREC, pages 2531–2536.
  74. Learning human multimodal dialogue strategies. Natural Language Engineering, 16(1):3.
  75. Evaluation of Emi interaction with non-disabled children in nursery school using Wizard of Oz technique. In Proceedings of IEEE Robio, pages 1147–1153.
  76. Applying the Wizard of Oz technique to the study of multimodal systems. In Proceedings of EWHCI, pages 219–230.
  77. A Wizard of Oz platform for the study of multimodal systems. In Proceedings of INTERACT and CHI, pages 95–96.
  78. Emotion recognition from speech using multi-classifier systems and RBF-ensembles. Studies in Computational Intelligence, 83:49–70.
  79. A flexible Wizard-of-Oz environment for rapid prototyping. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC), pages 958–961.
  80. Using Wizard of Oz to collect interaction data for voice controlled home care and communication services. In Proceedings of IASTED SPPRA, pages 511–518.
  81. WebWOZ: a Wizard of Oz prototyping framework. In Proceedings of the 2nd ACM SIGCHI symposium on Engineering interactive computing systems (EICS’10), pages 109–114. ACM.
  82. Speaker alignment in synthesised, machine translated communication. In International Workshop on Spoken Language Translation, pages 254–260. ISCA.
  83. Schneider, A. H. (2013). Intrinsic and Extrinsic Component Evaluation in Interactive Multilingual Speech Applications. PhD thesis, Trinity College, University of Dublin.
  84. Comparing intrinsic and extrinsic evaluation of MT output in a dialogue system. In Proceedings of IWSLT, pages 329–336.
  85. Séquin, C. H. (2005). Rapid prototyping: a 3d visualization tool takes on sculpture and mathematical forms. Communications of the ACM, 48(6):66–73.
  86. A Wizard of Oz component-based approach for rapidly prototyping and testing input multimodal interfaces. Journal of Multimodal User Interfaces, 3(3):215–225.
  87. Towards incremental speech generation in dialogue systems. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL ’10, pages 1–8, Stroudsburg, PA, USA. Association for Computational Linguistics.
  88. QuickWoZ: A multi-purpose Wizard-of-Oz framework for experiments with embodied conversational agents. In Proceedings of IUI, pages 427–428.
  89. User studies and the design of Natural Language Systems. In Proceedings of EACL, pages 116–123.
  90. Speech-to-speech translation services for the Olympic Games 2008. In Renals, S., Bengio, S., and Fiscus, J. G., editors, Proceedings of MLMI, pages 297–308.
  91. Universal speech tools: The CSLU toolkit.
  92. Jaspis - a framework for multilingual adaptive speech applications. In Proceedings of Sixth International Conference on Spoken Language Processing, ICSLP 2000 / INTERSPEECH 2000, pages 719–722.
  93. Spoken and multimodal bus timetable systems: Design, development and evaluation. In Proceedings of SPECOM, pages 389–392.
  94. DOMER: A Wizard of Oz interface for using interactive robots to scaffold social skills for children with autism spectrum disorders. In Proceedings of HRI, pages 279–280.
Citations (30)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets