Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Comparing Traditional and LLM-based Search for Image Geolocation (2401.10184v1)

Published 18 Jan 2024 in cs.IR and cs.HC

Abstract: Web search engines have long served as indispensable tools for information retrieval; user behavior and query formulation strategies have been well studied. The introduction of search engines powered by LLMs suggested more conversational search and new types of query strategies. In this paper, we compare traditional and LLM-based search for the task of image geolocation, i.e., determining the location where an image was captured. Our work examines user interactions, with a particular focus on query formulation strategies. In our study, 60 participants were assigned either traditional or LLM-based search engines as assistants for geolocation. Participants using traditional search more accurately predicted the location of the image compared to those using the LLM-based search. Distinct strategies emerged between users depending on the type of assistant. Participants using the LLM-based search issued longer, more natural language queries, but had shorter search sessions. When reformulating their search queries, traditional search participants tended to add more terms to their initial queries, whereas participants using the LLM-based search consistently rephrased their initial queries.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. How to Query Language Models? arXiv:2108.01928 [cs.CL]
  2. Anne Aula. 2003. Query Formulation in Web Information Search.. In ICWI. International Conference WWW/Internet, Algarve, Portugal, 403–410.
  3. The Linguistic Structure of English Web-Search Queries. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (Honolulu, Hawaii) (EMNLP ’08). Association for Computational Linguistics, USA, 1021–1030.
  4. Simon Bilel Jegham, dim fort. 2021. GeoGuess. MIT Licensed. https://github.com/GeoGuess/GeoGuess Accessed: 2023.
  5. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs.CL]
  6. Towards a Better Understanding of Query Reformulation Behavior in Web Search. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW ’21). Association for Computing Machinery, New York, NY, USA, 743–755. https://doi.org/10.1145/3442381.3450127
  7. Investigating query reformulation behavior of search users. In China Conference on Information Retrieval. Springer, China, 39–51.
  8. What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300705
  9. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, USA, 4171–4186. https://doi.org/10.18653/v1/n19-1423
  10. James J Gibson. 2014. The ecological approach to visual perception: classic edition. Psychology press, Online.
  11. Jutta Haider and Olof Sundin. 2019. Invisible search and online search engines: The ubiquity of search in everyday life. Taylor & Francis, London.
  12. James Hays and Alexei A Efros. 2008. Im2gps: estimating geographic information from a single image. In 2008 ieee conference on computer vision and pattern recognition. IEEE, CVPR, Alaska, USA, 1–8.
  13. Jeff Huang and Efthimis N. Efthimiadis. 2009. Analyzing and Evaluating Query Reformulation Strategies in Web Search Logs. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (Hong Kong, China) (CIKM ’09). Association for Computing Machinery, New York, NY, USA, 77–86. https://doi.org/10.1145/1645953.1645966
  14. Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23 Adjunct). Association for Computing Machinery, New York, NY, USA, Article 97, 3 pages. https://doi.org/10.1145/3586182.3615796
  15. CausalMapper: Challenging Designers to Think in Systems with Causal Maps and Large Language Model. In Proceedings of the 15th Conference on Creativity and Cognition (Virtual Event, USA) (C&C ’23). Association for Computing Machinery, New York, NY, USA, 325–329. https://doi.org/10.1145/3591196.3596818
  16. Angel Hsing-Chi Hwang and Andrea Stevenson Won. 2021. IdeaBot: investigating social facilitation in human-machine team creativity. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama,Japan, 1–16.
  17. Significant role of search engine in higher education. International Journal of Scientific & Engineering Research 2, 4 (2011), 1–5.
  18. Bernard J Jansen and Amanda Spink. 2006. How are we searching the World Wide Web? A comparison of nine search engine transaction logs. Information processing & management 42, 1 (2006), 248–263.
  19. PromptMaker: Prompt-Based Prototyping with Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 35, 8 pages. https://doi.org/10.1145/3491101.3503564
  20. Learning User Reformulation Behavior for Query Auto-Completion. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (Gold Coast, Queensland, Australia) (SIGIR ’14). Association for Computing Machinery, New York, NY, USA, 445–454. https://doi.org/10.1145/2600428.2609614
  21. How Can We Know What Language Models Know? arXiv:1911.12543 [cs.CL]
  22. Image Geolocation by Non-Expert Crowd Workers with an Expert Strategy. In 2022 IEEE International Conference on Big Data. IEEE xplore, Osaka, Japan, 4009–4013. https://doi.org/10.1109/BigData55660.2022.10020932
  23. Supporting Image Geolocation with Diagramming and Crowdsourcing. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 5, 1 (Sep. 2017), 98–107. https://doi.org/10.1609/hcomp.v5i1.13296
  24. Dirk Lewandowski. 2008. Search engine user behaviour: How can users be guided to quality content? Information Services & Use 28, 3-4 (2008), 261–268.
  25. Investigating queries and search failures in academic search. Information Processing & Management 53, 3 (2017), 666–683. https://doi.org/10.1016/j.ipm.2017.01.005
  26. Learning deep representations for ground-to-aerial geolocalization. In 2015 IEEE Conference on Computer Vision and Pattern Recognition. IEEE xplore, Boston, MA, USA, 5007–5015. https://doi.org/10.1109/CVPR.2015.7299135
  27. The exploration of objective task difficulty and domain knowledge effects on users’ query formulation. Proceedings of the Association for Information Science and Technology 53 (12 2016), 1–9. https://doi.org/10.1002/pra2.2016.14505301063
  28. Prompt Middleware: Mapping Prompts for Large Language Models to UI Affordances. arXiv:2307.01142 [cs.HC]
  29. An exploratory study of human performance in image geolocation tasks. In GroupSight Workshop on Human Computation for Image and Video Analysis, Vol. 308. HCOMP 2016, Austin, TX (USA), 3–4.
  30. Bo Pang and Ravi Kumar. 2011. Search in the lost sense of “query”: Question formulation in Web search queries and its temporal changes. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, USA, 135–140.
  31. Starting Conversations with Search Engines - Interfaces That Elicit Natural Language Queries. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval (Canberra ACT, Australia) (CHIIR ’21). Association for Computing Machinery, New York, NY, USA, 261–265. https://doi.org/10.1145/3406522.3446035
  32. How Context Affects Language Models’ Factual Predictions. arXiv:2005.04611 [cs.CL]
  33. Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, Vol. 5. McLean, VA, USA, 2–4.
  34. CROWD4EMS: A CROWDSOURCING PLATFORM FOR GATHERING AND GEOLOCATING SOCIAL MEDIA CONTENT IN DISASTER RESPONSE. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLII-3/W8 (08 2019), 331–340. https://doi.org/10.5194/isprs-archives-XLII-3-W8-331-2019
  35. Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). Association for Computing Machinery, New York, NY, USA, Article 314, 7 pages. https://doi.org/10.1145/3411763.3451760
  36. Use of Web search engines and personalisation in information searching for educational purposes. iRInformation Research.
  37. Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces. arXiv preprint arXiv:2309.14459 none, none (2023), 10 pages.
  38. TwitterSearch: A Comparison of Microblog Search and Web Search. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (Hong Kong, China) (WSDM ’11). Association for Computing Machinery, New York, NY, USA, 35–44. https://doi.org/10.1145/1935826.1935842
  39. David R Thomas. 2006. A general inductive approach for analyzing qualitative evaluation data. American journal of evaluation 27, 2 (2006), 237–246.
  40. Investigating searchers’ mental models to inform search explanations. ACM Transactions on Information Systems (TOIS) 38, 1 (2019), 1–25.
  41. GroundTruth: Augmenting expert image geolocation with crowdsourcing and shared representations. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–30.
  42. Search successes and failures in query segments and search tasks: A field study. Proceedings of the Association for Information Science and Technology 54, 1 (2017), 436–445.
  43. Emergent Abilities of Large Language Models. arXiv:2206.07682 [cs.CL]
  44. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., New Orleans, USA, 24824–24837. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
  45. PlaNet - Photo Geolocation with Convolutional Neural Networks. In Computer Vision – ECCV 2016. Springer International Publishing, Amsterdam, The Netherlands, 37–55. https://doi.org/10.1007/978-3-319-46484-8_3
  46. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 385, 22 pages. https://doi.org/10.1145/3491102.3517582
  47. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages. https://doi.org/10.1145/3544548.3581388
  48. Diagnose this if you can: On the effectiveness of search engines in finding medical self-diagnosis information. In Advances in Information Retrieval: 37th European Conference on IR Research, ECIR 2015, March 29-April 2, 2015. Proceedings 37. Springer, Vienna, Austria, 562–567.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Albatool Wazzan (2 papers)
  2. Stephen MacNeil (37 papers)
  3. Richard Souvenir (9 papers)
Citations (4)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets