Evaluating hardware differences for crowdsourcing and traditional recruiting methods
Abstract: The most frequently used method to collect research data online is crowdsouring and its use continues to grow rapidly. This report investigates for the first time whether researchers also have to expect significantly different hardware performance when deploying to Amazon Mechanical Turk (MTurk). This is assessed by collecting basic hardware parameters (Operating System, GPU, and used browser) from Amazon Mechanical Turk (MTurk) and a traditional recruitment method (i.e., snowballing). The significant hardware differences between crowdsourcing participants (MTurk) and snowball recruiting are reported including relevant descriptive statistics for assessing hardware performance of 3D web applications. The report suggests that hardware differences need to be considered to obtain valid results if the designed experiment application requires graphical intense computations and relies on a coherent user experience of MTurk and more established recruitment strategies (i.e. snowballing).
- H. Aguinis, I. Villamor, and R. S. Ramani, “Mturk research: Review and recommendations,” Journal of Management, vol. 47, no. 4, pp. 823–837, 2021.
- E. Estellés-Arolas and F. González-Ladrón-de Guevara, “Towards an integrated crowdsourcing definition,” Journal of Information science, vol. 38, no. 2, pp. 189–200, 2012.
- J. Y. Chen and J. E. Thropp, “Review of low frame rate effects on human performance,” IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, vol. 37, no. 6, pp. 1063–1076, 2007.
- Userbenchmark.com. (2022) Userbenchmark - effective fps methodology. https://www.userbenchmark.com/Faq/What-are-effective-frames-per-second-EFps/112 [last accessed: 2022-07-05].
- G. Paolacci, J. Chandler, and P. G. Ipeirotis, “Running experiments on amazon mechanical turk,” Judgment and Decision making, vol. 5, no. 5, pp. 411–419, 2010.
- L. Mohan, P. Raman, V. Choppella, and Y. R. Reddy, “A crowdsourcing approach for quality enhancement of elearning systems.” in ISEC, 2017, pp. 188–194.
- E. Dolstra, R. Vliegendhart, and J. Pouwelse, “Crowdsourcing gui tests,” in 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. IEEE, 2013, pp. 332–341.
- E. Y. Kan, W. K. Chan, and T. Tse, “Crowdadaptor: A crowd sourcing approach toward adaptive energy-efficient configurations of virtual machines hosting mobile applications,” in 2014 IEEE 38th Annual Computer Software and Applications Conference. IEEE, 2014, pp. 493–502.
- A. Evans, M. Romeo, A. Bahrehmand, J. Agenjo, and J. Blat, “3d graphics on the web: A survey,” Computers & graphics, vol. 41, pp. 43–61, 2014.
- T. Glander, A. Moreno, M. Aristizabal, J. Congote, J. Posada, A. Garcia-Alonso, and O. Ruiz, “Reweb3d: Enabling desktop 3d applications to run in the web,” in Proceedings of the 18th International Conference on 3D Web Technology, 2013, pp. 147–155.
- M. Gries, “Methods for evaluating and covering the design space during early design development,” Integration, vol. 38, no. 2, pp. 131–183, 2004.
- J. v. Kistowski, J. A. Arnold, K. Huppler, K.-D. Lange, J. L. Henning, and P. Cao, “How to build a benchmark,” in Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, 2015, pp. 333–336.
- A. ElBanna and N. Abdelbaki, “Browsers fingerprinting motives, methods, and countermeasures,” in 2018 International Conference on Computer, Information and Telecommunication Systems (CITS). IEEE, 2018, pp. 1–5.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
