Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
120 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Data Extraction, Transformation, and Loading Process Automation for Algorithmic Trading Machine Learning Modelling and Performance Optimization (2312.12774v1)

Published 20 Dec 2023 in cs.DC

Abstract: A data warehouse efficiently prepares data for effective and fast data analysis and modelling using machine learning algorithms. This paper discusses existing solutions for the Data Extraction, Transformation, and Loading (ETL) process and automation for algorithmic trading algorithms. Integrating the Data Warehouses and, in the future, the Data Lakes with the Machine Learning Algorithms gives enormous opportunities in research when performance and data processing time become critical non-functional requirements.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. S. Makridakis, F. Petropoulos, and Y. Kang, “Large language models: Their success and impact,” Forecasting, vol. 5, no. 3, pp. 536–549, 2023.
  2. A. Wong, S. Whang, E. Sagre, N. Sachin, G. Dutra, Y.-W. Lim, G. Hains, Y. Khmelevsky, and F. Zhang, “Short-term stock price forecasting using exogenous variables and machine learning algorithms,” arXiv preprint arXiv:2309.00618, 2023.
  3. S. Dhanjal, Y. Khmelevsky, M. Govorov, V. A. Ustymenko, and P. N. Sharma, “Security solutions for spatial data in storage - (Implementation case within oracle 9iAS),” 8th World Multi-Conference on Systemics, Cybernetics and Informatics, Vol Ii, Proceedings, pp. 318–323, 2004.
  4. M. Govorov, Y. Khmelevsky, V. Ustimenko, and A. Khorev, “Security for GIS N-tier architecture,” Developments in Spatial Data Handling, pp. 71–83, 2005.
  5. Y. Khmelevsky, “SW Development Projects in Academia,” WCCCE 2009 - Proceedings of the 14th Western Canadian Conference on Computing Education, vol. 1, no. 250, pp. 60–64, 2009. [Online]. Available: http://dl.acm.org/citation.cfm?id=1536292
  6. Y. Khmelevsky, M. Govorov, and L. Burge, “Okanagan College and Vancouver Island University educational joint projects results,” in Proceedings of the 14th Western Canadian Conference on Computing Education - WCCCE ’09, 2009, pp. 65–69. [Online]. Available: http://portal.acm.org/citation.cfm?id=1536274.1536293
  7. P. Sharma, M. Govorov, Y. Khmelevsky, and S. Dhanjal, “Oracle 9iAS Portal as a platform for Geographic Information Science distance and flexible learning at the University of the South Pacific,” WIT Transactions on Information and Communication Technologies, vol. 31, 2004.
  8. Y. Khmelevsky and V. Voytenko, “Cloud computing infrastructure prototype for university education and research,” in Computing.   ACM Press, 2010, pp. 1–5. [Online]. Available: https://doi.org/10.1145/1806512.1806524
  9. Y. Khmelevsky, “Research and Teaching Strategies Integration at Post-secondary Programs,” in Proceedings of the 16th Western Canadian Conference on Computing Education, ser. WCCCE ’11.   New York, NY, USA: ACM, 2011, pp. 57–60. [Online]. Available: http://doi.acm.org/10.1145/1989622.1989638
  10. Y. Khmelevsky, L. Burge, M. Govorov, and G. Hains, “Distance Learning Components in CS and GIS Courses,” in Proceedings of the 16th Western Canadian Conference on Computing Education, ser. WCCCE ’11.   New York, NY, USA: ACM, 2011, pp. 17–21. [Online]. Available: http://doi.acm.org/10.1145/1989622.1989627
  11. Y. Khmelevsky, V. Ustimenko, G. Hains, C. Kluka, E. Ozan, and D. Syrotovsky, “International collaboration in SW engineering research projects,” in Proceedings of the 16th Western Canadian Conference on Computing Education - WCCCE ’11, 2011.
  12. Y. Khmelevsky, G. G. Hains, and C. Li, “Automatic Code Generation Within Student’s Software Engineering Projects,” in WCCCE ’12, 2012, pp. 29–33.
  13. Y. Khmelevsky, V. Volodymyr, and D. Ph, “Strategies for Teaching Mobile Application Development,” 18th Western Canadian Conference on Computing Education, vol. 18, pp. 8–13, 2013.
  14. T. Alstad, J. R. Dunkin, R. Bartlett, A. Needham, G. Hains, and Y. Khmelevsky, “Minecraft computer game simulation and network performance analysis,” Second International Conferences on Computer Graphics, Visualization, Computer Vision, and Game Technology {(VisioGame 2014)}, 11 2014.
  15. Y. Khmelevsky and V. Voytenko, “Hybrid Cloud Computing Infrastructure in Academia.” in WCCCE 2015 - the 20th Western Canadian Conference on Computing Education, At May 8-9, 2015.   Vancouver Island University (VIU), Nanaimo, British Columbia, Canada., 2015.
  16. T. Alstad, J. R. Duncan, S. Detlor, B. French, H. Caswell, Z. Ouimet, R. Bartlett, A. Needham, Y. Khmelevsky, G. Hains, R. Bartlett, and A. Needham, “Minecraft computer game performance analysis and network traffic emulation by a custom bot,” in Proceedings of the 2015 Science and Information Conference, SAI 2015, 2015.
  17. T. Alstad, J. Riley Dunkin, S. Detlor, B. French, H. Caswell, Z. Ouimet, Y. Khmelevsky, G. Hains, J. R. Dunkin, S. Detlor, B. French, H. Caswell, Z. Ouimet, Y. Khmelevsky, J. Riley Dunkin, S. Detlor, B. French, H. Caswell, Z. Ouimet, Y. Khmelevsky, G. Hains, J. R. Dunkin, S. Detlor, B. French, H. Caswell, Z. Ouimet, and Y. Khmelevsky, “Game Network Traffic Emulation by a Custom Bot.” 2015 IEEE International Systems Conference (SysCon 2015) Proceedings, pp. 675–680, 4 2015.
  18. Y. Khmelevsky, “Ten Years of Capstone Projects at Okanagan College: A Retrospective Analysis,” in Proceedings of the 21st Western Canadian Conference on Computing Education.   New York, NY, USA: ACM, 2016, pp. 7:1–7:6. [Online]. Available: http://doi.acm.org/10.1145/2910925.2910949
  19. Y. Khmelevsky and V. Voytenko, “A New Paradigm for Teaching Mobile Application Development,” in Proceedings of the 21st Western Canadian Conference on Computing Education - WCCCE ’16, 2016.
  20. D. Atkinson, N. McDonald, and Y. Khmelevsky, “Reporting personal and corporate data for secure storage in cloud,” in 2016 IEEE International Conference on Cybercrime and Computer Forensic, ICCCF 2016, 2016.
  21. N. McDonald, D. Atkinson, Y. Khmelevsky, and S. McMillan, “Sport wearable biometric data encrypted emulation and storage in cloud,” in Canadian Conference on Electrical and Computer Engineering, 2016.
  22. Z. Ouimet, H. Caswell, Y. Khmelevsky, R. Bartlett, and A. Needham, “Game servers deployment automation case study,” in 2016 Annual IEEE Systems Conference (SysCon), 2016, pp. 1–7.
  23. N. McDonald, C. Frank, Y. Khmelevsky, R. Bartlett, and A. Needham, “GPN game users performance data gathering and analysis by a custom-built tool,” in Canadian Conference on Electrical and Computer Engineering, 2016.
  24. N. Mcdonald, D. Leader, C. K. Chiang, Y. Khmelevsky, R. Bartlett, and A. Needham, “A new online tool for gamer network performance analysis,” in 2016 IEEE International Conference on Cybercrime and Computer Forensic (ICCCF), 2016, pp. 1–6.
  25. N. Mcdonald, D. Atkinson, C. Frank, Y. Khmelevsky, and S. McMillan, “Biometric data emulation and encryption for sport wearable devices (A case study),” in 2016 Annual IEEE Systems Conference (SysCon), 2016, pp. 1–6.
  26. Y. Khmelevsky, H. Mahasneh, and G. J. D. R. Hains, “A stochastic gamer’s model for on-line games,” in 2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE).   IEEE, 2017, pp. 1–4.
  27. M. Cocar, R. Harris, and Y. Khmelevsky, “Utilizing Minecraft bots to optimize game server performance and deployment,” in Canadian Conference on Electrical and Computer Engineering, 2017.
  28. B. Ward, Y. Khmelevsky, G. Hains, R. Bartlett, A. Needham, and T. Sutherland, “Gaming network delays investigation and collection of very large-scale data sets,” in 11th Annual IEEE International Systems Conference, SysCon 2017 - Proceedings, 2017.
  29. Y. Khmelevsky, X. Li, and S. Madnick, “Software development using agile and scrum in distributed teams,” in 11th Annual IEEE International Systems Conference, SysCon 2017 - Proceedings, 2017.
  30. B. Amann, Y. Khmelevsky, and G. Hains, “State-of-the-art on query & transaction processing acceleration,” 2019.
  31. D. Joiner, M. Clement, S. T. Chan, K. Pereira, A. Wong, Y. Khmelevsky, J. Mahony, and M. Ferri, “DW vs OLTP Performance Optimization in the Cloud on PostgreSQL (A Case Study),” in 2022 IEEE International Conference on Recent Advances in Systems Science and Engineering (RASSE), 2022, pp. 1–8.
  32. “ The Digital Research Alliance of Canada. .” [Online]. Available: https://alliancecan.ca/en/about/alliance
  33. “ GitHub.com — Cloud based software development and version control .” [Online]. Available: https://github.com/
  34. “ Ionos.com — Web hosting plans for CA .” [Online]. Available: https://www.ionos.ca/hosting/web-hosting#plans
  35. “ Financial Modeling Prep — Stock Market Information .” [Online]. Available: https://site.financialmodelingprep.com
  36. “ HiDrive — Cloud storage by Ionos .” [Online]. Available: https://www.ionos.ca/office-solutions/hidrive-cloud-storage
  37. R. K. Dubey, “Algorithmic Trading: The Intelligent Trading Systems and Its Impact on Trade Size,” Expert Systems with Applications, vol. 202, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/abs/pii/S0957417422006479
  38. S. Martínez-Fernández, J. Bogner, X. Franch, M. Oriol, J. Siebert, A. Trendowicz, A. M. Vollmer, and S. Wagner, “Software Engineering for AI-Based Systems: A Survey,” ACM Transactions on Software Engineering and Methodology, vol. 31, no. 2, 4 2022.
  39. T. Théate and D. Ernst, “An application of deep reinforcement learning to algorithmic trading,” Expert Systems with Applications, vol. 173, p. 114632, 2021.
  40. E. M. Haryono, I. Gunawan, A. N. Hidayanto, U. Rahardja et al., “Comparison of the e-lt vs etl method in data warehouse implementation: A qualitative study,” in 2020 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS).   IEEE, 2020, pp. 115–120.
  41. A. A. Yulianto, “Extract transform load (etl) process in distributed database academic data warehouse,” APTIKOM Journal on Computer Science and Information Technologies, vol. 4, no. 2, pp. 61–68, 2019.
  42. O. Azeroual, G. Saake, and M. Abuosba, “Etl best practices for data quality checks in ris databases,” in Informatics, vol. 6, no. 1.   MDPI, 2019, p. 10.
  43. S. M. F. Ali, “Next-generation etl framework to address the challenges posed by big data.” in DOLAP, 2018.
  44. M. Patel and D. B. Patel, “Progressive growth of etl tools: A literature review of past to equip future,” Rising Threats in Expert Applications and Solutions: Proceedings of FICR-TEAS 2020, pp. 389–398, 2020.
  45. A. V. Michael and P. Ahirao, “Improved use of etl tool for updation and creation of data warehouse from different rdbms,” in Proceedings of the 3rd International Conference on Advances in Science & Technology (ICAST), 2020.
  46. N. Biswas, A. Sarkar, and K. C. Mondal, “Efficient incremental loading in etl processing for real-time data integration,” Innovations in Systems and Software Engineering, vol. 16, pp. 53–61, 2020.
  47. “ Github Repository — Algorithmic Trading Project .” [Online]. Available: https://github.com/youry/AlgorithmicTrading
  48. A. Wong, J. Figini, A. Raheem, G. Hains, Y. Khmelevsky, and P. Chu, “Forecasting of Stock Prices Using Machine Learning Models,” in IEEE International Systems Conference (SysCon) 2023, 2023.
  49. “ RoboForm — password manager and form filling solution .” [Online]. Available: https://www.roboform.com/
  50. “IONOS - Hosting Provider — Websites. Domains. Server.” [Online]. Available: https://www.ionos.com/
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets