2000 character limit reached
QI2 -- an Interactive Tool for Data Quality Assurance (2307.03419v2)
Published 7 Jul 2023 in cs.CY, cs.AI, cs.DS, and cs.LG
Abstract: The importance of high data quality is increasing with the growing impact and distribution of ML systems and big data. Also the planned AI Act from the European commission defines challenging legal requirements for data quality especially for the market introduction of safety relevant ML systems. In this paper we introduce a novel approach that supports the data quality assurance process of multiple data quality aspects. This approach enables the verification of quantitative data quality requirements. The concept and benefits are introduced and explained on small example data sets. How the method is applied is demonstrated on the well known MNIST data set based an handwritten digits.
- OPTICS: Ordering Points to Identify the Clustering Structure. In Proceedings of the 1999 ACM SIGMOD International Conference on Management of Data, SIGMOD ’99, 49–60. New York, NY, USA: Association for Computing Machinery. ISBN 1-58113-084-8. Event-place: Philadelphia, Pennsylvania, USA.
- LOF: Identifying Density-Based Local Outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, SIGMOD ’00, 93–104. New York, NY, USA: Association for Computing Machinery. ISBN 1-58113-217-4. Event-place: Dallas, Texas, USA.
- Safety Assurance of Machine Learning for Perception Functions. In Fingscheidt, T.; Gottschalk, H.; and Houben, S., eds., Deep Neural Networks and Data for Automated Driving, 335–358. Cham: Springer International Publishing. ISBN 978-3-031-01232-7 978-3-031-01233-4.
- Deng, L. 2012. The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]. IEEE Signal Processing Magazine, 29(6): 141–142.
- A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD’96, 226–231. AAAI Press. Event-place: Portland, Oregon.
- European Comission. 2021. LAYING DOWN HARMONISED RULES ON ARTIFICIAL INTELLIGENCE (ARTIFICIAL INTELLIGENCE ACT) AND AMENDING CERTAIN UNION LEGISLATIVE ACTS.
- Outliers detection and classification in wireless sensor networks. Egyptian Informatics Journal, 14(2): 157–164.
- Geerkens, S. 2021. Anwendung und Validierung des SHLQI² auf realen Beispielmengen und neuronale Netzwerke.
- Data quality certification using ISO/IEC 25012: Industrial experiences. Journal of Systems and Software, 176: 110938.
- Assessing data quality – A probability-based metric for semantic consistency. Decision Support Systems, 110: 95–106.
- Holcomb, Z. 2016. Fundamentals of Descriptive Statistics. Routledge, 0 edition. ISBN 978-1-351-97033-4.
- pointblank: Data Validation and Organization of Metadata for Local and Remote Tables. Https://rich-iannone.github.io/pointblank/, https://github.com/rich-iannone/pointblank.
- Jolliffe, I. T. 1990. PRINCIPAL COMPONENT ANALYSIS: A BEGINNER’S GUIDE - I. Introduction and application. Weather, 45(10): 375–382.
- Visualizing Data using t-SNE. Journal of Machine Learning Research, 9: 2579–2605.
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. ArXiv:1802.03426 [cs, stat].
- An Integrated Approach to a Safety Argumentation for AI-Based Perception Functions in Automated Driving. In Habli, I.; Sujan, M.; Gerasimou, S.; Schoitsch, E.; and Bitsch, F., eds., Computer Safety, Reliability, and Security. SAFECOMP 2021 Workshops, volume 12853, 265–271. Cham: Springer International Publishing. ISBN 978-3-030-83905-5 978-3-030-83906-2. Series Title: Lecture Notes in Computer Science.
- Data Quality Assessment. Commun. ACM, 45(4): 211–218. Place: New York, NY, USA Publisher: Association for Computing Machinery.
- Image Similarity Using Mutual Information of Regions. In Kanade, T.; Kittler, J.; Kleinberg, J. M.; Mattern, F.; Mitchell, J. C.; Nierstrasz, O.; Pandu Rangan, C.; Steffen, B.; Sudan, M.; Terzopoulos, D.; Tygar, D.; Vardi, M. Y.; Weikum, G.; Pajdla, T.; and Matas, J., eds., Computer Vision - ECCV 2004, volume 3023, 596–607. Berlin, Heidelberg: Springer Berlin Heidelberg. ISBN 978-3-540-21982-8 978-3-540-24672-5. Series Title: Lecture Notes in Computer Science.
- Enhanced efficient outlier detection and classification approach for WSNs. Simulation Modelling Practice and Theory, 120: 102618.
- Complete outlier detection and classification framework for WSNs based on OPTICS. Journal of Network and Computer Applications, 211: 103563.
- Automating large-scale data quality verification. Proceedings of the VLDB Endowment, 11(12): 1781–1794.
- DEEQU - Data quality validation for machine learning pipelines. In NeurIPS 2018.
- Data quality: A survey of data quality dimensions. In 2012 International Conference on Information Retrieval & Knowledge Management, 300–304. Kuala Lumpur: IEEE. ISBN 978-1-4673-1091-8 978-1-4673-1090-1.
- Sieberichs, C. 2021. Anwendung und Validierung des ECS auf reale Beispielmengen und neuronale Netzwerke.
- ECS - an Interactive Tool for Data Quality Assurance.
- Siemens. 2022. safe.trAIn. https://safetrain-project.de. Accessed: 2023-01-15.
- Tran Manh Thang; and Juntae Kim. 2011. The Anomaly Detection by Using DBSCAN Clustering with Multiple Parameters. In 2011 International Conference on Information Science and Applications, 1–5. Jeju Island: IEEE. ISBN 978-1-4244-9222-0.
- Beyond Accuracy: What Data Quality Means to Data Consumers. Journal of Management Information Systems, 12(4): 5–33.
- Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 13(4): 600–612.
- Waschulzik, T. 1999. Qualitätsgesicherte effiziente Entwicklung vorwärtsgerichteter künstlicher Neuronaler Netze mit überwachtem Lernen (QUEEN). Ph.D. thesis, Technische Universität München, München.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.