AI Driven Enhancement of ETL Workflows for Scalable and Efficient Cloud Data Engineering
This article explores the role of AI in enhancing Extract, Transform, Load (ETL) workflows to improve scalability, efficiency, and performance in cloud data engineering. Traditional ETL processes face challenges such as high latency, resource inefficiencies, and complex transformations. AI-driven optimizations, including intelligent workload management, automated schema evolution, and anomaly detection, are revolutionizing data pipeline efficiency. This article delves into key AI-driven enhancements, implementation strategies, and real-world use cases demonstrating improved data processing and operational efficiency.
Bathani, Ronakkumar. "Data Engineering for AI in Healthcare From ETL to Advanced Analytics on Cloud Platforms for Smart Education." In Smart Education and Sustainable Learning Environments in Smart Cities, pp. 173-190. IGI Global Scientific Publishing, 2025.
Tadi, Venkata. "Revolutionizing Data Integration: The Impact of AI and Real-Time Technologies on Modern Data Engineering Efficiency and Effectiveness."
Selvarajan, Guru Prasad. "Leveraging SnowflakeDB in Cloud Environments: Optimizing AI-driven Data Processing for Scalable and Intelligent Analytics." International Journal of Enhanced Research in Science, Technology & Engineering 11, no. 11 (2022): 257-264.
Galla, Eswar Prasad, Chandrababu Kuraku, Hemanth Kumar Gollangi, Janardhana Rao Sunkara, and Chandrakanth Rao Madhavaram. AI-DRIVEN DATA ENGINEERING TRANSFORMING BIG DATA INTO ACTIONABLE INSIGHT. JEC PUBLICATION.
Badgujar, Pooja. "Optimizing ETL Processes for Large-Scale Data Warehouses." Journal of Technological Innovations 2, no. 4 (2021).
Van der Putten, Chiara. "Transforming data flow: Generative AI in ETL pipeline automatization." PhD diss., Politecnico di Torino, 2024.
Pothineni, Balakrishna, Durgaraman Maruthavanan, Ashok Gadi Parthi, Deepak Jayabalan, and Prema kumar Veerapaneni. "Enhancing Data Integration and ETL Processes Using AWS Glue." International Journal of Research and Analytical Reviews 11 (2024): 728-33.
Joshi, Nikhil. "Optimizing Real-Time ETL Pipelines Using Machine Learning Techniques." Available at SSRN 5054767 (2024).
Maxwell, Mickael, and Albert Gilbert. "Enhancing Decision-Making with Reverse ETL and Data Streaming in Multi-Cloud Environments." (2024).
Uddin, Md Kazi Shahab, and Kazi Md Riaz Hossan. "A Review of Implementing AI-Powered Data Warehouse Solutions to Optimize Big Data Management and Utilization." Academic Journal on Business Administration, Innovation & Sustainability 4, no. 3 (2024): 10-69593.
Katari, Abhilash, and Anjali Rodwal. "NEXT-GENERATION ETL IN FINTECH: LEVERAGING AI AND ML FOR INTELLIGENT DATA TRANSFORMATION."
Paul, Charles. "Optimizing Data Pipelines with Advanced ETL Automation Techniques." (2022).
Naveen, Kumar KR, V. Priya, Rachana G. Sunkad, and N. Pradeep. "An overview of cloud computing for data-driven intelligent systems with AI services." Data-Driven Systems and Intelligent Applications (2024): 72-118.
Zahra, Fatima tu, Yavuz Selim Bostanci, Ozay Tokgozlu, Malik Turkoglu, and Mujdat Soyturk. "Big Data Streaming and Data Analytics Infrastructure for Efficient AI-Based Processing." In Recent Advances in Microelectronics Reliability: Contributions from the European ECSEL JU project iRel40, pp. 213-249. Cham: Springer International Publishing, 2024.
Copyright (c) 2024 International Journal of Engineering and Computer Science
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.