Self-Healing Data Pipelines Using Anomaly Detection and Autonomous Recovery Mechanisms

Certificate: View Certificate

Published Paper PDF: View PDF

Confirmation Letter: View

DOI: https://doi.org/10.63345/ijrsml.v11.i10.1

Sarvesh Kumar Gupta
Consulting Member of Technical Staff
Oracle
Saint Peters, Missouri -63376 , USA
ORCID: 0009-0008-7460-4874

Abstract— Modern organizations increasingly depend on large-scale data pipelines to support analytics, machine learning, business intelligence, and real-time decision-making. However, the growing complexity of distributed data-processing environments makes pipelines vulnerable to failures caused by schema changes, data-quality issues, infrastructure outages, resource exhaustion, network disruptions, and workload fluctuations. Traditional monitoring approaches primarily rely on static rules and manual intervention, resulting in prolonged recovery times and reduced operational reliability. This paper presents an experimental evaluation of self-healing data pipelines that integrate anomaly detection and autonomous recovery mechanisms to improve resilience and availability. A distributed data-processing environment was developed using Apache Kafka, Apache Spark, and Apache Flink, and multiple fault-injection scenarios were introduced to simulate realistic operational failures. The study compares threshold-based monitoring, machine-learning-based anomaly detection, and a self-healing architecture combining intelligent anomaly detection with automated recovery actions. Experimental results demonstrate that advanced anomaly-detection models, particularly LSTM Autoencoders, achieve superior precision, recall, and F1-score compared with traditional monitoring approaches. Furthermore, autonomous recovery mechanisms, including checkpoint restoration, workflow rerouting, automatic retries, and dynamic resource scaling, significantly reduce recovery time and improve system availability. The self-healing architecture achieved the highest operational performance, reducing downtime while maintaining near-continuous service availability. The findings indicate that combining intelligent anomaly detection with autonomous recovery provides an effective strategy for enhancing reliability, fault tolerance, and scalability in modern data-engineering ecosystems. Future research directions involving AI-driven root-cause analysis and reinforcement-learning-based recovery optimization are also discussed.

Keywords— Self-Healing Data Pipelines, Anomaly Detection, Autonomous Recovery, Data Quality Monitoring, Fault Tolerance, Data Engineering, Workflow Automation, Intelligent Data Systems.

References

• Kephart, J. O., & Chess, D. M. (2003). The vision of autonomic computing. Computer, 36(1), 41–50.
• Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), 1–58.
• Gupta, M., Gao, J., Aggarwal, C. C., & Han, J. (2014). Outlier detection for temporal data: A survey. IEEE Transactions on Knowledge and Data Engineering, 26(9), 2250–2267.
• Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4), 1–37.
• Krishnan, S., Wang, J., Wu, E., Franklin, M. J., & Goldberg, K. (2016). ActiveClean: Interactive data cleaning for statistical modeling. Proceedings of the VLDB Endowment, 9(12), 948–959.
• Rekatsinas, T., Chu, X., Ilyas, I. F., & Ré, C. (2017). HoloClean: Holistic data repairs with probabilistic inference. Proceedings of the VLDB Endowment, 10(11), 1190–1201.
• Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017). The ML test score: A rubric for ML production readiness and technical debt reduction. IEEE Big Data.
• Schelter, S., Lange, D., Schmidt, P., Celikel, M., Biessmann, F., & Grafberger, A. (2018). Automating large-scale data quality verification. Proceedings of the VLDB Endowment, 11(12), 1781–1794.
• Breck, E., Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2019). Data validation for machine learning. Proceedings of Machine Learning and Systems.
• Schelter, S., Grafberger, A., Dunning, T., & others. (2019). Unit testing data with Deequ. Proceedings of the ACM SIGMOD International Conference on Management of Data.
• Breck, E., Zinkevich, M., Polyzotis, N., Whang, S. E., & Roy, S. (2020). Data analysis and validation in continuous ML pipelines. Proceedings of the ACM SIGMOD International Conference on Management of Data.
• Psaier, H., & Dustdar, S. (2011). A survey on self-healing systems: Approaches and systems. Computing, 91, 43–73.