Towards a cloud native Big Data platform using MiCADO

Mosa, Abdelkhalik and Kiss, Tamas and Pierantoni, Gabriele and DesLauriers, James and Kagialis, Dimitrios and Terstyanszky, Gabor; (2020) Towards a cloud native Big Data platform using MiCADO. In: 2020 19th International Symposium on Parallel and Distributed Computing (ISPDC). IEEE, Piscataway, NJ, pp. 118-125. ISBN 9781728189468 (

[thumbnail of Mosa-etal-ISPDC-2020-Towards-a-cloud-native-Big-Data-platform-using-MiCADO]
Text. Filename: Mosa_etal_ISPDC_2020_Towards_a_cloud_native_Big_Data_platform_using_MiCADO.pdf
Accepted Author Manuscript
License: Strathprints license 1.0

Download (1MB)| Preview


In the big data era, creating self-managing scalable platforms for running big data applications is a fundamental task. Such self-managing and self-healing platforms involve a proper reaction to hardware (e.g., cluster nodes) and software (e.g., big data tools) failures, besides a dynamic resizing of the allocated resources based on overload and underload situations and scaling policies. The distributed and stateful nature of big data platforms (e.g., Hadoop-based cluster) makes the management of these platforms a challenging task. This paper aims to design and implement a scalable cloud native Hadoopbased big data platform using MiCADO, an open-source, and a highly customisable multi-cloud orchestration and auto-scaling framework for Docker containers, orchestrated by Kubernetes. The proposed MiCADO-based big data platform automates the deployment and enables an automatic horizontal scaling (in and out) of the underlying cloud infrastructure. The empirical evaluation of the MiCADO-based big data platform demonstrates how easy, efficient, and fast it is to deploy and undeploy Hadoop clusters of different sizes. Additionally, it shows how the platform can automatically be scaled based on user-defined policies (such as CPU-based scaling).