Towards a cloud native Big Data platform using MiCADO

Mosa, Abdelkhalik and Kiss, Tamas and Pierantoni, Gabriele and DesLauriers, James and Kagialis, Dimitrios and Terstyanszky, Gabor; (2020) Towards a cloud native Big Data platform using MiCADO. In: 2020 19th International Symposium on Parallel and Distributed Computing (ISPDC). IEEE, Piscataway, NJ, pp. 118-125. ISBN 9781728189468 (https://doi.org/10.1109/ISPDC51135.2020.00025)

[thumbnail of Mosa-etal-ISPDC-2020-Towards-a-cloud-native-Big-Data-platform-using-MiCADO]
Preview
 Text. Filename: Mosa_etal_ISPDC_2020_Towards_a_cloud_native_Big_Data_platform_using_MiCADO.pdf
Accepted Author Manuscript
License: Strathprints license 1.0
Download (1MB)| Preview

Abstract

In the big data era, creating self-managing scalable platforms for running big data applications is a fundamental task. Such self-managing and self-healing platforms involve a proper reaction to hardware (e.g., cluster nodes) and software (e.g., big data tools) failures, besides a dynamic resizing of the allocated resources based on overload and underload situations and scaling policies. The distributed and stateful nature of big data platforms (e.g., Hadoop-based cluster) makes the management of these platforms a challenging task. This paper aims to design and implement a scalable cloud native Hadoopbased big data platform using MiCADO, an open-source, and a highly customisable multi-cloud orchestration and auto-scaling framework for Docker containers, orchestrated by Kubernetes. The proposed MiCADO-based big data platform automates the deployment and enables an automatic horizontal scaling (in and out) of the underlying cloud infrastructure. The empirical evaluation of the MiCADO-based big data platform demonstrates how easy, efficient, and fast it is to deploy and undeploy Hadoop clusters of different sizes. Additionally, it shows how the platform can automatically be scaled based on user-defined policies (such as CPU-based scaling).

  • Item type:Book Section
    ID code:82380
    Dates:
    Date
    Event
    22 September 2020
    Published
    8 July 2020
    Published Online
    2 June 2020
    Accepted
    Notes:© 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
    Subjects:Technology > Electrical engineering. Electronics Nuclear engineering
    Department:UNSPECIFIED
    Depositing user: Pure Administrator
    Date deposited:21 Sep 2022 08:51
    Last modified:11 Jul 2024 01:38
    URI:https://strathprints.strath.ac.uk/id/eprint/82380
CORE (COnnecting REpositories)