Home Software and platforms Old Software Merkat

Merkat

Merkat

Merkat is a platform that allows users of an organization to automatically manage and scale their applications while maximizing the infrastructure’s utilization. Merkat is generic and extensible, allowing users to automate the application deployment and management process. Users have the flexibility to control how many resources are allocated to their applications and to define their own resource demand adaptation policies. Merkat applies an unique approach to multiplex the infrastructure capacity between the applications, by implementing a proportional-share market and allowing applications to adapt autonomously to resource price and their given performance objectives. The price of the acquired resources acts as a control mechanism to ensure that resources are distributed to applications according to the user’s value for them. Merkat was evaluated on Grid’5000 with several scientific applications.

The main features of Merkat are:

  • controlled software environments: By virtualizing resources, Merkat gives control to users over their software requirements. Merkat provides per-application isolation: each application runs in an environment customized to its needs.
  • autonomous SLO-driven application management: In Merkat applications can autonomously adapt their resource demands to meet their Service Level Objectives (SLOs). Each application runs in a controlled environment, called virtual platform. The resource management is decentralized: each virtual platform can take resource demand adaptation decisions independently from the others. Thus, Merkat becomes flexible and can support multiple application types and SLOs.
  • market-based resource regulation: To regulate user’s access to resources Merkat uses a market mechanism. This market relies on a proportional-share policy, providing efficient resource allocation: resources in terms of CPU and memory are allocated in a fine-grained manner to the provisioned VMs. The market’s currency distribution policy ensures that users receive limited budgets and value truthfully the execution of their applications. The combination of currency distribution and dynamic resource pricing ensures fair resource utilization.

. On top of the implemented proportional-share market, we provide two policies to scale the resource demand of the applications running on the infrastructure:

  • vertical scaling: a policy that adapts the application resource demand per VM to the current resource availability and resource price;
  •  horizontal scaling: a policy that adapts the application resource demand in terms of number of VMs to the current resource availability and resource price.

Using these mechanisms, users can scale their applications both horizontally and vertically in order to meet their SLOs (e.g., deadlines).

Project Members

Current Status

Merkat is implemented in Python and currently relies on three services: the application management system, the VM scheduler, and the virtual currency manager. To manage the infrastructure, these services interact with an IaaS Cloud Manager. The following figure gives a better overview of Merkat.

architecture
  • The Application Management System: The application management system is in charge of executing applications on the infrastructure. This component is composed of an applications manager and a set of independent entities, called virtual platforms. The applications manager acts as an entry point for running applications in our system. Each application runs in a virtual platform, composed of one or more virtual clusters, according to the application components, and managed by an application controller. Each virtual cluster can have its own virtual cluster monitor.
  • The VM Scheduler: The VM scheduler is in charge of allocating VMs on the infrastructure and limiting the resource consumption in terms of memory and CPU for every VM according to infrastructure load and VM bid. To regulate the resource consumption, the scheduler applies a periodical proportional-share policy. To ease application management, the scheduler exposes an API for virtual cluster management. The scheduler is designed to be generic enough to support different allocation policies. To ease the development of adaptation policies, the VM scheduler also provides additional information that helps the application controllers to take more informed decisions: (i) the current resource price; (ii) estimations regarding the cost of the virtual cluster if a certain allocation (i.e., number of VMs and resource amount required by each VM) is specified; (iii) and estimation of the allocation (i.e., number of VMs) obtained at the current bid.
  • The Virtual Currency Manager: The virtual currency manager manages the virtual currency and distributes it among users and their applications. Virtual currency is desirable in closed environments, i.e., private infrastructures shared by the users belonging to the same organization, as it provides incentives to users to request only as many resources as needed. Price inflation is bounded as there is no external currency introduced.
  • The IaaS Cloud Manager: The IaaS Cloud Manager manages the virtual machines and the physical infrastructure. It handles the user, VM image and network management as well as all the operations with VMs, e.g., start, migrate, stop.

Supported applications: We currently provide plugins to scale MPI applications, Torque and Condor frameworks and we are evaluating Merkat at EDF R&D using EDF high-performance applications.

Dependencies: Merkat depends on the following technologies:

  • OpenNebula (as an IaaS Cloud Manager)
  • ZeroMq
  • paramiko
  • libvirt, KVM and cgroups
  • MySql
  • Apache Zookeeper (soon)

Publications

Journals

  • Stefania Costache, Christine Morin; Nikos Parlavantzas. “Market-based Autonomous Resource and Application Management in the Cloud”. Under preparation.

International Conferences

  • Stefania Victoria Costache; Nikos Parlavantzas; Christine Morin; Samuel Kortas. “Merkat: A Market-based SLO-driven Cloud Platform”, 5th IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2013), December 2013, Bristol, United Kingdom.
  • Stefania Victoria Costache; Nikos Parlavantzas; Christine Morin; Samuel Kortas. “On the Use of a Proportional-Share Market for Application SLO Support in Clouds”, 19th International European Conference on Parallel and Distributed Computing (EuroPar 2013), August 2013, Aachen, Germany.
  • Stefania Victoria Costache; Nikos Parlavantzas; Christine Morin; Samuel Kortas.”Themis: Economy-Based Automatic Resource Scaling for Cloud Systems“, 14th IEEE International Conference on High Performance Computing and Communications (HPCC 2012), Jun 2012, Liverpool, United Kingdom.

Workshops

Posters

Comments are closed.