Efficient Resource Management techniques in Cloud Computing Environment: Review and discussion

Headnote

Abstract

The increasing number of cloud computing infrastructure and of the users' demands for services has made the resource management an impossible task to be manually performed by human operators. In this paper, we surveyed the state-of-the-art of resource management for infrastructure as a service in cloud computing. We provide an overview of the recent research findings and technologies by focusing on the resource management techniques such as resource provisioning, discovery, monitoring, mapping, allocation, consolidation, modeling, and scheduling. Then we putted forward the open challenge issues in the deployment of the virtual machines which may result into underutilized and wastage of resources or into a poor service delivery in cloud computing. This survey triggered innovative methods to handle the existing problems of resource management in cloud computing and we hope that it can be used as a source of interested readers to understand the existing methodologies in this research area for future enhancements.

Keywords: Cloud Computing, Resource management techniques, Resource provisioning, Resource monitoring, resource allocation, resource discovery

Copyright © 2017 Universitas Ahmad Dahlan. All rights reserved.

1.Introduction

The cloud computing infrastructure resources often sit in geographically dispersed data centers and the customers obtain services through network. A data center represents a physical area in which racks containing IT equipment such as disk enclosures, controller enclosure, servers, routers, switches, load balancers, firewalls, power delivery devices, cooling machines, unmanaged devices, and other security devices. The IT equipment are enclosed in racks and they could be dynamicaly managed by monitoring systems. The system monitors the equipment usage status and it can generate the alarms when it detects an abnormal usage which is greater than or equal to the specified alarm threshold. We use the figure 1 below to show the infrastructure resources in one data center where a rack contains a certain number of servers stacked one above the other. The rack server brings the advantage for easy cabling and saving floor space in data center. ToR (Top-of-Rack) is linked to AS (Aggregation Switch) for redundancy and provides connectivity to the servers mounted in a one rack. The aggregation switch in the aggregation layer forwards traffic from multiple ToR switches to the core layer. A core router in core layer is a router designed to operate in the Internet backbone and must support multiple communication interfaces of the fastest speed and must be able to forward IP packets at full speed on all of them.

Over the past decade, advances in commodity and virtualization technologies enabled the cost-effective realization of large-scale data centers that run large portion of today's internet applications and backend processing and allow data center infrastructure resources to be leased and released by third parties [1]. Since cloud computing resources are scattered across various places and its use continues to grow, companies, universities and research centers have high-performance needs which lead to the resource management problems. According to the NIST [2], cloud computing resources are classified as IaaS(infrastructure as a Service), SaaS(Sofware as a Servce) and PaaS (Platform as a Service). One of the key problem in cloud computing is the management of how to procure and release the resources [3]. Other cloud characteristics such the interdependancies of heterogeneous resource types, the variation and unpredictability of the workload all contribute to the complexity of cloud environment management.

To address the above mentioned problems, we referred to the works done by other researchers on this topic. Currently, we found that there exist few papers and most of them lack detailed working on each of the resource management technique and a limited critical discussion on the existing issues and challenges. Author in [4] has surveyed the state-of-the-art resource provisioning algorithms based on the techniques used to manage resources. Authors also mentioned some metrics such as cost, utility, and service quality. In [5], author studied an online auction framework for dynamic resource provisioning. The online auction framework is based on optimized system efficiency model dynamically and provisioning of heterogeneous rource types. The resource allocation based on IEDA (Intelligent Economic Approach) was suggested in [6]. Authors have proposed an IEDA based on auction protocol to let different kinds of resources traded between cloud consumers and cloud providers. Auction mechanisms have attracted attention of many researchers as an efficient approach for pricing and allocating resources. Another management of the resource allocation was studied in [7] using batch scheduling strategies. For most switable resources, paper [8] has suggested MADM(multiattribute decision making) technique for multiple resource discovery. Authors have focused on the methods of matching resources at a super-peer nodes based on matchmaker algorithm.

Authors in [9] conducted a survey study on VMs placement for cloud computing. Authors focused on scheduling and mapping resources using different optimization objectives. In the survey, their first priority is to analyze how the VMs can be mapped onto PMs to optimize different objectives. Also they analyzed the efficiency of different solutions in different situations. Authors in [10] have outlined the use of self-organizing algorithms in resource management such as bio-inspired computing, multi-agent system, bacterial foraging algorithm (BFA), shuffle flog algorithm leaping (SFLA), evolutionary techniques and have highlighted few of the main open challenges related to the area such as energy efficiency, and security issues. The security issues in cloud resources also were highlighted in [11]. Some techniques such as ant colony optmization algorithm (ACO) for workload consolidation, Honey Bees algorithm for load balancing, were also studied to make resources more robust and adaptable.

Our contributions in our study is to conduct a comprehensive survey of different techniques which can be applied to solve resource management problems. Among those techniques are resource provisioning, resource discovery, resource monitoring, resource mapping, resource allocation, resource consolidation, and resource scheduling. We want to convey the complexity of the problem by discribing the state-of-the-art outlining different metrics or parameters studied so far such as improved throughput, availability, scalability, quality of service (QoS), execution time, SLA violation, resource utilization, energy efficiency, and performance. For this paper, we reviewed 300+ recent research papers, which we selected about a third for this article.

The rest of the paper was organized as follows. In section2, we outlined the scope of the resource management in cloud computing. Resource management techniques were presented in section 3. Migration goals were discussed in section 4. Some issues and solutions for resource management were given in section 5. Section 6 shows the open problems, discussion and suggestions. Finally, we concluded in section 7.

2.The Scope of Resource Management in Cloud Computing

NIST defines cloud computing as a model for providing services to computing resources such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS). The resources are classified into physical resources, virtual resources, and logical resources.

2.1Physical, Logical, and Virtual Resource Management

The physical resources (PR) consist of the processors, memory, disk drive, Network Interface Controller (NIC), peripheral devices (floppy, keyboard), network resources (networking products), storage media, and other physical components. The physical resources are dispatched across multiple compute requests through virtualization and provisioning while the virtual resources (VR) are dynamically assigned and reassigned according to consumer demand. The physical resources and virtual resources are managed with resource management system (RMS). When a user orders resources, the RMS checks actual resource status to decide if there are sufficient resources that could satisfy user's request. The RMS is responsible to monitor continuously the user's allocated resource while optimizing the system [10]. Virtual and Physical servers in data-centers host many applications types such as webservers, databases, customer business apps, etc. The physical servers host the virtual servers called virtual machines and those VMs run OS or applications. Virtual resource management tools and technologies in cloud data-center as the cloud computing platforms shifts toward providing infrastructure as service(IaaS) where computing resources are provisioned on a per-demand resource basis regardless of location of the available computing resources [12].

The researches have been done to develop the cloud data-center infrastructure. Server consolidation is one of the approaches used to consolidate the physical servers and save space in data center. To consolidate the VMs on one physical server reduce the capital, operating expenses, and cut all unnecessary costs to maximize return on investment(ROI). A server consolidation technique is divided into three groups: physical consolidation, logical consolidation, and workloads consolidation. The physical consolidation means collocating servers at fewer locations [13]. Logical consolidation means no physical relocation of servers done and the implementation procedures are done across the server application. Logically, the servers are consolidated through a set of management tools and processes. Workload consolidation means implementing multiple applications on fewer, more powerful platforms through workload management and partitioning also called rationalized consolidation. Authors [14] have categorized the type of workload in cloud computing: Web server and SaaS workload based such as communication, file storage, processing, online shops, interactive, DBMS, scientific applications based such as Big Data, Workflows, learning algorithms, etc. Logical resources management are system abstractions which have temporary control over physical resources[15] i.e operating system, energy, network throughput/ bandwidth, information security, protocols, APIs, network loads, and delays.

2.2Requirements of Resource Management

According to the NIST[2], cloud resources exhibit five essential characteristics and their requirements:

2.3Issues in Resource Management

There exists several problems while managing cloud resources. In this section, we investigated the challenges, strategies, and the opportunities related to cloud resource management.

2.3.1.Challenges in Cloud Resource Management

Whenever a cloud user makes a request for the provisioning of one or more resources, the cloud provider's resource management system (RMS) schedules the resources by making them available to the cloud user. This decision helps to acheive the cloud provider's objectives but some challenges are still appearing such as the capabilities to minimize energy consumption, minimize the cost, and optimize the performance. The efficient strategies for resource management should be based on the challenges [16-19]: energy-efficient, bandwidth cost minimization, storage and performance optimization. We limit ourselves to the brief review of some works addressing power resources, networking resources, compute resources, storage resources challenges.

The power resource is a big challenge in cloud computing companies. According to the work in [20], authors pointed out that the utilization rate of server resources in most datacenters is 20% and the rest 80% represents the idle servers and those idle servers need 60% of the total consumed power and the rest of power coupled with cooling equipment, and the security system. Consider the CPU as the main components to consume energy. Turning unused hosts off improves the energy efficiency. According to [20], Dynamic Component Deactivation (DCD), Dynamic Performance Scaling (DPS), and Dynamic Voltage and Frequency Scaling (DVFS) are ones of the dynamic power management suggested solutions. DPS technique is for the automatic adjustment of the performance proportional to the power consumption. DVFS and DCD are applied to different computer component and at OS level such as CPU, Memory, disk, network interface, and other power-aware OS such as KVM, VMware solution, and Xen Hypervisor.

Kim et al. [21] studied different adaptive DVFS schemes to maximize energy savings and user profits. Their proposed algorithm selects the least expensive VM and placement that meets the required application throughput (MIPS rate) to minimize user's cost. The work in [22] proposed an energy-aware algorithm that supports DVFS. Their algorithm schedules deadlineconstrained virtual machines to the cores of the PMs (physical machines) to provide more computation resource within a certain power budget. The physical machines are prioritized depending to their performance-power ratio. As outlined in paper [23], a multi-agent-based resource management for energy minimization addressed the allocation of VMs to PMs in order to minimize the energy consumption. Instead, it is a traditional energy-aware VMs resource allocations because it either allocates VMs to PMs in a centralized manner or it implements VM migrations for energy reduction without considering the migration cost.

The network resource also is a big challenge in cloud computing. Author in [24, 25] proposed the VM resource scheduling algorithm based on auction mechanism which considered network bandwidth cost minimization. For computing resources optimization, Edwardo and Fabio [26] have presented a dynamic programming approach based on mathematical formulation to optimize the system reliability. The DPA approach was combined with successive approximations, branch-and-bound, and binary programming to get a good reliability index. Other methods have been suggested along years to compare the cost and the system reliability.

2.3.2.Cloud Resource Management Strategies

Resource management techniques are resource monitoring, resource allocation, resource consolidation (such as Virtualization Consolidation, storage consolidation, and Data center Consolidation), resource discovery, resource scheduling, resource brokering, resource modeling, and resource mapping. We reviewed various strategies developed so far for resource management. Topology-Aware Resource Allocation and Dynamic Resource Allocation for Parallel Data Processing are some of the techniques used for resource management design to increase the profit of cloud resource utilization, and minimize the response time. Network-Aware Resource Management technique was proposed in [27] to solve efficient network resource sharing issue, and network congestion issues based on SDN (Software-Defined Network) resource management strategy. The authors in [28] suggested a "skewness algorithm" to manage and reclaim the dynamic resources using virtualization concept. According to paper [29], T.R Gopalkrishnan Nair, et al., presented a model named RBRAM (Ruled Based Resource Allocation) to solve two critical resource management issues: resource arbitration and resource allocation. Authors discussed 3 main points: resource allocation strategies (RAS), complexity of the system allocation, and transition panorama of resource allocation. for better utilization of resources, in [30], authors focused on application level resource allocation and resource mapping between physical resources and virtual resources. In [31], authors proposed job scheduling algorithm based on Berger Model with dual fairness constraints. In [32], authors based on scheduling technique, they proposed a fault-tolerant scheduling for real-time scientific workflows with elastic resource provisioning in virtualized clouds based on PB (primary backup). Their technique uses the vertical scaling-down scheme to avoid unnecessary and ineffective resource changes due to fluctuated workflow requests to method to make full use of the idle resources. To solve monetary issues linked to under provisioning, an online auction framework for dynamic resource provisioning was presented in [5]. The resource provisioning and scheduling algorithm [33] techniques can be combined to solve the overall workflow execution cost and deadline constraints. There is a great concern on how to evaluate the performance of the resource provisioning. Paper in[34], suggested an approach for modeling and analyzing the performance of the resource provisioning. To evaluate the performance, the SPA framework was used. SPA (Stochastic Process Algebra) simulate the components involved in the resource provisioning. The theories of SPA can be found in [35-38].

Discovery of the cloud resource is critical issue in designing an enhanced and practical cloud system. The major challenge is to locate desired resources without the global knowledge about sharing resource information. Paper [39] introduces resource discovery technique named multi-valued distributed hash tables for efficient resource discovery. A new auction method was also suggested, by using a reserve bid formulated rationally by each user for the optimal allocation of discovered resources. The work[40]considers a fully decentralized resource discovery service based on an unstructured overlay. Paper [41]proposed an approach of managing dynamic and autonomous resources in cloud data centers. This technique adopts adjustments of the parameters in order to continuously manage the resources in cloud data centers with less human intervention. The figure 3 represents the resource discovery and resource allocation framework.

Monitoring of cloud resources is an serious issue. Researches have been conducted on how to monitor the performance of cloud resources. The measurement unit based on resource monitoring are total IOPS, read IOPS, write IOPS, total bandwidth, write bandwidth, total response time, read response time or write response time. The monitoring information includes CPU monitoring, capacity monitoring, network monitoring, power monitoring, and security monitoring. The capacity monitoring is to monitor the available total capacity, reserved capacity or reservation rate. The compute resource monitoring providers the monitoring of the VM status (running/stooped), CPU usage, memory usage, operations, and more. The network monitoring consists on monitoring the network traffic. The task tracing is the monitoring of task name, object name, start time, status, task end time , operator (admin, user, or system action). Alarm monitoring called security monitoring is the monitoring of the current alarms, alarm thresholds, alarm masking, alarm statistics, and alarm dump.

Event monitoring system provids the historical events. It checks all the past actions of the whole system. Energy-aware monitoring [42], considers energy as a QoS metric. Author presented service framework which allow to monitor the power consumption of a cloud infrastructure, calculated its energy efficiency to put in place an effective virtual machine management. Author calculated the energy consumed by virtual and physical hosts by applying power models over the utilization of the resource and use external sensors and devices for measuring the power consumption [43].

In cloud computing environment, the logical network must be deployed on to physical network. The current resource management methodologies are capable to map logical (i.e set of virtual resources) on to physical systems dynamically depends on workload.To map VMs onto PMs is a key problem for cloud providers since PM utilization impacts revenue and clients significantly. VM initial placement refers to the process of mapping VMs onto PMs with optimization objectives but the challenge is to know which VMs to allocate to which PMs to achieve the targeted goals. This action is formulated as an integer programming problem where various applications and system constraints may also be considered to deploy an intelligent resource mapping[9].There exists some challenges related to resource mapping such as designing the algorithm that can find a fast mapping using genetic algorithms to speed up the mapping process and ensures the respecting of all task deadlines; minimize the cost of mapping; evaluate the services providers as possible candidates for hosting the applications, etc.

2.3.3.Opportunities in Resource Management

Clouds are designed to deliver as much as computing capacity as any user wants. Cloud provider rent or lease resources to users in self-service manner with additional services such as resource scalability which provide greater flexibility for users since user is only paying for what needed. It has some benefits such as scalability, QoS (Quality of Service), cost effectiveness, and simplified interfaces.

3.Resource Management Techniques

The main objective of resource management is to reduce the service costs, improve the performance, security, and energy efficiency. Here, we considered different resource management techniques based on parameters like monetary parameters (service cost), application performance parameters (response time, execution time, delay, SLA violation, task type, number of processors needed to run tasks, throughput, resource availability, resource utlization, etc), security, and energy-efficiency parameters (overall power, energy consumption) [44]. The algorithms such as load balancing, Round Robin, Bin packing algorithm and gradient search algorithm were mentioned as the optimization methods to improve the performance, reduce the cost, and minimize the energy consumption in IaaS resources. For the bin packing algorithm [45, 46], the heuristic based VM migration scenario is partitioned as follows: (1) Divides machines into two pools - core nodes and accelerator nodes. (2) Adjusts the size of each pool to reduce cost or increase resource utilization. This manages the resource nodes allocated and decides when to add or remove them from the resource pool. This also monitors the storage system to estimate the incoming data capacity. The author in [47, 48] focused on maximizing the efficiency of the resource scheduling algorithm. The research in [49] considered a stochastic model based on load balancing and scheduling in cloud computing clusters, where tasks arrive according to a stochastic process and request resources like memory, CPU, and storage space.

In [50], Feng et al. presented a joint allocation and scheduling of network resource for multiple control applications in SDN.The authors have evaluated the learning algorithms by using the price paid by the user for both bandwidth and flow table capacities in order to guarantee fair allocation of the network resources. The bandwidth allocation was achieved by maximizing the sum of each control application in a logarithmic rate, where the control application rate is proportional to the bandwidth price for that particular application while flow table allocation favors flows with higher hit-rate to optimize OpenFlow switches' throughput.

Hierarchical network-aware placement algorithms based on Integer Linear Programming (ILP) [51] were discussed to solve the Cloud Application Placement Problems (CAPP). CAPP is used to determine how applications and services are allocated within the cloud. For instance, in which machine should a service be allocated in order to satisfy multiple constraints such as CPU, memory, bandwidth, and management policies. Another resource management approaches were proposed in [52] which use the Self-Organizing Clouds (SOC) to achieve the maximum of the resource utilization and to deliver optimal execution time. SOC having two main issues: (a) Locating a qualified node to satisfy a user task's resource demand with bounded delay; (b) Optimize a task's execution time by determining the optimal shares of the multi-attribute resources to allocate to the tasks with various QoS constraints, such as the expected execution time. Table 2 gives a summary comparison of cloud resources based on some metrics.

3.1 Virtualization Techniques

Virtualization technique [71] has become popular in IT industry as a software-based solution to build shared hardware infrastructures [72]. This concept of virtualization has started back in 1964 with IBM project called CP/CMS system and later spreaded all over in IT companies [73]. Virtualization technique is a framework to divide the resources of a computer into execution environments to make the physical machine to be productive for the resource utilization and resource management. Authors [74]elaborated the importance of virtualization in cloud computing to manage cloud resources. Virtualization supports migration of VMs from existing to other physical machines [75]. Virtualization technology brought technical basis in cloud computing. There exists different kind of virtualization technology used in cloud computing such as storage virtualization, server virtualization, network virtualization, client virtualization, and operation framework virtualization [76]. The network virtualization is based on virtualized network switch technology commonly known as vSwitch. Client Virtualization is a client virtualization technology based on creating a client desktop as a VM called VDI (virtual desktop infrastructure). A VDI uses rack-based servers distributed across the data center (DC) with a top of rack (ToR) switches at the network edge. A virtual processor also called virtual CPU (vCPU) is a share of a physical CPU which is assigned to a VM. By default, each VM is allocated one vCPU. Storage Area Network (SAN) is a storage architecture that connects the storage system to the application server over the network. SAN storage devices provide data storage space for VMs in the system using shared software to provide block-level data sharing service. Cloud storage virtualization is a solution which permits cloud users to save data or use storage resources on a unified cloud platform. Such a cloud storage system consolidates all types of storage devices on the network into a unified storage platform using various functions, such as cluster applications and distributed file system.

3.2 Migration Techniques

Migration technique is the process of moving an operating system instance from one node to another node without interrupting an operation in order to improve the system reliability and availability. Migration facilitates load balancing, fault tolerant management, low level system maintenance and energy efficiency in data-center [77]. Due to the sensitivity of the migration process, some performance metrics have to be considered such as preparation time, memory pages transferred estimation, total migration time, and application performance degradation.

a)Live migration: The live migration is the process of transferring a running OS and its associated applications from the source host to the destination host without disrupting the running services. It is used to increase the resource availability and decrease energy consumption in cloud computing. There are many techniques which attempt to reduce the downtime and to provide better performance in low bandwidth environment. We outline two techniques: pre-copy technique and post-copy technique. Pre-copy technique allows the source servers to transfer the memory to the target VMs over a series of iterations while in post-copy technique, the information of device state on the target machines are transferred at first and also uses the following metrics to measure the performance: preparation, service downtime, resume, total migration time, CPU and network consumption.

b) Offline migration: During the offline migration, the service is stopped before the migration and resumed after the migration while for the live migration the process can execute at the user unaware state. The disadvantage of offline migration is the larger downtime.

c) Cold migration: Cold Migration also called regular migration is the migration of a power-off virtual machine. With cold migration, you have the option of moving the associated disks from one data store to another[78].

d) Hybrid migration: Hybrid migration is a combination of both pre-copy and post-copy virtual machine migration. It presents five phases:preparation phase;bounded pre-copy rounds phase; virtual machine state transfer phase; virtual machine resume phase; and on demand paging phase.

e) Storage Migration: The storage migration can be performed while the VM still running. Whether a VM is reading or writing from virtual hard disk file for Microsoft Windows servers or whether they are OVMs or VMS. There exist many VM storage data migration solutions. In this paper, we only cite the following storage data migration solutions:

LVM mirroring based data migration solution: This migration solution is based on Linux LVM mirroring function.The disk space used by applications comes from logical volumes (LVs) created based on physical volumes (PVs) of the source storage. The storage model must be compatible with the Linux system. The host must be directly connected to a storage device (DAS networking mode) or it must be connected to the storage device using switch(SAN networking mode) and the number of idle ports on the switch must be greater than or equal to 2xn where n represents the number of controllers on the target storage. It supports both online and offline data migrations.

LDM Mirroring-based Data Migration Solution: The LDM(Logical Disk Manager) is a migration technique to migrate data between different storage systems using the logical disk manager function.

SmartVirtualization and SmartMigration based data migration solution: These migration solutions describe how to migrate data from the source storage to target storage using smartVirtualization and smartMigration features. SmartVirtualization supports both DAS and SAN networking mode and is used while migrating data on peer vendors' products.It provides the benefits such as Online LUN migration and ensuring service continuity and data consistency.

LUN copy based data migration: The LUN (Logical Unit Number) copy based migration is the migration technique based on LUN copy feature. LUN copy feature are used to copy data from remote LUNs on storage devices to the target storage using fibre channel links. The LUN Copy migration technique provides quick data distribution, and centralized, and generates multiple data duplicates to ensure data security.

- ASM based data migration solution: The basic principles of ASM data migration solution are as follows: the extents of files are moved among disks using ASM Rebalance to realize I/O balance on the ASM disk group.LUNs of the target storage are added to the ASM disk group of the database, and LUNs of the source storage are deleted from the ASM disk group.

VIS-based Data Migration Solution: VIS-based migration technique is used to migrate data from the source storage to the target storage. This technique consists of taking over the source storage and migrate data from source storage to the target storage.

FastCopy-based Data Migration Solution: The FastCopy-based data migration is the migration technique used to migrate data between windows server OS. The source and target file systems are mounted on the migration server. The FastCopy is used to synchronize files between the source and target systems, achieving data migration between the source and target storage.

There are other resource management functions to be considered in storage system such as SmartThin, smartTier, SmartQoS, SmartMotion, SmartPartition, etc. The SmartThin function automatically expands capacity, and help on improving disk utilization.The SmartTier function intelligently migrates data among different storage tiers based on the data access frequencies.It dynamically matches hotspot data and storage medium, improving system performance and reducing the TCO (total cost of ownership). The SmartQoS function intelligently schedules storage resources based on the priority of services, optimizing system resourceallocation. It dynamically allocates resources of storage systems, meeting specific performance requirements on the IOPS, bandwidth, or latency.The SmartMotion function dynamically relocates data based on the service changes to balance loads of storage systems. The SmartPartition function sets cache partition requirements for critical services, dynamically allocates cache resources to services based on the requirements, and isolates the cache resources between services, preventing unnecessary cache competition and ensuring performance of critical services.

4.Migration Objectives

4.1. Server Optimization

The optimization for server migration helps on improving the live migration and optimizing the performance metrics such as total migration time, and downtime. While providing uninterruptible services to applications running in VMs, some techniques have been proposed: (1) Memory Page Compression: Live migration performance is improved by minimizing the amount of data transferred to the destination using the technique called memory page compression which compress the source PM's memory pages and decompress the memory pages in target PM. (2) Delta page transfer: This technique reduces the network bandwidth consumption by maintaining a cache of already transferred memory pages. It improves the live migration process and reduces service interruption risks. (3) Data de-duplication: the data deduplication is a compression technique which is specialized to find the duplicates data inside the memory and disk of a single VM and removes them during the live migration process. It improves the storage data utilization and in network data transfer. (4) Post-copy technique: The post copy migration technique means the transfer of a VM's memory contents until after its processor state has been sent to the target host and resumed there[79]. The performance of this technique depends on the way and which the VM's memory contents are fetched from source machines during the live migration. (5)Hybrid pre and post copy: This technique performs single round of pre-copying which precedes the virtual CPU state transfer. This technique improves live migration process. (6) Server consolidation: To reduce server sprawl in data-centers, server consolidation algorithms should be the requirements. Among them are: stochastic bin packing[80],[81], multi-capacity bin packing [82],VM packing heuristics, and memory buddies algorithms.

The memory buddies algorithm[83] is a server consolidation technique which determines which candidate servers to be shutting down and attempts to migrate virtual machines to hosts with high sharing opportunities. The memory buddies algorithm comprises three steps: The first step is to identify server to consolidate by examining memory utilization statistics of every hosts. The second step is to identify the target hosts. After determining the server candidates, the algorithm has to determine the new physical server to house the VM but one of the existing challenges reside on how to find the proper PM in order to minimize the cost function. This can happen especially when the VM has a large CPU size, network or especially when the existing servers are heavily utilized. The third step concerned with the migration of the VMs to their target hosts. Once the new destinations have been discovered, the algorithm can perform VM migration process with minimal service downtime and with the minimum resource consumption. The server consolidation technique results in reduced power consumption, reduced overall operational costs, and increase overall resource usage.

4.2. Load Balancing with Data Migration

The more VMs can be placed on the same host, the greater the energy efficiency can be obtained. However, a large number of VMs at the same host will increase the VMs migration and degrade the system performance. Therefore, the hotspot mitigation algorithm[84]came up to help to determine which heavy-loaded VMs to migrate first. Other approaches such as roundrobin, minimal queue depth were also invoked. The advantage of load balancing algorithm is to enhance the performance in data migration and the resource utilization rate.

4.3Hotspot & Coldspot Migration

Hotspots and clodspots are based on the thresholds set by the cloud owner or based on the SLA specified by the clients. Hotspot uses a memory-to-memory stop-and-copy migration technique that copies the container's memory state from the memory of the destination VM without saving it to stable storage. Hotspot could define the migration policy that considers optimization of the cost.

Hotspot's migration policy focuses on minimizing cost, and doesn't consider the performance when deciding where and when to migrate, It may select a host VM that is underprovisioned and degrades an application performance.

Also, the migration policy determines when and where to migrate its resource container based on current resource usage. A higher resource usage value close to the maximum is set as the upper threshold and a very low resource usage value is set as the lower threshold [74]. Hotspot makes estimation of the resourcee utilization by considering two cases: Low utilization and high utilization depending on whether the PMs' resource usage values are upper or lower threshold [85].

5.Solution to Resource Management Issues

5.1 Resource Management Schemes

6.Summary, Open Problems and Suggestions

In this work, we have theoretically analyzed issues in resource management techniques and discussed their advantages such as availability, scalability, quality of Service, optimal utility, reduced latency, reduced overheads, specialized environment, communication cost, computational time, and energy efficiency. Many resource management approaches have been investigated by different researchers under different virtualized environments and the performance of those approaches has been evaluated in cited referenced papers. Here we are mentioning some of the unresolved issues to focus on.

a.Performance issues

Performance guarantee is the important issue between the cloud service providers and cloud users. As mentioned in the previous paragraphs, there are many challenges in terms of performance of the cloud resources. Suggestions were given on how to improve the performance of resources and applications running in cloud environment. There are possible ways to improve the performance of cloud to model the system more accurately by identifying parameters and metrics which cause the system less productive.

Evaluation metrics: These are the metrics used to evaluate the performance of cloud resources: IOPS including read IOPS, write IOPS, total response time, read response time or write response time, total bandwidth, write bandwidth, and so on.

Application performance metrics: It measures user satisfaction trying to capture some QoS level of performance which may include response time or execution time, the ratio of the capacity provided to the application to the maximum capacity at optimal allocation, the number of SLA violations and delay.

Energy Efficiency metrics: The good way to measure the energy improvements is to consider overall power and energy consumption. Some power saving methods were proposed such as DVFS [92-94] on/off [95-99], DVS [100, 101], VM migration [102], VM placement [103], Hibernation [104].

Monetary metrics: This evaluation is based on the service provider's cost, and user cost.

b. Security and privacy issue

According to [105], the security and privacy are the major concerns in cloud computing since the users have no control over their data. The cloud providers may convincy the third parties the safety of their data and the infrastructure against both exterior and domestic attacks. That hope is however rely on organizational procedures but it cannot be verified or remotely validated by the third parties [106]. In cloud security alliance reports [107, 108] and studies on cloud security issues [109] have mentioned several security threats such as data breaches, data loss, denial of service, and more other malicious which can occur in cloud system information management. The security solution approaches were proposed such as authentication and authorization, identity and access management, confidentiality, integrity, and availability, etc. The study on authentication solution with MiLAMob [110]. MiLAMob is a middleware-layer that handles the real-time authentication events on behalf of consumer devices with minimal HTTP traffic. Another approach on identification and authentication is Public key infrastructure (PKI) X.509 certificates [111]. We propose other cryptographic approach such as IBE, IBS, CP-ABE [112] to enforce identity and access management policies. Also there exists other proposed approaches such as CloudProof [113], trusted cloud computing platform (TCCP) [114, 115], Fuzzy authorization (FA) for cloud storage [116] for enforcing the confidentiality, integrity, reliability, and availability.

c. Resource scheduling

Processor resources scheduling: Efficient processor resource scheduling leads to the reduction of dynamic energy consumption. Currently, researches on single and multiprocessor scheduling were conducted to reveal techniques to reduce energy consumption such as poweraware task scheduling (PTS) based on dynamic voltage scaling (DVS) [117], real time DVS algorithms (RT-DVS) [118], dynamic-voltage scaling-adaptive body biasing(DVS-ABB) [119] , etc.

Server resource scheduling: The purpose for server resource scheduling is to find the idle machines to be turn off when no services running on them to save the energy. The VMs can dynamically start, hang, or turn off. We categorized the energy-saving solution into two categories: The VMs based servers and PMs based servers. The resource assigned to the VMs can dynamically be adjusted and monitored.

7.Conclusion

In our study, we have surveyed different techniques which can be applied to manage cloud resources. Among those techniques are resource provisioning, resource discovery, resource monitoring, resource mapping, resource allocation, and resource consolidation. We also find the opening problems which can be solved to make the cloud system more manageable and efficient. Those open challenges are resource performance issues, security and privacy issue, and server, processor, workload resources scheduling. The following metrics were applied: CPU utilization, VMs allocation SLA violation, total cost, and profit. In our next paper we will focus on cloud resource scheduling based on meta-heuristic techniques.

Acknowledgement

This work was supported by the National Science Foundation of China (Grant Nos. 61202508, 61272432, 61370132, 61472033, 61370092, Fundamental Research Funds for the Central Universities [FRF-TP-14-045A2]), and Rwanda Education Board.

References

References

[1] Jennings, B, R Stadler. Resource management in clouds: Survey and research challenges. Journal of Network and Systems Management, 2015. 23(3): p. 567-619.

[2] Mell, P, T Grance. The NIST definition of cloud computing. 2011.

[3] Arianyan, E, H Taheri, S Sharifian. Novel energy and SLA efficient resource management heuristics for consolidation of virtual machines in cloud data centers. Computers & Electrical Engineering. 2015; 47: 222-240.

[4] Zhang, J, H Huang, X Wang. Resource provision algorithms in cloud computing: A survey. Journal of Network and Computer Applications. 2016; 64: 23-42.

[5] Shi, W, et al. An online auction framework for dynamic resource provisioning in cloud computing. IEEE/ACM Transactions on Networking. 2016; 24(4): 2060-2073.

[6] Wang, X, et al., An intelligent economic approach for dynamic resource allocation in cloud services. IEEE Transactions on Cloud Computing. 2015; 3(3): 275-289.

[7] Jintao, J, Y Wensen, G Lei. Research on Batch Scheduling in Cloud Computing. TELKOMNIKA (Telecommunication Computing Electronics and Control). 2016; 14(4): 1454-1461.

[8] Kaur, M, SS Kadam. Discovery of resources using MADM approaches for parallel and distributed computing. Engineering Science and Technology, an International Journal. 2017.

[9] Pietri, I, R Sakellariou. Mapping virtual machines onto physical machines in cloud computing: A survey. ACM Computing Surveys (CSUR). 2016. 49(3): 49.

[10] Endo, PT , et al. Self-organizing strategies for resource management in Cloud Computing: State-ofthe-art and challenges. in Cloud Computing and Communications (LatinCloud), 2nd IEEE Latin American Conference on. 2013. IEEE.

[11] Singh, A, K Chatterjee. Cloud security issues and challenges: A survey. Journal of Network and Computer Applications. 2017; 79: 88-115.

[12] Nguyen Van, H, F Dang Tran, J-M. Menaud. Autonomic virtual resource management for service hosting platforms. in Proceedings of the 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing. 2009. IEEE Computer Society.

[13] Mazumdar, S and M Pranzo. Power efficient server consolidation for Cloud data center. Future Generation Computer Systems. 2017; 70: 4-16.

[14] Ullrich, M, J Lässig, M Gaedke. Towards Efficient Resource Management in Cloud Computing: A Survey. in Future Internet of Things and Cloud (FiCloud). 2016 IEEE 4th International Conference on. 2016. IEEE.

[15] Manvi, SS, GK Shyam. Resource management for Infrastructure as a Service (IaaS) in cloud computing: A survey. Journal of Network and Computer Applications. 2014; 41: 424-440.

[16] Parikh, SM. A survey on cloud computing resource allocation techniques. in Engineering (NUiCONE). 2013 Nirma University International Conference on. 2013. IEEE.

[17] Madni, SHH, et al. Resource scheduling for infrastructure as a service (IaaS) in cloud computing: Challenges and opportunities. Journal of Network and Computer Applications. 2016; 68: 173-200.

[18] Vakilinia, S, MM Ali, D Qiu. Modeling of the resource allocation in cloud computing centers. Computer Networks. 2015; 91: 453-470.

[19] Han, D, W Sun, X Fan. Dynamic energy management in smart grid: A fast randomized first-order optimization algorithm. International Journal of Electrical Power & Energy Systems; 2018; 94: 179187.

[20] Lee, L-T, et al. A dynamic resource management with energy saving mechanism for supporting cloud computing. International Journal of Grid and Distributed Computing. 2013; 6(1): 67-76.

[21] Kim, N, J Cho, E Seo. Energy-credit scheduler: an energy-aware virtual machine scheduler for cloud systems. Future Generation Computer Systems. 2014; 32: 128-137.

[22] Ding, Y, et al. Energy efficient scheduling of virtual machines in cloud with deadline constraint. Future Generation Computer Systems. 2015; 50: 62-74.

[23] Wang, W, Y Jiang, W Wu. Multiagent-Based Resource Allocation for Energy Minimization in Cloud Computing Systems. IEEE Transactions on Systems, Man, and Cybernetics: Systems. 2016.

[24] Younge, A J , et al. Efficient resource management for cloud computing environments. in Green Computing Conference, 2010 International. 2010. IEEE.

[25] Kong, W, Y Lei, J Ma. Virtual machine resource scheduling algorithm for cloud computing based on auction mechanism. Optik-International Journal for Light and Electron Optics. 2016; 127(12): 50995104.

[26] Bacalhau, ET , FL Usberti, C Lyra. A dynamic programming approach for optimal allocation of maintenance resources on power distribution networks. in Power and Energy Society General Meeting (PESJ. 2013 IEEE.

[27] Abdelaal, MA , GA Ebrahim, WR Anis. Network-aware resource management strategy in cloud computing environments. in Computer Engineering & Systems (ICCES), 2016 11th International Conference on. 2016. IEEE.

[28] Verma, M , et al. Dynamic resource demand prediction and allocation in multi-tenant service clouds. Concurrency and Computation: Practice and Experience, 2016.

[29] Nair, TG, M Vaidehi. Efficient resource arbitration and allocation strategies in cloud computing through virtualization. In Cloud Computing and Intelligence Systems (CCIS), 2011 IEEE International Conference on. 2011. IEEE.

[30] Lin, W, et al. A threshold-based dynamic resource allocation scheme for cloud computing. Procedía Engineering. 2011; 23: 695-703.

[31] Xu, B, et al. Job scheduling algorithm based on Berger model in cloud environment. Advances in Engineering Software; 2011; 42(7): 419-425.

[32] Zhu, X, et al. Fault-tolerant scheduling for real-time scientific workflows with elastic resource provisioning in virtualized clouds. IEEE Transactions on Parallel and Distributed Systems. 2016; 27(12): 3501-3517.

[33] Rodriguez, MA, R Buyya. Deadline Based Resource Provisioningand Scheduling Algorithm for Scientific Workflows on Clouds. cloud computing. 2014; 2(2): 222-235.

[34] Shawky, DM. Performance evaluation of dynamic resource allocation in cloud computing platforms using Stochastic Process Algebra. in Computer Engineering & Systems (ICCES), 2013 8th International Conference on. 2013. IEEE.

[35] Bernardo, M, L Donatiello, P Ciancarini. Stochastic process algebra: From an algebraic formalism to an architectural description language. in IFIP International Symposium on Computer Performance Modeling, Measurement and Evaluation. 2002. Springer.

[36] Baeten, JC, WP. Weijland. Process algebra, volume 18 of Cambridge tracts in theoretical computer science. 1990. Cambridge University Press Cambridge, UK.

[37] Hermanns, H, U Herzog, V Mertsiotakis. Stochastic process algebras as a tool for performance and dependability modelling. in Computer Performance and Dependability Symposium. 1995. Proceedings. International. 1995. IEEE.

[38] Herzog, U. Formal description, time and performance analysis a framework, in Entwurf und Betrieb verteilter Systeme. 1990. Springer. 172-190.

[39] Khethavath, P, et al. Introducing a distributed cloud architecture with efficient resource discovery and optimal resource allocation. in Services (SERVICES), 203 IEEE Ninth World Congress on. 2013. IEEE.

[40] Chung, W-C, et al. Direction-aware resource discovery service in large-scale grid and cloud computing. in Service-Oriented Computing and Applications (SOCA). 2011. IEEE International Conference on. 2011. IEEE.

[41] Yazir, YO , et al. Dynamic resource allocation in computing clouds using distributed multiple criteria decision analysis. in Cloud Computing (CLOUD). 2010 IEEE 3rd International Conference on. 2010. IEEE.

[42] Katsaros, G , et al. A service framework for energy-aware monitoring and VM management in Clouds. Future Generation Computer Systems. 2013; 29(8): 2077-2091.

[43] Rong, H, et al. Optimizing energy consumption for data centers. Renewable and Sustainable Energy Reviews. 2016; 58: 674-691.

[44] Lee, G, RH Katz. Heterogeneity-Aware Resource Allocation and Scheduling in the Cloud. in HotCloud. 2011.

[45] Ngenzi, A, SR Nair. Dynamic resource management in Cloud datacenters for Server consolidation. arXiv preprint arXiv:1505.00577. 2015.

[46] Coffman, EG , MR Garey, DS Johnson. Dynamic Bin Packing. SIAM Journal on Computing. 2006; 12(2): 227-258.

[47] Koneru, S , VR Uddandi, S Kavuri. Resource Allocation Method using Scheduling methods for Parallel Data Processing in Cloud. International Journal of Computer Science and Information Technologies [IJCSIT]. 2012; 3(4): 4625-4628.

[48] Jiang, W , et al. A Resource scheduling strategy in cloud computing based on multi-agent genetic algorithm. Indonesian Journal of Electrical Engineering and Computer Science. 2013; 11(11): 65636569.

[49] Maguluri, ST , R Srikant, L Ying. Stochastic models of load balancing and scheduling in cloud computing clusters. in international conference on computer communications. 2012.

[50] Feng, T, J Bi, K Wang. Joint allocation and scheduling of network resource for multiple control applications in SDN. in Network Operations and Management Symposium (NOMS). 2014. IEEE.

[51] Moens, H , et al. Hierarchical network-aware placement of service oriented applications in clouds. in Network Operations and Management Symposium (NOMS); 2014 IEEE.

[52] Di, S, C-L Wang. Dynamic optimization of multiattribute resource allocation in self-organizing clouds. IEEE Transactions on parallel and distributed systems. 2013; 24(3): 464-478.

[53] Han, G , et al. Resource-utilization-aware energy efficient server consolidation algorithm for green computing in IIOT. Journal of Network and Computer Applications. 2017.

[54] He, L, et al. Developing resource consolidation frameworks for moldable virtual machines in clouds. Future Generation Computer Systems. 2014; 32: 69-81.

[55] Mitra, A, N O'Regan, D Sarpong. Cloud resource adaptation: A resource based perspective on value creation for corporate growth. Technological Forecasting and Social Change. 2017.

[56] Hummaida, AR, NW Paton, R Sakellariou. Adaptation in cloud resource configuration: a survey. Journal of Cloud Computing. 2016; 5(1): 1-16.

[57] Bhattacharya, A, P De. A survey of adaptation techniques in computation offloading. Journal of Network and Computer Applications. 2017; 78: 97-115.

[58] Jung, G, KM Sim. Agent-based adaptive resource allocation on the cloud computing environment. in Parallel Processing Workshops (iCpPW). 2011 40th International Conference on. 2011. IEEE.

[59] Shi, W, et al. An online auction framework for dynamic resource provisioning in cloud computing. ACM SIGMETRICS Performance Evaluation Review. 2014; 42(1): 71-83.

[60] Singh, A, D Juneja, M Malhotra. A novel agent based autonomous and service composition framework for cost optimization of resource provisioning in cloud computing. Journal of King Saud University-Computer and Information Sciences. 2015.

[61] Alkalbani, AM, FK Hussain. A comparative study and future research directions in cloud service discovery. in Industrial Electronics and Applications (ICIEA). 2016 IEEE 11th Conference on. 2016. IEEE.

[62] Wright, P, et al. A constraints-based resource discovery model for multi-provider cloud environments. Journal of cloud computing: advances, systems and applications. 2012; 1(1): 6.

[63] Singh, S, I Chana. A survey on resource scheduling in cloud computing: Issues and challenges. Journal of grid computing. 2016; 14(2): 217-264.

[64] Ma, F, Y Yang. An energy sentient service brokering strategy in cloud computing. in Control And Decision Conference (CcDc). 2017 29th Chinese. 2017. IEEE.

[65] Awasthi, C, P Kanungo. Client Requirement Modeling Using Resource Broker Architecture in Cloud Computing Environment. in Cloud Computing in Emerging Markets (CCEM). 2014 IEEE International Conference on. 2014. IEEE.

[66] Roy, DG, et al. Multi-cloud scenario based QoS enhancing virtual resource brokering. in Recent Advances in Information Technology (RAIT). 2016 3rd International Conference on. 2016. IEEE.

[67] Devi, KV, K Mehata. Modelling resource estimation and scheduling for processing in cloud computing. in Advanced Computing (ICoAC). 2013 Fifth International Conference on. 2013. IEEE.

[68] Fan, G, H Yu, L Chen. A formal aspect-oriented method for modeling and analyzing adaptive resource scheduling in cloud computing. IEEE Transactions on Network and Service Management. 2016; 13(2): 281-294.

[69] Song, Y, R Routray, R Jain. Virtual-to-physical mapping inference in virtualized cloud environments. in Cloud Engineering (IC2E). 2014 IEEE International Conference on. 2014. IEEE.

[70] Di, H, et al. Reliable virtual infrastructure mapping with efficient resource sharing. in Communications. Circuits and Systems (ICCCAS), 2013 International Conference on. 2013. IEEE.

[71] Leelipushpam, PGJ, J Sharmila. Live VM migration techniques in cloud environment-a survey. in Information & Communication Technologies (ICT), 2013 IEEe Conference on. 2013. IEEE.

[72] Rabbani, M. Resource Management in Virtualized Data Center. 2014.

[73] Rodriguez-Haro, F, et al. A summary of virtualization techniques. Procedía Technology. 2012; 3: 267-272.

[74] Mohan, A, S Shine. Survey on live vm migration techniques. International Journal of Advanced Research in Computer Engineering & Technology. 2013; 2.

[75] Sonkar, S, M Kharat. A review on resource allocation and VM scheduling techniques and a model for efficient resource management in cloud computing environment. in ICT in Business Industry & Government (ICTBIG), International Conference on. 2016. IEEE.

[76] Singh, G, S Behal, M Taneja. Advanced Memory Reusing Mechanism for Virtual Machines in Cloud Computing. Procedía Computer Science.2015; 57: 91-103.

[77] Kale, RMCO. Virtual Machine Migration Techniques in Cloud Environment: A Survey.

[78] Vyas, N, A Chauhan. A Survey on Virtual Machine Migration Techniques In Cloud Computing.

[79] Hines, MR, K Gopalan. Post-copy based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning. in Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments. 2009. ACM.

[80] Jin, H, et al. Efficient VM placement with multiple deterministic and stochastic resources in data centers. in global communications conference. 2012.

[81] Zhang, L, et al. Moving Big Data to The Cloud: An Online Cost-Minimizing Approach. IEEE Journal on Selected Areas in Communications. 2013; 31(12): 2710-2721.

[82] Hwang, I, M Pedram. Hierarchical Virtual Machine Consolidation in a Cloud Computing System. in international conference on cloud computing. 2013.

[83] Wood, T, et al. Memory buddies: Exploiting page sharing for server consolidation in virtualized data centers. Citeseer. Tech. Rep. 2007.

[84] Wood, T, et al. Black-box and gray-box strategies for virtual machine migration. in networked systems design and implementation. 2007.

[85] Xu, Y, Y Sekiya. Virtual machine migration strategy in federated cloud. in Internet Conference. 2010.

[86] Javadi, B, J Abawajy, R Buyya. Failure-aware resource provisioning for hybrid Cloud infrastructure. Journal of Parallel and Distributed Computing. 2012; 72(10): 1318-1331.

[87] Nghiem, PP, S Figueira. Towards efficient resource provisioning in MapReduce. Journal of Parallel and Distributed Computing. 2016; 95: 29-41.

[88] Kaur, P, S Mehta. Resource provisioning and work flow scheduling in clouds using augmented Shuffled Frog Leaping Algorithm. Journal of Parallel and Distributed Computing. 2017; 101: 41-50.

[89] Yang, S, et al. Energy-Aware Provisioning in Optical Cloud Networks. Computer Networks. 2017; 118: 78-95.

[90] Ghobaei-Arani, M, S Jabbehdari, M A Pourmina. An autonomic resource provisioning approach for service-based cloud applications: A hybrid approach. Future Generation Computer Systems. 2017.

[91] Hadji, M, D Zeghlache. Minimum cost maximum flow algorithm for dynamic resource allocation in clouds. in Cloud Computing (CLOUD). 2012 IEEE 5th International Conference on. 2012. IEEE.

[92] Kaur, S, A. Kaur, M Gobindgarh. Energy Aware Resources Allocation Heuristic For Efficient Management of Data Centers For Cloud Computing. 2016.

[93] Beloglazov, A, J Abawajy, R Buyya. Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing. Future generation computer systems. 2012; 28(5): 755-768.

[94] Garg, SK, et al. Environment-conscious scheduling of HPC applications on distributed cloudoriented data centers. Journal of Parallel and Distributed Computing. 2011; 71(6): 732-749.

[95] Burge, J, P Ranganathan, JL Wiener. Cost-aware scheduling for heterogeneous enterprise machines (CASHEM). in Cluster Computing. 2007 IEEE International Conference On. 2007. IEEE.

[96] Bradley, DJ, RE Harper, SW Hunter. Workload-based power management for parallel computer systems. IBM Journal of Research and Development. 2003; 47(5.6): 703-718.

[97] Lefěvre, L, A-C Orgerie. Designing and evaluating an energy efficient cloud. The Journal of Supercomputing. 2010; 51(3): 352-373.

[98] Salfner, F, M Lenk, M Malek. A survey of online failure prediction methods. ACM Computing Surveys (CSUR). 2010; 42(3): 10.

[99] Wang, C-F, W-Y Hung, C-S Yang. A prediction based energy conserving resources allocation scheme for cloud computing. in Granular Computing (GrC). 2014 IEEE International Conference on. 2014. IEEE.

[100] Chen, Y, et al. Managing server energy and operational costs in hosting centers. in ACM SIGMETRICS performance evaluation review. 2005. ACM.

[101] Mezmaz, M, et al. A parallel bi-objective hybrid metaheuristic for energy-aware scheduling for cloud computing systems. Journal of Parallel and Distributed Computing. 2011; 71(11): 1497-1508.

[102] Tesfatsion, SK, E Wadbro, J Tordsson. A combined frequency scaling and application elasticity approach for energy-efficient cloud computing. Sustainable Computing: Informatics and Systems. 2014; 4(4): 205-214.

[103] Khosravi, A, SK Garg, R Buyya. Energy and carbon-efficient placement of virtual machines in distributed cloud data centers. in European Conference on Parallel Processing. 2013. Springer.

[104] Subrata, R, AY Zomaya, B Landfeldt. Cooperative power-aware scheduling in grid computing environments. Journal of Parallel and Distributed Computing. 2010; 70(2): 84-91.

[105] Sen, J. Security and privacy issues in cloud computing. Architectures and Protocols for Secure Information Technology Infrastructures. 2013: 1-45.

[106] Nagesh, OS, T Kumar, VR Vedula. A Survey on Security Aspects of Server Virtualization in Cloud Computing. International Journal of Electrical and Computer Engineering (IJECE). 2017; 7(3): 1326-1336.

[107] Islam, T, D Manivannan, S Zeadally. A classification and characterization of security threats in cloud computing. Int. J. Next-Gener. Comput. 2016; 7(1).

[108] Sutikno, T, D Stiawan, IMI Subroto. Fortifying big data infrastructures to face security and privacy issues. TELKOMNIKA (Telecommunication Computing Electronics and Control). 2014; 12(4): 751752.

[109] Gholami, A, E Laure. Security and privacy of sensitive data in cloud computing: a survey of recent developments. arXiv preprint arXiv:1601.01498. 2016.

[110] Lomotey, RK, R Deters. Saas authentication middleware for mobile consumers of iaas cloud. in Services (SERVICES). 203 IEEE Ninth World Congress on. 2013. IEEE.

[111] Kim, H, SC Timm X. 509 authentication and authorization in fermi cloud. in Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing. 2014. IEEE Computer Society.

[112] Nzanywayingoma, F, Q Huang. Securable personal health records using ciphertext policy attribute based encryption. in E-Health Networking, Applications and Services (Healthcom). 2012 IEEE 14th International Conference on. 2012. IEEE.

[113] Popa, RA, et al. Enabling Security in Cloud Storage SLAs with CloudProof in USENIX Annual Technical Conference. 2011.

[114] Garfinkel, T, et al. Terra: A virtual machine-based platform for trusted computing. in ACM SIGOPS Operating Systems Review. 2003. ACM.

[115] Santos, N, Kp Gummadi, R Rodrigues. Towards Trusted Cloud Computing. HotCloud. 2009; 9(9): 3.

[116] Zhu, S, G Gong. Fuzzy authorization for cloud storage. IEEE Transactions on Cloud Computing. 2014; 2(4): 422-435.

[117] Kim, KH, A Beloglazov, R Buyya. Power-aware provisioning of cloud resources for real-time services. in Proceedings of the 7th International Workshop on Middleware for Grids, Clouds and eScience. 2009. ACM.

[118] Pillai, P, KG Shin. Real-time dynamic voltage scaling for low-power embedded operating systems. in ACM SIGOPS Operating Systems Review. 2001. ACM.

[119] Martin, SM, et al. Combined dynamic voltage scaling and adaptive body biasing for lower power microprocessors under dynamic workloads. in Proceedings of the 2002 IEEE/ACM international conference on Computer-aided design. 2002. ACM.