2.1. VM Migration Techniques

Virtualization platforms generally utilize either a pre-copy live migration approach [1,2] or a post-copy live migration approach [3,4]. Of the two methods, the pre-copy method results in a shorter downtime, while the post-copy method results in a shorter total migration time. In Akoush et al. [5] and Elsaid et al. [6], the authors showed that the overall migration time for Stages 0–5 of the pre-copy method comprises four components, namely the Pre-Migration Overhead (Stages 0 and 1), the Stage 2 Overhead, the Stage 3 Overhead, and the Post-Migration Overhead (Stages 4 and 5). Among these components, the Stages 0/1/4/5 overhead time is static but not constant. Consequently, under the same VM migration conditions, the overhead times of the pre- and post-copy migration methods are slightly different.

2.2. Migration Performance Indicators

Common migration performance indicators include the total migration time, the migration down time, and the total number of VM migrations. The total migration time measures the elapsed time between the start of the first VM migration process and the end of the final VM migration process. Akoush et al. [5], Elsaid et al. [6], and Salfner et al. [7] investigated the effects of various factors on the total migration time, including the migration link bandwidth, the frequency of dirty memory pages, the VM memory size, and so on. The migration down time measures the time for which service is interrupted during the migration process, and should be minimized in order to maintain user satisfaction. Finally, the number of VM migrations indicates the total number of VMs migrated, and should be reduced to the minimum number possible to minimize the impact of the migration process.

2.3. TCP Window Size

Virtual machineware [8] prescribes the use of the transmission control protocol (TCP) to establish a connection for VM migration and transmit the corresponding migration traffic. TCP is a synchronous transmission protocol with good reliability and completeness. The TCP window size used to transmit the migration packets is assigned a minimum value of Congestion Window (CWND), receiver window (RWND), i.e., sendwin = min(cwnd, rwnd). Phanishayee et al. [9] presented a scheme for lowering the value of slow start threshold (ssthresh) in order to enter the congestion avoidance stage early. However, the system suffered an insufficient startup problem accordingly. Osada et al. [10] considered a serial transfer process rather than parallel server request units (SRU) transmissions. However, the method resulted in a low bandwidth utilization rate during the slow start cycle.

3.1. Sequential Method

Figure 1a shows a typical example of the sequential VM migration process. Let bandwidth delay product (BDP) be the product of the link bandwidth and the round-trip time. Assume that VM1 and VM2 have the same memory size and page dirtying rates (i.e., t_ss + t₁ is constant). Furthermore, assume that TCP ssthresh is larger than BDP. Thus, sendwin is equal to BDP in the slow start phase. The sum of sendwin for the two VMs is then obtained as shown in Figure 1b.

Figure 1

(a) Sequential method. (b) Sum of sendwin.

In the sequential migration method, there is no overlap between the migrations of VM1 and VM2, respectively. Consequently, the sum of sendwin in Figure 1b does not exceed BDP. As a result, the throughput is not degraded by overflows at the switch buffer. However, the bandwidth cannot be fully utilized during the slow start phase and the migration overheads of the two VMs cannot be processed at the same time. Consequently, the total migration time is increased. Furthermore, the operator cannot explicitly configure the time at which each VM migration task is to be performed. In other words, the sequential migration method lacks versatility.

3.2. Parallel Method

As shown in Figure 2a, in the parallel migration method, VM1 and VM2 are located on the same server and are transmitted at the same time over a single connection. The sendwin size of each VM is thus reduced to (1/2)BDP. Consequently, the sum of sendwin is equal to BDP, as shown in Figure 2b. It is seen in Figure 2b that t′_ss is less than t_ss. In other words, the parallel migration method improves the bandwidth usage during the slow start phase. In addition, the migration overheads of the two VMs can be processed at the same time. Hence, the total migration time is reduced.

Figure 2

(a) Parallel method. (b) Sum of sendwin.

3.3. Simultaneous Method

Figure 3a illustrates the simultaneous migration method, in which the migrations of VM1 and VM2, which are located on different servers, are executed in a many-to-one communication mode. Assuming that the links between the source servers and the switch have the same bandwidth, the sum of sendwin reaches BDP after just t′, as shown in Figure 3b. Notably, t′ is less than t_ss. In other words, the migration process results in a long overlap time between the transmissions of the two VMs. Consequently, the switch buffer easily overflows; resulting in packet drops and a significant reduction in the throughput. Thus, the total migration time of the simultaneous migration process is longer than that of the parallel migration process.

Figure 3

(a) Simultaneous method. (b) Sum of sendwin.

3.4. Delay Method

As shown in Figure 4a, the delay migration method is similar to the simultaneous migration method other than for the fact that a short delay is introduced between the migrations of the two VMs. The delay method reduces the overlap time of the multiple TCP connections (see Figure 4b), and therefore smooths the traffic volume and improves the throughput. However, the total migration time is usually much longer than that of the simultaneous migration method.

Figure 4

(a) Delay method. (b) Sum of sendwin.

Table 1 lists the difference of the performance and total migration time in the four approaches.

4.1. Experiment Environment

Migration experiments were performed using three HP ProLiant DL380 G6 physical servers (two sources and a destination) and hypervisor VMware vSphere 5.5 software (Palo Alto, California, USA). The experiments replicated the network architectures and migration methods shown in Figures 1a–4a. Each VM was configured to have eight virtual CPUs and 14 GB of virtual RAM. Memory testing tool was used to increase the memory loads of the VMs in the range of 2–11 GB. The total migration time of the VMs under different loads was measured.

4.2. Experiment Results

Figure 5 shows the experimental results obtained for the variation of the total migration time with the VM memory load for each of the considered migration methods. The results support the following main conclusions:

1. (1)

   The migration methods can be ranked in terms of the total migration time (shortest to longest) as follows: Parallel < Simultaneous < Delay < Sequential.

2. (2)

   For a memory load of 11 GB, the total migration time of the simultaneous method is close to that of the sequential method.

3. (3)

   The delay method generally results in a longer total migration time than the simultaneous method. However, the performance of the delay method depends on the delay time applied. For example, given a memory load of 11 GB and a delay time of 120 s, the delay method results in a shorter total migration time than the simultaneous method.

Figure 5

Total migration times of different VM migration methods.

Figure 6 shows the throughputs obtained for each of the four migration methods given a VM memory load of 4 GB. The migration methods can be ranked in terms of the throughput performance (high to low) as follows: Parallel ≈ Sequential > Delay > Simultaneous.

Figure 6

Throughputs obtained under different VM migration methods.

4.3. Migration Selection Strategy

Based on the experimental results presented above for the total VM migration time, the following migration selection strategies are proposed:

1. (1)

   Selection strategy 1: If the switch-server 3 link bandwidth is much higher than that of the source-switch links:

   Selection order: Simultaneous → Parallel → Delay → Sequential method.

2. (2)

   Selection strategy 2: If the bandwidths of the links between the switch and the servers (sources and destination) are the same:

   Selection order: Parallel → Simultaneous → Delay → Sequential method.