Virtualization and Cloud Provider Comparison

We are going to start by covering some of the core terms which are used when talking about virtualization, just to ensure everyone is on the same page.

Next, we will cover some of the main technologies that are used with VMs.

Hardware assisted virtualization is a common term used to talk about the technologies that have been added to system architectures, such as Intel and AMD processors, system board chipsets, etc. This was done to reduce the software overheard of VMs when addressing hardware such as network cards to support more PCIe devices in the guest OS and to streamline IO from the host machines.

Over 10 years ago, Intel released the first two CPUs that supported VT-x which were Pentium 4s. AMD followed not long after in 2006 when they released their Athlon 64 processors that supported AMD-V. These technologies have been enhanced and extended since then to bring us to where we are now.

We come to the standard virtualization stack.

A host at the bottom, which is a physical server box or blade-type server in a chassis and contains the virtualization technologies that were talked about.

Running on this is the hypervisor, such as ESX, Hyper-V or Xen, etc.

Then we have our various guest operating systems, which can be Windows or Linux, etc.

Finally, we have the applications or services those guests are providing, such as email, web, Cinegy, etc.

What benefits does virtualization offer you, and why would you deploy these types of environments?

Now we move onto talking about the various offerings available in the marketplace. We are going to concentrate on the main three vendors in this area, which are:

Obviously, as mentioned before, there are others in this space, such as the Linux KVM platform. However, this doesn’t tend to get used in the same large scale deployments as these three and doesn’t come with the same type of features, and so we won’t be including it here today.

If we use our stack model from earlier, then we can compare the offerings at each of these levels to give us some idea of the differences between them.

At the bottom, we had the host or physical box, so we will start by comparing the three offerings at this level.

We will start with Microsoft as they have the most general of requirements.

Choosing hardware that you wish to use for your virtualization platform should be reasonably straight forward.

There is plenty of information to allow you to ensure that hardware will be suitable and compatible with XenServer and vSphere and their compatibility guides.

Microsoft ‒ you need to make sure you have correct hardware capabilities in the CPU, system board and peripherals to allow you to use the features you wish and install the product.

VMware and Citrix have specific community resources for additional compatibility information which can be a great help.

Citrix offer a self-certification scheme for hardware vendors, so you could ask them to go through this process for you.

VMWare also have the partner verified and supported products, which although doesn’t have many items could also be helpful.

Citrix also have their Citrix-Ready Marketplace covering a wide range of both hardware and software.

Our stack picture again and we are now moving up to the Hypervisor level.

The next level up on our stack is the hypervisor.

Now this is either a straight installation such as vSphere, XenServer or Hyper-V Server or the addition of the Hyper-V role to an installation of Windows Server 2016.

We are going to list out some of the features of the hypervisor that are available. You will get the best availability of features with generation 2 VMs running the latest versions of the guest operating system. Also, some features rely on hardware capabilities or associated infrastructure.

vSphere is VMWare’s product which uses their ESX hypervisor.

This comes in three different versions that offer access to different features and different maximum amounts. These are Standard, Enterprise Plus, and Operations Management Enterprise Plus.

We will list some of the features again, and highlight what the differences may be with the different versions of vSphere.

Some of the features are:

The Xen hypervisor comes in two versions: standard or enterprise.

Our stack picture again and we are now moving up to the guest level:

Obviously, Hyper-V supports a lot of Windows versions. vSphere and XenServer also have similar levels of support.

Hyper-V also now supports a wide variety of Linux distributions. To make the most of the features, it is best to use Linux Integration Services or the FreeBSD Integration Services drivers collection. This have been integrated into the kernel in new releases and is updated.

vSphere and XenServer support for Linux is better than Hyper-V, but there are only some minor differences in the distribution list.

We are not going to list out all the Linux versions supported, but you can run distributions from:

Again, the newest version of these OS’s allows use of the most number of features from the hypervisor.

There are four key factors affecting your possible choice of hypervisor to use:

We have looked at the features, and what guest OS’s are available for the hypervisors. Now let’s look at some number comparisons.

Here we have some of the numbers for the maximums supported on hosts for some of the hypervisor versions. We have included 3 versions for vSphere as they represent different price points for deployment.

Next, we have some of the maximums that are possible for guests running on the hypervisors.

So, we have covered the capabilities of the hypervisors and some of the maximums that are possible with them. The last comparison method that we can use is the cost.

Total cost for Hyper-V datacenter with management is $9,762 for a 2 processor – 16 core model which would allow you to run as many VMs as the box would support along with management of them.

Total cost for the Enterprise Plus version of vSphere which gives us the graphics pass-through feature we want on a 1 CPU server with 24x7 support and a management server to go along with it is $13,997.

For a 2 CPU server, we add on another $6000 to the price that gives us a total of basically $20,000 which is double the Hyper-V cost.

Obviously, Enterprise Plus is the most expensive version of vSphere available, but this is the only version which provides the graphics pass-through capability.

It depends! Some of the questions that you need to ask are:

One of the aspects that needs to be considered when running Cinegy in a virtualization environment is that you will want to make use of our H.264 offloading as much as possible to maximize the software capabilities.

NVIDIA brought in some changes when they introduced that GRID 2.0 initiative.

The three software licensing models are virtual applications, virtual PC, and virtual workstation:

They introduced a new licensing cost which originally brought some consternation from the community due to the amounts. This has been revised down since then to a lower level.

Each of these costs is for a concurrent user license which you need 1 of per active session on the GPU.

Annual subscription includes license and Support, Updates and Maintenance (SUMS) for that year.

The perpetual license is just that, the license for ever, but SUMS is only required in the first year and then purchasable annually.

For a VM environment, you are going to need a Virtual Workstation license to give GPU pass-through and allow access to CUDA and OpenCL.

Finally, you have the hardware platform:

The two main Cloud providers that people tend to use are Amazon and Microsoft. Whilst Google do have their own offering, in our experience customers don’t tend to consider it.

For an apples to apples type comparison, it is best to use the cost of compute services from the public cloud providers. This tends to make up the biggest chunk of spend for people in those environments.

These instances are running Linux in the us-east region with SSD storage. The underlying hardware is:

So, here we can see that using the on-demand costs Azure comes out as cheaper for each of these types.

You can improve these costs by using things like Reserved Instances for AWS, or if you have a Microsoft Enterprise Agreement for Azure. However, these lock you in to a certain degree.

Changing over to running Windows on these instances attracts a premium from each of the vendors as you would expect. The increase per year for the two vendors is about $1,100.

The other comparison that can be used is the level of services available per region. GPU-based instances, for example, are not available in AWS newest regions, such as Mumbai and Canada; and for Azure the availability is even less with only some of the USA regions having availability.

What have we learnt from our time of putting things into the cloud and conversation we have had with customers?

One of the main ones is the level of drift you can get with the clocks of the machines running.

The second one which has shown itself to pose a challenge is a packet loss. Because cloud providers use software-defined networking, which must be reconfigurable to provide the connectivity required to the various VMs, the packet loss is something which should be factored in. This is a problem when using UDP for your traffic, and packets need to arrive in the same order as sent. Some numbers demonstrating this are shown below.

Having said that, you should build for failure. It is possible that some packets might get lost, it is possible that an underlying host will go down taking your instance with it. It is possible that your instance will develop a fault and become unresponsive.

As we mentioned earlier, you need to check that the service you want to use is available in the areas of the world that you wish to deploy into. The GPU-backed instance is one example of this.

Don’t think that because you can’t do something in the cloud today that you won’t be able to do it in the cloud next month. New services are being added by the providers at quite a high rate and existing services are being enhanced or added to as well.

Keeping up with all the changes that this technology brings is quite the task.

In this type of environment, you need to rely on either the underlying real time clock of the hosts or sync your OS clock against another accurate time source.

There are various methods of ensuring the accuracy of the OS clock, and one of these is to use a third-party application which would run on your machine and ensure the accuracy.

This is the drift graph of the Domain Time client running on a standard G2 instance in the Frankfurt region.

It is set to query the various NTP server pools run by Amazon and to achieve a target maximum of 5 ms of difference for this machine.

As you can see, when software starts, there is a large amount of difference between the system clock and the reference servers, and whilst this is brought down, it still takes quite a few checks before the clock is within the configured limit.

We then move into a period of minor adjustments needed and relative stability, then again, some larger variance in the clock.

This is something of great importance when you are running playout from cloud to ensure that the packets leave the machine in a consistent manner. It also allows sending and receiving machines to agree regarding packets sent.

The two blue lines are the upper and lower limits of the difference detected, and the green line is the current average offset that the machine clock has to the reference clock.

Here we can see the results of software running for a prolonged period and time. This software provides a feature which adapts an ongoing clock difference between the machine and the reference clock on the machine it is running on:

So, now the green line which is the average time difference being achieved is much closer to zero; the level of variance is much less, and the number of large differences is also reduced, but they are still present, so this is something that needs to be monitored.

Therefore, it would be best to ensure that the machine has achieved a standard clock accuracy before you move onto running it in a production environment.

Packet loss is another challenge which would need to be considered when operating in the cloud environment.

We have found that the amount of loss you experience does vary between different regions of the provider and could be attributed to both the age of the datacenter installation and the load that the hosts in the datacenter are under.

We ran some standard iPerf tests between instances running in the AWS US-EAST-1 region, and the type of loss that we experienced was most pronounced when we were using 8K packet sizes.

Here you can see the way in which the packet loss manifests itself. We get a drop in the throughput, then a complete absence of any traffic, and then an attempt to catch up which then generates the loss in packets:

The total sent and loss for this 24-hour test was 316 packets loss out of a total 19,775,391 sent. Whilst this may not seem like much, it is the way in which the loss occurred that was problematic as we had a one-second period where no packets appeared to flow:

When we reran these tests using a 1K packet size, the loss either disappeared (so no packets were lost when 158,202,970 packets were sent) or the number was the odd one or two here or there, which is fixable using mechanisms such as forward error correction.

That brings us to the end of this rather long post. We hope that some of the information found here is useful and helps you make a more informed decision on whether virtualization or the public cloud, or a hybrid of the two is a good fit for your plans.