New Performance Papers from VMware

Two new performance whitepapers from VMware:

1. VMware vSphere 4: The CPU Scheduler in VMware ESX 4
   
2. Understanding Memory Resource Management in VMware ESX Server:
   

Both apply controlled measurements ("Experimental Environment"), as described in my latest Linux Technical Review article.

VMware CEO Walks

VMware CEO, Diane Greene, stepped down abruptly on Tuesday, sending shares down nearly 25%. No reason was given but the company had warned revenue for the quarter ending June 30 would be below its predicted 50% growth compared with last year. It could also be partly a reaction to the commercial marketplace heating up with Microsoft, Sun Microsystems, Hewlett-Packard and Citrix (backed by IBM) having all built or acquired similar virtualization technologies.

On the FOSS scene, we have VirtualBox as a replacement for VMWare Fusion or Parallels.

Paravirtualization in VMWare Server

Guerrilla alumnus Peter Lauterbach provides some updates on improved performance transparency in VMware ESX Server 3.5 (the bare metal virtualization product).

"Paravirtualization has been available in the VMware Workstation product since version 6, but it also requires guest kernel changes. Linux kernel 2.6.20 and above support pv, look for CONFIG_VMI=y in kernel config. The latest Ubuntu release has this enabled by default. You also need to check the paravirtualized kernel flag in the VMware Workstation and ESX Server Advanced settings under the Options tab.

Paravirtualization in Linux has quite a few moving parts to it, and affects mostly performance metrics and timekeeping functions. The best example is %steal time, which is the time that the virtual guest wanted to run, but was not able to, usually due to contention with other VM guests. This time was previously only visible to the virtual host, but can now be seen by the guest. Paravirtualized guests do run faster, and certainly feel more responsive when using a Linux GUI like KDE on a virtual guest.

Other advances from VMware are things like "VMDesched", which is an experimental feature in VMware Tools that creates a thread that is charged all the involuntary preemption time from the physical CPU."

Thanks, Peter.

Beware VMWare!

Hot on the heels of the Cisco hyper-mega-switch announcement and Yahoo announcing a reduction in workforce despite increasing profits, comes the nose-dive of VMWare's stock price; losing one-third it's value in a single day. Keeping in mind that Wall Street is capable of either irrational exuberance or irrational pessimism, the events pertaining to Yahoo and VMWare are most likely an expression of the latter. Both companies have solid business plans.

From the performance angle, however, it may also be that the honeymoon period is now over in part because VMWare is a victim of its own success. Not only has all the hype surrounding virtualization and consolidation worn a bit thin these days, but it has also attracted the big guns---Microsoft and Oracle---into the market. And customers are not doubt becoming aware that consolidation doesn't always translate into less MIPS or more greenness. The overheads of virtualization can be very significant. The problem for us performance weenies is knowing what are those overheads in a QUANTITATIVE way. As I've stated before:

All virtualization is about illusions and although it is perfectly reasonable to perpetrate such illusions onto a user, it is entirely unreasonable to propagate those same illusions to the performance analyst.

So, here's an opportunity for VMWare to differentiate itself in the madding crowd of new VMM vendors; focus on providing more whistles and less bells. And you (Dear reader), as John Q. Customer/Analyst, should demand it (even if by proxy). If you don't make the ultimate performance issue known to management, how can they be expected to pressure the vendors?

My Message to Virtualization Vendors

Virtualization is about creating illusions (see Chapter 7 in Guerrilla Capacity Planning). However, vendors need to recognize that virtualization is a double-edged sword.

Constructing illusions by hiding physical information from users is one thing, propagating that illusion to the performance analyst or capacity planner is quite another, and considered harmful.

Presumably, it's also potentially bad for business, in the long run. This unfortunate situation has arisen for one of the following reasons:

1. The performance data that is available is incorrect.
   

   
   Example: Enabling hyperthreading on a Xeon processor misleads the operating system, and thereby performance tools, into treating the single core as 2 virtual processors. This means that many performance management tools will conclude and report that your system has 200% processor capacity available. But, this is an illusion, so you will never see 200% processor capacity.
   

   
   
2. The correct performance data is not made available.
   

   
   Example: With hyperthreading enabled, there should be a separate register or port that allows performance management tools to sample the actual utilization of the single physical core (AKA the execution unit). The IBM Power-5 has something called the PURR register that performs this role.
   

   
   

There are many examples of this kind of mangled performance shenanigans in the virtualized world, especially in what I call the meso-VM (PDF) level such as VMware and XenSource. The good news there is, since it's software, it's easier to modify and therefore more likely that actual performance data will become exposed to the analyst.

In other words:

Fewer bells, more whistles

should be the watchword for virtualization vendors.