Monitoring CPU Utilization Under Hyper-threading

The question of accurately measuring processor utilization with hyper-threading (HT) enabled came up recently in a Performance Engineering Group discussion on Linked-in. Since I spent some considerable time looking into this issue while writing my Guerrilla Capacity Planning book, I thought I'd repeat my response here (slightly edited for this blog), in case it's useful to a broader audience interested in performance and capacity management. Not much has changed, it seems.

In a nutshell, the original question concerned whether or not it was possible for a single core to be observed running at 200% busy, as reported by Linux top, when HT is enabled.

---

This question is an old canard (well, "old" for multicore technology). I call it the "Missing MIPS" paradox. Regarding the question, "Is it really possible for a single core to be 200% busy?" the short answer is: never! So, you are quite right to be highly suspicious and confused.

You don't say which make of processor is running on your hardware platform, but I'll guess Intel. Very briefly, the OS (Linux in your case) is being lied to. Each core has 2 registers where inbound threads are stored for processing. Intel calls these AS (Architectural State) registers. With HT *disabled*, the OS only sees a single AS register as being available. In that case, the mapping between state registers and cores is 1:1. The idea behind HT is to allow a different application thread to run when the currently running app stalls; due to branch misprediction, bubbles in the pipeline, etc. To make that possible, there has to be another port or AS register. That register becomes visible to the OS when HT is enabled. However, the OS (and all the way up the food chain to whatever perf tools you are using) now thinks twice the processor capacity is available, i.e., 100% CPU at each AS port.

New Performance Papers from VMware

Two new performance whitepapers from VMware:

1. VMware vSphere 4: The CPU Scheduler in VMware ESX 4
   
2. Understanding Memory Resource Management in VMware ESX Server:
   

Both apply controlled measurements ("Experimental Environment"), as described in my latest Linux Technical Review article.

Performance und Virtualisierung

Autor: Neil J. Gunther
Erschienen: 15.09.2009
Umfang: 13 Seiten

Woran denkt man, wenn man den Begriff Virtualisierung hört? An einen Hypervisor wie Citrix XenServer oder den ESX-Server von VMware? Oder an virtualisierte Services wie beim Cloud Computing? Oder an Multicore-CPUs mit Hyperthreading, die virtuelle Prozessoren ermöglichen? Am besten betrachtet man all diese Erscheinungsformen von Virtualisierung nicht isoliert, sondern als Teile eines einzigen Performance-Management-Puzzles. Dieser Beitrag erklärt wieso und er unterstreicht, wie wichtig es ist, durch kontrollierte Performance-Messungen Daten zu sammeln.

"Virtualization in the Enterprise from the Performance Management Perspective"

When you see the word virtualization, what do you think of? Hypervisors like Citrix XenServer or VMware ESX? Perhaps you thought of virtualized services like cloud computing? What about hyperthreaded multicores that facilitate virtual processors? Rather than thinking of all these forms of virtualization as being completely different from one another, this article explains why it's better to think of them as being pieces of the same performance management puzzle.The importance of doing controlled performance measurements is also emphasized.

Linux Technical Review

Learning to Live with Virtualization

Comment at ArsTechnica:

"One of the big things that I've learned and that's been recently reinforced for me, is that you need a whole mindset change about how you build servers, how you evaluate what goes into your standard builds, how you monitor...

Think, for example, about a daemon that wakes up once an hour to check on something (it doesn't matter what...). Once an hour's not much, right? Except you have 100 guests, so that's once every 36 seconds, on average. Still sound like a lightweight process? Still need to be in your standard build? Likewise a daemon that eats 100 MB of RAM, or blindly installing packages because you "might need to use gcc some time". Yeah, maybe you might. Is it worth the storage you just ate?"

Couldn't agree more; especially the part about different monitoring requirements for VMMs.

VMware CEO Walks

VMware CEO, Diane Greene, stepped down abruptly on Tuesday, sending shares down nearly 25%. No reason was given but the company had warned revenue for the quarter ending June 30 would be below its predicted 50% growth compared with last year. It could also be partly a reaction to the commercial marketplace heating up with Microsoft, Sun Microsystems, Hewlett-Packard and Citrix (backed by IBM) having all built or acquired similar virtualization technologies.

On the FOSS scene, we have VirtualBox as a replacement for VMWare Fusion or Parallels.

Pay per VPU Application Development

So-called "Cloud Computing" (aka Cluster Computing, aka Grids, aka Utility Computing, etc.) is even making the morning news these days, where it is being presented as having a supercomputer with virtual processor units just a click away on your home PC. I don't know too many home PC users who need a supercomputer, but even if they did and it was readily available, how competitive would it be given the plummeting cost of multicores for PCs?

Beware VMWare!

Hot on the heels of the Cisco hyper-mega-switch announcement and Yahoo announcing a reduction in workforce despite increasing profits, comes the nose-dive of VMWare's stock price; losing one-third it's value in a single day. Keeping in mind that Wall Street is capable of either irrational exuberance or irrational pessimism, the events pertaining to Yahoo and VMWare are most likely an expression of the latter. Both companies have solid business plans.

From the performance angle, however, it may also be that the honeymoon period is now over in part because VMWare is a victim of its own success. Not only has all the hype surrounding virtualization and consolidation worn a bit thin these days, but it has also attracted the big guns---Microsoft and Oracle---into the market. And customers are not doubt becoming aware that consolidation doesn't always translate into less MIPS or more greenness. The overheads of virtualization can be very significant. The problem for us performance weenies is knowing what are those overheads in a QUANTITATIVE way. As I've stated before:

All virtualization is about illusions and although it is perfectly reasonable to perpetrate such illusions onto a user, it is entirely unreasonable to propagate those same illusions to the performance analyst.

So, here's an opportunity for VMWare to differentiate itself in the madding crowd of new VMM vendors; focus on providing more whistles and less bells. And you (Dear reader), as John Q. Customer/Analyst, should demand it (even if by proxy). If you don't make the ultimate performance issue known to management, how can they be expected to pressure the vendors?

Cisco Systems: "It's the switch, stupid!"

Today, Cisco Systems (San Jose, California) announced its mother of all switching platforms, the Nexus 7000 Series, aimed at what it calls Data Center 3.0 (analogous to Web 2.0, I presume. I missed Data Center 2.0). Cisco is essentially trying to eliminate the need for separate storage networks, server networks, routing, switching and virtualization, by combining them all into a single unified fabric and managing it through Cisco's new proprietary NX-OS ("nex-os", get it?) operating system.

The Nexus 7000 will deliver up to 15 Tbps of switching capacity in a single chassis, with 512 ports for 10 Gbps ethernet, and eventually it is slated to be delivered with 40 Gbps and 100 Gbps ports. Some of the claimed performance speeds-and-feeds appear rather breathtaking:

• Copy the entire Wikipedia database in 10 milliseconds.
  
• Copy the entire searchable Internet in 7.5 Minutes.
  
• Download all 90,000 Netflix movies in 38.4 seconds.
  
• Send a high-resolution 2 megapixel photo to everyone on earth in 28 minutes.
  
• Add a Web server in 9 seconds rather than 90–180 Days.
  
• Transmit the data in all U.S. academic research libraries (estimated at more than 2,000 TB) in 1.07 seconds.
  

If nothing else does it, the 3 significant digits in the last claim tells you this is marketing-speak (read: calculated using max bandwidth assumptions), so a liberal dusting of sodium chloride is recommended.

The concept of a "data center" is currently undergoing a serious transformation and it will be interesting to see how this kind of mega-switch stacks up against alternative approaches, such Google's Data Center in a Box.

Virtualization Rootkit Wars

VMM malware is another side-effect of creating illusions (See my previous blog entry on the danger of illusions). It turns out that still waters run very deep. Here's a potted summary of some recent events in the world of stealth that have impinged on both VMM security issues and performance analysis. (The following contains a lot of acronyms, for which I've provided a glossary at the end).

Last year at BlackHat, some Polish security experts announced a proof-of-concept for a VME rootkit called "Blue Pill " (BP) that they claimed was undetectable. For BlackHat 2007, some U.S. security experts challenged the Polish team to a Detect-A-Thon (my term). This caused the Polish team to go into defensive posture and make a list of run-rules (my term) for how the Detect-A-Thon was to be carried out. Since BP is only a virtual rootkit (if I can use that term), one of the proposed run-rules was payment (up front?) of almost $500,000 for development costs to make a real implementation of BP battle ready. Nice work if you can get it.

Quite apart from all these claim-counter-claim machinations, what got my attention was one of the ways by which the U.S. team claimed that BP would be detectable (there are plausibly many) viz., counting execution cycles. The CPUID instruction, in particular, is supposed to only take 200 cycles (as root), not 5000 cycles (non-root). I saw a certain irony in the fact that, although I've been complaining about VMM illusions masking correct performance analysis, performance analysis is one method for detecting HVM malware. The procedure is analogous to the analysis in Section 3.2.2. of my CMG 2006 paper "The Virtualization Spectrum from Hyperthreads to GRIDs" where I showed that the increase in thread execution time is due mostly to an inflation of the thread service time on a dual-core. There, I had to infer the effect from system-level measurements whereas here, they are talking about reading the actual cycle counter/register directly. It turns out that this technique is not totally foolproof either, because the timings can be masked with the appropriate trap. Looking for changes in the TLB is another method that has been proposed. Naturally, in this kind of game, the beat goes on and although rootkit detectors are already available, there will be many more as VMM stealth techniques evolve.

Glossary

• BP: "Blue Pill". An HVM rootkit.
  
• CPUID: x86 instruction to identify the CPU type.
  
• Guest: VMWare lingo for a native O/S that runs on a VMM.
  
• HVM: Hardware-Assisted Virtual Machine.
  
• Hyperjacker: Hypervisor hijacking.
  
• Hypervisor: See VMM.
  
• Malware: Malicious software. A stealthy rootkit in this context.
  
• Rootkit: A set/kit of tools/executibles with root access (highest privilege).
  
• TLB: Translation Look-aside Buffer.
  
• VME: Virtual Machine Emulators e.g, "Blue Pill", "Vitriol".
  
• VMM: Virtual Machine Monitor e.g., VMWare, Xen.
  

My Message to Virtualization Vendors

Virtualization is about creating illusions (see Chapter 7 in Guerrilla Capacity Planning). However, vendors need to recognize that virtualization is a double-edged sword.

Constructing illusions by hiding physical information from users is one thing, propagating that illusion to the performance analyst or capacity planner is quite another, and considered harmful.

Presumably, it's also potentially bad for business, in the long run. This unfortunate situation has arisen for one of the following reasons:

1. The performance data that is available is incorrect.
   

   
   Example: Enabling hyperthreading on a Xeon processor misleads the operating system, and thereby performance tools, into treating the single core as 2 virtual processors. This means that many performance management tools will conclude and report that your system has 200% processor capacity available. But, this is an illusion, so you will never see 200% processor capacity.
   

   
   
2. The correct performance data is not made available.
   

   
   Example: With hyperthreading enabled, there should be a separate register or port that allows performance management tools to sample the actual utilization of the single physical core (AKA the execution unit). The IBM Power-5 has something called the PURR register that performs this role.
   

   
   

There are many examples of this kind of mangled performance shenanigans in the virtualized world, especially in what I call the meso-VM (PDF) level such as VMware and XenSource. The good news there is, since it's software, it's easier to modify and therefore more likely that actual performance data will become exposed to the analyst.

In other words:

Fewer bells, more whistles

should be the watchword for virtualization vendors.

Overview of Virtualization Performance

As the Guest Editor for this month's MeasureIT e-zine on the topic of virtualization, a compliation of articles is presented from both earlier MeasureIT authors as well as some papers from the CMG conference proceedings. Titles include:

• Visualizing Virtualization


• It May Be Virtual - But the Overhead is Not


• A Realistic Assessment of the Performance of Windows Guest Virtual Machines


• Measuring CPU Time from Hyper-Threading Enabled Intel Processors


• Hyperthreading - Two for the Price of One?


• To V or Not to V: A Practical Guide To Virtualization


• The Virtualization Spectrum from Hyperthreads to GRIDs

This issue of MeasureIT is unique in my mind because it is rare to find, in one place, such a broad collection of performance perspectives centered on the intensely hot topic of virtualization.