SOA Scalability and Steady-State

Guerrilla alumnus Peter Lauterbach just brought to my attention an article in SOA World entitled "Load Testing Web Services". I have to commend these authors for performing their SOA load tests in steady state. Elsewhere, I've discussed how wrong things can go when you don't adhere to this procedure. In their online article, these authors show the response time (R) as a time-series plot, more or less as it would appear in a measurement tool like say, LoadRunner. Although they don't show it, the throughput measurements would also look similar when plotted as a function of time (t).

Best Practices Are An Admission of Failure

Six Sigma: Quite a list.

ITIL: Best Practice is defined as "good working practice developed through consensus that helps organizations to achieve better performance.”

Sounds good, but ...

Ludwig Wittgenstein: "Just because we all agree on something, doesn't make it true."

Therefore ...

Guerrilla Manual 1.21: Best Practice is tantamount to not trying to understand the problem. Merely copying someone else's apparent success is like cheating on a test. You might make the grade, but how far is the bluff going to take you?

So ...

Thomas Edison: "There's a better way. Find it!"

Black Swans, Instantons, Hedge Funds and Network Collapse

On my flight to Europe last July, I read The Black Swan: The Impact of the Highly Improbable by N. Taleb. Unfortunately, I found the book irksome for several reasons:

• I already knew the mathematical underpinnings of the metaphors used in the book (more on that below).
  
• Taleb's writing style is unnecessarily condescending toward others mentioned in the book and to the reader.
  
• Some rather obvious points are labored. The weirdest of these comes in the form of a entirely fictitious character to which an entire chapter is devoted.
  
• Many of his often poor and sometimes inaccurate examples kept reminding me of something a Stanford mathematician once told me: "Economists are mathematically unsophisticated."
  
• He describes a general problem or syndrome related to how people assess risk incorrectly, but he doesn't really offer any solutions (or maybe I missed it in the chapter entitled, "How to Look for Bird Poop" ... seriously).
  

I must say this book was a disappointment because it was a stark contrast to seeing him interviewed months earlier on PBS, where he came across as more thoughtful and measured. My opinion notwithstanding, you might find the book worth reading because it's an easy read, it covers many topics (mostly with a financial slant—the author's background), and he's also warning the reader about the dangers of things like high-risk hedge funds. Moreover, as I shall try to demonstrate here, these same concepts also impinge on performance analysis (not that Taleb is aware of that) and whereas they might otherwise be impenetrable to the non-mathematician, possibly they are made a little more accessible in a book like this. In a nutshell, I believe he is saying: Think wild, not mild; easy to say, hard to do, as I shall try to explain.

Virtualization Rootkit Wars

VMM malware is another side-effect of creating illusions (See my previous blog entry on the danger of illusions). It turns out that still waters run very deep. Here's a potted summary of some recent events in the world of stealth that have impinged on both VMM security issues and performance analysis. (The following contains a lot of acronyms, for which I've provided a glossary at the end).

Last year at BlackHat, some Polish security experts announced a proof-of-concept for a VME rootkit called "Blue Pill " (BP) that they claimed was undetectable. For BlackHat 2007, some U.S. security experts challenged the Polish team to a Detect-A-Thon (my term). This caused the Polish team to go into defensive posture and make a list of run-rules (my term) for how the Detect-A-Thon was to be carried out. Since BP is only a virtual rootkit (if I can use that term), one of the proposed run-rules was payment (up front?) of almost $500,000 for development costs to make a real implementation of BP battle ready. Nice work if you can get it.

Quite apart from all these claim-counter-claim machinations, what got my attention was one of the ways by which the U.S. team claimed that BP would be detectable (there are plausibly many) viz., counting execution cycles. The CPUID instruction, in particular, is supposed to only take 200 cycles (as root), not 5000 cycles (non-root). I saw a certain irony in the fact that, although I've been complaining about VMM illusions masking correct performance analysis, performance analysis is one method for detecting HVM malware. The procedure is analogous to the analysis in Section 3.2.2. of my CMG 2006 paper "The Virtualization Spectrum from Hyperthreads to GRIDs" where I showed that the increase in thread execution time is due mostly to an inflation of the thread service time on a dual-core. There, I had to infer the effect from system-level measurements whereas here, they are talking about reading the actual cycle counter/register directly. It turns out that this technique is not totally foolproof either, because the timings can be masked with the appropriate trap. Looking for changes in the TLB is another method that has been proposed. Naturally, in this kind of game, the beat goes on and although rootkit detectors are already available, there will be many more as VMM stealth techniques evolve.

Glossary

• BP: "Blue Pill". An HVM rootkit.
  
• CPUID: x86 instruction to identify the CPU type.
  
• Guest: VMWare lingo for a native O/S that runs on a VMM.
  
• HVM: Hardware-Assisted Virtual Machine.
  
• Hyperjacker: Hypervisor hijacking.
  
• Hypervisor: See VMM.
  
• Malware: Malicious software. A stealthy rootkit in this context.
  
• Rootkit: A set/kit of tools/executibles with root access (highest privilege).
  
• TLB: Translation Look-aside Buffer.
  
• VME: Virtual Machine Emulators e.g, "Blue Pill", "Vitriol".
  
• VMM: Virtual Machine Monitor e.g., VMWare, Xen.
  

MVA, Upgrades, and Other Visitations Upon PDQ

Guerrilla alumnus Sudarsan Kannan asks:

"I'm trying to understand concepts behind performance modeling (analytical modeling) based on MVA algorithm. ...I'm also trying to understand WHAT IF scenarios such as: What if I increase my CPU speed or upgrade Disk I/O subsystem? What impact will the increase in CPU speed have on throughput, response time and more variables? What if I increase number of users. I have couple of questions to get a better picture of MVA algorithm:

1. How to find visit ratios for CPU?
   
2. Can I vary service time (S) for a resource (CPU or disk) if I increase/decrease the processor/disk speed to answer WHAT IF scenarios?"
   

Why Doesn't PDQ Have a GUI?

Recently I was asked if I planned to create a GUI (Graphical User Interface) for PDQ. I've thought a lot about this over the years and my answer (still) is negatory and here's why.

Darwin’s Dictum and Performance Monitoring

Darwin’s Dictum:

"All observation must be for or against some view if it is to be of any service." (Source: Scientific American magazine, April 2001)

Translation:

All performance monitoring must either agree or disagree with some performance model* if it is to be of any use.

* A performance model could be any or all of a SWAG, a back-of-the-envelope calculation, an Excel spreadsheet, a Mathematica notebook, an S-plus script, a PDQ model, a SAS model, a SimPy simulation, a LoadRunner script, etc.

I'll have a lot more to say about this during my CMG Sunday Workshop entitled Moving Beyond Monitoring, Pretty Damn Quick! (December 2, 2007).

To BEA or Not to BEA?

For all you WebLogic and Tuxedo lovers, here's another fine example of why I keep saying, we performance weenies can't afford to operate in a computing cloister. We have to keep a weather eye on the machinations of the marketplace.

Billionaire activist investor Carl Icahn (Wasn't he in the movie "Corporate Raiders of the Last Deutschmark"?) has called for the sale of BEA Systems Inc., whose stock price has sagged with the growth in open-source software and under pressure from larger competitors such as IBM Corp. and Oracle Corp. Analysts said BEA has failed to stir enthusiasm among investors. For example, Rob Enderle, principal analyst with Enderle Group, a technology consulting firm in San Jose, Calif. stated, "This class of software, because of open source, it's much harder to get people interested in it unless you're doing phenomenally well in sales...which BEA has not been."

Also on Friday, BEA said it has received an additional notice from the Nasdaq that it remains out of compliance because of the delayed filings and its shares remain in danger of being delisted.