Guerrilla 2018 Classes Now Open

All Guerrilla training classes are now open for registration.

1. GCAP: Guerrilla Capacity and Performance — From Counters to Containers and Clouds
2. GDAT: Guerrilla Data Analytics — Everything from Linear Regression to Machine Learning
3. PDQW: Pretty Damn Quick Workshop — Personal tuition for performance and capacity mgmt

The following highlights indicate the kind of thing you'll learn. Most especially, how to make better use of all that monitoring and load-testing data you keep collecting.

• How to save millions of dollars with a one-line performance model (video)
• How to minimize chargeback after you lift and shift to the cloud (video)
• How to correctly emulate web traffic on a load-testing rig (PDF)

See what Guerrilla grads are saying about these classes. And how many instructors do you know that are available for you from 9am to 9pm (or later) each day of your class?

Who should attend?

• IT architects
• Application developers
• Performance engineers
• Sysadmins (Linux, Unix, Windows)
• System engineers
• Test engineers
• Mainframe sysops (IBM. Hitachi, Fujitsu, Unisys)
• Database admins
• Devops practitioners
• SRE engineers
• Anyone interested in getting beyond performance monitoring

As usual, Sheraton Four Points has bedrooms available at the Performance Dynamics discounted rate. The room-booking link is on the registration page.

Tell a colleague and see you in September!

PDQ 7.0 Dev is Underway

The primary goal for this release is to make PDQ acceptable for uploading to CRAN. This is a non-trivial exercise because there is some legacy C code in the PDQ library that needs to be reorganized while, at the same time, keeping it consistent for programmatically porting to other languages besides R—chiefly Perl (for the book) and Python.

To get there, the following steps have been identified:

1. High Priority

   1. Migrate from SourceForge to GitHub.
   2. Change the return type for these functions from int to void:
      • PDQ_CreateOpen()
      • PDQ_CreateClosed()
      • PDQ_CreateNode()
      • PDQ_CreateMultiNode()
      Using the returned int as a counter was deprecated in version 6.1.1.
   3. Convert PDQ-R to Rcpp interface.
   4. Clean out the Examples directory and other contributed code directories leaving only Examples that actually use the PDQ C library.
   5. Add unit tests for PDQ C library, as well as the Perl, Python, and R languages.
   6. Get interface accepted on CRAN
   7. Add the ability to solve multi-server queueing nodes servicing an arbitrary number of workloads.

2. Low Priority

   1. Get interface accepted on CPAN and PyPI.
   2. Convert to build system from makefiles to Cmake.

Stay tuned!

—njg and pjp

PDQ Version 6.2.0 Released

PDQ (Pretty Damn Quick) is a FOSS performance analysis tool based on the paradigm of queueing models that can be programmed natively in

• R
• Python
• Perl
• C and several other languages.

This minor release is now available for download.

PDQ 6.0.1 is Released

As already described previously, the main purpose of Release 6.0.1 Build 121512 is improved compatibility and stability between PDQ and the R statistical environment. For example, many of the PDQ models, previously found in the ../examples/ directory, can now also be accessed via the demo() command in the R-console. Testing was carried out using R version 2.15.2 (2012-10-26).

Operationally, PDQ, in any of the supported languages, should appear cosmetically the same as Release 5.0; no additional programming required. Since the PDQ-R source can be compiled separately, this release will be of special interest to Microsoft Windows users.

If you're new to PDQ, here's a simple PDQ-R model you can paste directly into the R-console:

library(pdq)
# input parameters
arrivalRate <- 0.75
serviceRate <- 1.0
## Build and solve the PDQ model
Init("Single queue model")         # initialize PDQ
CreateOpen("Work", arrivalRate)    # open workflow
CreateNode("Server", CEN, FCFS)    # single server
SetDemand("Server", "Work", 1/serviceRate) # service time
Solve(CANON)                       # solve the model
Report()                           # tabulated output

Please see the online release notes for the download links and more detailed information, as well as the top-level README file in the distribution. Beyond that, check out the relevant books and training classes.

Merry Xmas from the PDQ Dev Team!

How to Generate Exponential Delays

This question arose while addressing Comments on a previous blog post about exponentially distributed delays. One of my ongoing complaints is that many, if not most, popular load-test generation tools do not provide exponential variates as part of a library of time delays or think-time distributions. Not only is this situation bizarre, given that all load tests are actually performance models (and who doesn't love an exponential distribution in their performance models?), but without the exponential distribution you are less likely to observe such things as buffer overflow conditons due to larger than normal (or uniform) queueing fluctuations. Exponential delays are both simple and useful for that purpose, but we are often left to roll our own code and then debug it.

My Year in Review 2011

Some days I wonder if I ever actually accomplish anything anymore. Maybe it's time to just pack it in and become a greeter at Walmart. I know a bit about how queues work, so that should put me a few notches ahead of the competition. And I would expect the competition to be fierce because it's a pretty cushy job; but not every day, apparently.

Before taking the big leap, I decided it might be a good idea to note down some of the technical projects I've worked on this year (over and above the daily grind):

PDQ 5.0 Test Suite or ... How I Spent My Weekend

I was planning to blog about the amazing time I had at Velocity 2009 last week, when this landed in my mailbox (edited for space and privacy):

Subject: Seeking help with PDQ-R ...
Date: Thu, 25 Jun 2009 15:51:21 -0500

My name is James and I've been trying to learn to properly use PDQ after reading two of your books, "Guerrilla Capacity Planning" and "Analyzing Computer System Performance with Perl::PDQ." I'm still getting a grip on PDQ-R. ... I decided to set about of re-creating the queue circuit in the study with PDQ-R as an exercise. ...
The output of my code yields:
[1] "Manual response time for class 1 is 0.864179 seconds"
[1] "PDQ-R response time for class 1 is 0.313637 seconds"
[1] "Manual response time for class 2 is 6.105397 seconds"
[1] "PDQ-R response time for class 2 is 3.552873 seconds"
[1] "Manual response time for class 3 is 4.535833 seconds"
[1] "PDQ-R response time for class 3 is 4.535833 seconds"
If you could give my code a look over and give me some hints I would really appreciate it.

Queues, Schedulers and the Multicore Wall

The other day, I came across a blog post entitled "Server utilization: Joel on queuing", so naturally I had to stop and take a squiz. The blogger, John D. Cook, is an applied mathematician by training and a cancer researcher by trade, so he's quite capable of understanding a little queueing theory. What he had not heard of before was a rule-of-thumb (ROT) that was quoted in a podcast (skip to 00:26:35) by NYC software developer and raconteur, Joel Spolsky. Although rather garbled, as I think any Guerrilla graduate would agree, what Spolsky says is this:

If you have a line of people (e.g., in a bank or a coffee shop) and the utilization of the people serving them gets above 80% , things start to go wrong because the lines get very long and the average amount of time people spend waiting tends to get really, really bad.

No news to us performance weenies, and the way I've sometimes heard it expressed at CMG is:

No resource should be more than 75% busy.

Personally, I don't like this kind of statement because it is very misleading. Let me explain why.

Plotting PDQ Output with R

One the nice things about PDQ-R (coming in release 5.0) is the ability to plot PDQ output directly in R. Here's a PDQ-R script, together with the corresponding graphical output, that I knocked up to show the effect on the throughput curve of adding more queueing delay stages (K), with everything else held constant.

PDQ From SourceForge

Thoughly fed up doing mind-numbing company income taxes for 2008 (Yes, I have to do them earlier than most to get my 1099s out to sub-contractors), I decided to take a break and see if I could compile PDQ (Pretty Damn Quick) by downloading it from our SourceForge project onto my new Macbook.

Building Perl PDQ with VisualStudio

PDQ user Alex Podelko reports a gotcha when building Perl PDQ with VisualStudio. His solution has been posted to the PDQ download page. Thanks, Alex!

Perl::PDQ Corrigenda Updated

It's been a while, but reader Peter Altevogt (Germany) spotted some numerical inconsistencies in Table D.2 on p. 403 of my Perl:PDQ book. This turns out to be the result of an Excel gotcha when copying cells and pasting them to a different spreadsheet location; the cell reference gets silently incremented. Sigh! The correct values are now available on the corrigenda page. Thank you, Peter!

PDQ Gets Tickled

I recently stumbled across this reference to PDQ in Tcl. The author (Todd Coram) correctly notes that we use SWIG to generate the Perl and Python wrappers and this also facilitates Tcl, apparently. I don't know if he has completed the Tcl port or plans to offer it to us for release to the world at large. Maybe he'll let me know. Surely, The Father of Tcl (a big fan of scripting tools) would approve.

Postscript: Todd Coram responded via email and stated that his attempt had gone into limbo some time ago. So, it looks like open season for anyone interested in doing a Tcl port of PDQ (or any other language for that matter).

Linux Instrumentation: Is Linus Part of the Problem?

I was interested to read in a LinuxWorld article entitled "File System, Power and Instrumentation: Can Linux Close Its Technical Gaps?", that Linus Torvalds believes the current kernel instrumentation sufficiently addresses real-world performance problems.

This statement would be laughable, if he weren't serious. Consider the following:

• How can current Linux instrumentation be sufficient when older UNIX performance instrumentation is still not sufficient?
  
• UNIX instrumentation was not introduced to solve real-world performance problems. It was a hack by and for kernel developers to monitor the performance impact of code changes in a light-weight O/S. We're still living with that legacy. It might've been necessary, but that doesn't make it sufficient.
  
• The level of instrumentation in Linux (and UNIX-es) is not greatly different from what it was 30 years ago. As I discuss in Chap. 4 of my Perl::PDQ book, the idea of instrumenting an O/S goes back (at least) to c.1965 at MIT.
  
• Last time I checked, this was the 21st century. By now, I would have expected (foolishly, it seems) to have at my fingertips, a common set of useful performance metrics, together with a common means for accessing them across all variants of UNIX and Linux.
  
• Several attempts have been made to standardize UNIX performance instrumentation. One was called the Universal Measurement Architecture (UMA), and another was presented at CMG in 1999.
  
• The UMA spec arrived DOA because the UNIX vendors, although they helped to design it, didn't see any ROI where there was no apparent demand from users/analysts. Analysts, on the other hand, didn't demand what they had not conceived was missing. Such a Mexican standoff was cheaper for the UNIX vendors. (How conveeeeenient!) This remains a potential opportunity for Linux, in my view.
  
• Rich Pettit wrote a very thoughtful paper entitled "Formalizing Performance Metrics in Linux", which was resoundingly ignored by Linux developers, as far as I know.
  

Elsewhere, I've read that Linus would like to keep Linux "lean and mean". This reflects a naive PC mentality. When it comes to mid-range SMP servers, it is not possible to symmetrize the kernel without making the code paths longer. Longer code paths are necessary for control and scalability. That's a good thing. And improved performance instrumentation is needed a fortiori for the layered software architectures that support virtual machines. Since Linus is the primary gate-keeper of Linux, I can't help wondering if he is part of the solution or part of the problem.