Guerrilla 2018 Classes Now Open

All Guerrilla training classes are now open for registration.

1. GCAP: Guerrilla Capacity and Performance — From Counters to Containers and Clouds
2. GDAT: Guerrilla Data Analytics — Everything from Linear Regression to Machine Learning
3. PDQW: Pretty Damn Quick Workshop — Personal tuition for performance and capacity mgmt

The following highlights indicate the kind of thing you'll learn. Most especially, how to make better use of all that monitoring and load-testing data you keep collecting.

• How to save millions of dollars with a one-line performance model (video)
• How to minimize chargeback after you lift and shift to the cloud (video)
• How to correctly emulate web traffic on a load-testing rig (PDF)

See what Guerrilla grads are saying about these classes. And how many instructors do you know that are available for you from 9am to 9pm (or later) each day of your class?

Who should attend?

• IT architects
• Application developers
• Performance engineers
• Sysadmins (Linux, Unix, Windows)
• System engineers
• Test engineers
• Mainframe sysops (IBM. Hitachi, Fujitsu, Unisys)
• Database admins
• Devops practitioners
• SRE engineers
• Anyone interested in getting beyond performance monitoring

As usual, Sheraton Four Points has bedrooms available at the Performance Dynamics discounted rate. The room-booking link is on the registration page.

Tell a colleague and see you in September!

Crib Sheet for Emulating Web Traffic

Our paper entitled, How to Emulate Web Traffic Using Standard Load Testing Tools (PDF) is now available online and will be presented at the upcoming CMG conference in November.

Presenter: James Brady (co-author: Neil Gunther)
Session Number: 436
Subject Area: APM
Session Date: Wed, November 9, 2016
Session Time: 1:00 PM - 2:00 PM
Session Room: PortofinoB

How to Emulate Web Traffic Using Standard Load Testing Tools

The following abstract has been submitted to CMG 2016:

How to Emulate Web Traffic Using Standard Load Testing Tools

James Brady (State of Nevada) and Neil Gunther (Performance Dynamics)

Conventional load-testing tools are based on a fifty year old time-share computer paradigm where a finite number of users submit requests and respond in a synchronized fashion. Conversely, modern web traffic is essentially asynchronous and driven by an unknown number of users. This difference presents a conundrum for testing the performance of modern web applications. Even when the difference is recognized, performance engineers often introduce virtual-user script modifications based on hearsay; much of which leads to wrong results. We present a coherent methodology for emulating web traffic that can be applied to existing test tools.

Keywords: load testing, workload simulation, web applications, software performance engineering, performance modeling

Related blog posts:

1. Emulating Web Traffic in Load Tests
2. Mapping Virtual Users to Real Users
3. How to Extend Load Tests with PDQ

Load Testing with Uniform vs. Exponential Arrivals

In a couple of recent blog posts about generating exponential loads and why that is important for load testing and performance testing, it was not completely clear to some readers what was motivating my remarks. In this post, I will try to provide a more visual elaboration of that aspect.

My fundamental point is this. When it comes to load testing^*, presumably the idea is to exercise the system under test (SUT). Otherwise, why are you doing it? Part of exercising the SUT is to produce significant fluctuations in the number of requests residing in application buffers. Those fluctuations can be induced by the pattern of arriving requests issued by the client-side driver (DVR): usually implemented as a pile of PCs or blades.

How to Generate Exponential Delays

This question arose while addressing Comments on a previous blog post about exponentially distributed delays. One of my ongoing complaints is that many, if not most, popular load-test generation tools do not provide exponential variates as part of a library of time delays or think-time distributions. Not only is this situation bizarre, given that all load tests are actually performance models (and who doesn't love an exponential distribution in their performance models?), but without the exponential distribution you are less likely to observe such things as buffer overflow conditons due to larger than normal (or uniform) queueing fluctuations. Exponential delays are both simple and useful for that purpose, but we are often left to roll our own code and then debug it.

Webinar: Load Testing Meets Data Analytics

This Thursday, October 27 at 10 am PDT^*, I'll be participating in a webinar sponsored by SOASTA, Inc. They make a new breed of load-testing product called CloudTest® which, despite its name, is not restricted to load testing cloud-based apps, although it can do that too.

Load Testing Think Time Distributions

One of my gripes about some commercial load testing tools is that they only provide a think time distribution (Z) that is equivalent to uniform variates in the client-script. If you want some other distribution, you have to code it and debug it yourself. Load test generators are essentially very expensive workload simulators; especially when you take into account the cost of the SUT platform. At those prices, a selection of distributions should be provided as a standard library—like they are in event-based simulators.

To make this point a bit clearer, I used the very convenient variate-generation functions in R to compare some of the distributions that I consider should be included in such a library for the convenience of workload-test designers and performance engineers. The statistical mean (i.e., the average think delay) is the same in all these plots and is shown as the red vertical line, but pay particular attention to the spread around the mean on the x-axis.

Emulating Web Traffic in Load Tests

One of the recurring questions in the GCaP class last week was: How can we make web-application load tests more representative of real Internet traffic? The sticking point is that conventional load-test simulators like LoadRunner, JMeter, and httperf, represent the load in terms of a finite number of virtual user (or vuser) scripts, whereas the Internet has an indeterminately large number of real users creating load.

GCaP Class Highlights

It was sunny outside but we ended up staying in the shade for better lighting balance. Other grads had to catch earlier flights home by the time this shot was taken.

Some graduates of the May 2010 GCaP class. Photo courtesy Manu M.
Here are some of the interesting topics that popped up in class discussions this week:

• How to emulate Internet traffic with load test tools like LoadRunner
• Contol Groups (cgroups) for fair-share resource allocation in Linux containers—not to be confused with CFS (completely fair scheduler)
• Demo of JXinsight by the tool architect (and now Guerrilla graduate) William Louth
• DTrace for Solaris and Mac OS X
• Instruments in Mac OS X (Leopard and higher)
• The performance and capacity implications of cloud computing
• How to get started doing GCaP with VAM: Visualize, Analyze, Modelize
• httperf web workload generator

This is why you too should consider attending an upcoming Guerrilla class.

Using Think Times to Determine Arrival Rates

This question came up at the NorCal CMG meeting last week. Hugh S. asked me: Is there is a relationship between the choice of think time (Z) in a load-test client script and the rate at which requests will arrive into the system under test? The answer is, yes, and it's easy to understand how by using the preceding blog post about mapping virtual users to real users.

Mapping Virtual Users to Real Users

In performance engineering scenarios that use commercial load testing tools, e.g., LoadRunner, the question often arises: How many virtual users (vusers) should be exercised in order to simulate by some expected number of real users? This is important, more often than not, because the requirement might be to simulate thousands or even tens of thousands of real users, but the stiff licensing fees associated with each vuser (above some small default number) makes that cost-prohibitive. As I intend to demonstrate here, we can apply Little's law to map vusers to real users.

A commonly used practical approach to ameliorate this circumstance is to run the load test scenarios with zero think time (i.e., Z = 0) in the client scripts on the driver (DVR) side of the test rig. This choice effectively increases the number of active transactions running on the system under test (SUT), which might include apps servers and database servers. These two subsystems are usually connected by a local area network, as shown in the following diagram.

Data + Models == Insight

Al Bundy, of the TV show Married with Children, understood it and performance engineers should too. What am I talking about? The theme music for that show is the tune "Love and Marriage" as sung by Frank Sinatra. Just like the song says about love and marriage, so it is with measurements and models ... You can't have one without the other.

Assessing USL Scalability with Mixed Business Functions

Professional capacity planner, Raja C., has been applying my Universal Scalability Law (USL) in some fascinating and progressive ways. By that I mean, fascinating to me, because I hadn't thought about applying the USL model in the same way he has; I don't have a real job, you understand. On the other hand, this may well represent the situation that many of you are faced with on a day-to-day basis, so I'd like to present and discuss Raja's question here in some detail.

Plotting PDQ Output with R

One the nice things about PDQ-R (coming in release 5.0) is the ability to plot PDQ output directly in R. Here's a PDQ-R script, together with the corresponding graphical output, that I knocked up to show the effect on the throughput curve of adding more queueing delay stages (K), with everything else held constant.

How to Recover the Missing X(1) for the USL Scalability Model

When it comes to assessing application scalability, controlled measurements of the type that can be obtained with tools like Grinder or LoadRunner, are very useful because they provide a direct measurement of the throughput, X(N), as a function of the vuser/generator load, N. These data can be input easily into my universal scalability model (USL). To apply USL, however, you need to normalize all the X(N) data to the X(1) value. It often happens that the X(1) value may not have been measured by your QA group or it simply may not be measurable easily. What do you do in that case? You estimate it (carefully).

SOA Scalability and Steady-State

Guerrilla alumnus Peter Lauterbach just brought to my attention an article in SOA World entitled "Load Testing Web Services". I have to commend these authors for performing their SOA load tests in steady state. Elsewhere, I've discussed how wrong things can go when you don't adhere to this procedure. In their online article, these authors show the response time (R) as a time-series plot, more or less as it would appear in a measurement tool like say, LoadRunner. Although they don't show it, the throughput measurements would also look similar when plotted as a function of time (t).

How to Extend Load Tests with PDQ

Suppose you want to assess the scalability of a web application based on measurements from a test rig using a load-test tool like LR, WAS or Grinder. There's one slight problem. The load-test tools are limited to running a finite number (N) of client-side load generators, and that number will always be much much smaller than the number of actual web users submitting transactions when the application goes live. How can you bridge that gap?