How to Quantify Scalability

How to Quantify Scalability: Synopsis of The Universal Scalability Law (USL), has just been posted in PDF format for improved readability. I did this post haste so, reporting any typos, broken links, etc., is always appreciated.

Performance Ponderings Updated

Just updated my "Performance Ponderings" page. It covers my performance analysis papers published over the past 35 yrs. 😳 You may be thinking: Who cares about performance analysis cases that is that dated?

Amazingly, performance analysis (done right) is often not technology-dependent, per se. The right abstraction can remain invariant into perpetuity. For example, my 2025 analysis of the GPT Efficient Computer Frontier (ref. 8 in paper 1) is based partly on my 1989 analysis of virtual memory thrashing in paper 53. Same paradigm; vastly different context.

The connection between the two papers is by no means obvious but, quite striking (not to mention satisfying) when you realize it.

Solving a Functional Equation with Functional Programming

Functional equation question seen on Twitter:

This is my general solution, computed in Mathematica: a functional programming language. Scroll through the PDF for details.

The Sommerfeld-Dirac Paradox Reexamined

This talk will be presented at the APS Global Physics Summit on Thursday, March 20, 2025 at 03:15 PM.

Does the Efficiency Compute Frontier Represent New Physics?

This talk will be presented at the APS Global Physics Summit on Monday, March 17, 2025 at 11:30 AM.

Resolving the Dirac-Sommerfeld Paradox via the Path to Quantum Computers

Upcoming online presentation.

This topic goes back to my M.Sc. thesis, when my supervisor, Prof. Christie Eliezer (2nd from left in the photo), who was one of Dirac's very few research students, told me about the Dirac-Sommerfeld paradox. He thought it might be pertinent to my thesis.

Being a physics "genius", I didn't pursue it because I assumed it had been resolved by more modern developments like, Quantum Field Theory — a subject that neither Eliezer nor I knew much about. I was wrong. Very wrong!

Extended Microbenchmarks for Modern Disk Drives

I learnt about this interesting blog post entitled, "Discovering Hard Disk Physical Geometry through Microbenchmarking" by Henry Wong (2019) via @TanelPoder on Twitter. As the title suggests, it is about applying so-called disk drive microbenchmarking to reveal information about the internal mechanical configuration and data layout that is not typically available with other HDD microbenchmark codes; many of which are cited by the author in the References section (something most bloggers are too lazy to do).

While reading it I had some thoughts that turned out to be too long for tweeting a response. So, here I am revisiting my traditional blog, mostly for the formatted space it provides, but also to remind myself that it still lives!

The Article

The article contains a massive amount of very admirable work (~1 year of effort, including benchmark code development) that is exceptionally well written, considering all the details that are covered. I didn't come across any typos or misspellings, either. There are actually TWO articles in one: an extensive summary analysis (not an Executive Summary) and an equally long Appendix containing the detailed measurements obtained from each disk model that was benchmarked.

Quite apart from my subsequent remarks below, the inferences about internal disk structure drawn from the microbenched timing-data are quite remarkable. And the beautiful plots generated from those data (e.g., mapping thousands of sector defects) are something to behold. Understandably, HDD manufacturers are not keen to include such plots in their marketing collateral.

The author states the following as his prime motivations for developing the extended microbenchmark algorithms.

"I had initially set out to detect the number of platters in a disk without opening up a disk, but in modern disks this simple-sounding task requires measuring several other properties before inferring the count of recording surfaces."

"Many of the prior works had practical applications in mind (as opposed to pure curiosity), and were willing to trade measurement speed for a small loss in accuracy. Although [my microbenchmark] algorithms are not fundamentally different, prior work likely avoided exploring these because these measurements can take many hours and there is little practical benefit for applications such as performance modelling or improving data layouts in filesystems."

Although a truly laudible effort, that is largely driven by scientific "curiosity", I can't help but wonder if the author is straining at a gnat.

With a bit of perseverance, I could find several corresponding drive specs online. And, although manufactuers never lie (smirk) the search terms can vary widely.

Platters = disks = discs...
Surfaces = heads = actuators...

Applying the author's terminology:

1. The number of discrete platters has to be an integer.
2. The number of surfaces is an even integer (each platter having two sides).

But, that's not what you see in the Summary table of the 17 HDD models that he tested. Let's look at a few examples.

Example Geometries

The following disk models were compared against their respective manufacturer specs that I was able to find online.

• Tosh X300 5GB has 5 platters and therefore 10 surfaces (1 Gig per platter).
• Seagate ST1 5GB has 1 platter and therefore 2 surfaces.
• WD S25, however, has 2 platters but only 3 surfaces. How can that be?

Western Digital built this disk with only 3 actuator arms, not the expected 4. The fourth surface isn't used. Only WD knows why. Cost cutting is one plausible explanation. There is no way to anticpate such quirks. You either have to measure it or read about it.

The author infers the same number of WD S25 used surfaces (3) based on his benching data which, in itself, is quite remarkable. On the other hand, this same information is generally included in the disk manufacturer specs (or similar docs). Worst case, you could actually contact the manufacturer and they would very likely tell you; especially if they thought they could sell you some disks.

Bitter Pill

However, even when you determine all these interesting extended disk metrics (measured, calculated or specified), you are no better off when it comes to using that information for better storage performance — as the author admits (see quote above) — or predicting future HDD internal geometries. Just like with modern multicore microprocessors,

1. control (especially that related to performance) has progressively been taken away from you and placed inside a silicon module. You can't blue-wire that. Similarly for HDD packaging.
2. manufacturers tend to implement the cheapest solutions not the best performing solutions, which you can't correct. See WD S25 above. Similarly for HDD controller logic.

The HDD microbenching measurements described in the 2019 article would have to be repeated on each new model release. Any mechanical tweaks or other changes inside the new disk would have to be discovered and then incorporated into updated microbenchmark code ... if possible.

Computer manufacturers of all stripes are in business to make a profit and to avoid being eaten by the competition. They will do whatever it takes (good, bad or ugly), whenever it's deemed necessary, to maintain or improve their market position. An unfortunate casuality of this ongoing commercial warfare is that any painstakingly acquired scientific analysis can become an historical footnote overnight.

Performance Modeling

On the topic of performance modeling of disks (to which the author also alludes), his article is a good reminder of the mind-boggling complexity of data layout in modern disks. And that's essentially just the static picture. Along with that goes the complex dynamics of local request-chain caching and optimization.

In that vein, it's a fool's errand to try and model individual modern disks. In principle, you would need to construct a simulation model to handle all the potential transient behaviors. But that, in turn, requires deciphering the embedded controller algorithms, which are not only highly complex but proprietary. You'll never figure them out and no manufacturer will ever reveal them. So, a performance simulation is actually a non-starter.

On a more positive note, there is a surprisingly counterintuitive technique for constructing accurate performance models of collective disks, like SANs, using the PDQ (Pretty Damn Quick) performance analyzer. That's something I demonstrate in my Guerrilla Capacity and Performance (GCAP) classes.

Related Posts

• Henry Wong's microbenchmark code written in C++
• ATTO Disk Benchmark (2019) — not cited by Henry Wong but I saw it used in other disk microbenchmark reports.
• The SSD World Will End in 2024
• Green Disk Sizing
• Disk Storage Myth Busters
• All competitive benchmarking is institutionalized cheating (Gaphorism 1.21)

PDQ Online Workshop, May 17-21, 2021

PDQ (Pretty Damn Quick) is a free, open source, performance analyzer available from the Performance Dynamics web site.

All modern computer systems, no matter how complex, can be thought of as a directed graph of individual buffers that hold requests until to be serviced at a shared computational resource, e.g., a CPU or disk. Since a buffer is just a queue, any computer infrastructure, from your laptop up to Facebook.com, can be represented as a directed graph of queues.

The directed arcs or arrows in such a graph correspond to workflows between different queues. In the parlance of queueing theory, a directed graph of queues is called a queueing network model. PDQ is a tool for predicting performance metrics such as, waiting time, throughput, optimal user-load.

Two major benefits of using PDQ are:

1. confirmation that monitored performance metrics have their expected values
2. predict performance for circumstances that lie beyond current measurements

Find out more about the workshop and register today.

PDQ 7.0 is Not a Turkey

Giving Thanks for the release of PDQ 7.0, after a 5-year drought, and just in time for the PDQW workshop next week.

New Featues

1. The introduction of the STREAMING solution method for OPEN queueing networks. (cf. CANON, which can still be used).
2. The CreateMultiNode() function is now defined for CLOSED queueing networks and distinguished via the MSC device type (cf. MSO for OPEN networks).
3. The format of Report() has been modified to make the various types of queueing network parameters clearer.
4. See the R Help pages in RStudio for details.
5. Run the demo(package="pdq") command in the R console to review a variety of PDQ 7 models.

Maintenance Changes

The migration of Python from 2 to 3 has introduced maintenance complications for PDQ. Python 3 may eventually be accommodated in future PDQ releases. Perl maintenance has ended with PDQ release 6.2, which to remain compatible with the Perl::PDQ book (2011).

PDQ Online Workshop, Nov 30-Dec 5, 2020

PDQ (Pretty Damn Quick) is a queueing graph performance analyzer that comes as:

1. free open source software package
2. with an online user manual

As shown in the above diagram, any modern computer system can be thought of as a directed graph of individual buffers where requests wait to be serviced at some kind of shared computational resource, e.g., a CPU or disk. Since a buffer is just a queue, any computer infrastructure, from a laptop to Facebook, can be represented as a directed graph of queues. The directed arcs or arrows correspond to workflows between the different queues. In the parlance of queueing theory, a directed graph of queues is called a queueing network model. PDQ is a tool for calculating the performance metrics, e.g., waiting time, throughput, optimal load, of such network models.

Some example PQD applications include models of:

• Cloud applications
• Packet networks
• HTTP + VM + DB apps
• PAXOS-type distributed apps

Two major benefits of using PDQ are:

1. confirmation that monitored performance metrics have their expected values
2. predict performance for circumstances that lie beyond current load-testing

This particular PDQ workshop has been requested for the Central European Time zone.

• All sessions are INTERACTIVE using Skype (not canned videos)
• Online sessions are usually 4 hours in duration
• A typical timespan is 2pm to 6pm CET each business day
• A nominal 5-10 minute bio break at 4pm CET
• Attendees are encouraged to bring there own PDQ projects or data to start one

1. Find out more about the workshop.
2. Here is the REGISTRATION page.

Hope to see you at PDQW!

Converting Between Human Time and Absolute Time in the Shell

This is really more of a note-to-self but it may also be useful for other readers.

Converting between various time zones, including UTC time, is both a pain and error-prone. A better solution is to use absolute time. Thankfully, Unix provides such a time: the so-called Epoch time, which is the integer count of the number of seconds since January 1, 1970.

Timestamps are both very important and often overlooked: the moreso in the context of performance analysis, where time is the zeroth metric. In fact, my preferred title would have been, Death to Time Zones but that choice would probably have made it harder to find the main point here, later on, viz., how to use the Unix epoch time.

Although there are web pages that facilitate such time conversions, there are also shell commands that are often more convenient to use, but not so well known. With that in mind, here are some examples (for future reference) of how to convert between human-time and Unix time.

Examples

The following are the bash shell commands for both MacOS and Linux.

MacOS and BSD

Optionally see the human-readable date-time:

[njg]~% date
Tue Feb 11 10:04:32 PST 2020

Get the Unix integer time for the current date:

[njg]~% date +%s
1581444272

Resolve a Unix epoch integer to date-time:

[njg]~% date -r 1581444272
Tue Feb 11 10:04:32 PST 2020

Linux

Optionally see the human-readable date-time:

[njg]~% date
Tue Feb 11 10:04:32 PST 2020

Get the Unix integer time for the current date:

[njg]~% date +%s
1581444272

Resolve a Unix epoch integer to date-time:

[njg]~% date -d @1581444272
Tue Feb 11 18:04:32 UTC 2020

Book Review: How to Build a Performance Testing Stack From Scratch

Writing a technical book is a difficult undertaking. As a technical author myself, I know that writing well is both arduous and tedious. There are no shortcuts. Over the last 40 years or so, computer-based tools have been developed to help authors write printed textbooks, monographs and technical articles. LaTeX (pronounced lar-tek) reigns supreme in that realm because it's not just word processing software but, a full-blown digital typesetting application that enables authors to produce a single camera-ready PDF file (bits) that is fit for direct printing. Not only does LaTeX correctly typeset characters and mathematical symbols but it can generate the Table of Contents and the corresponding Index of terms, together with correctly cross-referenced callouts for numbered chapters, sections, figures, tables and equations.

In the meantime, over the past 20 years, the nature of the book itself has become progressively more digital. The ability to render the book-block on digital devices as an "e-book" has made printed hardcopies optional. Although purely digital e-books make reading more ubiquitous, e-book file formats and display quality varies across devices, i.e., laptops, phones tablets, and various e-readers. Any reduction in display quality is offset by virtue of e-books being able to include user interaction and even animation: features entirely beyond the printed book. In that sense, there really has been a revolution in the publishing industry: books are no longer about books, they're now about media.

Recently, I became aware of an undergraduate calculus textbook that makes powerful use of animation and audio. The author is a academic mathematician and he published it on YouTube! Is that a book or a movie? Somehow, it's a hybrid of both and, indeed, I wish I'd been able to learn from technical "books" like that. Good visuals and animations can make difficult technical concepts much easier to comprehend. Progressive as all that is, when it comes to technical e-books, I'm not aware of any single authoring tool that can match the quality of LaTeX, let alone incorporate user-interaction and animation. If such a thing did exist, I would be all over it. And Markdown doesn't cut it. But, digital authoring tools are continually evolving.

Matt Fleming (@fleming_matt on Twitter), the author of How to Build a Performance Testing Stack From Scratch, opted to use a static e-book format—not because it produces the most readable result, but because it is the best way to reach a wider audience at lower cost than a more expensive print publisher. The e-publisher in this case is (the relatively unknown) Ministry of Testing Ltd in Brighton, UK and is available on Amazon for the Kindle reader. I was also able to read it using iBooks on Mac OS X.

The range of topics covered is very extensive. I've included the Table of Contents here because it is not viewable on Amazon:

Part 1

• Step 1: Identify Stakeholders
• Step 2: Identify What to Measure
• Step 3: Test Design
• Step 4: Measuring Test Success and Failure
• Step 5: Sharing Results

Part 2

• Understanding Statistics
• Latency
• Throughput
• Statistical Significance

Part 3 The Benchmark Hierarchy Picking Tests Validating Tests Part 4

• Use a Performance Framework
• Ensure the Test Duration is Consistent
• Order Tests by Duration
• Keep Reproduction Cases Small
• Setup The Environment Before Each Test
• Make Updating Tests Easy
• Errors Should Be Fatal

Part 5

• Format
• Use Individual Sample Data
• Detecting Results in the Noise
• Outliers
• Result Precision
• If All Else Fails Use Test Duration
• Delivering Results

The overall e-book presentation of "Performance Testing Stack" seems underdeveloped. Most topics could have been greatly expanded. But, as Matt informed me, this is probably due to the content amounting to a concatenation of previously written blog posts. As I said earlier, there are no shortcuts to writing well. The paucity of detail, however, is offset by the shear enthusiasm the Matt brings to his writing: an aspect that deserves separate acknowledgement because performance testing is a very complex subject which can otherwise appear dry and mind-boggling to the uninitiated. And it's the uninitiated that Matt wants to reach. He has written this book in order to encourage the uninitiated reader to seriously consider entering the field.

Some points that could have been developed further, include:

• Plots and tables can be expanded for legibility by double clicking on them
• The term "benchmark" needs better explaination
• Section 2.1 discusses the Harmonic mean but there's no discussion of the Geometric mean.
• Section 3.3 on Distributions does not clearly distinguish between analytic (parametric) distributions and sample distributions (which usually have no analytic form).
• Section 5 (p.85) on Result Precision needs to discuss the difference between accuracy, precision, and error.

Conversely, Matt's enthusiasm may have gone a bit overboard in his choice of title. The book promises:

This book will walk you through designing and building a performance testing stack from scratch; step by step from planning and running performance tests through to understanding and analysing the results. If you’re new to performance testing or looking to expand your understanding of this topic then this book is for you!

Unfortunately, this book doesn't provide enough details to actually build a test stack—which would've been very cool. Rather, it presents a comprehensive overview of all the major concepts that one needs to absorb in order to develop a running performance testing stack. But even with the more limited scope, this book is still important because, off hand, I don't know of any other source where one can be introduced to performance testing without drowning in a sea of terminology, procedures and architectures.

Ultimately, this e-book is a great starting point for newbies, as well was being a good reminder for seasoned testers about what should be done in good performance tests.

DSConf 2019 Featured Talk

"Applying The Universal Scalability Law to Distributed Systems"

DSConf'19 - Distributed Systems Conference (scroll down)
Pune, India
11am IST
February 16

I'm very much looking forward to this event and I thank @ShrivedAgashe for the invitation.

Guerrilla 2018 Classes Now Open

All Guerrilla training classes are now open for registration.

1. GCAP: Guerrilla Capacity and Performance — From Counters to Containers and Clouds
2. GDAT: Guerrilla Data Analytics — Everything from Linear Regression to Machine Learning
3. PDQW: Pretty Damn Quick Workshop — Personal tuition for performance and capacity mgmt

The following highlights indicate the kind of thing you'll learn. Most especially, how to make better use of all that monitoring and load-testing data you keep collecting.

• How to save millions of dollars with a one-line performance model (video)
• How to minimize chargeback after you lift and shift to the cloud (video)
• How to correctly emulate web traffic on a load-testing rig (PDF)

See what Guerrilla grads are saying about these classes. And how many instructors do you know that are available for you from 9am to 9pm (or later) each day of your class?

Who should attend?

• IT architects
• Application developers
• Performance engineers
• Sysadmins (Linux, Unix, Windows)
• System engineers
• Test engineers
• Mainframe sysops (IBM. Hitachi, Fujitsu, Unisys)
• Database admins
• Devops practitioners
• SRE engineers
• Anyone interested in getting beyond performance monitoring

As usual, Sheraton Four Points has bedrooms available at the Performance Dynamics discounted rate. The room-booking link is on the registration page.

Tell a colleague and see you in September!

Chargeback in the Cloud - The Movie

If you were unable to attend the live presentation on cost-effective defenses against chargeback in the cloud, or simply can't get enough performance and capacity analysis for the AWS cloud (which is completely understandable), here's a direct link to the video recording on CMG's YouTube channel.

The details concerning how you can do this kind of cost-benefit analysis for your cloud applications will be discussed in the upcoming GCAP class and the PDQW workshop. Check the corresponding class registration pages for dates and pricing.

USL Scalability Modeling with Three Parameters

NOTE: Annoyingly, the remote mathjax server often takes it's sweet time rendering LaTex equations (like, maybe a minute!!!). I don't know if this is deliberate on the part of Google or a bug. It used to be faster. If anyone knows, I'd be interested to hear; especially if there is a way to speed it up. And no, I'm not planning to move to WordPress.

Update of Oct 2018: Wow! MathJax performance is back. Clearly, whinging is the most powerful performance optimizer. :)

The 2-parameter USL model

The original USL model, presented in my GCAP book and updated in the blog post How to Quantify Scalability, is defined in terms of fitting two parameters $\alpha$ (contention) and $\beta$ (coherency). \begin{equation} X(N) = \frac{N \, X(1)}{1 + \alpha \, (N - 1) + \beta \, N (N - 1)} \label{eqn: usl2} \end{equation}

Fitting this nonlinear USL equational model to data requires several steps:

1. normalizing the throughput data, $X$, to determine relative capacity, $C(N)$.
2. equation (\ref{eqn: usl2}) is equivalent to $X(N) = C(N) \, X(1)$.
3. if the $X(1)$ measurement is missing or simply not available—as is often the case with data collected from production systems—the GCAP book describes an elaborate technique for interpolating the value.

The motivation for a 2-parameter model arose out of a desire to meet the twin goals of:

1. providing each term of the USL with a proper physical meaning, i.e., not treat the USL like a conventional multivariate statistical model (statistics is not math)
2. satisfying the von Neumann criterion: minimal number of modeling parameters

Last year, I realized the 2-paramater constraint is actually overly severe. Introducing a third parameter would make the statistical fitting process even more universal, as well as simplify the overall procedure. For the USL particularly, the von Neumann criterion should not be taken too literally. It's really more of a guideline: fewer is generally better.

The Geometry of Latency

... AKA hyperbolae.

Here's a mnemonic tabulation based on dishes and bowls:

Hopefully this makes amends for the more complicated explanation I wrote for CMG back in 2009 entitled: "Mind Your Knees and Queues: Responding to Hyperbole with Hyperbolæ", which I'm pretty sure almost nobody understood.

Virtual cloudXchange 2018 Conference

Our abstract has been accepted for presentation at the FREE cloudXchange online event to be held by CMG on June 19th at 10am Pacific (5pm UTC). [Extended slides]

Exposing the Cost of Performance
Hidden in the Cloud

Neil Gunther
Performance Dynamics, Castro Valley, California

Mohit Chawla
Independent Systems Engineer, Hamburg, Germany

10am Pacific Time on June 19, 2018

Whilst offering lift-and-shift migration and versatile elastic capacity, the cloud also reintroduces an old mainframe concept—chargeback—which rejuvenates the need for performance analysis and capacity planning. Combining production JMX data with an appropriate performance model, we show how to assess fee-based EC2 configurations for a mobile-user application running on a Linux-hosted Tomcat cluster. The performance model also facilitates ongoing cost-benefit analysis of various EC2 Auto Scaling policies.

WTF is Modeling, Anyway?

A conversation with performance and capacity management veteran Boris Zibitsker, on his BEZnext channel, about how to save multiple millions of dollars with a one-line performance model (at 21:50 minutes into the video) that has less than 5% error. I wish my PDQ models were that good. :/

The strength of the model turns out to be its explanatory power, rather than prediction, per se. However, with the correct explanation of the performance problem in hand (which also proved that all other guesses were wrong), this model correctly predicted a 300% reduction in application response time for essentially no cost. Modeling doesn't get much better than this.

Footnotes

1. According to Computer World in 1999, a 32-node IBM SP2 cost $2 million to lease over 3 years. This SP2 cluster was about 6 times bigger.
2. Because of my vain attempt to suppress details (in the interests of video length), Boris gets confused about the kind of files that are causing the performance problem (near 26:30 minutes). They're not regular data files and they're not executable files. The executable is already running but sometimes waits—for a long time. The question is, waits for what? They are, in fact, special font files that are requested by the X-windows application (the client, in X parlance). These remote files may also get cached, so it's complicated. In my GCAP class, I have more time to go into this level of detail. Despite all these potential complications, my 'log model' accurately predicts the mean application launch time.
3. Log_2 assumes a binary tree organization of font files whereas, Log_10 assumes a denary tree.
4. Question for the astute viewer. Since these geophysics applications were all developed in-house, how come the developers never saw the performance problems before they ever got into production? Here's a hint.
5. Some ppl have asked why there's no video of me. This was the first time Boris had recorded video of a Skype session and he pushed the wrong button (or something). It's prolly better this way. :P

CPU Idle Is Not Like White Space

This post seems like it ought to be too trite to write but, I see the following performance gotcha cropping up over and over again.

Under pressure to consolidate resources, usually driven by management and especially regarding processor capacity, there is often an urge to "use up" any idle processor cycles. Idle processor capacity tends to be viewed like it's whitespace on a written page—just begging to be filled up.

The logical equivalent of filling up the "whitespace" is absorbing idle processor capacity by migrating applications that are currently running on other servers and turning those excess servers off or using them for something else.