Converting Between Human Time and Absolute Time in the Shell

This is really more of a note-to-self but it may also be useful for other readers.

Converting between various time zones, including UTC time, is both a pain and error-prone. A better solution is to use absolute time. Thankfully, Unix provides such a time: the so-called Epoch time, which is the integer count of the number of seconds since January 1, 1970.

Timestamps are both very important and often overlooked: the moreso in the context of performance analysis, where time is the zeroth metric. In fact, my preferred title would have been, Death to Time Zones but that choice would probably have made it harder to find the main point here, later on, viz., how to use the Unix epoch time.

Although there are web pages that facilitate such time conversions, there are also shell commands that are often more convenient to use, but not so well known. With that in mind, here are some examples (for future reference) of how to convert between human-time and Unix time.

Examples

The following are the bash shell commands for both MacOS and Linux.

MacOS and BSD

Optionally see the human-readable date-time:

[njg]~% date
Tue Feb 11 10:04:32 PST 2020

Get the Unix integer time for the current date:

[njg]~% date +%s
1581444272

Resolve a Unix epoch integer to date-time:

[njg]~% date -r 1581444272
Tue Feb 11 10:04:32 PST 2020

Linux

Optionally see the human-readable date-time:

[njg]~% date
Tue Feb 11 10:04:32 PST 2020

Get the Unix integer time for the current date:

[njg]~% date +%s
1581444272

Resolve a Unix epoch integer to date-time:

[njg]~% date -d @1581444272
Tue Feb 11 18:04:32 UTC 2020

Chargeback in the Cloud - The Movie

If you were unable to attend the live presentation on cost-effective defenses against chargeback in the cloud, or simply can't get enough performance and capacity analysis for the AWS cloud (which is completely understandable), here's a direct link to the video recording on CMG's YouTube channel.

The details concerning how you can do this kind of cost-benefit analysis for your cloud applications will be discussed in the upcoming GCAP class and the PDQW workshop. Check the corresponding class registration pages for dates and pricing.

Load Average in FreeBSD

In my Perl PDQ book there is a chapter, entitled Linux Load Average, where I dissect how the load average metric (or metrics, since there are three reported numbers) is computed in shell commands like uptime, viz.,

[njg]~/Desktop% uptime
16:18  up 9 days, 15 mins, 4 users, load averages: 2.11 1.99 1.99

For the book, I used Linux 2.6 source because it was accessible on the web with convenient hyperlinks to navigate the code. Somewhere in the kernel scheduler, the following C code appeared:

#define FSHIFT    11          /* nr of bits of precision */ 
#define FIXED_1   (1<<FSHIFT) /* 1.0 as fixed-point */ 
#define LOAD_FREQ (5*HZ)      /* 5 sec intervals */ 
#define EXP_1     1884        /* 1/exp(5sec/1min) fixed-pt */ 
#define EXP_5     2014        /* 1/exp(5sec/5min) */ 
#define EXP_15    2037        /* 1/exp(5sec/15min) */ 

#define CALC_LOAD(load,exp,n) \
 load *= exp; \
 load += n*(FIXED_1-exp); \
 load >>= FSHIFT;

where the C macro CALC_LOAD computes the following equation

Monitoring CPU Utilization Under Hyper-threading

The question of accurately measuring processor utilization with hyper-threading (HT) enabled came up recently in a Performance Engineering Group discussion on Linked-in. Since I spent some considerable time looking into this issue while writing my Guerrilla Capacity Planning book, I thought I'd repeat my response here (slightly edited for this blog), in case it's useful to a broader audience interested in performance and capacity management. Not much has changed, it seems.

In a nutshell, the original question concerned whether or not it was possible for a single core to be observed running at 200% busy, as reported by Linux top, when HT is enabled.

---

This question is an old canard (well, "old" for multicore technology). I call it the "Missing MIPS" paradox. Regarding the question, "Is it really possible for a single core to be 200% busy?" the short answer is: never! So, you are quite right to be highly suspicious and confused.

You don't say which make of processor is running on your hardware platform, but I'll guess Intel. Very briefly, the OS (Linux in your case) is being lied to. Each core has 2 registers where inbound threads are stored for processing. Intel calls these AS (Architectural State) registers. With HT *disabled*, the OS only sees a single AS register as being available. In that case, the mapping between state registers and cores is 1:1. The idea behind HT is to allow a different application thread to run when the currently running app stalls; due to branch misprediction, bubbles in the pipeline, etc. To make that possible, there has to be another port or AS register. That register becomes visible to the OS when HT is enabled. However, the OS (and all the way up the food chain to whatever perf tools you are using) now thinks twice the processor capacity is available, i.e., 100% CPU at each AS port.

Linux Per-Entity Load Tracking: Plus ça change

Canadian capacity planner, David Collier-Brown, pointed me at this post about some more proposed changes to how load is measured in the Linux kernel. He's not sure they're on the right track. David has written about such things as cgroups in Linux and I'm sure he understands these things better than I do, so he might be right. I never understood the so-called CFS: Completely Fair Scheduler. Is it a fair-share scheduler or something else? Not only was there a certain amount of political fallout over CFS but, do we care about such things anymore? That was back in 2007. These days we are just as likely to run Linux in a VM under VMware or XenServer or the cloud. Others have proposed that the Linux load average metric be made "more accurate" by including IO load. Would that be local IO, remote IO or both? Disk IO, network IO, etc., etc?

Sex, Lies and Log Plots

From time to time, at the Hotsos conferences on Oracle performance, I've heard the phrase, "battle against any guess" (BAAG) used in presentations. It captures a good idea: eliminate guesswork from your decision making process. Although that's certainly a laudable goal, life is sometimes not so simple; particularly when it comes to performance analysis. Sometimes, you really can't seem to determine unequivocally what is going on. Inevitably, you are left with nothing but making a guess—preferably an educated guess, not a random guess (the type BAAG wants to eliminate). As I say in one of my Guerrilla mantras: even wrong expectations (or a guess) are better than no expectations. In more scientific terms, such an educated guess is called a hypothesis and it's a major way of making scientific progress.

Of course, it doesn't stop there. The most important part of making an educated guess is testing its validity. That's called hypothesis testing, in scientific circles. To paraphrase the well-known Russian proverb, in contradistinction to BAAG: Guess, but justify^*. Because all hypothesis testing is a difficult process, it can easily get subverted into reaching the wrong conclusion. Therefore, it is extremely important not to set booby traps inadvertently along the way. One of the most common visual booby trap arises from the inappropriate use of logarithmically-scaled axes (hereafter, log axes) when plotting data.

Linear scale:

Each major interval has a common difference $(d)$, e.g., $200, 400, 600, 800, 1000$ if $d=200$:

Log scale:

Each major interval has a common multiple or base $(b)$, e.g., $0.1, 1, 10, 100, 1000$ if $b=10$:

The general property of a log axis is to stretch out the low end of the axis and compress the high end. Notice the unequal minor interval spacings. Hence, using a log scaled axis (either $x$ or $y$) is equivalent to applying a nonlinear transformation to the data. In other words, you should be aware that introducing a log axis will distort the visual representation of the data^†, which can lead to entirely wrong conclusions.

The Backstory on Time-Share Computing

In chapter 4 of my Perl::PDQ book, "Linux Load Average—Take a Load Off" and Appendix B "A Short History of Buffers," I briefly refer to the history of Unix and ultimately Linux via Multics, starting with the original MIT project called CTSS (Compatible Time-sharing System). My purpose there was to point out that the load average metric is the earliest example of O/S performance instrumentation. Naturally then, the following 5-part series in the NYT on the development of time-share computers caught my attention:

• Part one
  
• Part two
  
• Part three
  
• Part four
  
• Part five
  

These accounts are noteworthy because they are written by the brother of one of the developers (of early email—depending on how you define email) and the author is a journalist, so he interviewed some of the personalities (who are now getting on a bit).

There are also lots of fascinating photos.

GCaP Class Highlights

It was sunny outside but we ended up staying in the shade for better lighting balance. Other grads had to catch earlier flights home by the time this shot was taken.

Some graduates of the May 2010 GCaP class. Photo courtesy Manu M.
Here are some of the interesting topics that popped up in class discussions this week:

• How to emulate Internet traffic with load test tools like LoadRunner
• Contol Groups (cgroups) for fair-share resource allocation in Linux containers—not to be confused with CFS (completely fair scheduler)
• Demo of JXinsight by the tool architect (and now Guerrilla graduate) William Louth
• DTrace for Solaris and Mac OS X
• Instruments in Mac OS X (Leopard and higher)
• The performance and capacity implications of cloud computing
• How to get started doing GCaP with VAM: Visualize, Analyze, Modelize
• httperf web workload generator

This is why you too should consider attending an upcoming Guerrilla class.

Googling Google + Linux

Honorary Guerrilla alumnus, GB, sent me a link to "How Google uses Linux" which, although it provides an interesting view inside The Goog's datacenter management, looks like it's supposed to be available only to LWN subscribers:

"The following subscription-only content has been made available to you by an LWN subscriber." Eh? 

Not wishing to let any cats out of their subscription bag, I checked with the editor and he said it's ok to blog the link.

This Apple Does Fall Far From The B-Tree

That old adage: the apple doesn't fall far from the tree, doesn't apply to Apple the corporation. According to ArsTechnica today, Apple abruptly abandoned its open-source project to port Sun's ZFS as the filesystem for Mac OS X, on October 23rd.

The speculation is that Sun licensing fees may have been viewed as a roadblock to adoption or possibly there are growing concerns that Oracle's acquisition of Sun could cause other problems. In the meantime, Apple is hiring engineers to build its own advanced filesystem, instead of adopting either ZFS or its Linux derivative BtrFS.

Is SCO Waiting for Godot?

Bankruptcy Judge Kevin Gross remarked in his recent ruling that ongoing SCO Group litigation attempts were like a bad version of Samuel Beckett's play, Waiting for Godot. The almost decade-long legal saga gained publicity in the FOSS community for targeting Linux as illegally containing licensed AT&T UNIX System V source code.

Performance und Virtualisierung

Autor: Neil J. Gunther
Erschienen: 15.09.2009
Umfang: 13 Seiten

Woran denkt man, wenn man den Begriff Virtualisierung hört? An einen Hypervisor wie Citrix XenServer oder den ESX-Server von VMware? Oder an virtualisierte Services wie beim Cloud Computing? Oder an Multicore-CPUs mit Hyperthreading, die virtuelle Prozessoren ermöglichen? Am besten betrachtet man all diese Erscheinungsformen von Virtualisierung nicht isoliert, sondern als Teile eines einzigen Performance-Management-Puzzles. Dieser Beitrag erklärt wieso und er unterstreicht, wie wichtig es ist, durch kontrollierte Performance-Messungen Daten zu sammeln.

"Virtualization in the Enterprise from the Performance Management Perspective"

When you see the word virtualization, what do you think of? Hypervisors like Citrix XenServer or VMware ESX? Perhaps you thought of virtualized services like cloud computing? What about hyperthreaded multicores that facilitate virtual processors? Rather than thinking of all these forms of virtualization as being completely different from one another, this article explains why it's better to think of them as being pieces of the same performance management puzzle.The importance of doing controlled performance measurements is also emphasized.

Linux Technical Review

Anti Log Plots

A sure sign that somebody doesn't know what they're doing is, when they plot their data on logarithmic axes. Logarithmic plots are almost always the wrong thing to use. The motivation to use log axes often arises from misguided aesthetic considerations, rather than any attempt to enhance technical understanding, e.g., trying to "compress" a lot of data points into the available plot width on a page. The temptation to use log plots is also too easily facilitated by tools like Excel, where re-scaling the plot axes is just a button-click away.

Block The Emergency Exit for Faster Evacuations

NewScientist reports that Japanese physicists timed a crowd of 50 women(?) as they exited as fast as possible through a door. They then repeated the experiment with a 20 cm (10 inch) wide pillar placed 65 cm (2 feet) in front of the exit to the left-hand side. The obstacle increased the exit throughput by an extra seven people per minute.

"Usually, the exit becomes clogged by people competing for the small space, and the crowd is slowed. The pillar blocks pedestrians arriving at the exit from the left so effectively that the number of people attempting to occupy the space just in front of the exit is reduced, says Yanagisawa. With reduced crowding there are fewer conflicts and the outflow rate increases."

How does this apply to computer performance? Think polling systems, where there are multiple waiting lines or buffers but only one service facility, like a grocery store with the usual checkout lanes but only one cashier running between them! Would you want to shop in that store? In the physics experiment, the exit is the single server and the lines are the streams of people (women?) approaching the exit from all angles. The asymmetric placement of the pillar effectively reduces the number of exit streams that can form (I'm guessing).

Oracle Buys Sun Microsystems (Really!?)

I just read it (7am) and ... I'm speechless.

Thinks ....

• Larry doesn't do hardware.
  
• Decimation à la PeopleSoft?
  
• Oracle still runs on IBM, and HP, et al.
  
• Wherefore MySQL? Just a cheap shoehorn for the Oracle RDBMS?
  
• Solaris (vs. Linux, which Oracle Corp has been pushing)? Ah! SMP scalability
  
• And Java? (that made sense for IBM but...) Ah ha! Larry also owns Weblogic!
  
• Can't think... Need coffee ...
  
• Wait! What about OpenOffice? Oh oh!
  

Post café noir, this EETimes article seems to hit the salient points (modulo my JVM/Weblogic/J2EE observation). Update (April 24): The Oracle @cringely weighs in on the Sunset. [ He needs to read my blog. :) But he does have the IBM memo ]

Gunther Receives A.A. Michelson Award

At the annual CMG International Conference in Las Vegas last week, Dr. Neil Gunther was the recipient of the prestigious A.A. Michelson Award; the industry's highest honor for computer performance analysis and capacity planning. As he said in his acceptance speech, it was the fulfillment of a dream he had entertained at his first CMG Conference in 1993.

Linux Meets Zipf's Law

It's amazing how power laws seem to be seen in everything these days:

A team of researchers from ETH Zürich (Einstein's alma mater) in Switzerland used four decades of data detailing the evolution of open source software applications created for a Linux operating system to confirm the adherence to Zipf's law. The team studied Debian Linux, as it is continuously being developed by more than 1,000 volunteers from around the world. Beginning with 474 packages in 1996, Debian has expanded to include more than 18,000 packages today. The packages form an intricate network, with some packages having greater connectivity than others, as defined by how many other packages depend on a given package.

It's also amazing what passes for physics these days:

T. Maillart.; D. Sornette; S. Spaeth, and G. von Krogh. “Empircal Tests of Zipf’s Law Mechanism in Open Source Linux Distribution.” Physical Review Letters 101 218701 (2008).

Guerrilla Data Analysis Class - Seats Still Available

Most operating systems are capable of collecting hundreds of system and performance metrics every second. Even if you only record them once an hour, after a week you will have more than 50,000 samples; after a month you will be staring at almost a quarter of a million samples! But data are just data. How do you extract performance information out of all those data? Easy! You attend our new and expanded 5-day Guerrilla Data Analysis class, here in California.

In this class, computer engineering and statistics expert Prof. David Lilja presents an easy introduction to statistical methods and finally leads us into the topic of Design of Experiment (DOE) methods applied to performance and capacity planning data.

Having established the foundation theory, R expert, Jim Holtman will show you how to apply DOE and other statistical techniques using example case studies.

You can register for the class, and book your hotel room, online. Book early, book often! We look forward to seeing you in August.

Instrumentierung – die Chance für Linux?

My latest article for the German publication Linux Technical Review appears in Volume 8 on "Performance und Tuning" and discusses a possible roadmap for future Linux instrumentation and performance management capabilities. The Abstract reads:

• German: Linux könnte seine Position im Servermarkt ausbauen, wenn es dem Vorbild der Mainframes folgte und deren raffiniertes Performance-Management übernähme.
• English: Linux could be in a position to expand its presence in the server market by looking to mainframe computer performance management as a role model and adapting its instrumentation accordingly.

Topics discussed include a comparison of time-share scheduling (TSS) with fair-share scheduling (FSS) and the Linux Completely fair scheduler (CFS), how to achieve a more uniform interface to performance and capacity planning measurements, and the kind of advanced system management capabilities available on IBM System Z for both their mainframes and clusters.

Pay per VPU Application Development

So-called "Cloud Computing" (aka Cluster Computing, aka Grids, aka Utility Computing, etc.) is even making the morning news these days, where it is being presented as having a supercomputer with virtual processor units just a click away on your home PC. I don't know too many home PC users who need a supercomputer, but even if they did and it was readily available, how competitive would it be given the plummeting cost of multicores for PCs?