Gone Guerrill_ R on Our Data

Here's a summary of some things we learnt about applying R to computer performance and capacity planning data in the GDAT Class last week.

• Neural nets pkg nnet applied to CPU performance data in the Ripley and Venables book (see Section 8.10).
  
• How to do stacked plots that Jim calls "spark plots."
  
• Jim told us that ggplot has a nice GUI but considerably slower than using the base plot routines.
  
• Use of POSIXct to convert timestamps.
  
• Handling multi-line headers.
  
• Handling multi-word fields in headers.
  
• To make getwd() like the UNIX shell command: pwd<-function(){cat(getwd())}.
  
• Think of lapply as a vectorized for-loop.
  
• Calculating confidence intervals, which David explained earlier in the week, is available as the CI function in gmodels pkg on CRAN.
  
• Fourier Transform Your Data. This was done using Mathematica but the same thing can be accomplished with the fftw pkg on CRAN.
  
• VAMOOS your data.
  

If you want to learn things like this, then consider putting this GDAT class on your calendar for next year.

GDAT: Fourier Transform Your Data

The Fourier theorem essentially states that any arbitrary continuous function can be constructed by adding together sine and cosine functions with appropriately chosen amplitudes, frequencies and phases. This is what distinguishes two musical instruments, e.g., a violin and a trumpet, when they are both tuning to the same concert-A pitch. Each is playing the same fundamental frequency (440 Hz) but the higher harmonics; the additional sines and cosines overlaid on top of that fundamental tone, are what allows your ear to distinguish the violin sound from the trumpet sound.

Here is a nice little video demonstration of the Fourier theorem in action using a Hammond B3 electronic organ—an instrument capable of mimicking other instruments through the use of the Fourier theorem.

What would happen if we tried to apply the Fourier theorem to performance or capacity planning data?

GDAT Visualization: Black Friday at eBay

This animated heatmap visualization of Black Friday transaction volumes at eBay


was brought to our attention in the GDAT class today, compliments of Matt C. from PayPal.

Florence Nightingale was a Statistician

Florence Nightingale was elected the first female member of the Royal Statistical Society in 1859 and she later became an honorary member of the American Statistical Association. She also travelled with a pet owl in her pocket. [Source: Graham Farmelo]


Moreover, she was also a pioneer in data visualization by virtue of developing a form of pie chart known today as the polar area diagram.

World Datacenter Storage at 1 ZB

Heard on the BBC World Service:

"The world is drowning in a sea of data. Facebook users alone are uploading more than a thousand photos a second. We're now seeing an exponential explosion of information. So how much information are we really storing?"

Go Guerrill-R on Your Data in August

Only one month to go! Register now for the Guerrilla Data Analysis Techniques (GDAT) class to be held during the week of August 9-13, 2010. The focus will be on using R and the PDQ-R for computer performance analysis and capacity planning.



(Click on the image for details)
For those of you coming from international locations, here is a table of currency EXCHANGE rates. We look forward to seeing all of you in August!

Prime Parallels for Load Balancing

Having finally popped the stack on computing prime numbers with R in Part II and Part III, we are now in a position to discuss their relevance for computational scalability.