Velocity 2010 The Aftermathglow

I was so impressed with Velocity 2009, I really wanted to present something at Velocity 2010.


Thread-limited scalability of memcached
Working with Shanti and Stefan of Oracle (née Sun Microsystems), I was able to accomplish that goal. Our session was rated 92.4%, which is an A+ in anyone's books. Congrats to us and the Velocity organizers and thank you, crowd.

Of course, for someone like me, it's often more about the social than the sessions. So, it was great to meet and chat with some new people:

• Tom Cook (FB)
• Jason Dixon and Theo Schlossnagle (omniti)
• Randy Shoup (eBay)
• John Allspaw (etsy)

as well as to revisit with the people I met last year:

• John Adams (Twitter)
• Jesse Robbins (Opscode)
• Bob Lane (Yahoo)
• Jeremy Bingham (Daily Koz)
• Chetan Ahuja (Google and Guerrilla alumus)
• Alan Lapthorn (freelancer)

and a whole bunch of other people who came up to me in the hallways. It was intense.

OK, that's enough name-dropping. The tech session that I found the most interesting was the one by Tom Cook from FB. The truth is, I was not expecting to be impressed because I had already been so impressed by Bobby Johnson's talk on FB scalability, last year. Naturally, being a Facebook session, the room was packed. So, I deliberately stood at the back, near the door, in order to make a quick escape. Well, I didn't escape ... because I didn't want to.

Although Tom's presentation was more about daily ops issues, it became clear to me after a while that FB ops people have developed some intriguing scalability methodologies. They're totally a Linux shop, coding in PHP compiled with HipHop. Code changes are pushed using BitTorrent. That intrigues me. Does it work well because BT seems to configure itself as a 20-dimensional virtual hypercube on the internet? Well, as I learnt afterwards, Tom doesn't know and nor do I, but we could probably figure it out from the monitored metrics they already collect. I'd like to see more room for presentations of this type at Velocity.

Another aspect of Tom's talk that really caught my attention was that FB forces their devops to be responsible for their own performance. This sounds good but for me, it translates to producing many local optimizations, and it is well known mathematically, that local optimization does not and cannot guarantee global optimization. In other words, optimizing your local apps code will most probably de-optimize your entire web site. I discuss several war stories of this type in my Guerrilla classes. It's also anithetical to Knuth's mantra: premature optimization is the root of all evil.

Despite this, the fact remains that FB is very rarely down, so they seem to be doing something right (even if it's hiding their fails somehow Nah!). What is it that FB webops is doing that does not de-optimize their site? I would really like to know. When I raised the question with Tom later, I believe he would like to know too. Once again, I believe it would be good to have a Velocity track for this level of information.

So, why is it not there? Apparently, there is some collective wisdom at Velocity that all server-side performance and scalability issues have already been solved or are no longer interesting to attendees. I think that's a myth that needs to be busted. Our own talk on unrecognized scalability limitations in memcached provides demonstrable evidence already. And judging by the standing-room-only audience reaction, there is an interest in this depth of presentation. Similarly, the more that client-side performance gets improved, the more server-side becomes the bottleneck. Based on what I heard and saw of the browser performance track, a lot of progress has really been made there.

My guess is that this myth arose because many of the Velocity crowd don't have a framework and track for this slightly deeper level of discussion about server-side performance and capacity; like the FB topics I mentioned above. Based on my discussions with various folks, I'm quite sure there would be a high degree of interest. It's just a matter of how to get there. Providing such an avenue might be worthwhile for the organizers to consider implementing at future Velocity conferences.

Finally, I was impressed with myself for being able to drink with the best of 'em until 2am and yet, stand and deliver my talk the next day. :)