“We ran into a small snag, which hopefully won’t turn into a big snag," he said.
"We have several supercomputer-class compute clusters. One of our tests was to use one cluster to simulate traffic and run it against the other cluster. And when we did that last night, we found that the through-put we got degraded horribly when we increased the amount of traffic that we were pushing from one cluster to the other.
"So we don’t quite understand that, and that would very much degrade the through-put that we could get."
All of which begs the question: Why are you just now doing that level of testing, this close to launch day?
Anyway, the description sounds horribly familiar. Let's throw the USL, expressed in Wolfram's Mathematica, at WolframAlpha performance, shall we?
In other words, the coherency penalty (β) is an order of magnitude worse than the level of contention (α), in this example, and that produces the severe peak in the throughput curve. See my previous post for more background. I'm not sure why this would be, because I would've expected the Wα workload to be read-intensive rather than write-intensive. Reads can be made highly parallel, if you know what you're doing. So, the coherency problem could be elsewhere in the system.
They claim it will be running in 5 data centers totaling 10,000 CPUs. That's only moderately Google-ish. It will be interesting to see what happens when it finally does go live. These things have a tendency to behave themselves, until you let the public get a hold of them.