What happened at HealthCare.gov?

On Oct. 6th Federal officials admitted the online marketplace needed design changes, as well as more server capacity to improve efficiency on the federally run exchange that serves 36 states. More details in this WSJ article.

And finally, from the PR horse's mouth on Oct 20th:

"Initially, we implemented a virtual 'waiting room,' but many found this experience to be confusing. We continued to add more capacity in order to meet demand and execute software fixes to address the sign up and log in issues, stabilizing those parts of the service and allowing us to remove the virtual 'waiting room.' "

Quite apart from the bizarre architectural description, a "virtual waiting room" implies a buffer or buffers where pending requests must wait for service because the necessary resources to complete those requests are not available due to being either busy or failed. A certain amount of waiting time can be tolerated by users (both applicants and providers) but if it becomes too long or simply fails to complete, that kind of poor performance points to grossly under-scaled capacity in the original design.

It helps if you design scalability into the architecture.

Tracking Press Updates

• October 21, 2013: Healthcare.gov: It Could be Worse
• October 22, 2013: Builders Of Obama's Health Website Saw Red Flags
• October 25, 2013: Why the Government Never Gets Tech Right
• October 26, 2013: Other, less discussed, constraints.
  1. Insufficient funding by Congress. It seems that certain congressional members are less than totally supportive of "Obamacare.'
  2. Lack of readiness and cooperation for integration with the existing and heterogeneous computer systems of the actual insurance providers.
  3. Booz Allen Hamilton is one of the contractors. Where have I heard that name before?
  4. Beware Brooks' law. "Our team is bringing in some of the best and brightest from both inside and outside government to scrub in with the team and help improve HealthCare.gov."
• October 30, 2013: Breakout of the App Stack as an Infographic
• November 6, 2013: CGI Federal (largest contractor) with contracts totaling more than $200 million
• December 1, 2013: Another way to find out what went wrong is to see what they fixed.
  1. Site now supports 50,000 concurrent users.
  2. Average system response time is lower than a second.
  3. Error rate below 1%, down from 6%.
  4. Uptime now more than 90%.
• December 2, 2013: Unrealistic Technology Expectations. The website was doomed by an inordinately complex setup that tried to link disparate databases in real time. “You don’t want to query 10 downstream systems and be reliant on their performance, because you are only going to be as good as the slowest one.” [aka scalability bottleneck]