Become a fan of Slashdot on Facebook


Forgot your password?
United States Medicine IT

Lessons From the Fiasco 501

Nerval's Lobster writes "In theory, the federal government's Health Insurance Marketplace was supposed to make things easy for anyone in the market for health insurance. But fourteen days after the Website made its debut, the online initiative—an integral part of the Obama administration's Affordable Care Act—has metastasized into a disaster. Despite costing $400 million (so far) and employing an army of experienced IT contractors (such as Booz Allen Hamilton and CGI Group), the Website is prone to glitches and frequent crashes, frustrating many of those seeking to sign up for a health-insurance policy. Unless you're the head of a major federal agency or a huge company launching an online initiative targeted at millions of users, it's unlikely you'll be the one responsible for a project (and problems) on the scale of the Health Insurance Marketplace. Nonetheless, the debacle offers some handy lessons in project management for Websites and portals of any size: know your IT specifications (federal contractors reportedly didn't receive theirs until a few months ago), choose management capable of recognizing the problems that arise (management of was entrusted to the Medicare and Medicaid agency, which didn't have the technical chops), roll out small if possible, and test, test, test. The Health Insurance Marketplace fiasco speaks to an unfortunate truth about Web development: even when an entity (whether public or private, corporation or federal government) has keen minds and millions of dollars at its disposal, forgetting or mishandling the basics of successful Web construction can lead to embarrassing problems."
This discussion has been archived. No new comments can be posted.

Lessons From the Fiasco

Comments Filter:
  •     That sounds familiar.. I've said the same thing here and elsewhere. But it's not like my analysis is unique. There are lots of people who have done large implementations in the past. This one turned out with the expected results. They'll get it working right in a few months.

    • by khasim ( 1285 ) <> on Monday October 14, 2013 @03:07PM (#45124455)

      And you have to realize that not everyone on the team has the same goals.

      How much do the contractor companies get paid for overtime or change requests?

      When I'm a contractor I will tell you what problems there could be that I can see. But if you tell me to do it your way I'll do it your way and collect my check.

  • by zooblethorpe ( 686757 ) on Monday October 14, 2013 @02:59PM (#45124367)

    There's a cream for that.

    But we can't tell yet if your insurance will cover it.

  • by 0100010001010011 ( 652467 ) on Monday October 14, 2013 @03:00PM (#45124377)

    Remember that this is only for people that live in states that tried to stall off the inevitable. I live in Kentucky and despite being a pretty red state we have a Democratic governor and he saw the writing on the wall. Rather than try and delay and delay it we have our own. Numerous other states did the same thing. I haven't heard anything about ours being down.

    • Your Democrat governor (and several others) did not see the writing on the wall. He was simply not opposed to the system itself the way some other governors were, and worked to build it at a state level. Other states have governors who fought the program and the result is those living in those states have to deal with the broken federal one.

      The "writing on the wall" idea is nonsense. The reactions of governors have been political, not practical, as far as whether to set up state systems. One needs only

      • by jedidiah ( 1196 ) on Monday October 14, 2013 @03:18PM (#45124579) Homepage

        The political rhetoric is irrelevant. The point is that states implemented their own systems and none of them have been declared a disaster. You don't hear about any of them because they are working as intended. All of these other systems are just too boring to make the news.

        Each of them stands as an example of why the problem is not an insurmountable one and perhaps not even a terribly difficult one.

        Each one of them shames the federal government.

        • by Obfuscant ( 592200 ) on Monday October 14, 2013 @03:44PM (#45124879)

          The point is that states implemented their own systems and none of them have been declared a disaster. You don't hear about any of them because they are working as intended.

          Sadly, this is simply not true.

          Oregon had been running ads for CoverOregon [] for months prior to Oct. 1. Cute ads, catchy music, but no indication of exactly what "Cover Oregon" was. Unicorns and pixie dust.

          Come Oct. 1, the website went live. Unfortunately, they hadn't yet implemented the details of how to sign up, ignoring the basics of "how much will you have to pay" based on income, etc. That part of the website is, to this day "Coming Soon".

          You can sign up, but you have to contact a "Community Partner" (new name for "Insurance Agent") on your own. They'll point you at one, but interestingly, the law doesn't require that "partner" to tell you about anything other than the plans his company sells. Lowest rate? Well, look here at my glossy brochure, ...

          And no, this isn't how it was intended to be. It just wasn't finished, and still isn't. It did make the news, but only in Oregon media. Who cares about failures of the health care exchanges in someone else's state? BTW, Gov. Kitzhaber is a Democrat and a physician who is fully behind taxpayer-funded health care for all.

  • by schneidafunk ( 795759 ) on Monday October 14, 2013 @03:01PM (#45124391)

    I'm just curious if anyone knows of an alternative way to sign up without using the website. How many homeless have access to computers & the knowledge to use them anyway?

    • Re:Alternatives? (Score:4, Informative)

      by bobdehnhardt ( 18286 ) on Monday October 14, 2013 @03:20PM (#45124603)

      For states that set up their own exchanges, there are generally offices available as well as phone lines people can call. Many of the states that opted out are also trying their damnedest to block any perceived successes for the ACA, and have taken steps to hinder their establishment. How much help someone can expect in signing up depends entirely on what state you're talking about.

    • by jlechem ( 613317 )
      How many homeless file a tax return to begin with? I bet most of them don't have a drivers license or know their SSN number. You have to some kind of mailing address or permanent residence for these things. The people I think it would hit hardest by being online are the elderly and working poor. But they can always call the toll free number and talk to a person.
  • "Unless you're the head of a major federal agency or a huge company launching an online initiative targeted at millions of users, it's unlikely you'll be the one responsible for a project (and problems) on the scale of the Health Insurance Marketplace."

    Going by budget, even if you are the head of Facebook and Twitter, you are still not going to be responsible for a project on the scale of the Health Insurance Marketplace.

    This farce is wholly, completely, and unarguably inexcusable.

    • Re: (Score:3, Insightful)

      by ebno-10db ( 1459097 )

      Facebook and Twitter started out small and grew. That's also true of Google, Amazon, and just about any other very high volume site. This is different because they had to build a site and then, on the first day, turn on the fire hydrant all the way. I'm sure there are plenty of things they did wrong, and it was probably very badly organized. Nevertheless I'm curious what the best way to handle something like this would be. How many people have worked on a project where they had to go from zero to millions o

      • Seems like most of the state-run sites are doing OK. At the very least, they are functional if not final.

        Why are you defending a group that had 3 years to build a website, yet only decided to test it the week before it went live?

  • by djbckr ( 673156 ) on Monday October 14, 2013 @03:09PM (#45124483)
    I don't see how anybody could build a semi-complicated system from scratch in a few months. A system this big would take at least a year to get right, and that's if everything was spec'd appropriately, and the coders were good, and the project was managed well. Since the project actually got under way only several months ago, I knew it would be - at the very least - quirky. If it ran at all.
    • Re: (Score:3, Insightful)

      by jedidiah ( 1196 )

      It's essentially an e-commerce site with a government subsidy element added to it. There are any number of similar sites that already existed. They were created to fill the same basic market need by people interested in making a buck.

      Since this whole thing was a gift to the insurance industry, perhaps the feds should have considered that the industry may have made a useful partner. Let all of the insurance sales men out there be honorary do-gooders helping themselves while helping Obama's agenda.

      • It's essentially an e-commerce site with a government subsidy element added to it.

        .......integrated with over a dozen other government systems (IRS, department of homeland security, etc). It's not nearly as simple as it seems on the surface.

    • From what I heard you pretty much just register on the site, other than needing to support a rather large number of concurrent users, there are already suits of programs that do everything that this projects needs.

      You could set up a Drupal forum in a week with the required content, and then just buy some ultra plan on Amazon cloud servers.

      • by pspahn ( 1175617 ) on Monday October 14, 2013 @04:23PM (#45125393)

        I wouldn't be so sure.

        In my experience of building e-commerce sites (over roughly five years) the actual *building* of the site isn't the difficult part. I've taken a barebones install of Magento (or Prestashop, etc) and themed the front-end, by myself, in just two/three weeks. Looking at this site, I can't imagine the front-end would take any longer.

        The proverbial iceberg, though, is what you're looking at. The bits that take the most time are all the logistics bits like shipping, payment processing, which customers can purchase what, how are discounts handled, tiered pricing, product entry, admin training, etc. I would guess that 90% of my conversations with clients over the years involve some logistics bit, not whether the buttons on the checkout page are the correct color of blue.

        And then to top all that off, you have the infrastructure to worry about. You aren't necessarily dealing with a web server and that's it. You might have a cluster of web servers that need to talk to a cluster of SOLR servers. You might have to implement solutions for payment processing servers.

        In the end, these items all take a great amount of time not because of how complex they are to implement, but instead it has everything to do with the *people* that are organizing this information. Hell, can you just imagine the nightmare it must have been to get all the insurance companies to provide all their data/plans in a standardized format so they could be integrated to the store front?

        In the end, though not unexpectedly, they ran out of time and testing was shat upon. Every relatively complex site I have ever built or worked on has had testing shat upon. Now that I have just a single site that I develop and work on, testing happens all the time since I am my own boss as far as deciding what I need to work on. For every other project out there where the developers aren't the ones that even have a say in what areas are focused on, testing will always be a second-class citizen.

  • Those who fail to learn the lessons of history are doomed to repeat them in summer school.
  • they should have just sold policies through eBay and/or Amazon.
    • by lgw ( 121541 ) on Monday October 14, 2013 @03:20PM (#45124601) Journal

      You may be joking, but I think you're right.

      Most states have this worked out pretty well for car insurance. There's a great market, you buy it just like anything else with no exchanges needed, it's just another service. If you're especially high risk, the insurers are required to take you as a customer at government-limited rates, but in return everyone is required to buy insurance. Seems to work well for all involved, and for 90% of customers the government is just not involved in the purchase process (nor is your employer, nor any other third party).

      • but in return everyone is required to buy insurance.

        Only if you want to drive a motor vehicle on the public streets, and the mandate is at the state level. Two critical differences.

  • Contractors (Score:5, Insightful)

    by Todd Knarr ( 15451 ) on Monday October 14, 2013 @03:12PM (#45124521) Homepage

    Part of the problem is the usual problems with large-scale IT projects: it's not until you're well into it that you really get a grasp of what's involved. Nothing government-specific there, that plagues all large IT projects in private industry. Part of the problem, though, lies exactly in the fact that contractors were used. Contractors are mercenaries. They're here to deliver this project, and once they get their paycheck they're on to other work. They won't be around to deal with the fall-out and maintenance headaches from their work, and they don't have any vested interest in the quality of their work as long as it's good enough to pass review and get their payment check cut. In fact, poor quality is actually an opportunity to get paid twice since fixing the problems is a new project. Full-time permanent employees may not be as efficient as contractors, but on the other hand they've got a vested interest in making sure the system doesn't create any more problems than necessary because they know they're the ones who're going to have to clean up the messes. Long-term employees also have a better grasp of what's already involved in the current system, which translates directly into a better grasp of what the new system will need to do. They're less likely to miss major complications because they already have to deal with them.

    Part of the problem with contractors is also the fact that large organizations like governments limit themselves to Tier 1 contractors. And there aren't a lot of those. So it rapidly becomes a situation where the Tier 1 contractors aren't really concerned about quality and results, because they know their customers will by policy refuse to consider any alternatives outside a small set and those others aren't any better about quality. If the government switches from contractor A to B, that means B can't take on another customer who takes their business to A (because A and B are the only Tier 1 firms and the customer can't consider anyone who isn't a Tier 1 firm) and it's a net wash for A.

    • Contractors are mercenaries. They're here to deliver this project, and once they get their paycheck they're on to other work. They won't be around to deal with the fall-out and maintenance headaches from their work, and they don't have any vested interest in the quality of their work as long as it's good enough to pass review and get their payment check cut. In fact, poor quality is actually an opportunity to get paid twice since fixing the problems is a new project. Full-time permanent employees may not be

  • Why the FUCK was this broadly released instead of tested and stepped? Even people firing up PrestaShop sites aren't this cavalier.

    I can hear someone screaming about how to choose to/for whom and where to release first: how about by acceptance/ratification/support of the program?

  • Now I'm not saying incompetence isn't plausible, or even likely. But I also wonder if this wouldn't be somewhat intentional on the part of a few people as a political maneuver, whether via who the contracts went too, artificial delays, etc etc, in order to make the project become politically embarrassing. Sabotaging a co-workers project is not unheard of in the corporate world to get ahead or inhibit their credibility, so why would the government be any different...

  • by GGardner ( 97375 ) on Monday October 14, 2013 @03:16PM (#45124557)
    As a software engineer, I'm very curious about where this $400 million went. In all the articles about this project, I've never seen a breakdown of where the money was spent, at least at the granularity of people/hardware/software. Typically software projects spent most of their budgets on people, but a $400 M project that is basically a year old implies on the order of thousands of employees. That can't be right? Did they get dinged by ridiculous licensing fees from the usual suspects? Where did the money go?
    • by DNS-and-BIND ( 461968 ) on Monday October 14, 2013 @04:35PM (#45125537) Homepage
      Seeing that the current administration is, by their own admission, the most transparent administration in history, it shouldn't be difficult. Moreover an entire legion of crusading journalists are just waiting for leads to fall into their laps so they can take down high-ranking Democratic government officials for corruption. There are multiple Pulitzer Prizes waiting for those who do these deeds. So, don't worry!
    • by Jason Pollock ( 45537 ) on Monday October 14, 2013 @05:18PM (#45126035) Homepage

      It's a website that needs to be able to handle 3million visitors per day, with the majority of them being signups, or at least hitting the calculator. That's a lot of deep hits that can't be cached.

      Then, add on a back-end that has to talk to insurance companies. These guys still have a tonne of Cobol code running around. There's nothing wrong with that (Seinfeld!), but I think it might indicate that their systems aren't necessarily built for online, real-time querying.

      To recap, it is a multi-tier system:

      1) Front end, performing user signup, and calculator.
      2) Back end database. HIPA compliant, Sarbanes-Oxley compliant and able to deal with 100m customer records.
      3) Feeds to remote systems, also HIPA compliant, Sarbanes-Oxley and other stuff.

      So, you've got something that looks a lot like twitter (the back-end links), only more expensive because it needs to be Capital S secure, along with something that looks like an insurance company (the middle tier) and finally something that looks like a dot-com (front end calculator).

      That's already a lot of hardware and software. "Free" open source doesn't actually save a lot of money here, since most of the money is in support (over 1/2 the 5year cost!). Now, triple it do deal with hot site failover, backups and other various disaster recovery plans.

      Although they've had 3 years to get the system complete, the software was probably only developed in the last 10-12 months (at most). The rest of the time would have been spent in getting agreement on the data exchange formats with the insurance companies, deciding on a vendor to use for each part, and standing up an internal team to manage it. Then add in several parties involved playing schedule chicken with Congress, hoping for the whole thing to either be delayed or scrapped. Fun!

      Finally, they went for a nationwide rollout for political reasons, which was guaranteed to result in peak traffic on day 0.

      • So, something similar to what I've implemented twice in the past at different companies for $10 million in software development and $10 million in hardware? (monthly operational costs for bandwidth can get expensive, but that's not included here).

        So what does the other $600 million+ goes towards? Bribes and other assorted waste?

        Most people don't realize how bad news stories are until they see one they have personal knowledge of. Guess, what, the others are generally just as bad.

        Most people don't realize of

  • by BenEnglishAtHome ( 449670 ) on Monday October 14, 2013 @03:20PM (#45124595)


    ...experienced IT contractors (such as Booz Allen Hamilton...

    made me laugh.

    I've been in the unfortunate situation of working for a government agency when Booze Allen Hamilton came in to help make changes and improve things. They did much of the former and none of the latter.

    Typically, dealing with whoever was going to actually use the process they were changing was something the Boozers did just to check off an item on a list. They did not listen to users because they assumed government employees were all idiots and could tell them nothing they really needed to know about the processes they were about to change.

    Personally, if I were going to change business processes that had been in place for decades I'd want to talk to the people who work the current processes and find out how they work before I started trying to think up better ways of doing things. BAH never did that. They brought in workers for planning sessions, listened for a couple of days, then distilled the results of those discussions into a document of findings that was obviously written before the research ever started and contained exactly zero input from the field workers who truly understood job requirements.

    Boozers, in my organization, were almost universally so convinced that their shit didn't stink that they were worse than useless. In the course of years of contacts with them, I met exactly ONE who listened, learned, and improved things.

    Based on those past experiences, I can only surmise that the folks responsible for this current fiasco simply said "Oh, we don't need to talk to anyone from the government about how they run web sites that stand up to incredible traffic swings. We know what we're doing."

    And some idiot government executives trusted them.

    I don't know who to be more disgusted with.

  • It seems like many times when a large government entity spends billions of dollars on a large IT project to consolidate or make more efficient the handling of lots of data, it frequently ends up in massive amounts of wasted money and failed projects, with lots of pork doled out to consultancies and middlemen, and in the worse cases ends up with the project abandoned entirely with all the money down the toilet. Many examples have been posted to /. in the last 10 years.

    Are there some good cases of where the

  • by fahrbot-bot ( 874524 ) on Monday October 14, 2013 @03:21PM (#45124621)

    I don't know about you, but does the site really need links and JS from ad sites (like doubleclick, chartbeat), YouTube, and Facebook, as well as whatever googletagmanager and optimizely provide - as noticed by what I had to temporarily allow in NoScript - to simply make the site work to, you know, helps people get access to healthcare insurance?

  • Campaign team (Score:4, Insightful)

    by curunir ( 98273 ) * on Monday October 14, 2013 @03:22PM (#45124635) Homepage Journal

    I find it interesting that the team behind the technical aspects of Obama's presidential campaign were so capable (more here []'s a great read) and yet he still chose the tried and false alternate model of outsourced government contractors to handle this.

    A methodology more similar to what was used on his campaign would have been far more successful and cost significantly less.

  • >> when an entity (whether public or private, corporation or federal government) has keen minds and millions of dollars at its disposal

    Not sure there's any evidence of "keen minds" here, but I'd suspect that the root of the problem is that there were millions of dollars allocated to the project. With that kind of money, the incentive is probably to put as many billable bodies on the thing, regardless of qualifications or result.

  • What fiasco? (Score:5, Interesting)

    by rsilvergun ( 571051 ) on Monday October 14, 2013 @03:24PM (#45124669)
    This happens every time a major new internet service is launched. And it _always_ will. See, here's the problem: at launch everyone is interested and wants in. After a few weeks/months the interest dies off and the site hits a BAU point. So if you're designing one of these sites you're stuck either:

    a. Spending billions on infrastructure for 3 months tops of high volume and then getting ripped to shreds in the press for 'wasting' all that money. or...

    b. Taking your lumps up front and waiting a few months for people to forget about it.

    The guys running opted for 'b.', and I would too. The kinds of people that just want to say bad things about the ACA would have a field day with 'a.', with 'b.' they'll have to acknowledge (or at least ignore) the fact that in a few months it'll be working more or less as intended.
    • Why? I would opt for:

      c. Failing gracefully (i.e., having a plan for the suspected onslaught).

      d. Growing a new service incrementally from old, proven parts.

      I'm sure a smarter person could figure out `e', etc.

  • "Keen minds and millions of dollars".
    Maybe millions of dollars (ours), but keen minds? Not so much.

  • It's funny all the finger pointing, how the government screws up IT, etc.
    I've seen dozens of major web site projects and many other major IT projects totally screwed up. It's not government, it the human people involved and they are everywhere!
  • Central Planning does NOT work.

    Successful giant endeavors evolve organically out of small, working endeavors.

    • What you call "Central Planning" is also known as top-down design. It's a valid design method, it certainly works and is used regularly. Done properly, it produces cleaner designs than bottom-up alternative. You seem to argue for the bottom-up design to be the single correct method because it fits your right-wing ideology.

  • by jamesl ( 106902 ) on Monday October 14, 2013 @03:34PM (#45124775)

    Nerval's Lobster has a firm grasp of the obvious.

    A successful project requires ...
    1. A detailed and unchanging specification.
    2. Experienced and qualified managers.
    3. Incremental releases.
    4. Test.

    He forgot ...
    5. Realistic schedule.

  • by rbrander ( 73222 ) on Monday October 14, 2013 @03:43PM (#45124857) Homepage

    It's pushing 20 years since I first saw an academic study showing that IT project failure probability increases dramatically - the latest was 2005:

    The Challenges of Large-Scale IT Projects []

    You're darn right I won't be put in charge of such - not without a gun to my head. I'd want to de-scale anything down to a size where you could reasonably spec and test it. As the article says, "test, test, test". A formative experience in my programming was FORTH, a language that strongly rewards small incremental experiments, compiling as you go, building from small functions up to large ones. I'm not saying use FORTH, but the philosophy of getting the basics working and building up has really worked for me for a whole career.

    By contrast, all the large-scale projects I've worked on have all taken a philosophy like building a skyscraper or 747 - no one person can comprehend it, design everything before the first screwdriver is picked up, so the design process goes on for months and years, etc. And then you have "crunch time" from then on as the fond beliefs of the design team smack into reality, and the specs are proven to not match needs. Incremental building and testing tends to reveal these problems.

    The fear that drives these philosophies is that you'll have the thing mostly built...and discover it doesn't meet every need and can't without some huge rebuild, because you didn't think of everything up front. Rather like an old system that's been patched to death and has to be tossed because it just can't keep up with changes. I think the fear exaggerated, particularly if the design-build team is at least roughly aware of the whole project dimensions.

    The advantage of more-incremental projects that are never large because you take one part at a time is you develop in priority order. The 80/20 rule suggests that 80% of the clients will want about 20% of the options available - so get 20% of the offering working, and working well, first.

    Canada has this story of medical records: []
    As /. covered it, "open source" would have saved 95% of project costs, but I think it was also about the open-source development was in small increments, no large projects.

  • by Merk42 ( 1906718 ) on Monday October 14, 2013 @03:46PM (#45124905)
    Clearly this is all because of {current President} and all the rest of the {President's affiliated party}. If only this were done by {opposing party to that of the President} instead none of this would have happpened!

"Hey Ivan, check your six." -- Sidewinder missile jacket patch, showing a Sidewinder driving up the tail of a Russian Su-27