Tech Titans Oracle, Red Hat and Google To Help Fix Healthcare.gov 404
wjcofkc writes "The United States Government has officially called in the calvary over the problems with Healthcare.gov. Tech titans Oracle, Red Hat and Google have been tapped to join the effort to fix the website that went live a month ago, only to quickly roll over and die. While a tech surge of engineers to fix such a complex problem is arguably not the greatest idea, if you're going to do so, you might as well bring in the big guns. The question is: can they make the end of November deadline?"
Answer: No. (Score:5, Insightful)
Nine women cannot make a baby in one month.
Re:Answer: No. (Score:5, Insightful)
Nine women cannot make a baby in one month.
True, but the website already exists. If it's a case of fixing defects rather than re-architecting from scratch, there's no reason why multiple teams can't work on different parts of the system. And multiple people within a team can't work on different defects.
Defect fixing is indeed somewhat scalable.
Re:Answer: No. (Score:5, Insightful)
It all depends on the quality of the existing code base. More often than not, it's better to start from scratch.
Why can't they start over ? (Score:3, Insightful)
Instead of fixing a bunch of hopeless code, why can't they start over the damn thing - with a properly designed paradigm ?
Re:Why can't they start over ? (Score:5, Insightful)
That's my idea. Government hired an incompetent contractor to build something. They built a freaking MESS. Just clear it all out. Sure, examine the code, see what the ideas were when they built the site. Take the best ideas, and rebuild the ideas, from the ground up.
Years ago, I was called in to a construction job, where the previous foreman had really screwed up. He built a foundation and wall in the wrong place. We didn't try to make the wall fit into the plan - we wrecked the frigging wall, poured a new footer, and built the wall on top of our new footer.
The site designers need to do the equivalent. Consider the "blueprint", see where everything went wrong, tear out the screwups, and build from the ground up. If that should happen to mean that not one single line of code remains, then so be it. If it means that 1/4 or 1/2 or even 3/4 of the code can be reused - fine. Just get it working. And, do it for less than another half billion freaking dollars!!
Re:Why can't they start over ? (Score:5, Funny)
Oracle's involved, so good luck with that.
Re:Why can't they start over ? (Score:5, Insightful)
Actually, government hired far too many contractors as everybody wanted a piece. Now they are doing the same again. Have one competent entity fix this mess, not a lot of them and especially not a lot of them that are not used to cooperating.
Re:Why can't they start over ? (Score:5, Insightful)
HHS was supposed to provide the supervisory role. Problem was they didn't have the experience to do such a thing. In a way, they were stuck. If they'd've hired a single contractor, they'd still be in litigation because the others would have sued. Hiring many meant they couldn't use a single company to ride shotgun because companies don't play well together in shotgun marriages.
They should have had the NSA do it. I hear they are quite good a building large systems.
Re:Answer: No. (Score:5, Interesting)
But all 3 companies listed will have those rock stars that will:
a. look at the code and call is rubbish.
b. ask to rewrite the whole thing
c. charge an arm and a leg to do it within time.
d. run it under agile (so THEY control the requirements, not the domain experts).
Really they should have hired the guys that do turbotax and such.... it works for the type of users on this healthcare system. The above 3 will struggle through it as well... but will milk it for all it's worth.
All I say to the Obamacare management team & Obama: TAKE A STEP BACK, WAIT.... ASSESS THE PROBLEMS one by one, THEN HIRE THE RIGHT FOLKS. This is a knee jerk reaction and will go down in flames. Of course, the valley and wall street is loving it....
Young MBA folks: this is your Y2K computer problem moment. Remember those times: the panic, the flooding of cash, and nothing happened afterall? Yeah, get ready for another internet boom/bust.
Re: (Score:3)
b. ask to rewrite the whole thing
Seeing as it's Oracle, Redhat, and Google.... the application will probably be:
Rewritten to run on Oracle Java, throwing away that old Visual-Basic code.
Leverage Google AppEngine and BigTable for data storage, instead of the Microsoft Access-based backend
Run on 64-bit Redhat Enterprise Linux servers, instead of 32-bit Windows 2003 and XP servers running IIS
In other words..... it ought to be a smashing success
Re: (Score:3)
Comment removed (Score:5, Interesting)
Re:Answer: No. (Score:5, Insightful)
For working, debugged, stable code that looks messy, that's almost always true. But this site failed immediately when put under load.
Now, if the site logic for ONE user is sound, then they could preserve that and put in the infrastructure needed to handle the sheer electricity of thousands of requests per minute. That's what Google is known for, with their vast datacenters and ability to load balance. Oracle is known for databases able to handle high concurrent transaction loads. Red Hat can provide support on a reliable, robust operating system (Linux).
For 654 million dollars, hopefully the government got the logic and blueprints down for how one user is supposed to progress. Now, the folks who know how to handle the sheer electrical volume of the massive numbers of connections can perhaps install that missing, essential portion of the website. IF of course, the design and logic of the site for one user is sound.
Re: (Score:3)
Re:Answer: No. (Score:5, Interesting)
Almost universally in software development, starting from scratch is a stupid fucking idea repeated by inexperienced developers.
When the code is an unsalvageable pile of crap; sometimes it does make more sense, to reevaluate the design, and re-implement the entire application properly, using the old code only as a reference; than to try and repair.
Re:Answer: No. (Score:5, Insightful)
The last time I had to "re-architect" an existing website, I ended up putting in roughly twice the amount of time as the original "architects" (and I use that word very very loosely). Believe me, there's a lot of shit out there that will require a lot more effort to fix than originally went into building it.
Re: (Score:3)
Yes, because it's likely they'll have to put as much effort into fixing it as the original designers, if not more.
Re: (Score:2)
Yes, because it's likely they'll have to put as much effort into fixing it as the original designers, if not more.
Lets hope so.
Or lets hope they have the common sense to start over.
Comment removed (Score:5, Insightful)
Re:Answer: No. (Score:5, Interesting)
Rare? Nearly every government contract offered to the private sector since Bush took office has been no-bid. Remember the deals made during the Iraq War? Every single one of those was no-bid to Halliburton. This kind of cronyism is NOT rare at all; it is the norm, and has been for over a decade.
Re:Answer: No. (Score:5, Insightful)
there's no reason why multiple teams can't work on different parts of the system
You've never worked on anything with multiple teams working on different parts, have you?
It never fucking works. You need knowledgeable oversight.
Re:Answer: No. (Score:5, Insightful)
Exactly. I think the Linux kernel is a damned good example of how a large number of developers working in very different kinds of development environments, some working in side-projects like Netfilter, are coordinated by one guy intimately acquainted with the kernel.
You can say what you like about Linus's attitude at times, but the fact that the Linux kernel is running on everything from supercomputers to be Nexus 7 tablet tells you that there is a way to successfully and productively organize multiple teams to produce a successful software product.
Re: (Score:3)
Oh BS. A kernel is a helluva lot more complex. Big or small, sites like Healthcare.gov are, no matter how you look at it, scripts gluing together database queries. A big job, to be sure, but certainly not one invoking the complexities of a modern-day kernel.
Re:Answer: No. (Score:5, Informative)
> True, but the website already exists. If it's a case of fixing defects rather than re-architecting from scratch, there's no reason why multiple teams can't work on different parts of the system. And multiple people within a team can't work on different defects.
You are assuming that there is a detailed (and accurate) functional spec, design spec, and that the code is organized and well-documented - and that it is architected in such a manner that throwing more engineers at it will actually fix the problem. More often than not, that is not the case.
Re: (Score:3)
More likely than not, there are likely components which will have to be entirely rewritten. While many of the bugs are trivial defects, it looks like in many areas, the design is just inherently flawed at the root. Particularly seeing the performance issues, I can't help but believe that it's just fundamentally architected in a very poor way, and while there might be quick hacks to at least get it standing, it won't really be functioning properly without seriously ripping out the internals.
But let's take a
Re:Answer: No. (Score:5, Funny)
No but I heard 18 Women can do it in two weeks. The guy from Infosys told me so.
Re: (Score:2)
That only works in Bangalore.
Re: (Score:2)
That only works in Bangalore.
Bang-a-Lot?
Re:Answer: No. (Score:4, Funny)
Re:Answer: No. (Score:5, Insightful)
Re:Answer: No. (Score:5, Funny)
Re: (Score:3)
.... with a vengeance. And this time, its personal .... health insurance that's at stake.
At least the stakes are low. No worries.
Obama Officials In 2010: 93 Million Americans Will Be Unable To Keep Their Health Plans Under Obamacare [forbes.com]
Re:Answer: No. (Score:5, Insightful)
That article is so full of contradictory statements it's ridiculous. Which isn't to say I'm defending the excessively sugar-coated defenses the administration made in 2010. But 94 million is an upper limit, and it's mostly composed of private insurers and private companies purposefully choosing to change coverage, not because the law mandates it.
And let's not forget about the 20-40 million people who will be unable to keep their lack of insurance coverage. What's the difference between being uninsured and underinsured? Maybe I should be allowed to get a car insurance policy with a $100 limit. I mean, freedom, right?
If you want to diss the ACA, then diss it on its merits.
I hate taxes as much as the next guy. More, in fact. My combined income is over $240k/year, almost all earned income, so its taxed heavily. It's a gigantic bitch. But you know what? I grew up in poverty, in foster homes. I benefited from a safety net. And the elder members of my family all depend on some sort of government assistance. So I just suck it up, because as the extremely conservative Justice Holmes once said, taxes are the price of civilization. And this civilization let's me make almost a quarter of a million per year. You think I could make that in Mexico, Brazil, or China?
The penalties for having no insurance are is like $150/year. If you can't afford that, then you have bigger problems--and in any event, if you couldn't afford it the government would pay for it.
Re: (Score:3, Informative)
Taxes may very well be the price of civilization, but what those taxes are spent on may be efficient and valuable, or destructive and wasteful. They can build bridges that are needed, and in a useful place, or expensive bridges to nowhere. The ACA is proving to be badly thought out, badly implemented, justified by lies, and seems to be headed towards being a train wreck for the American people, the economy, the healthcare industry, and even the Democratic party. It is already driving many jobs out of the
Re: (Score:3)
the Mythical Man Month returns
It never went away.
Re:Answer: No. (Score:5, Insightful)
Nine women cannot make a baby in one month.
But I bet even one woman could spell cavalry, and know the difference.
Slashdot editors wanted. No Experience needed. We wouldn't know what to do with experience if we tripped over it.
Re:Answer: No. (Score:5, Funny)
They didn't mean like horses and stuff... Wow, they like totally meant Calvary [wikipedia.org] - cause that's like the most common saying ever, you know, calling in the ancient name for Golgotha, the place just outside Jerusalem. You and your horses.... cavalry indeed. Preposterous!
*sips coffee*
Re:Answer: No. (Score:5, Funny)
Someone is going to be crucified before this is over...
Re:Answer: No. (Score:4, Funny)
But be back with more wiseass comments in three days...
Re:Answer: No. (Score:4, Insightful)
Interestingly said.
Even if by some miracle, they bring something up, it doesn't fix the actual problems. Ridiculously increased rates, it's a new tax on everyone, lies about keeping one's old policy and a general over-all burdon on the remaining who are employed above the poverty line.
Re: (Score:3)
That's a valid analogy for babies.
One person could have designed, programmed, and coded the whole site, including back end in about six months, If that person were skilled.
Instead, HHS and CMS paid multi-million dollar contracts to 3 foreign corporations, who had a year and still couldn't do it.
The site was doomed by salespeople and politicians.
QSSI, who got the contract for the EIDM in 2012, evidently got it working for medicare and medicaid, but this site wasn't even coded or tested right.
CGI Federal exec
Comment removed (Score:5, Informative)
Re: (Score:3)
Demand that further payments using taxpayer money not be made to CGI.
Re:Vermont's Site is Toast (Score:5, Interesting)
All Vermont needs to do is buy a copy of Kentucky's system. Kentucky's system works fine.
Calvary? Really? (Score:2, Insightful)
I think it's cavalry.
Calvary? (Score:5, Funny)
It's a Biblical reference -- and at this rate it would take divine intervention.
Re:Calvary? (Score:5, Funny)
It's a Biblical reference -- and at this rate it would take divine intervention.
This is government, nobody gets crucified, they all get promoted.
Re: (Score:3)
It's a Biblical reference -- and at this rate it would take divine intervention.
Minus that it's going to hell in a checkout basket.
Re: (Score:2)
Re:Calvary? Really? (Score:4, Funny)
Of course maybe it was a literary illusion. ;D
Re: (Score:2)
I think you mean "allusion". Unless that was some illusion passing me by, going "whoosh"...
Amazon (Score:5, Interesting)
I think they should have just listed the plans on Amazon. Almost everyone already knows how to buy stuff from them and their servers would have handled it.
Re:Amazon (Score:5, Informative)
Prices and availability vary hugely for the same insurance plan for different people. Amazon has no way of handling that.
Re: (Score:3)
Amazon on occasion has posted different prices for different customers.
As I remember they got in trouble for doing that, so I think you're right, they probably don't still have a way of handling it.
Re: (Score:2)
You could already buy all of the available plans through einsurance.com. I do not see what value the government website adds.
Re:Amazon (Score:4, Funny)
Just what we need. One-click insurance from Amazon. :P
if only they could fix healthcare.gov by (Score:2, Insightful)
bombing the hell out of it!
Let's see.. (Score:5, Funny)
In two months the site will be using Oracle and Ellison will charge the Feds a fortune for the license fees.
Google will start mining every piece of data it can get off the website, of course the NSA will be stealing that and stashing it in Utah.
Red Hat will push it all to RHEL which isn't necessarily a bad thing.
Re: (Score:2)
In two months the site will be using Oracle and Ellison will charge the Feds a fortune for the license fees.....
Yea you gotta believe Larry is foaming at the mouth at this. Selling them a named-user license and getting paid for every man, woman, and child in the US? Cha-Ching!!!
Re: (Score:3)
He has to do something since his pay package was not approved by shareholders today. Out of the 1.6 Billion shares that voted for his pay he owns 1.1 Billion... He made $78 Mil last year (Fiscal Year ended in May) I guess this government deal will generate some license fees and the shareholders definitely sent a message to the board that you can't keep paying this sleeze that kind of money unless he produces results.
http://dealbook.nytimes.com/2013/10/31/oracle-shareholders-oppose-compensation-for-elliso [nytimes.com]
Red Hat? (Score:4, Funny)
No Microsoft? lol :)
Re: (Score:2)
No open source, either.
Oracle! YES!! (Score:2)
Re:Oracle! YES!! (Score:4, Interesting)
Re:Oracle! YES!! (Score:4, Interesting)
One of the many problems is that most people do not know how to tune Oracle. Properly tuned Oracle, even when running on inadequate hardware, oracle can support TPS levels that many DB's only dream about with full ACID as a matter of course on the same hardware. I have watched Postgres, MS-SQL Server and DB2 just hit the floor while Oracle kept chugging right along, not always mind you, but more often then not.
I am currently running 11gR2 on hardware that is at best adequate and can assimilate the entire output of 80% of the state of California's highway loop detectors ( approximately 50,000 raw data rows inserted every 30 seconds 24/7/365 ) and that into a rather poky 15TB drive array with 7500rpm 2TB drives, in raid 5 no less, then query all of that data filter,clean and analyze it and shove that data into another table all in the same 30 second period.
The DMV project was a nightmare of never ending changes of requirements. When you think about the basic project, it aint that hard, but when there is no point at which you could say it was stable because the target just kept moving, I don't care who takes it on or who's DB engine you throw at it, it will fail.
When it comes to scaling something out, you take you best guess at what you load will be. When your prospective load might be a large percentage of 300 million people it is a hard target to pin down and that is what ( along with a few bugs that escaped unit testing ) was their ultimate undoing. No one knows who's DB engine was behind it but I doubt it was any of the "web scale" DB's since they don't support ACID very well and this was one of those when it was absolutely essential.
Re: (Score:3)
1,700 inserts per second is pretty easy for any modern RDBMS on decent hardware. Here's PostgreSQL doing 14,000 per second [blogspot.com] on a laptop, for example. My company routinely handles 40,000 inserts per second into Cassandra on midrange AWS virtual servers.
Properly tuned Oracle is probably pretty quick, but these days so is properly tuned everything else.
Comment removed (Score:5, Interesting)
Re: (Score:3)
At the same time, it's kind of entertaining to watch the general public start to grapple and become aware of the same project management issues I've had to deal with for the last decade.
Re: (Score:2)
My guess would be that healthcare.gov runs on top of an Oracle DB already, so Oracle probably has a few engineers that can be brought in to help identify and restructure problematic queries and/or tweak server settings to eek out a bit more performance. It's also a distinct possibility that the back end is in Java, so Oracle has a few knowledgeable Java engineers too.
Oracle's one-word answer for the DB -- Exadata...
Brooks Law (Score:5, Insightful)
Re: (Score:2)
Brooks Law states "adding manpower to a late software project makes it later".
+6 man, +6. That is exactly what first came to mind when they went for that "surge" mentaphor.
Second thing that comes to mind is that the surge didn't work, it just happened to coincide with a change of local Iraqi politics (locals got sick of extremists killing locals instead of just americans so they started outing the extremists so the americans finally knew who to kill).
Re: (Score:3)
"Failure to allow enough time for system test is peculiarly disastrous. Since the delay comes at the end of the schedule, no one is aware of schedule trouble until almost the delivery date."
"False scheduling to match the patron's desired date is much more common in our discipline than elsewhere."
"Take no small slips. That is, allow enough time in the new schedule to ensure that the work can be carefully and thoroughly do
Will they teach Economics? (Score:2)
I, for one, am glad to see government doing something right. They have fallen short of privatizing the site, but....
Will the three tech giants also teach Economics?
Re:Will they teach Economics? (Score:5, Insightful)
The government should have done it in-house, using directly hired citizens as developers and project managers. Use top developers that fully understand the selected technology. This site is something that will be changing a lot over many years, so continued staff where most developers already know how it's built would keep it upgraded.
Re: (Score:3)
That's the kind of idea that sounds great until you get to the details. Who actually employs them, how do they get hired, who watches over the project managers as a stakeholder?
The reason contractors get used is they offload all of these problems.
"I know someone who is employed by the government therefore they can hire people directly." "My brother works in IT, they can just shore up the team and have them do the website."
No, these do not work. Adding infrastructure to handle these employed people is an
Re: (Score:3)
Really? When you go shopping (possibly on a tight budget), you don't care about knowing the prices until you reach the register? "Don't worry, that box of pasta says $150, but it'll probably cost somewhere between $0.29 and $48 when you reach checkout. Just toss a dozen in the cart." To the end user, being presented *the amount they'll pay* while they're shopping is pretty important --- tacking on a random-number discount at the very end wouldn't make a helpful system.
What is it originally coded in? (Score:2)
The choice of these companies makes it obvious that it is not an asp.net fix. Being from Canada I have no idea what front end the site is using in the first place. But if it is not a based upon Microsoft style asp.net in the first place then you can bet that the choice of who gets government contracts will be effected in the future.
Here in Canada the government has completely sold out to Microsoft and in some cases if you need to access government services on the net it is all coded in asp.net especially t
huh? (Score:5, Insightful)
I can understand Google and Redhat... but Oracle? Talk about having a fox in the hen-house.
Re: (Score:3)
Too Many Cooks Spoil the Soup (Score:3, Funny)
Spread out the demand (Score:2)
Why does everyone in the country need to use the website at once? Couldn't the problem be fixed with a little javascript function:
1. Enter your Social Security Number
2. Based on your Social Security number, your enrollment date is 1-Nov-2013 - 7-Nov-2013 or anytime after 31-Dec-2013. If you do not know or do not have an SSN, your enrollment date is after 15-Jan-2014. Click here to have an email reminder sent on your enrollment date.
They could instantly cut the website demand by 90% by dividing enrollments
Re: (Score:3)
They could instantly cut the website demand by 90% by dividing enrollments up by the last digit of the SSN of the primary enrollee.
There aren't enough people as it is to pay double and triple for health plans that they don't need. I, personally, have no desire to even visit "that website," whatever URL it may have. I can pay for my own healthcare without involvement of moneychangers.
Re: (Score:3)
They could instantly cut the website demand by 90% by dividing enrollments up by the last digit of the SSN of the primary enrollee.
There aren't enough people as it is to pay double and triple for health plans that they don't need. I, personally, have no desire to even visit "that website," whatever URL it may have. I can pay for my own healthcare without involvement of moneychangers.
Unless, of course you suffer a catastrophic illness or injury. I know someone whose husband slipped while getting out of the shower, he hit his head on the floor, and ended up with a brain injury and needing brain surgery and months of rehabilitation. So far it's cost over half a million dollars. He was in his 30's, a triathlete in perfect health. Fortunately, he had insurance and his wife was able to take 3 months leave to care for him and can support the household on her income.
Few people can afford a $50
Re: (Score:3)
So, to summarize...
Your utopi
Re: (Score:3)
Nazis killed their insane patients; that campaign preceded the rounding up and killing of Jews.
My view of the problem comes from purely financial side. Consider the following: (1) Healthcare costs money, and (2) you do not have money. You can have only two solutions: (a) you don't get healthcare, or (b) you do get healthcare, but someone else pays for you.
The (b) is traditionally reserved to those who the state officially considers to be unable to work. Those would be children, and adults with injuries
Google?.... (Score:5, Funny)
Oracle's involved? (Score:2)
And we thought it was expensive and past deadline NOW.
Good Luck (Score:2)
But in all seriousness, the reason this web site is in shambles is because the developers weren't given nearly enough time to implement a product this complex. And if years of development wasn't enough time, the government thinks that a few big tech
Called in the calvary? (Score:3, Funny)
Do we know what the current architecture is? (Score:2)
Who here has used the site? (Score:3)
I have. It's not that bad. Really.
Now I don't need insurance as I already have it from my employer, but I was curious how bad the site was. But it didn't turn out being difficult or error prone at all to sign up. It took about 15 minutes total and I had the eligibility report for me and my daughter. Some nit picks:
1) The confirmation email was one of three emails i got from healthcare.gov when signing up. That could confuse some people.
2) One required field on one page was scrolled off the bottom, and no scroll bar appeared to indicate that. Mouse wheel scrolling down solved that, but if there are many pages with that problem it could be confusing.
That's about it. Maybe I just lucked out, bit it was an easy site to use.
Re:Why not IBM (Score:4, Informative)
IBM certainly made sure the Nazi's CRM system worked right.
Re:Why not IBM (Score:5, Interesting)
No, they wanted it done and not outsourced to India.
Re: (Score:2)
As an Anonymous Coward, I am very concerned that proper language be used only when it places me in a position of higher authority.
The word for "soldiers who fought on horseback" is cavalry.
The word for "a hill near Jerusalem where Jesus was crucified" is calvary.
The word for "ice cream that's really expensive and super fucking creamy" is "Carvel".
The word for "fibrous green shit that children and guinea pigs eat" is "celery".
The word for "the poison center of a Milk Dud" is "caramel".
The word for "that shit you bruised when you gave your wife a raging tsunami" is "clavicle".
Re: (Score:2)
He has already succeeded at that, and if ObamaCare gets off the ground it will be like 10x over all prior presidents combined.
Re:Just say no (Score:4, Insightful)
In that scenario, we'd actually be worse off - the ones with principles wouldn't be working on it...
Re: (Score:2)
Nah, just buy it in the Google Play Store.
They could have written an Android and Apple app by now.
Re: (Score:2)
And Elon Musk hasn't even been MENTIONED yet?!
He already designed a solution based on "a series of tubes", but it was dismissed as being impractical in the real world because it didn't involve enough contractors for implementation.
Re:And... (Score:5, Informative)
It's not 600 million.
http://www.washingtonpost.com/blogs/fact-checker/wp/2013/10/24/how-much-did-healthcare-gov-cost/?wprss=rss_politics [washingtonpost.com]
Re: (Score:3)
North Korea has Democratic and Republic in its official name, this does not implies that it is a democractic republic and it does not implies democracy and republic are totalitarism.
Same fro NSDAP: the fact that someone grabbed and kinked a concept does not invalidates it universally. And we we talk about "social" in the US, it has nothing to do with soviet Russia.