BSOD Issues On Deepwater Horizon 383
ctdownunder passes along this excerpt from a NY Times article about a rig worker's testimony concerning the April 20 accident at the Deepwater Horizon well:
"The emergency alarm on the Deepwater Horizon was not fully activated on the day the oil rig caught fire and exploded, triggering the massive spill in the Gulf of Mexico, a rig worker on Friday told a government panel investigating the accident. ... On Friday, Mr. Williams added several new details about the equipment on the vessel, testifying that another Transocean official turned a critical system for removing dangerous gas from the drilling shack to 'bypass mode.' When he questioned that decision, Mr. Williams said, he was reprimanded. ... Problems existed from the beginning of drilling the well, Mr. Williams said. For months, the computer system had been locking up, producing what the crew deemed the 'blue screen of death.' 'It would just turn blue,' he said. 'You’d have no data coming through.' Replacement hardware had been ordered but not yet installed by the time of the disaster, he said."
The article doesn't mention whether it was specifically a Windows BSOD, or just an error screen that happened to be blue.
Re:Why didn't they fix it? (Score:2, Informative)
RTFA, the workers ordered the parts and were still waiting for them when the place blew up and sank.
Re:Egregious (Score:2, Informative)
This is not the first disaster for BP that ended in the loss of life.
What's it got to do with BP? [dailymail.co.uk] The rig was owned and operated by a company called Transocean. BP (and others) just leased it off them to do the drilling (and no BP employee was involved in the actual work).
Incidentally, the company working on the well head was a company called Halliburton. They were pumping cement into the well [nowpublic.com] to prepare it when things went bad.
and at the end, its a group of companies, all blaming each other and each one trying desperately to avoid paying out. BP, to its credit, has accepted responsibility even though its almost certainly not to blame. Perhaps the US government won't be able to blame the Swiss-located Transocean (for tax reasons, 50 Transocean people work in the Swiss HQ, whereas the rest work in the USA - all 26,000 of them).
Re:BSOD (Score:5, Informative)
There is no evidence that BSODs contributed to this disaster. What is know to have contributed is the cheap cement job, plugged pressure sensors on the blowout preventer, possible damage to the blowout preventer during drilling (rubber fragments observed), and using seawater instead of drilling mud. None of these were automated.
Re:Egregious (Score:5, Informative)
I found this episode of 60 minutes quite interesting:
http://www.cbsnews.com/video/watch/?id=6490509n&tag=api
Apparently, BP was putting on a lot of pressure to do things quickly, since they were running behind schedule and it was costing them money.
Specifically, on the day of the accident, there was an argument between representatives of Transocean and BP on how to close the well (in preparation for later exploitation by another ship). Transocean was in favor the slower, safer procedure. BP wanted things to be done more quickly. They did it the BP way, which was the point when the accident happened. So, according to this report, there were BP emplyes on the Deepwater Horizont, and they influenced the procedures by pressuring their subcontractors.
According to the report, several other things had to happen as well in order for things to go wrong so badly, but I would not so easily let BP of the hook.
It was Windows NT (Score:5, Informative)
Re:Don't throw Bill under the bus (Score:4, Informative)
If it turns out these crashes are Windows blue screens, the media will jump all over Microsoft
Well, before all the Microsoft Haters pile on, according to this [deepwater.com] the Control System in place was something called Cameron Multiplex Control System [cedip.edu.mx], which I've also seen referred to as Cameron MUX and CAMITROL [c-a-m.com]. I am not pretending to be an expert in these things, just thought I'd share what little Googling turned up.
In short, it looks pretty unlikely that there's going to be a red hot poker headed toward Redmond over this.
Re:Egregious (Score:5, Informative)
Protip for all you people saying "They could have reconfigured the routers, etc." (on Childs refusal to hand over passwords) - not so much.
Why? Because Childs had either disabled serial consoles, disabled password recovery, or configured devices to -never- save configuration, only to run in RAM.
Well, shit, you say, restore the config from backups. Guess what, SF owned no backups of the configuration files, or network maps. The only configuration files Childs kept were on his personal laptop, encrypted with a key known only to him, and configured such that his laptop was the only device capable of updating configs. Network maps? Same. Sitting on his personal laptop. Nowhere else.
The guy viewed SF's network as his personal playground, and believed no-one else worthy to take the reins of it - guess what, he had no authority to decide that, and when he got nicely obstinate about it, he crossed a fairly clear line in the sand.
Stop the martyred geek defending valiantly our security creed. It bares little resemblance to reality.
Re:Why didn't they fix it? (Score:1, Informative)
For months?
Problems existed from the beginning of drilling the well, Mr. Williams said. For months, the computer system had been locking up, producing what the crew deemed the “blue screen of death.”
Re:They didn't fix a lot of things (Score:3, Informative)
BP's safety procedures are based on industry standards, which were so good that there had never been a spill in the 40 years prior to the BP spill. 40 years without incident, think about that.
Heh, heh, heh...
http://kyon.pl/img/16258,oil,map,.html [kyon.pl]
Re:They didn't fix a lot of things (Score:4, Informative)
Out of curiosity, where are you getting your information? You mean the last time there was a spill in the Gulf?
The oil spill off the Australian coast [nola.com] in August last year would seem to counter your claim entirely. Their procedures were lax and probably are still lax because those procedures are expensive. I'm not sure why people are so quick to dismiss the Gulf spill as a series of calamitous events when there is a ton of evidence to the contrary. Maybe people just want cheap gas too much and are willing to ignore all evidence so they don't have to face any consequences.
The sad reality is that oil is spilled everyday, Shell spills every year as much oil as the Exxon Valdez disaster in Alaska. Of course they spill it in Nigeria so no one cares.
Re:Safety List (Score:2, Informative)
Ada is a language that allows competent programmers to write safety-critical code. It's not idiot-proof, but average competance and dilligence will see you through.
C++ can be made as safe as Ada, but only if ou have a few top-notch coders to build some basic libraries for everyone else to use - basically, to re-invent Ada in C++.
So why would you use C++ for a project like this? You might if you just couldn't find enough comptenent Ada programmers, but you could find a few great C++ coders. I don't have any direct experience with the Ada job market, but from what I hear it's fading fast, while C++ has been very steady for 15 years (in absolute figures, not as a percentage). So this might actually make sense, if they're just adapting the project to the available talent.
There's little-to-no C++ in "Windows" BTW. The kernel stuff is written in C, and even asking why C++ isn't used might provoke violence (and the kernel stuff is really pretty solid: Dave Cutler knew his stuff, and few of the security flaws have been kernel-related). Office was developed mostly in VBA, with a bit of C, but now most of the work is in C#. Perhaps by "Windows" you meant "Internet Explorer"? How un-geeky of you.
Re:that's right! (Score:3, Informative)
but you've already agreed that regulatory regimes don't work, and don't prevent any of the problems you've mentioned.
Do you think Chernobyl was caused by greed and profit, and insufficient government oversight? (PS: It happened in _the Soviet Union_, land of small government and evil capitalists, right?)
Everything _does_ have a price. Everything.
Regarding Haiti: no, Haiti is nothing like what libertarians want. Haiti has an oppressive government with rampant corruption at every level, all the way down to the traffic cop. Authority invites corruption. Libertarians want fewer authorities. The result should be less corruption.
People, even basically good people, will behave predictably in the face of incentives. You've not addressed how you will change the incentives for corner cutting, non-compliance, and incompetence in regulatory regimes, so arguing that these regulatory regimes should be kept, strengthened, or expanded, isn't a reflection of a solution, but of an ideology.
That you are also unwilling to discuss things without resorting to insults does nothing to solidify your position.
I've read many of your posts; you're very angry, and very dismissive, but very short on arguments. It's fine to be angry and dismissive from time to time, but please show your work.
Re:Why didn't they fix it? (Score:3, Informative)
I call bullshit, a real windows admin would have enough servers that he would see blue screens from bad ram every so often.
Re:Interesting (Score:3, Informative)
Even if Windows had a BSOD it would not hurt anything. Control systems and especially safety systems don't run on Windows, they run on dedicated hardware. All the windows box is there to do is monitor what is going on with the control systems and make any configuration changes if necessary. Most of it is going to be automated, with logic running on the controllers themselves, but most of those can be overridden from the console.
The BSOD could be caused by windows, or it could be caused by the control system software running on the machine. Either way it doesn't matter, you can bring the controls software up on any machine on the network and you're back in business. Worst case scenario you lose visibility for a couple of minutes, which is about as long as it would take to recognize that conditions are changing to where a problem could be mounting.
Bypassing alerts is also common, though it usually isn't done in safety systems (which should be any controls relating to any part of the system that could potentially put the system in an unsafe condition) unless you have a very, very, very good reason. And no "the alarm is annoying and it won't shut off" is not a good reason. It sounds like they may have bypassed a critical alarm, or the part of the system that the alarm was for was not classified as safety critical when it should have been.
So it could be the operators and their supervisors fault (whoever told them to bypass the alarm) or it could be the engineering group that set up the system and didn't designate that particular part as safety-critical and make it part of the safety system.
Re:BSOD (Score:5, Informative)
Halliburton did the cementing job. http://online.wsj.com/article/SB10001424052748703572504575214593564769072.html [wsj.com]
http://www.huffingtonpost.com/2010/05/10/halliburton-deepwater-hor_n_570733.html [huffingtonpost.com]
Lack of competition is to blame too (Score:2, Informative)
I've worked on the computer systems that many of these rigs run and they are generally done using one of the following Operating Systems:
Solaris 9/10 SPARC
Windows 2003
None of them, at least none of the ones made by Seimens, Honeywell, Invensys, or Emerson run on Linux, BSD, or any other OS. The Solaris versions are being phased out in favor of Windows derivatives because developers for them and support/training personnel are cheaper to come by than those who can write code for or comprehend the workings of *NIX systems.
It is, in all honesty, a bit scary to think there are a growing number of both Power Plants (Gas, Coal, Oil, Hydro, and Wind) and Offshore Oil Rigs that run entire solutions based around Microsoft Windows platforms. While these companies state in their marketing that they offer full redundancy, options to run 2 or more "backup" stations, and even 5 x 9's SLA, both design control automation and system controls solutions are quite flawed.
The main issue here is that many of these companies offer products that have nearly no competition from others in the market. Each company has their strengths and weaknesses that a majority of the customers know about, so it's not a matter of bidding against competition so much as it is about "How quickly can you get it done and can you do it for this price?"
I hope that something good comes out of this for that Industry, they have been needing a shot in the foot for quite some time over it.
Re:the regulators were the regulated (Score:3, Informative)