US/Canada Power Outage Task Force Event Timeline

bofus writes "The U.S./Canada Power Outage Task Force issued the Aug. 14, 2003 Sequence of Events at noon today. While no conclusions are drawn at this point, it does paint a pretty good picture of what happened and when it happened."

Re:It's not the end of the world

[edrugtrader]

its not funny to bite other comedians... namely lewis black.

get a life.

Re:MSBlaster.exe

[insecuritiez]

A lot of people suspect that. I personally think it's the best explanation. However, even if Blaster caused the outage and every "expert" at the plant knew it, it would NEVER be published that way. That would open up a whole new can of worms in the public eye. A security and publicity nightmare. No, if Blaster caused anything that issue will be quietly swept under the rug. Maybe Microsoft will suddenly not get a contract with the power generators anymore, but that's as far as we'll ever hear of it.

Underfrequency Relaying

[Anonymous Coward]

Finally someone gets to the heart of the matter. As someone who works in the power industry, and knows about UFR's, I can say that they do clear the lines quite quickly in an event. :)

"A place I worked" went dark a few years ago in a similar event. A large generator's main breaker tripped, system went unstable and underfrequency, and UFR's (which are set in multiple stages) cleared the lines. Took 8 hours to recover the 2000 MW that were lost (light load).

It's all part of standard protection equipment doing its job. You see 59 Hz? Open the breakers! 58 Hz? Open more breakers! 0 Hertz? Uh oh, here come the guys in suits...

Re:Imagine if...

[NerveGas]

I'm all for it.

I have a propane camping stove, plenty of propane, and plenty of canned food around. The servers I'm responsible for will stay running, the data center has its own generators, but our office will be dead.

That means I'd get to sit at home and play Monopoly all day. If I get bored of that, there are about a hundred things I can do for fun that don't require a bit of electricity.

Shoot, it would probably even be beneficial to people in my neighborhood. I'll bet that the park down the street from me would be teeming with people outside, enjoying wholesome activities and human interaction.

Instead of neighbors walking around the block looking for code violations to report to the city, they'd probably be actually interacting with each other, maybe even solving their problems without running to a baby-sitting city government!

I'm sure there would be consequences. Analysts would talk about how many hundreds of trillions of dollars were lost, but in the end, we'd all go back to work, take care of the stuff that didn't get done, and we'd have had a good time while it lasted.

Now, if the power outtage also included me being somewhere like the island of Manhatten or on a subway when it hit, that might be a bit less enjoyable, but unbeknownst to New Yorkers and Californians, the rest of the country works a whole lot differently than they do.

steve

Re:MSBlaster.exe

[Nucleon500]

Well, there isn't much evidence, but would there be? That's the type of thing that gets covered up. (Notice how the media is careful to call msblaster.exe blaster, stripping the ms?) Yeah, my guess is someone brought an infected laptop onto the SCADA net. What would happen?

The systems are designed to run without monitoring. However, without human intervention, the systems aren't very good at staying up in exceptional circumstances. My guess is a computer failure made the grid much more vulnerable, to the extent where something routine brought it down.

Something else to add: a while ago in Ohio, a nucler power plant had its control systems down [securityfocus.com] for a while as a result of msblaster.exe. Thankfully, the plant was off anyway, but it shows that sysadmins are just as bad in power infrastructure as they are in the rest of the commercial sector.

Re:Bloomberg

[ergo98]

To be fair, before Bloomberg made his idiotic statement (which included the words "absolutely certain") various levels of the Canadian government had blamed it on a New York power station, and then a Pennsylvania nuclear power plant fire (information they got from the US Department of Defense, as a sidenote, but nonetheless it was irresponsible to repeat it verbatim so early on). I think a big difference though is that when I hear that it's a New York power plane or a nuclear station in Penn, most Canadians don't think "Those damn Americans!", or any "us versus them" nonsense. Instead it's a power plant going down, and why the hell is this grid so fragile? I suspect for Bloomberg and crew, though, it's nothing of the sort: Not only is it not "Ontario", it's "Canada" (which gives you an idea of the perception right there), but there's a definite slant of "Well that explains it right there!". This is par for the course for the various levels of government in New York state, though: Hillary Clinton has made countless nonsensical statements about how Canada is to blame for every fault in her little fantasy world.

Ohms and amps

[Anonymous Coward]

Most writeups of this event blather about "power flow" and "electricity sloshing". I wish someone would explain this in terms of resistance, impedance, and current.

"Suddenly the impedance in Michagan dropped. With Ontario as a constant-current source, the current through Niagara increased ..."

That'd make more sense, no?

Correction: MONITORING system went down

[Spamalamadingdong]

Something else to add: a while ago in Ohio, a nucler power plant had its control systems down for a while as a result of msblaster.exe.

The article you cites says it was a safety monitoring system (the backup digital one, not the old analog one running in parallel) that went down. This had nothing to do with control systems.

While it is worrisome that any system at a nuke plant could be disabled by worm traffic (appalling breach of good network operating practices), the criticality of a backup radiation monitoring system is very, very low.

Power

[Orne]

No. The Voltage component of Power is important when managing the health of the bulk power grid. For those that don't remember, Power is Volts Amps, and because current is directional, power is directional also. This is why people often use "power flow"...

The impedence in line in a function of the amount of energy flowing through it; as current increases, capacitive losses increase, causing the voltages at the ends to drop. This is sometimes called surge impedence loading. Impedence across a power line is constantly changing, and it is easier to wrap both variables into Power.

Next, the use of transformers makes amps by themselves meaningless. Power is near constant across a transformer, so High Amps Low Voltage can become Low Amps High Voltage. By talking about everything in the form of Power, then you can easily measure the transfer of energy between the various voltage levels of your system, which eases explaining the system.

Finally, Power is an easily understood market concept. If I run a generator at a low voltage (13kV), and produce 10 Amps, I'm generating 130 kW. I pipe that through a large number of transformers & lines, and deliver it to a load running at 23kV, and maybe tomorrow I sell it to someone at 9kV. By keeping everyone running in Power notation, we can all agree that money is exchanged for work, and the proper energy is delivered and paid for.

Could it have been an EMP?

[asscroft]

could it have been an electro magnetic pulse, like in that movie where they break into the vault in the vegas hotel. Only it went a little haywire and gave them more than a 20 second window. That was my first thought. Probably a little imaginative, but it's still always nice to hear someone explain why this isn't the case.

yes, yes, I know with that attitude we'd have to hear people explaining about why it's not aliens, and not due to sun spots and any other thing some wacko can think up. But at least my idea was in a recent movie, so that gives explaining it away some entertainment value.

Or are they not explaining this away for the same reason they haven't explained away the crash over pennsylvania NOT being caused by an ordered shot. I.E., we'd rather not lie to you so we'll just not answer that question.

Anyone missing some money following that blackout?

Re:The blame game

[Mashiki]

Now you get into the fun stuff you see, this is called "politics".

I'm a real fan of wondering "what if", in this case "what if" Jean and idiot boy(that would be the minister), hadn't opened their mouths and blamed the US first. Would NY having been first target by Canada for the cause have reacted quite so in blaming Ontario? Maybe, maybe not.

But, with less then friendly relations between Canada and the US well I'm not surprised that the first reaction out of Ottawa...I mean from our Priminister sitting in Quebec with Hydro at his cottage while Ontario was without power and not saying a word except "blame the US".