US/Canada Power Outage Task Force Event Timeline

bofus writes "The U.S./Canada Power Outage Task Force issued the Aug. 14, 2003 Sequence of Events at noon today. While no conclusions are drawn at this point, it does paint a pretty good picture of what happened and when it happened."

The blame game

[trolman]

The really bizarre part that has not been explained is that the frequency [meinberg.de] deviated. I saw frequency deviation alarms on my equipment (ups and pdus) on the data center floor in NY, NJ, and CT and this is the first time in my 20 years that this has happened. A glaring ommission is is that PJM [pjm.com] stayed up as an entity. Kudos to PJM.

The blame will be put onderegulation and lack of government oversight.

Future Prevention

[LamerX]

Hopefully this is something that they can actually use to learn from. Seems to me that they should put safeguards in between each one of these events. Because if just ONE of these events could have been stopped, it would have stopped the whole east coast from losing power.

Re:The blame game

[twofidyKidd]

Well, what exactly causes frequency deviation? I'm not terribly familiar with it, and I can only imagine that there are a few things associated with it?

Re:Future Prevention

[CoffeeCrusader]

yeah, but that would have meant, that the power providers had to provide money too. All of this could have been prevented with a bit more modern equipment. And modern equipment is expensive. In a deregulated market the power companies don't have the money to buy it. (well, usually) And to refer to a point made a bit further up: yes, this stuff matters, because there are people who care about what the masters of the power are doing. Because i like my screen to have another color than a forced black. Thus I like power. And I want to know why people cut me off of my power.

MSBlaster.exe

[devphaeton]

Has anyone followed up or concluded anything regarding the possibility of the power grid's SCADA systems (which habitually run a stripped down Win2K) getting nailed by the Blaster worm? The timing is right, and there are a number of indications thereof:
See:

this [pbs.org]or
this [automationtechies.com]or
this. [heise.de]

Bloomberg

[oZZoZZ]

Mayor of NYC decided to publicly blame Canada for this before any facts surfaced.. while Lastman, the mayor of Toronto said something along the lines of "Do you expect the US to take blame for anything?", after *some* facts surfaced
Neither responses were politically acceptable, however the media coverage of the blame game seemed to evaporate as soon as it was clear that it wasn't Canada's fault.
I found that more than a little interesting.

Late Night "Wacko" Talk Radio

[GSpot]

A couple of days after the blackout, I was staying up late and unfortunatley listed to some late night radio. Two different kooks were speculating on the cause of the blackout. One was positive that the Federal government was testing out some "advanced" weapon and the other freak was convinced that the culprit was the power companies looking for an edge to justify raising rates to build more infrastructure.

just my .02$

Re:The blame game

[Anonymous Coward]

The frequency changed as generators tried to cope with overloads. In general, the only way to increase a generator's power output is to make it rotate faster- i.e. increase the frequency. Normally this happens on a very slow scale (generally variations of +/- 200ppm excursions over the course of day, with an average of 10pmm or so during the same period to keep clocks accurate). Obviously this is somewhat coordinated among different generation plants, since they need to be more or less in sync.

Anyway, severe load changes are likely to cause severe frequency changes, severe being +/- 10000 ppm or more. Naturally, this isn't helpful, and weird low-frequency feedback effects can happen as generators in different areas try to react to each other, and to the radically changing loads.

Disturbances Before Outage Itself

[shoemakc]

Did anyone else notice a strobing effect in their fluorescent lighting in those 20-30 seconds before the full power outage? My understanding is that any sort of arc lamp (fluorescent, metal halide) will extinguish if the voltage sags beyond a certain point, so I doubt it could have been a voltage sag before the full blackout.

It almost seemed as if the power frequency itself had gone unstable...say from a nominal 60Hz to like 5Hz. Then again, with the modern electronic ballasts used today, who knows how they respond to a voltage sag. Maybe they strobe. Any one have any thoughts on this?

-Chris

Re:MSBlaster.exe

[00420]

The best part is they decided it wasn't the Blaster worm before they even started the investigation.

Re:MSBlaster.exe

[gregmac]

Has anyone followed up or concluded anything regarding the possibility of the power grid's SCADA systems (which habitually run a stripped down Win2K)

I've still never understood this. I think most systems are actually based on NT, but maybe they are migrating to 2k now. Either way, the fact that these automation systems are based on a system like windows is very strange to me. OPC (the protocol used to communicate between sensors and databases) is based on DDE (or OLE), which seems so incredibly strange to me.

I've been developing a linux-based SCADA system. I took a look at quite a few systems, and I just didn't feel comfortable running any of them for a number of reasons. Stability and security being two major issues. Another was cost - these are being deployed in small installations, mostly for remote monitoring, which wouldn't typically have a SCADA system due to the cost. Between a mixture of existing open source software, some nice hardware, and in-house development (mostly me), the system has cost us about $20k to develop, which is less than it would cost to licence most software per site.

Anyways, that was a bit OT, but the point is, very early on we decided that deploying on windows would be a pain. These are all remote installations, with no one on site that can service them. If something goes down, I want to be able to remotely fix it as much as possible. I just don't feel comfortable deploying a remote windows system and relying on it to stay running, not to mention the fact that people's health could be affected (water treatment). To me, windows is not the proper platform to be using for this situation.

Re:The blame game

[PD]

You can try this for yourself. Get a little bitty electric motor. DC will work fine. An electric is basically the same as a generator, and if you turn the motor, it will generate an electric current.

Spin the thing by hand and see how easy it is to turn. Pretty easy. Now, short out the connections to the motor and you will find that it's considerably harder to turn it. There's some resistance to the turning there.

When you shorted out the leads on the motor (which is operating as a generator), you've increased the load on the generator from nothing to something very large. And, as that load increases, the generator gets harder to turn.

Since the frequency of AC power coming out of a generator is set by the speed at which the generator rotates, it's simple to see that if the load makes the generator harder to turn, it will slow down because of that load. That makes the frequency of the AC coming out go down.

Re:MSBlaster.exe

[MagikSlinger]

Maybe Microsoft will suddenly not get a contract with the power generators anymore, but that's as far as we'll ever hear of it.

Obligatory response: Woo-hoo! Linux all the way baby!

BTW, the EULA specifically prohibits NT and up from being used in "critical" situations where life and environmental damage are on the line. So it would be the utilities' fault, not Microsoft's.

Re:Future Prevention

[AKnightCowboy]

The trick is to build the power plants in the same area that the load is located in. This reduces the amount of power that must be imported or exported from any given area.

No, the trick is to USE the power plants in the same area that the load is located in. First Energy has at least 2 nuclear power plants in Northern Ohio and one of them was shut down. Energy companies are finding it cheaper to buy electricity on the open market instead of generating their own. Yes kids, de-regulation was a horrible mistake when you do it to public utility monopolies.

The blame game

[Mashiki]

Your forgetting that the Priminister of Canada was the first one along with one of his ministers to directly blame the US, that being New York(first) and Pennsylvania(second), NY blaming Ontario(third) for the blackout.

Tit for Tat.

Re:Disturbances Before Outage Itself

[gid]

I live an North Eastern Ohio. I had two CRT monitors on and plugged into normal power. Both of my computers are on a UPS. I specifically remember my monitors dimming and then comming back a few times, all the awhile my UPS is going nuts, supplying extra battery power to keep the voltage up. My lights remained on, but dimmed. Then a few seconds later, maybe 10-15, the power totally went out.

So of course I shut down my Windows machine as fast as I could, as this isn't a beastly UPS by any means, and two computers on it won't last long. I'm skilled in the art of windows keyboard shortcuts, so shuting down was no problem. As far as writing code goes, I probably save the file I'm working on once at least every 60 seconds out of habbit, ctrl-s is your friend. :) So then a few minutes later my UPS starts really going nuts, so I shutdown my linux box, by ctlr-alt-backspacing out of x, and then giving it the three finger salute. Anyway, enough rambling. :)

Kind of odd, when the power came back, the next day my central air conditioner wasn't working after I noticed it being hotter than blazes in my computer room. The circuit breaker for that had been tripped, none of the others were.

We had a flourescent tube that had been...

[AzrealAO]

We had a flourescent tube that had been "burned out" for about 6 months suddenly come on about 2 minutes before we lost power here in Toronto.

Quebec's seperate and distinct power grid

[Anonymous Coward]

Did you notice that the effect stopped at the Quebec border? They also have a seperate and distinct power grid (in addition to a seperate and distinct culture) as a remnant of the repairs after the 98(?) Ice storm. They were quite pleased that the effect could not propogate through the DC buffers at the NY and Ontario borders

CNN

[Anonymous Coward]

It was a laugh tuning in the American networks. Canadian media was saying that there was a large power outage, CNN was saying 'We have no confirmation that this is a terorist attack' LOL They just love their hype and sensationalism

Re:The blame game

[Mashiki]

Neither were justified, and all could be considered childish. Neither knew what was going on, and neither country still really knows what "exactly happened"; as it stands the statements were tit for tat. Or politics, nothing more nothing less.

Roadmap?

[dpeltzm1]

Is it my imagination or did we just give the bad guys a roadmap on how to do this on purpose? I understand the public 'needs to know' but on the other hand i need to be able to sleep at night! and while im sure this report will fade into memory as most things in our country seem to do, i'll bet someone else will remember it just fine.