US/Canada Power Outage Task Force Event Timeline 303
bofus writes "The U.S./Canada Power Outage Task Force issued the Aug. 14, 2003 Sequence of Events at noon today. While no conclusions are drawn at this point, it does paint a pretty good picture of what happened and when it happened."
text version (aka karma whoring) (Score:5, Informative)
2:02:00 - 2:02:00 PM - Transmission line disconnects in southwestern Ohio
3:05:41 - 3:41:33 PM - Transmission lines disconnect between eastern Ohio and northern Ohio
3:45:33 - 4:08:58 PM - Remaining transmission lines disconnect from eastern into northern Ohio
4:08:58 - 4:10:27 PM - Transmission lines into northwestern Ohio disconnect, and generation trips in central Michigan
4:10:00 - 4:10:38 PM - Transmission lines disconnect across Michigan and northern Ohio, generation trips off line in northern Michigan and northern Ohio, and northern Ohio separates from Pennsylvania
4:10:40 - 4:10:44 PM - Four transmission lines disconnect between Pennsylvania and New York
4:10:41 - 4:10:41 PM - Transmission line disconnects and generation trips in northern Ohio
4:10:42 - 4:10:45 PM - Transmission paths disconnect in northern Ontario and New Jersey, isolating the northeast portion of the Eastern Interconnection
4:10:46 - 4:10:55 PM - New York splits east-to-west. New England (except Southwestern Connecticut) and the Maritimes separate from New York and remain intact.
4:10:50 - 4:11:57 PM - Ontario separates from New York west of Niagara Falls and west of St. Lawrence. Southwestern Connecticut separates from New York and blacks out.
Re:text version (aka karma whoring) (Score:5, Funny)
Think he must have watched a bit too much South Park!
The blame game (Score:3, Interesting)
Tit for Tat.
Re:The blame game (Score:3, Funny)
Sometimes our elected officials act like school children. "I'm telling MOMMYYYYY"!!! I'm glad they aren't representative of those of us here on slash. . . oh . . .wait.
Re:The blame game (Score:2)
Re:The blame game (Score:3, Interesting)
Re:text version (aka karma whoring) (Score:5, Funny)
Re:text version (aka karma whoring) (Score:5, Funny)
Re:text version (aka karma whoring) (Score:2, Informative)
Ahhh...
I always heard they could have had French culture, British law and American economy. I guess the key being Government instead of Technology. French technology includes Dassault - who make the core avionics software in every modern military aircraft , including U.S. They also have a decent
...the class system? (Score:2)
Re:text version (aka karma whoring) (Score:2)
let's hope there isn't one for electricity too!
Re:text version (aka karma whoring) (Score:5, Funny)
Friday September 12, @03:58PM: Server goes down due to unusually high traffic.
The last entry: (Score:5, Funny)
First Energy the real problem (Score:3, Funny)
Re:First Energy the real problem (Score:2)
<RANT type="paranoid>
The power outage occurred after the SEC requested anything. The outage was First Energy's way of asking, "Are you sure you want to mess with us?"
</RANT>
It's not the end of the world (Score:3, Funny)
Re:It's not the end of the world (Score:3, Insightful)
get a life.
I CAN'T READ IT. MY SCREEN IS ALL BLACK. (Score:4, Funny)
The blame game (Score:5, Interesting)
The blame will be put onderegulation and lack of government oversight.
Re:The blame game (Score:2, Interesting)
Re:The blame game (Score:5, Informative)
The trick is that the control system can only react so fast - suddenly disconnect an entire town, and the load drops, causing the power in to spin the generator too fast. If the control system overcorrects, then you'll get too low of a frequency. If a far-away generator drops out and you've got to supply more current to your local region, then the demand has gone up, slowing the frequency.
If you've been around generators, you can hear this exact phenonemoa - if the load changes suddenly, the motor will hunker down a little and then catch back up to normal speed. Usually a flywheel can damp out extremely short transients, but it would be prohibitively big if it were sized to handle transients as large as the control system (throttle) will allow.
Re:The blame game (Score:2, Interesting)
Anyway, s
Re:The blame game (Score:5, Funny)
what exactly causes frequency deviation? I'm not terribly familiar with it
Glad you asked.
It's a highly technical term that refers to the frequency of the AC sine wave of electric power that is delivered over your lines.
This frequency is related to the 60 Hz rate of revolution maintained by most generators.
Occassionally, something will happen to upset this steady rate of revolution of the generators.
In particular, Homer, a donut that falls into the space between the stator and armature will cause this problem.
Lubrication from jelly-filling in the donut can help stave off the inevitable disaster of frequency deviation, but it is simply an old-wive's tale to believe that pouring coffee into the space will ungunk the works in time.
Re:The blame game (Score:2)
Re:The blame game (Score:2, Interesting)
Spin the thing by hand and see how easy it is to turn. Pretty easy. Now, short out the connections to the motor and you will find that it's considerably harder to turn it. There's some resistance to the turning there.
When you shorted out the leads on the motor (which is operating as a generator), you've increa
Re:The blame game (Score:2)
The important thing in the investigation is the root cause of it and the reason why one problem caused a domino effect.
Underfrequency Relaying (Score:2, Insightful)
"A place I worked" went dark a few years ago in a similar event. A large generator's main breaker tripped, system went unstable and underfrequency, and UFR's (which are set in multiple stages) cleared the lines. Took 8 hours to recover the 2000 MW that were lost (light load).
It's all part of standard protection equipment doi
Re:The blame game (Score:5, Informative)
The Eastern Interconnection (everything in North America east of the Rockies and north of Texas) is tied together at many stations, such that there are many parallel paths to deliver energy to a customer load, providing an excellent level of stability. Simply put, the frequency is the prime measure of the balance between energy production and consumption. Energy generation is not a smooth process, it spikes as fuel is delivered and burned. If enough generators are synchronized with one another, they can automatically cover for each other's dips, and thus the frequency stays balanced.
Now, when the system split, imagine you had all of the generation on the west side, and all the load on the east side. For those of us in PA, we saw a huge loss of load, and the frequency shoots up. For those on the wrong side of the blackout, you suddenly lost your generation source, and your frequency drops.
Transmission equipment is easily damaged at low frequencies, so many are equipped with underfrequency relays that open breakers to protect themselves. What happened is that lines tripped and load sheds, forming smaller and smaller zones, until there were only small pockets of load and generators remaining (see the notes on western NY). Without the rest of the interconnection to syncronize with, your local generator was trying to maintain the frequency by itself as best it could, and was probably all over the map due to uneven fuel burn. Then, a few minutes later, you might have auto-reclosing of breakers (try-backs). If a line trips, some are programmed to auto-reclose, which, in an event like this, can suddenly add thousands of MW of load to an already stressed system, pulling the frequency down even more until everything is black.
The deal with the frequency (Score:5, Informative)
There are two things you need to keep in mind here. The first is that phase in AC systems performs much the same function as voltage in DC systems; just as power flows from higher voltage to lower voltage across a DC connection, power flows from leading phase to lagging phase along an AC connection. (This has to do with reactance; all power lines are inductive.) Counterintuitively, voltage helps move power but it mostly balances VARs (volt-amperes reactive); if you have a local low-voltage situation, you can connect a capacitor to add some VARs and the voltage will come up. This is part of why big inductive loads cause line voltage to dip.
The second thing is that frequency variation is just a phase change over time. If the local frequency falls for a bit, it means that the local phase is moving behind the rest of the grid. This is what you would expect if some large load was added (or a generator lost) and more power had to come from elsewhere on the grid; the delta-phase across the interconnecting lines has to shift to allow more power to flow. What little energy buffering there is is mostly the rotational energy of generators and motors, so phase changes don't quite happen instantaneously.
If you had a serious local power shortage leading to shutdown, under-frequency is exactly what you would expect. Generators trip off-line, and the phase of the local grid backs off to pull more power from outside. It would take a full second at 59 Hz to shift one cycle, so this can go on for a fair fraction of a second. If the phase change over a transmission line increases past 90 degrees it will have to trip off-line, and once the local grid is an island you can have just about any frequency that the system will try to operate at. It's my understanding that most generators trip off-line at more than a fractional Hz off 60, if for no other reason than that they aren't designed or certified to operate on a grid that's obviously malfunctioning and such a condition means trouble. Mechanical resonances at off-operating rotational speeds are another reason to shut down.
Last, I suspect your conclusion is correct.
Re:The deal with the frequency (Score:3, Informative)
Just to correct you:
Future Prevention (Score:4, Interesting)
Re:Future Prevention (Score:2, Funny)
Build more power plants!!!!
Re:Future Prevention (Score:2)
Re:Future Prevention (Score:2, Interesting)
No, the trick is to USE the power plants in the same area that the load is located in. First Energy has at least 2 nuclear power plants in Northern Ohio and one of them was shut down. Energy companies are finding it cheaper to buy electricity on the open market instead of generating their own. Yes kids, de-regulation was a horrible
Re:Future Prevention (Score:2)
Re:Future Prevention (Score:3, Informative)
If a supplier finds it cheaper to buy from someone else rather than produce his own, it indicates either: (a) the supplier is a less efficient producer. Someone else is able to do it more cheaply., or (b) the cost of building new capacity is so high that the cash flow from the new capacity does not justify the cost to build it.
If the problem is (a), then as a consumer, you want the supplier to
Re:Future Prevention (Score:2, Interesting)
Nobody wants to do the work of prevention (Score:2)
the timeline... (Score:5, Funny)
12.04 - power on
14.11 - power off
MSBlaster.exe (Score:5, Interesting)
See:
this [pbs.org]or
this [automationtechies.com]or
this. [heise.de]
Re:MSBlaster.exe (Score:5, Insightful)
Re:MSBlaster.exe (Score:3, Interesting)
Obligatory response: Woo-hoo! Linux all the way baby!
BTW, the EULA specifically prohibits NT and up from being used in "critical" situations where life and environmental damage are on the line. So it would be the utilities' fault, not Microsoft's.
Next on slashdot.org (Score:3, Funny)
All east coast power generating stations, from nuclear reactors to hydroelectric dams, have suddenly decided to switch to Linux. When asked for the reason behind this decision, the response was:
"Well, erm, it was because of bl*cough* erm, I mean just a financial decision from the boys upstairs, I don't know anything about it"
Re:MSBlaster.exe (Score:2)
Usually the "experts" are responsible professionals, perhaps engineers. They know they have a responsibility to the public good and let them know that systems out there (even if now their own systems are fixed) are vulnerable to this "flaw". It would be worth it to protect the X millions of dollars to the economy.
How hard would it be to say "Yes we got hit just l
Re:MSBlaster.exe (Score:2, Interesting)
Re:MSBlaster.exe (Score:5, Interesting)
I've still never understood this. I think most systems are actually based on NT, but maybe they are migrating to 2k now. Either way, the fact that these automation systems are based on a system like windows is very strange to me. OPC (the protocol used to communicate between sensors and databases) is based on DDE (or OLE), which seems so incredibly strange to me.
I've been developing a linux-based SCADA system. I took a look at quite a few systems, and I just didn't feel comfortable running any of them for a number of reasons. Stability and security being two major issues. Another was cost - these are being deployed in small installations, mostly for remote monitoring, which wouldn't typically have a SCADA system due to the cost. Between a mixture of existing open source software, some nice hardware, and in-house development (mostly me), the system has cost us about $20k to develop, which is less than it would cost to licence most software per site.
Anyways, that was a bit OT, but the point is, very early on we decided that deploying on windows would be a pain. These are all remote installations, with no one on site that can service them. If something goes down, I want to be able to remotely fix it as much as possible. I just don't feel comfortable deploying a remote windows system and relying on it to stay running, not to mention the fact that people's health could be affected (water treatment). To me, windows is not the proper platform to be using for this situation.
Re:MSBlaster.exe (Score:3, Informative)
I didn't actually write any OPC code, though that would have been handy. What I wrote is basically a messaging server, that uses a fairly simple protocol to talk over TCP sockets. Clients hook in, and can provides inputs, outputs and variables (there's really not much distinction between them, except variables have the option of being stored in an SQL database, and inputs can only be set by the client that provided them). Clients can request any value at any time, or reques
Re:MSBlaster.exe (Score:3, Insightful)
The systems are designed to run without monitoring. However, without human intervention, the systems aren't very good at staying up in exceptional circumstances. My guess is a computer failure made the grid much more vulnerable, to
Correction: MONITORING system went down (Score:3, Insightful)
The article you cites says it was a safety monitoring system (the backup digital one, not the old analog one running in parallel) that went down. This had nothing to do with control systems.
While it is worrisome that any system at a nuke plant could be disabled by worm traffic (appalling breach of good network operating practices), the criticality of a backup radiation m
Re:MSBlaster.exe (Score:2)
It's in my journal [slashdot.org].
The important bits are:
Win2K/XP
Commercial Internet used on these systems
- and now: -
I didn't write it at the time, but a major power generator in Akron, Ohio got bit in the ass by SQL.Slammer earlier this year. Now that we know this came out of Ohio, it's more significant. Follow the link from my story for more.
Re:MSBlaster.exe (Score:2)
I find it entirely more plausable that they were having network issues on their local networks that communicated with their mainframe, not the mainframe communicating with SC
Re:MSBlaster.exe (Score:2)
Re:MSBlaster.exe (Score:2)
You'd think so (used Ann Coulter's definition of the word). To be fair, the SCADA systems are not officially considered part of the safety or reliability system. If they were, they'd have to run a grown up operating system that is certified for use in power generation and distribution. Wasn't this QNX's niche?
Bloomberg (Score:5, Interesting)
Neither responses were politically acceptable, however the media coverage of the blame game seemed to evaporate as soon as it was clear that it wasn't Canada's fault.
I found that more than a little interesting.
Re:Bloomberg (Score:5, Insightful)
Re:Bloomberg (Score:2)
Since when did Colarado become a foreign country?
Re:Bloomberg (Score:2)
Late Night "Wacko" Talk Radio (Score:5, Interesting)
just my
Re:Late Night "Wacko" Talk Radio (Score:2)
Actually, of those three conspiracy theories, I'd pick #2 (jack up prices). Remember California!
What they *are* confident of... (Score:5, Funny)
"But what we *can* say is that we feel strongly that we feel something different should have happened. An appropriate amount of blame will be laid... oh yes... and we'll make the bad people pay."
Re:What they *are* confident of... (Score:2)
I know what really happened... (Score:2, Funny)
4:10:50 - 4:11:57 PM
Technician 1: "Hey I wonder what this big red button does?" {Click}
Technician 2: "NOOOOOO!!!!"
Imagine if... (Score:2)
Re:Imagine if... (Score:4, Insightful)
I'm all for it.
I have a propane camping stove, plenty of propane, and plenty of canned food around. The servers I'm responsible for will stay running, the data center has its own generators, but our office will be dead.
That means I'd get to sit at home and play Monopoly all day. If I get bored of that, there are about a hundred things I can do for fun that don't require a bit of electricity.
Shoot, it would probably even be beneficial to people in my neighborhood. I'll bet that the park down the street from me would be teeming with people outside, enjoying wholesome activities and human interaction.
Instead of neighbors walking around the block looking for code violations to report to the city, they'd probably be actually interacting with each other, maybe even solving their problems without running to a baby-sitting city government!
I'm sure there would be consequences. Analysts would talk about how many hundreds of trillions of dollars were lost, but in the end, we'd all go back to work, take care of the stuff that didn't get done, and we'd have had a good time while it lasted.
Now, if the power outtage also included me being somewhere like the island of Manhatten or on a subway when it hit, that might be a bit less enjoyable, but unbeknownst to New Yorkers and Californians, the rest of the country works a whole lot differently than they do.
steve
Re:Imagine if... (Score:2)
Signed:
Hank Hill, seller of propane and propane accessories.
Re:Imagine if... (Score:2)
Done near peak load, everything goes bye-bye again :-(
Snipers (Score:2)
I guess those georgian (or perhaps chechen) snipers used August/14 as an idea for their attacks. Or did they wait until their American brothers will do the job first?
So what
Re:Imagine if... (Score:2)
That was tried in the early 1980's on Vancouver Island by the Squamish Five [wikipedia.org]
BitTorrent link (Score:5, Informative)
Timeline of today's slashdotting... (Score:2, Funny)
2:02:00 - 4:50 PM - Business as usual. Interested parties are viewing the published timeline.
04:58:00PM - Existence of timeline is revealed to slashdot.org
05:00:03PM - energy.gov? what's that?
I wonder if it's bad when we slashdot effect gov't sites...
Re:Timeline of today's slashdotting... (Score:3, Funny)
Disturbances Before Outage Itself (Score:5, Interesting)
Did anyone else notice a strobing effect in their fluorescent lighting in those 20-30 seconds before the full power outage? My understanding is that any sort of arc lamp (fluorescent, metal halide) will extinguish if the voltage sags beyond a certain point, so I doubt it could have been a voltage sag before the full blackout.
It almost seemed as if the power frequency itself had gone unstable...say from a nominal 60Hz to like 5Hz. Then again, with the modern electronic ballasts used today, who knows how they respond to a voltage sag. Maybe they strobe. Any one have any thoughts on this?
-Chris
Re:Disturbances Before Outage Itself (Score:2)
Re:Disturbances Before Outage Itself (Score:2)
Re:Disturbances Before Outage Itself (Score:2)
Actually maybe it's too bad it didn't fry my computer, I could've assembled a decent G5 rig with that money...
Re:Disturbances Before Outage Itself (Score:2)
Re:Disturbances Before Outage Itself (Score:2, Interesting)
So of course I shut down my Windows machine as fast as I could, as this isn't a beastly UPS by any means, and t
Re:Disturbances Before Outage Itself (Score:2)
Always freaks Windows users when I say - look, you want proof linux is better? just reach over there and pull the plug out of the wall. They refuse to, and look at me like I'm crazy w
We had a flourescent tube that had been... (Score:3, Interesting)
Alternate Source (Score:5, Informative)
http://www.nrcan-rncan.gc.ca/media/documents/Blac
I think I will be fair and equitable and allow Slashdot to take out a Canadian website as well. Please be kind to Natural Resources Canada.
Re:Alternate Source (Score:2)
Ohms and amps (Score:3, Insightful)
"Suddenly the impedance in Michagan dropped. With Ontario as a constant-current source, the current through Niagara increased
That'd make more sense, no?
Power (Score:5, Insightful)
The impedence in line in a function of the amount of energy flowing through it; as current increases, capacitive losses increase, causing the voltages at the ends to drop. This is sometimes called surge impedence loading. Impedence across a power line is constantly changing, and it is easier to wrap both variables into Power.
Next, the use of transformers makes amps by themselves meaningless. Power is near constant across a transformer, so High Amps Low Voltage can become Low Amps High Voltage. By talking about everything in the form of Power, then you can easily measure the transfer of energy between the various voltage levels of your system, which eases explaining the system.
Finally, Power is an easily understood market concept. If I run a generator at a low voltage (13kV), and produce 10 Amps, I'm generating 130 kW. I pipe that through a large number of transformers & lines, and deliver it to a load running at 23kV, and maybe tomorrow I sell it to someone at 9kV. By keeping everyone running in Power notation, we can all agree that money is exchanged for work, and the proper energy is delivered and paid for.
It couldn't be greed, could it? (Score:2)
Meanwhile, my local electric utility stands acused of pulling power off-line in an attempt to jigger the price of power. Pioneer Press Article [twincities.com]
The What?!?! (Score:2)
We ACUTALLY have a task force dedicated to this?
So long for a timeline? (Score:4, Funny)
the twelve oclock effect (Score:3, Funny)
Re: The Blame Game (Score:5, Funny)
"It's a highly technical term that refers to the frequency of the AC sine wave of electric power that is delivered over your lines."
Wait a minute, could that be the problem? I hear in Canada they use cosine waves. ;-)
-Chris
Re: The Blame Game (Score:2)
Quebec's seperate and distinct power grid (Score:2, Interesting)
Re:moron smoke&mirrors, deep voodoo & chic (Score:2)
Blame timeline (Score:2)
Canada Dept. of Natl Defense says NO
Canadian Prime Minister says it was a fire at a Con Edison power plant in New York
Canadian Defense Minister says it's the nuclear power plant in Pennsylvania
PA's Emergency Management Agency says NO
6pm NYC Mayor Michael Bloomberg holds a news conference
Re:Hey they forgot... (Score:2)
Energy.Gov seems to be down (Score:2)
As of 5 PM CST (6 PM EST) I'm unable to connect to energy.gov and am getting a IE "This page cannot be displayed" error. A quick hop over to netcraft shows that no uptime data is available on energy.gov, which may or may not mean anything.
Did we manage to slashdot a g
Re:Energy.Gov seems to be down (Score:2)
Re:Energy.Gov seems to be down (Score:2)
Re:Canada topic/icon? (Score:2, Funny)
Re:Canada topic/icon? (Score:2)
Re:Canada topic/icon? (Score:2)
Re:Canada topic/icon? (Score:4, Funny)
As in "NOT USA"