US/Canada Power Outage Task Force Event Timeline 303
bofus writes "The U.S./Canada Power Outage Task Force issued the Aug. 14, 2003 Sequence of Events at noon today. While no conclusions are drawn at this point, it does paint a pretty good picture of what happened and when it happened."
text version (aka karma whoring) (Score:5, Informative)
2:02:00 - 2:02:00 PM - Transmission line disconnects in southwestern Ohio
3:05:41 - 3:41:33 PM - Transmission lines disconnect between eastern Ohio and northern Ohio
3:45:33 - 4:08:58 PM - Remaining transmission lines disconnect from eastern into northern Ohio
4:08:58 - 4:10:27 PM - Transmission lines into northwestern Ohio disconnect, and generation trips in central Michigan
4:10:00 - 4:10:38 PM - Transmission lines disconnect across Michigan and northern Ohio, generation trips off line in northern Michigan and northern Ohio, and northern Ohio separates from Pennsylvania
4:10:40 - 4:10:44 PM - Four transmission lines disconnect between Pennsylvania and New York
4:10:41 - 4:10:41 PM - Transmission line disconnects and generation trips in northern Ohio
4:10:42 - 4:10:45 PM - Transmission paths disconnect in northern Ontario and New Jersey, isolating the northeast portion of the Eastern Interconnection
4:10:46 - 4:10:55 PM - New York splits east-to-west. New England (except Southwestern Connecticut) and the Maritimes separate from New York and remain intact.
4:10:50 - 4:11:57 PM - Ontario separates from New York west of Niagara Falls and west of St. Lawrence. Southwestern Connecticut separates from New York and blacks out.
From Department of Energy Secretary: +1, Patriotic (Score:1, Informative)
That was a reassuring comment in the stream of
idiotic statements emanating from The Stacked Deck [whitehouse.org]
Thanks for nothing,
W00t
Re:The blame game (Score:5, Informative)
The trick is that the control system can only react so fast - suddenly disconnect an entire town, and the load drops, causing the power in to spin the generator too fast. If the control system overcorrects, then you'll get too low of a frequency. If a far-away generator drops out and you've got to supply more current to your local region, then the demand has gone up, slowing the frequency.
If you've been around generators, you can hear this exact phenonemoa - if the load changes suddenly, the motor will hunker down a little and then catch back up to normal speed. Usually a flywheel can damp out extremely short transients, but it would be prohibitively big if it were sized to handle transients as large as the control system (throttle) will allow.
BitTorrent link (Score:5, Informative)
Alternate Source (Score:5, Informative)
http://www.nrcan-rncan.gc.ca/media/documents/Blac
I think I will be fair and equitable and allow Slashdot to take out a Canadian website as well. Please be kind to Natural Resources Canada.
Re:The blame game (Score:5, Informative)
The Eastern Interconnection (everything in North America east of the Rockies and north of Texas) is tied together at many stations, such that there are many parallel paths to deliver energy to a customer load, providing an excellent level of stability. Simply put, the frequency is the prime measure of the balance between energy production and consumption. Energy generation is not a smooth process, it spikes as fuel is delivered and burned. If enough generators are synchronized with one another, they can automatically cover for each other's dips, and thus the frequency stays balanced.
Now, when the system split, imagine you had all of the generation on the west side, and all the load on the east side. For those of us in PA, we saw a huge loss of load, and the frequency shoots up. For those on the wrong side of the blackout, you suddenly lost your generation source, and your frequency drops.
Transmission equipment is easily damaged at low frequencies, so many are equipped with underfrequency relays that open breakers to protect themselves. What happened is that lines tripped and load sheds, forming smaller and smaller zones, until there were only small pockets of load and generators remaining (see the notes on western NY). Without the rest of the interconnection to syncronize with, your local generator was trying to maintain the frequency by itself as best it could, and was probably all over the map due to uneven fuel burn. Then, a few minutes later, you might have auto-reclosing of breakers (try-backs). If a line trips, some are programmed to auto-reclose, which, in an event like this, can suddenly add thousands of MW of load to an already stressed system, pulling the frequency down even more until everything is black.
The deal with the frequency (Score:5, Informative)
There are two things you need to keep in mind here. The first is that phase in AC systems performs much the same function as voltage in DC systems; just as power flows from higher voltage to lower voltage across a DC connection, power flows from leading phase to lagging phase along an AC connection. (This has to do with reactance; all power lines are inductive.) Counterintuitively, voltage helps move power but it mostly balances VARs (volt-amperes reactive); if you have a local low-voltage situation, you can connect a capacitor to add some VARs and the voltage will come up. This is part of why big inductive loads cause line voltage to dip.
The second thing is that frequency variation is just a phase change over time. If the local frequency falls for a bit, it means that the local phase is moving behind the rest of the grid. This is what you would expect if some large load was added (or a generator lost) and more power had to come from elsewhere on the grid; the delta-phase across the interconnecting lines has to shift to allow more power to flow. What little energy buffering there is is mostly the rotational energy of generators and motors, so phase changes don't quite happen instantaneously.
If you had a serious local power shortage leading to shutdown, under-frequency is exactly what you would expect. Generators trip off-line, and the phase of the local grid backs off to pull more power from outside. It would take a full second at 59 Hz to shift one cycle, so this can go on for a fair fraction of a second. If the phase change over a transmission line increases past 90 degrees it will have to trip off-line, and once the local grid is an island you can have just about any frequency that the system will try to operate at. It's my understanding that most generators trip off-line at more than a fractional Hz off 60, if for no other reason than that they aren't designed or certified to operate on a grid that's obviously malfunctioning and such a condition means trouble. Mechanical resonances at off-operating rotational speeds are another reason to shut down.
Last, I suspect your conclusion is correct.
Re:text version (aka karma whoring) (Score:2, Informative)
I always heard they could have had French culture, British law and American economy. I guess the key being Government instead of Technology. French technology includes Dassault - who make the core avionics software in every modern military aircraft , including U.S. They also have a decent home-grown telecommunications satellite network - which the U.S. depended on sub-leasing to meet the command-and-control bandwidth requirements during Guerre d'Gulf Deux.
Now- as for French Government...
Re:Future Prevention (Score:3, Informative)
If a supplier finds it cheaper to buy from someone else rather than produce his own, it indicates either: (a) the supplier is a less efficient producer. Someone else is able to do it more cheaply., or (b) the cost of building new capacity is so high that the cash flow from the new capacity does not justify the cost to build it.
If the problem is (a), then as a consumer, you want the supplier to outsource the work to someone else, because you get a cheaper product.
If the problem is (b) and the demand for the product is increasing in the long run, then as a consumer you will likely start to face spot shortages and price spikes caused by those shortages, since the outside supplier will eventually have less and less excess capacity to sell.
California has had experience with (b) in two markets, the electric power market and the gasoline market. In the former, the power outages were caused by high demand and insufficient generation capacity (only part of which was due to attempts to manipulate the spot price of energy). The major culprit was the length of time it took to get State regulatory approval to build new power plants. When Gov. Davis finally realized there was a problem, he relaxed the approval process and dozens of new plants were built.
The gasoline market suffers from regulatory restraint as well. Since 1985, not a single new oil refinery has been built in CA. Ten have closed. (See this [ca.gov] for info. Now, refineries are running at over 90% capacity, rather than the 65% capacity they ran at in the mid-80s. Every time a refinery has a problem, the supply is out-stripped by demand and prices spike. The refiners, needless to say, love this. The are making huge profits because the government has imposed an almost complete barrier to new competition. (Presumably there is some price of gas at which it would be profitable to run the regulatory gauntlet.) The situation will get worse, since it is expected that in the new 5 years about 10% of CA refining capacity will be closed down.
As the two examples suggest, regulation of the market between suppliers and consumers is not the issue. The issue is government regulation by of the suppliers which restricts capacity. The government has some legitimate reasons for those regulations (such as protecting the environment), but their citizens must accept the consequences of growing demand and static supply. Those consequences will include outages, shortages and higher prices.
I know who I am going to blame (for fun)... (Score:2, Informative)
A friend of mine works at NASA's Glenn Research Center near Cleveland (about where this whole mess started). GRC is where NASA has a very large, supersonic wind tunnel (large as in can accommodate full size mock-ups of fighter planes). When they run it during peak hours (rarely), it costs upwards of a million dollars per hour and causes localized blackouts in Cleveland. So I will have to make some sort of comment along the lines of "You just HAD to push to big, green button, didn't you?"
I also know that it wasn't the tunnel I work with in Chicago; we can only cause localized brownouts from the transient currents when we start the motor (rated for 1.5MW, starting power-draw is several times that). On an unrelated note, the sound of the capacitors charging is very similar to the noise you would expect from an evil villain's death ray.
Re:MSBlaster.exe (Score:3, Informative)
I didn't actually write any OPC code, though that would have been handy. What I wrote is basically a messaging server, that uses a fairly simple protocol to talk over TCP sockets. Clients hook in, and can provides inputs, outputs and variables (there's really not much distinction between them, except variables have the option of being stored in an SQL database, and inputs can only be set by the client that provided them). Clients can request any value at any time, or request to be notified of certain values.
I have a client written to gather input from my hardware (opto22), provide output, another that logs values to a database, one that graphs values to rrdtool, one that compares values against alarm setpoints, one that dispatches to people based on time-of-day when an alarm is activated, etc.
Writing an OPC bridge would not be too difficult, I don't think, depending on how hard it is to use the OPC protocol. I started on a modbus interface (never finished it since I don't need it right now), and have plans to make a Profibus interface.
The system is pretty simple, but at the same time it's pretty powerful. I started developing it in March, and have had the vast majority of it running for a couple months now. Since it's so modular, it's very easy to test incrementally, a lot of the design was based on a rapid development time.
The user interface is web-based, and it's designed from the start to be remotely configurable and viewable. Alarms, variables, settings, logging and graphing options, etc, can all be set from the GUI. Some of the system setup is also done from the GUI, but control programs (ie, the things that actually 'do' system control) have to be written and set up manually, although I'm sure that could be done through GUI as well, I just haven't bothered yet.
I've been too busy lately to put any effort into thinking about how to release it to open source, but it's quite possible I will. Now- most of the code I've written is in PHP, which I'm sure will turn a lot of people off. In reality, using PHP has allowed me to write solid code very rapidly, and running on linux on modern hardware, performance is not an issue at all. Python would be my next choice, but I didn't know it well enough and thought it would take to long to learn and develop the system at the same time. C is out mostly for the fact that it would take at least twice as long to write the same quality code (and I'm sure some people will aruge that point).
PuffinPLC has quite a bit of work done, and I really did consider using and contributing to their stuff, but last I checked, they had been going for couple years, and were still only predicting the first usable release would be in a year. I was looking for a SCADA system for a specific task - they're trying to build one that can replace big commercial packages.
Anyways, if there is enough interest, I would definately be willing to open source it. Likely there could be some performance boosts by rewriting some stuff, espessaily the server, in another language, but I'm going for a stable, working solution first.
Re:The deal with the frequency (Score:3, Informative)
Just to correct you:
Re:Why your A/C tripped (Score:2, Informative)
Motors initially draw an excess of current as they start up (they would also do this if the mechanical load were suddenly increased). If the voltage drops so low as to not allow the motor actually get going, it would be "stuck" in the high current situation for quite some time. This is why you see the lights dim briefly in many places when motors like A/C compressors start up.
In 2-phase circuits, the transformer is center tapped on the secondary, and the primary is on just one phase of the three phase source. You get 120v between either leg and the center (ground/neutral), and 240v between the opposing legs. A phase difference would certainly lower the voltage, but that's not going to happen on a 2-phase circuit unless you have other reactive loads there.
But, it may have actually been a 208v circuit between 2 legs of a 3-phase circuit. Most modern A/C systems are rated to operate even on 208 volts, so this kind of thing is not uncommon. It's cheaper to supply 208v instead of 240v to comemrcial buildings which already have a need for 3-phase motors (larger motors and A/C systems) than it is to supply genuine 240v (because this would require making the 3 transformers center tapped, and running 7 wires (6 point star and neutral) rather than 4 (3 point star and neutral). If a leg changed phase relative to the others, it could certainly upset the voltage balance, and the A/C could get undervoltage or overvoltage.
A frequency change can also cause problems, especially with 3-phase motors. A sudden phase shift (a brief frequency change) can cause a motor to go out of sync and draw more current to get back in sync.
Re:Disturbances Before Outage Itself (Score:2, Informative)
The working 'gas' in your typical 40W tube is actually a small amount of mercury, heated and vaporized by the high voltage generated by the ballast and starter. (about 10kV at low current to start it off)
The mercury vaporizes and ionizes, dropping the voltage across the tube to about 80V; it flashes 120 times a second, as the 60Hz is full-wave rectified to make the strobe effect less obvious, as well as keeping everything warm.
The mercury gives off mostly UV light, which is absorbed by the coating inside the tube, and that gives off the visible light we see.
HPS lamps work much the same way, but sodium is used as the phosphor material instead
Argon is the fill gas for both.
High frequency ballasts exist, but are expensive compared to the old standard, but are coming into wider use. If its running from dc, it is likely running at higher frequency for greater efficiency.
BTW, the outer envelope on a HID or HPS lamp is to block the UV emission; if it breaks, the lamp will still work, it will just dust all your plastic stuff (as well as you) with UV exposure. (UV is bad for plastic.)
So; None of our modern, indoor lights is an arc lamp, those are too inefficient, as well as burning the electrodes as they burn.
Most all are all electric column discharge lamps with the resulting UV converted to visible by a phosphor of some kind.