Insufficient Redundancy? Light-Pole Installation Cut Fiber Line, Triggered Three-State 911 Outage (apnews.com) 90
"Workers installing a light pole in Missouri cut into a fiber line," reports the Associated Press, knocking out 911 phone service "for emergency agencies in Nebraska, Nevada and South Dakota, an official with the company that operates the line said Thursday."
In Kansas City, Missouri, workers installing a light pole for another company Wednesday cut into a Lumen Technologies fiber line, Lumen global issues director Mark Molzen said in an email to The Associated Press. Service was restored within 2 1/2 hours, he said. There were no reports of 911 outages in Kansas City...
The Dundy County Sheriff's Office in Nebraska warned in a social media post Wednesday night that 911 callers would receive a busy signal and urged people to instead call the administrative phone line. About three hours later, officials said mobile and landline 911 services had been restored. In Douglas County, home to Omaha and more than a quarter of Nebraska's residents, officials first learned there was a problem when calls from certain cellphone companies showed up in a system that maps calls but didn't go through over the phone. Operators started calling back anyone whose call didn't go through, and officials reached out to Lumen, which confirmed the outage. Service was restored by 4 a.m.
Kyle Kramer, the technical manager for Douglas County's 911 Center, said the outage highlights the potential problems of having so many calls go over the same network. "As things become more interconnected in our modern world, whether you're on a wireless device or a landline now, those are no longer going over the traditional old copper phone wires that may have different paths in different areas," Kramer said. "Large networks usually have some aggregation point, and those aggregation points can be a high risk."
Kramer said this incident and the two previous 911 outages he has seen in the past year in Omaha make him concerned that communications companies aren't building enough redundancy into their networks.
South Dakota officials called the state-wide outage "unprecedented," with their Department of Public Safety reporting the outage lasted two hours (though texting to 911 still worked in most locations — and of course, people could still call local emergency services using their non-emergency lines.) America's FCC has already begun an investigation.
The article notes that "The outages, ironically, occurred in the midst of National Public Safety Telecommunicators Week."
Thanks to long-time Slashdot reader davidwr for sharing the article.
The Dundy County Sheriff's Office in Nebraska warned in a social media post Wednesday night that 911 callers would receive a busy signal and urged people to instead call the administrative phone line. About three hours later, officials said mobile and landline 911 services had been restored. In Douglas County, home to Omaha and more than a quarter of Nebraska's residents, officials first learned there was a problem when calls from certain cellphone companies showed up in a system that maps calls but didn't go through over the phone. Operators started calling back anyone whose call didn't go through, and officials reached out to Lumen, which confirmed the outage. Service was restored by 4 a.m.
Kyle Kramer, the technical manager for Douglas County's 911 Center, said the outage highlights the potential problems of having so many calls go over the same network. "As things become more interconnected in our modern world, whether you're on a wireless device or a landline now, those are no longer going over the traditional old copper phone wires that may have different paths in different areas," Kramer said. "Large networks usually have some aggregation point, and those aggregation points can be a high risk."
Kramer said this incident and the two previous 911 outages he has seen in the past year in Omaha make him concerned that communications companies aren't building enough redundancy into their networks.
South Dakota officials called the state-wide outage "unprecedented," with their Department of Public Safety reporting the outage lasted two hours (though texting to 911 still worked in most locations — and of course, people could still call local emergency services using their non-emergency lines.) America's FCC has already begun an investigation.
The article notes that "The outages, ironically, occurred in the midst of National Public Safety Telecommunicators Week."
Thanks to long-time Slashdot reader davidwr for sharing the article.
Maybe It's Documentation On Location. (Score:4, Insightful)
That is wrong. I'm sure someone with half a brain can figure out what the real issue is.
Re:Maybe It's Documentation On Location. (Score:4, Insightful)
Or, perhaps, the documentation is fine but someone simply made a mistake.
Or there are just too many things crammed into a small space at that location, and the light pole installers knew going in it was gonna be a dicey job to begin with.
Sometimes things just happen.
Re:Maybe It's Documentation On Location. (Score:5, Interesting)
Sometimes things just happen.
I will put it to you that things are just happening pretty much all the time.
Re: (Score:2)
But the superfluity of occurrence is not an excuse to let them continue to grow. The noise increases as a function of grown of population, and technologies serving them.
Inter-disciplinary education helps. Examples:
Better 511/utility search services, disciplined procedure, careful installation site survey techniques might have helped.
Communications network infrastructure additions, including redundancy, faster outage detection through hearbeat fault sensing, rapid deployment for fiber fixes, all these could
Re: (Score:2)
I meant that things, in general, are happening all the time. I was trying to be cheeky and a little philosophical.
Re: (Score:3)
and the light pole installers knew going in it was gonna be a dicey job to begin with.
Sometimes things just happen.
Those two statements are at odds with each other. As someone who has spent plenty of time authorising electrical excavations around far more sensitive things than a crappy fibre (the kind of things that could potentially go boom, or just silently kill everyone), when you *know* the job is dicey you put in additional controls around it.
Sometimes things just happen is a statement used on a normal day at a normal job site without any advanced knowledge. It's not a statement used for when you know the job is di
Re: (Score:2)
Somehow, somewhere, in all this discussion of redundancy, there is a joke reference to a 6001-hull oil tanker [youtube.com]. I just can't seem to find it, though.
Re: (Score:1)
corruption leads to incompetency, this is the inevitable result of classism and economic exploitation
Re: (Score:2)
I'm sure someone with half a brain can figure out what the real issue is.
I have at least half a brain, so here it is:
1. Kansas City's Subtropolis and nearby areas are host to a large number of data centers. The next closest data centers of comparable size are in Chicago, Dallas and Colorado, quite some distance away. And it's not clear that Kansas City isn't bigger.
But while the area is dense with data centers, it's not dense with communications pathways. The customers aren't in Kansas City.
2. It is notoriously difficult to assess whether two network services you wish to buy wil
Re: (Score:2)
Access to 911 is critical infrastructure - it should not be vulnerable to a single-point failure.
This should be impossible (Score:4, Interesting)
If three states can have a critical communications outage simultaneously due to work going wrong in a single location... that's just piss poor design and a few people need to be lined up against the wall and shot.
Mind you, I live in a country where a routing table update took down approximately half the nation's internet for 15 hours.
Re: (Score:2)
that's just piss poor design and a few people need to be lined up against the wall and shot.
During a 911 outage or otherwise?
Mind you, I live in a country where a routing table update took down approximately half the nation's internet for 15 hours.
Sounds about right, lol. The contrast between how big of a deal that is and how unsurprised I am by it is really something.
Re: (Score:2)
Re: (Score:1)
I hate replying to myself, but after basically 25 years Slashdot still doesn't have an edit function.
It's kind of like how vulnerable most of the world is to an EMP attack. Think it through. Someone blows off an EMP above North America. It fries everything imaginable, including our electrical grid. We don't have the spares in stock to fix it, and in fact we have to go to China to manufacture them with an ungodly lead time. Meanwhile, the majority of the world's advanced semiconductor production is in Taiwa
Re:This should be impossible (Score:5, Informative)
Thankfully such an attack is extremely difficult because of two things - shielding and the inverse square law.
Even the largest nukes known on the planet, the EMP area is rather small - an EMP large enough to take out everything in North America would basically destroy the globe in which case I think we have bigger problems than our cars being dead.
The other problem is shielding - those metal cans on everything do a really good job at blocking EMPs as well - by orders of magnitude. If an unshielded device can be affected by an EMP at say, 2km, wrapping it metal foil reduces the range to around 2m or so.
So your car's ECU, which generally lives in a very hostile environent is already sealed inside very thick metal boxes designed to keep out lots of electrical noise and other things but also protects it from the EMP. Chances are, if your car dies from an EMP, you won't have much of a car left anyhow.
Also, the most effective protection against an EMP is basically turning it off. With enough warning, it's possible to shut down the electrical grid to protect it.
Of course ,let's also not forget that gas pumps require electricity, so if it ever should come to that kind of scenario, you're still pretty screwed without having to siphon gas endlessly.
The biggest threat to the electrical grid is a CME (coronal mass ejection) because those wobble the Earth's magnetic field, and because power lines are long, those wobbles induce currents in them which can burn out transformers if the protective equipment doesn't react fast enough. But your car will be just fine other than the compass might be slightly messed up.
Re: (Score:3)
It's been a while since I looked into EMP and nuclear weapons, but I think you may underestimate the dangers. Your processor may be safe inside a metal box, but EMP will induce a voltage spike in the wires is uses to get power and to talk to the outside world. Are *all* of those wires shielded? Almost certainly not.
Your point about a CME is accurate, but just imagine the voltage that EMP can induce in power lines. I'm less worried about the high voltage ones, and more worried about the low-voltage lines di
One can only hope... (Score:5, Funny)
To be devastating, EMP doesn't have to take out everything. A random 5% would already be devastating.
One can only hope the printers will sacrifice themselves first, so that others can survive.
Re: (Score:3)
Your processor may be safe inside a metal box, but EMP will induce a voltage spike in the wires is uses to get power and to talk to the outside world. Are *all* of those wires shielded? Almost certainly not.
Equipment is incredibly resilient to such spikes. We're not going to fry entire processors here, in many cases the spike will simply be shunted in the power through a VAR or MOV designed precisely under the assumption that incoming power actually regularly does experience such spikes. Large enough spikes are likely to blow the fuse / trip circuit breakers, but your computer will be back up and running in a jiffy. You've over estimating the level of damage that would be done.
but just imagine the voltage that EMP can induce in power lines. I'm less worried about the high voltage ones, and more worried about the low-voltage lines directly supplying houses and businesses
I'd be more worried about high vo
Re:This should be impossible (Score:4, Informative)
Equipment is incredibly resilient to such spikes. We're not going to fry entire processors here, in many cases the spike will simply be shunted in the power through a VAR or MOV designed precisely under the assumption that incoming power actually regularly does experience such spikes. Large enough spikes are likely to blow the fuse / trip circuit breakers, but your computer will be back up and running in a jiffy. You've over estimating the level of damage that would be done.
Fuses, circuit breakers are way too slow to be of any use during highest energy components of a nuclear EMP. Even MOVs are too slow. Would need something like TVS diodes to respond quickly enough.
I think the best source of data is still the EMP commission report because they actually do real world testing of control systems, computers, network cables, vehicles..etc rather than just calculations and conjecturbation.
https://apps.dtic.mil/sti/pdfs... [dtic.mil]
Re: (Score:2)
At least the equipment that would be fried on the local distribution side is easy to come by. The transformers that would need a rebuild on the distribution side would have to be rebuilt since there are no spares. There's also nobody prepared to do such a rebuild in the U.S. currently.
If Congress is REALLY worried about any sort of strategic resiliency, that needs to be addressed. There should be spares and on-shore capability to manufacture and re-manufacture that equipment.
Re:I ahve to correct you on EMP (Score:4, Informative)
Seems like a few things in Hawaii 1,450km away were damaged, and some burglar alarms went off.
None of the equipment or boats, or anything around the site were damaged by the EMP, despite it being so much bigger than expected that it exceeded the scales in the instruments designed to measure it.
It also took out 3 LEO satellites, although not from the EMP, but from the charged particles stuck in the earths magnetic field. Good bye starlink et al. That's a global problem, not a local one.
Re:This should be impossible (Score:4, Interesting)
I don't believe much thought goes into making equipment resilient. I suffered equipment damage from a nearby lightning strike. Two PoE VoIP phones lost their boot memory. The utility meter had SPD type 1, the load center SPD type 2, and the PoE power was on a UPS. I suspect that the damage entered via the Cat 5E Ethernet cables.
Re: (Score:2)
Turning something off totally may be a problem for some devices with built in batteries - they don't always go to a hard off and use a trickle of power to still do various things - from checking if power is switching, keeping track of internal time, watching for a network connection to do an actual power on, etc.
Mobile devices, laptops, etc are pretty commonly build without removable batteries nowadays.
And I don't know if everything does a hard shut down anymore.
Re: (Score:3)
Someone blows off an EMP above North America. It fries everything imaginable, including our electrical grid.
Many knowledgeable people will tell you that's a wildly exaggerated scenario.
The grid is built to withstand lightning strikes, which are obviously more localized but otherwise stronger than an EMP.
Cell phones have bypass diodes. TVs have surge protection. Many critical systems have shielding. Etc.
Solar flares have a similar effect. There was a big one two months ago. The biggest ever recorded was in 2001. Do you remember it? Neither do I.
Re: (Score:2)
I hate replying to myself, but after basically 25 years Slashdot still doesn't have an edit function.
Skill issue. Write comment, use preview button, proofread, click submit.
I've been into places that have had a power failure and they literally had to shut down because the retards that they hired as cashiers can't do basic math and check people out.
At a simple business with simple goods you could do the work with a calculator. But you wouldn't have a mechanical one of those, either. At any more complex business the register is doing additional calculations which you wouldn't expect a cashier to do. And it's not realistic to expect cashiers to tabulate costs for dozens of items. It's very easy to get errors. The customer also has a right to a receipt, so those will have to be hand
Re: (Score:2)
Your new Teslas (or anything made after the early 90s, including ICEs) are bricks
Most vehicles since the early eighties have carburetors with electronic mixture control, yes also for emissions reasons. This means that they will never run right without a new carb. Anything newer than the mid seventies also has electronic ignition, so even it will need to be retrofit with an older distributor with points (usually possible, because most of those vehicles had engines designed in the sixties with emissions equipment added to them in the late seventies) if you want it to be secured against fu
Re: (Score:2)
Time to switch back to mechanical injection diesel engines, where the only way to stop them is to cut the fuel.
They were passing California emissions until at least 2002
Re: (Score:2)
They were passing California emissions until at least 2002
For your personal vehicle it is theoretically still possible to pass emissions with a mechanical diesel, and in a few regions you do not have recurring testing at all for light vehicles. Vehicles with diesels with Bosch M, MW, or P pumps can be (or be retrofit) to have a manual fuel cut, and can be roll or tow started if the transmission and situation allow.
Re: (Score:2)
No that's not how a EMP attack works (well only in movies). It's a nuclear bomb exploding about 100 feet above ground, that's a localized blast not country wide it would only effect blocks not miles. For something to effect the entire continent said people on the continent wouldn't exist any longer so no one would care about the lack of electronics.
Re: (Score:2)
Look up Starfish Prime [wikipedia.org]. A nuclear test at an altitude of 250 miles. It caused some unexpected EMP damage in Hawaii, 900 miles away.
Re: (Score:2)
Checking out in a power failure has only gotten harder over time. Now it's well beyond just having someone who can do arithmetic. None of the prices are actually on the items so without the scanner and the POS looking it up in a database the cashier has no way to know the price other than have someone go look at the shelf (assuming they can FIND the correct price there). Once it's all totaled up (perhaps an hour or 2 later), there's no way to accept a card payment. If the power outage is generalized, the cu
Re: (Score:2)
Obviously, these "engineers" who designed this mess have never considered the concept of redundancy.
More likely they were strong-armed by business and sales people giving them requirements to make it centralized and/or cloud-based, in order to resell the same platform to multiple customers/states while retaining control over it.
Re:This should be impossible (Score:5, Insightful)
Obviously, these "engineers" who designed this mess have never considered the concept of redundancy.
More likely they were strong-armed by business and sales people giving them requirements to make it centralized and/or cloud-based, in order to resell the same platform to multiple customers/states while retaining control over it.
Cloud-based emergency services. You know, even a few years ago that would have invoked widespread ridicule and been a total joke in the industry and a political nightmare. Now, people just emphasize that it's "in the cloud" and that seems to make untold degrees of incompetence acceptable. Even more recently, you can just claim that "AI" did it and therefore all us mere humans are somehow stupid.
I actually think back to the days of old copper POTS and how reliable it was.
Re: (Score:1)
Maybe they just bought into the idea the TCP/IP routes around damage flawlessly - without understanding that's only true if multiple routes exist.
Even when data centers have multiple redundant connections, how many of those physical wires aren't bundled together? And even if you did route one set of cables north of your building to utility poles, and another set south of your building to different utility poles, who knows if the utility company ends up merging those two sets of cables together at some poin
Re: (Score:1)
Maybe they just bought into the idea the TCP/IP routes around damage flawlessly - without understanding that's only true if multiple routes exist.
There is zero excuse for having a state's internal emergency system have any dependency upon infrastructure outside of that state. Same goes for towns.
Re: (Score:1)
There is zero excuse for having a state's internal emergency system have any dependency upon infrastructure outside of that state. Same goes for towns.
Note: this includes labor, not just hardware.
Re: (Score:1)
I actually think back to the days of old copper POTS and how reliable it was.
Yeah, we need to get rid of all this digital nonsense and go back to vacuum tubes, shit just worked.
Re: (Score:1)
For all their faults, Ma Bell was exceeding five 9's uptime/availability back in the 1970's regardless of any underlying technology or layer of infrastructure. As to your comment, - false dichotomy. There is a wide range in between the two extremes.
Re: (Score:2)
There is no such thing as "the cloud." It's just someone else's server.
Re: (Score:1)
There is no such thing as "the cloud." It's just someone else's server.
This used to be true and still is in many cases, but it's hard to look at AWS or Azure and not see them as more than just hosting, for better or worse they are platforms of tightly-coupled and interconnected services.
Re: (Score:2)
We really need the cloud to butt extension more than ever.
Re: (Score:2)
If it were properly "cloud based" it would have been connected to other facilities and been able to transition to routing through those in a matter of moments. If your "cloud" option doesn't support migration to other physical locations and multipath connections, it's not really "cloud" it's just "in someone's data center".
Re: (Score:1)
Re: (Score:2)
Your funny there were no "engineers" designing anything this is just a case of a piss poor backhoe asshat that knew where the cable was but didn't want to get out and hand shovel the last foot or two. That's how this works, those lines have a rfid tags every 10 feet someone walked the line and planted flags that the asshat didn't adhere to.
I've worked with large infrastructure for years and fiber cuts come in two flavors:
1.) Some idiot backhoe asshat that doesn't want to get off the machine and do his job
Re: (Score:3)
You're missing the point. Any decent engineer KNOWS that fiber cuts happen. Whether they should or should not is irrelevant, they happen all the time. Having a state's 911 service depending on a single cable not being damaged is piss poor engineering at best.
Side note, always keep a short length of fiber with you. If you get lost or stranded, bury the length of fiber in the ground. When the backhoe shows up to break it, ask the operator for a lift into town.
Re: (Score:2)
Now back when I was shopping for fiber at a job there were specifically 2 fiber lines into the property run in a double ring or some odd, I remember the sales guy telling me it was for this exact scenario. I always assumed for businesses and critical spots this was sortof standard procedure.
Re: (Score:2)
Now back when I was shopping for fiber at a job there were specifically 2 fiber lines into the property run in a double ring or some odd, I remember the sales guy telling me it was for this exact scenario. I always assumed for businesses and critical spots this was sortof standard procedure.
Obviously this wouldn't help you if your little hub took a hit. (Think, the thing in your back yard so to speak.) But I'm curious to know if they sent the fiber off in different directions. I mean, one line that went through one major street and another that went through another major street. That would mitigate a lightning strike or some kind of weird localized disruption.
I'm lucky enough to be on a power subgrid that includes a few "high priority" sites like nursing homes, so even when the power went o
Re: (Score:2)
He did say that sometimes they run off in different directions, or rather that they basically have 2 rings and pull from customer location or service box to the next but specifically said they are always a certain distance apart and never in the same conduit with the idea that no single line cut will bring your network down.
During install the technician actually did a test on the phone with their NOC of actually unplugging each line and seeing that the network was still up but that they could detect the bre
Re:This should be impossible (Score:4, Interesting)
When I was supporting 911 call centres, there were always multiple call centres using each other for fail over.
A covered region would have a primary and backup call centre, and another region would temporarily take the calls during a migration between the two.
In terms of interruption of service... there was none. Though every time we did the fail over testing I was always stressed the switch wouldn't work.
Re: (Score:1)
If three states can have a critical communications outage simultaneously due to work going wrong in a single location... that's just piss poor design and a few people need to be lined up against the wall and shot.
Not at all wrong, this was likely a move to a cloud-based provider, those states are quite a ways from Missouri. Each state needs its own infrastructure.
Re: (Score:2)
If three states can have a critical communications outage simultaneously due to work going wrong in a single location... that's just piss poor design and a few people need to be lined up against the wall and shot.
Not at all wrong, this was likely a move to a cloud-based provider, those states are quite a ways from Missouri. Each state needs its own infrastructure.
And what a shit show. I find it highly disturbing that the people in charge of this kind of thing have so little vision, and so little anticipation of what can go wrong. I have to assume that the reality is that they just don't care. Nobody can accidentally be this incompetent.
Re: (Score:1)
Re:This should be impossible (Score:5, Interesting)
I'd venture a guess that it started out with someone offering the service of providing 911 access, and it worked fine at small scale. Then they found more customers to connect, because traffic was very low for the infrastructure they already had. Then more and more jumped on the bandwagon, to offload their "911 responsibility" to this one popular provider. Adding more users was practically "plug-and-play", they just had to adjust some routing tables and bam, their 911 responsibility had been outsourced and now they can forget about that annoying and expensive thing the government mandates them to do "because we have this other outfit taking care of all that for us".
Then over time this one scrappy little outfit found itself the sole 911 service provider in three states. But instead of using all that new capital from all their new customers to upgrade their now over-extended systems and add redundancy, they just pocketed most of it. Their customers were happy, their shareholders were happy, life was good. Nobody thought there was anything wrong because there hadn't been any problems, and the 911 calls were getting through. There were no reliability or single-point-of-failure audits because there had never been any problems with it in the past.
Then a wild posthole digger appears, and suddenly everyone is vividly aware of just how fragile of a system everyone in the area is relying on. And the finger-pointing begins. "We never could have imagined something like this could possibly happen!" Oh yeah, ignorance is bliss.
I wonder if circuits like this are being overused due to low prices? Like if you have a "weak link" circuit that is one of the few that connects popular points X and Y together, and "in theory" there's a lot of users for that circuit, but the smart ones realize they can't put all their eggs in one basket through there (like their primary AND their backup system, you can't run BOTH through the same node, for fault-tolerance reasons) and so the owners of that circuit find they're having trouble selling capacity through it, so they lower prices. This attracts outsourcers and resellers to reroute through there for the low prices. Then some groups, looking to cut costs, find this new outfit offering bandwidth from X to Y and switch to it, without realizing that now their backup service is running through a node that their primary service does. I'm sure there's processes in place to catch when this might happen, but I wonder if the lower prices (due to those 'aware of the danger' avoiding the node) is what's leading to unexpected loss of redundancy for the less diligent groups?
910 (Score:3)
Re: (Score:3)
Re: This should be impossible (Score:2)
A good design, but in the corporate world, you would not be given the funds to build it.
Re: (Score:2)
You may recall that AT&T suffered a much more major outage a few years ago at Christmastime as some loony tried to blow up a building housing positively ancient infrastructure which by all rights, should have been transitioned to a newer and much fancier building a few blocks away *well over a decade ago*, and... precisely nothing happened.
So... probably nothing is going to happen this time, either. Nevermind that even junior techs know what redundant connections are for.
if the phone company s had their way it may be cal (Score:2, Offtopic)
if the phone company s had their way it may be call 1-900-911 for only $9.99 per min you can get an multi highest priory link to 911
Re: (Score:3)
if the phone company s had their way it may be call 1-900-911 for only $9.99 per min you can get an multi highest priory link to 911
Don't you mean 0118 999 881 999 119 725 3?
redneck bat signal (Score:1)
They can just use the redneck bat signal, fire your shotgun into the air 3 times.
Well, of course. (Score:3)
For Nebraska, South Dakota and Missouri, there's only the one pole.
Well, if it's ONLY 2 hours... (Score:3)
golly!, probably not too many people died because 911 service was down...
yeah, and probably not too many people had the effects of a stroke become permanent because they could not get care in that first "golden" hour where the damage might have possibly been reversed...
Is it the company's fault? (Score:2)
Kramer said this incident and the two previous 911 outages he has seen in the past year in Omaha make him concerned that communications companies aren't building enough redundancy into their networks.
When I was in the telco business, diversity meant multiple carriers, entering the building on different cables, thru different sides of the building.
You didn't rely on a single vendor providing diversity, you paid for it by having redundant connections to multiple providers on multiple connections.
They relied on a single vendor providing a diverse connection? Really? It is the custoner's responsibility, plain and simple.
Re:Is it the company's fault? (Score:5, Informative)
Many people who carefully designed such redundancy later found it gone due to mergers and acquisitions. Lumen has built one of the largest telecom companies in the world through M&A. So for example, you might have built your redundancy by getting links through Qwest, Savvis, and Level 3... all of whom are now part of Lumen. Over time, those links get "re-groomed" into the same fiber (DWDM may mean your "redundancy" ends up as just different wavelengths in the same strand).
It turns out to even be hard to hold anybody accountable, because contracts weren't written to say this can't be done (e.g. a contract with Qwest didn't say "and you can't carry this in the same physical route as Level 3")... and even it had been, custom information like that doesn't fit into the cable map databases and so doesn't filter down to the techs making design and change decisions years later.
Bet: they had redundancy (Score:3)
While I have no way of knowing, I will bet that the system had redundancy. Separate providers, separate contracts. However, it is insanely difficult to know where the data actually, physically flows. Even if the providers have completely different networks, at some point those networks may run on separate fibers in the same bundle.
The college where I teach has campuses in various locations. We have two endpoints, in different cities: If one goes down, the other one ought to still be alive. It has still happened once that a single cable cut took us offline.
Re: (Score:2)
While I have no way of knowing, I will bet that the system had redundancy.
Even if the system "had redundancy" it clearly didn't have any effective redundancy. The headline begins "Insufficient Redundancy?" and not "No Redundancy?".
The college where I teach has campuses in various locations. We have two endpoints, in different cities: If one goes down, the other one ought to still be alive. It has still happened once that a single cable cut took us offline.
Yes, we call that "insufficient redundancy" — especially for critical safety systems, not quite so much for your college where a 911 call could still be made with a cellular phone.
Just call 0118 999 881 999 119 725 .. three (Score:2)
They're your emergency services.
Re: (Score:2)
Murka Has The Best... (Score:2)
What other nation on the planet can match the United States for technological wizardry!
Silver lining (Score:2)
The repair crew had a bright light to work under.
Qwest, Again. Just now they're called "Lumen". (Score:5, Interesting)
They keep renaming the company, but it's still the same old Qwest, now Lumen, formerly Mountain Bell all outsourcing E911 services and all going down in 4-14 states MANY times in the last ten years.
This is nothing new, and has nothing to do with a "light pole" or a "fiber optic cut."
Each time they get a free pass and do nothing to fix the problem. They have outsourced crucial emergency response services to remote datacenters that are not redundant, not given E911 data (address and callback# of caller) and no modern logging capability (as in who called in when for how long, how long on hold, etc.)
Lumen (that's Qwest''s new name) is the same piece of shit southwest US network operator that's been flailing for decades.
End of story.
Re:Qwest, Again. Just now they're called "Lumen". (Score:4, Informative)
Lumen (that's Qwest''s new name) is the same piece of shit southwest US network operator that's been flailing for decades.
Qwest no longer exists, they were bought out by CenturyLink after the CEO refused to allow the NSA to install a tap [wikipedia.org] and was done for insider trading on a bullshit pretext [eff.org] and even denied bail. Qwest WAS the result of an insider deal, but not insider trading; the founder Philip Anschutz owned Southern Pacific Transportation Company (think Southern Pacific Railways) and he set up the arrangement of burying fiber with special railcars along the Southern Pacific line. This enabled Qwest to lay the first coast to coast fiber installed in over a decade.
You can blame CenturyLink and the federal panopticon for your Qwest woes.
Re: (Score:2)
You can blame CenturyLink and the federal panopticon for your Qwest woes.
Hey man, that panopticon is REALLY important. So important that losing some 911 calls isn't even considered water under the bridge. If thousands have to die in order for Law and Order to prevail, that is a small sacifice that I am willing to make for you.
"Insufficient"? Sounds more like "no redundancy". (Score:2)
You know, the definition of redundancy is that you can remove a component and things keep working. That as obviously not the case here. Greed at work.
Redundancy?! That's a laugh (Score:1)
Yes. Lacks redundancy (Score:2)
Critical infrastructure should have robust redundant infrastructure.
It's stupid that this single point of failure knocked out 911 services.
No matter how well locations are documented, it is inevitable that there will be failures.
Redundancy Is Not Enough (Score:2)
A single point of failure killed 911 service in three states. Why was there such a centralized system for 911? Critical services should be DECENTRALIZED!
Facepalm (Score:2)
Fiber rings typically have redundancy (Score:2)
The big fiber rings usually have redundancy built into them where traffic is transmitted in two directions simultaneously and if one side goes into LOS ( Loss of Signal ) the end node instantly switches to the secondary path. ( Yes, I worked with these systems for a large Telco for many years )
Said secondary path typically* rides a completely different fiber within a different sheath.
There were always dual paths between any two nodes.
This is for standard traffic. Telcos usually give 911 traffic even more
At this point - regulation is the only good option (Score:1)