Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
United States The Internet IT Technology

CenturyLink Outage Led To a 3.5% Drop in Global Web Traffic (zdnet.com) 27

Posted by msmash from the closer-look dept.
US internet service provider CenturyLink has suffered a major technical outage on Sunday after a misconfiguration in one of its data centers created havoc all over the internet. From a report Due to the technical nature of the outage -- involving both firewall and BGP routing -- the error spread outward from CenturyLink's network and also impacted other internet service providers, ending up causing connectivity problems for many more other companies. The list of tech giants who had services go down today because of the CenturyLink outage includes big names like Amazon, Twitter, Microsoft (Xbox Live), EA, Blizzard, Steam, Discord, Reddit, Hulu, Duo Security, Imperva, NameCheap, OpenDNS, and many more. Cloudflare, which was also severely impacted today, said CenturyLink's outward-propagating issue led to a 3.5% drop in global internet traffic, which would make this one of the biggest internet outages ever recorded.
This discussion has been archived. No new comments can be posted.

CenturyLink Outage Led To a 3.5% Drop in Global Web Traffic More

CenturyLink Outage Led To a 3.5% Drop in Global Web Traffic

Comments Filter:

  • And nothing of value was lost? (Score:3)

    by bobbied ( 2522392 ) on Monday August 31, 2020 @12:26PM (#60459356)

    I didn't even get my laptop out of it's bag on Sunday, so I didn't even know.. Nothing of value was lost.

    • There was a lot down.

      Not to mention if CenturyLink was your ISP, then everything was lost.

      • Re: (Score:2)

        by sheph ( 955019 )
        I have CenturyLink, and it was not completely down for me. I got up Sunday morning and tried to play WoW. It connected to Blizzard, I could log into the game, it loaded the realms, but then timed out connecting to the game server. Hotspotting from my phone worked fine. When I checked my firewall I saw denies going to a new DNS server (three actually). I run my own DNS and have forwarders to fetch records I don't know about. I added the new DNS servers to the list of forwarders, and to the network obje

    • Re: (Score:3)

      by pavon ( 30274 )

      Meanwhile the call center for the COVID-19 / Nurse Advice Hotline in my state was completely packed on Sunday as most of the employees that normally work from home had to come on-site at the last minute.

    • Many B2B cloud services were also down due to this. We had a debrief call earlier today and a many of our customers were affected all over the world. Thankfully it wasn't exactly peak time but some businesses do need to keep running on the weekends to.

      I gues this is where on-prem would be safer but the customers are definitely capable of screwing that up all by themselves anyway :)

    • Re: (Score:2)

      by antdude ( 79039 )

      I didn't even notice anything weird that day as an Internet addict.

  • How hard is it to run a big site without one? What challenges do they solve which a normal.server cannot?

    • CF routed around it pretty quickly, but only continued to have errors for destinations whose only peer was CenturyLink.

  • The old days... (Score:3)

    by beheaderaswp ( 549877 ) * on Monday August 31, 2020 @01:46PM (#60459426)

    Gosh I miss the old days....

    This isn't the internet I thought we would get. Most of the backhaul providers are incompetent. Consumer ISPs even worse.

    We did a lot better when the main routing point for the east coast was in a closet in a parking garage. A lot better.

    • Back in the 80s and 90s, we had pings of 1-6 ms to nearest hosts. Hell, I owned by own class C.
      Now, we are looking at 10-60. Why? Because IPv4 routing has been bastardized, as well as scaling.
      With IPv6, we return back to the simple days and simple routing.
      This is really why we need to get off IPv4 and over to IPv6.
      A lot of routing issues will go to the by-side with that. No more need for NAT, and I say thank god.

  • A day and a half later, and only now were they down.

  • I have a gigbit fiber from them. When I woke up in the morning it wasn't working. I reset all the equipment and it worked fine after that. Though admittedly, It could have been working far slower than the usual 960 MB or so up/down I usually get and I won't have noticed at all.

  • CenturyLink: sad (Score:5, Interesting)

    by PuddleBoy ( 544111 ) on Monday August 31, 2020 @03:48PM (#60459896)

    There are/were a lot of great people at CenturyLink (CTL). And they have/had a great network. (I used to work there)

    But they made a series of expensive acquisitions (not including Level3) over the last several years that were not in their best interest. (ie the acquired company's products were far from CTL's core strengths and, I believe, were geared toward making investors think they could be everything to everybody) With the L3 acquisition, a few people walked away with a lot of money. (I believe one top exec cashed a check for something in the neighborhood of $300M) And, though publicly it was CTL that acquired L3, most of the CTL upper management is gone replaced by L3 management. Many CTL products have been EOL'd, replaced with L3 products. (I understand that some product line consolidation must occur, but it has been somewhat lopsided, considering who officially acquired who)

    Today, they stand at $36B in debt (https://www.fool.com/investing/2019/03/19/where-will-centurylink-be-in-5-years.aspx) They have had a number of layoffs in the last couple years that they have managed to keep mostly out of the press. Lots of experienced people left, including a lot of engineers. Though I cannot confirm this, I assume that some of their cost-cutting may ultimately affect quality of service or at least response to outages.

    I wish them well, but they're in a tough spot with a lot of competition (often cable companies) and a lot of debt.

    • CTL has been a HORRIBLE joke. They might have been OK, when small, but CTL took over L3, they totally messed things up. L3 was DECENT and had TOP NOTCH ENGINEERS working there. CTL fired them, and outsourced to India. Since then, CTL had been a total joke.

  • "Amazon, Twitter, Microsoft (Xbox Live), EA, Blizzard, Steam, Discord, Reddit, Hulu,..."

    I can't imagine how people have suffered. The atrocity, the hopelessness...

  • 1. Announce changes in routing policy 30 days in advance if possible, 24 hrs if not.
    2. Have a testing methodology in place to ensure routes advertised before the change are advertised after.
    3. Have a rollback strategy so if the test is unsuccessful roll back to previous configure and try again tomorrow.
    4. Communicate to customers change time, change purpose, and follow up with "done, all hands off" or "not done"
    5. Don't schedule it in the middle of the day.
    6. Have peer review of everything above and at leas

  • I hope that my local area will go with its own fiber. These utilities are total nightmares.
  • When it comes to Enterprise routers, juniper is great because you can commit confirm a change or rollback a bad change. :) Other vendor require checking your configs a few times, apply and pray your rollback commands work when you manually apply them.

  • We observed the outage in Trinocular, an Internet-wide outage detection system---the animation is at https://ant.isi.edu/outage/ani/centurylink202008/index.html [isi.edu]. This is one of the largest U.S. outages in some time!

Slashdot Top Deals

It is not best to swap horses while crossing the river. -- Abraham Lincoln

Close