Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
Canada The Internet

A Major Rogers Outage Has Cut Off 25 Percent of Canada's Internet Traffic (theverge.com) 89

Posted by msmash from the great-start-to-the-day dept.
Canadian telecom Rogers is suffering a major outage affecting landline phones, cellular connections, and internet connectivity throughout Canada that started early this morning. DownDetector listed thousands of reports for the issue as people started to get up around 5AM ET and couldn't get online. From a report: Rogers first addressed the outage in a tweet from its official support account just before 9AM ET, and then went silent for a couple of hours. In a statement given to The Verge and tweeted at around 11:30AM ET, Rogers said, "We are currently experiencing an outage across our wireline and wireless networks and our technical teams are working hard to restore services as quickly as possible. On behalf of all of us at Rogers, we sincerely apologize to our customers, and we will continue to keep you updated as we have more information to share, including when we expect service to be back up. Thank you for your patience as we work to resolve this issue."
This discussion has been archived. No new comments can be posted.

A Major Rogers Outage Has Cut Off 25 Percent of Canada's Internet Traffic More

A Major Rogers Outage Has Cut Off 25 Percent of Canada's Internet Traffic

Comments Filter:

  • Sparse (Score:4, Informative)

    by fluffernutter ( 1411889 ) on Friday July 08, 2022 @01:26PM (#62684970)
    Unfortunately we are a very large country with very few customers. We don't get much choice who to use.

    • Re: (Score:2, Insightful)

      by Anonymous Coward

      Unfortunately we are a country with regulatory capture

      • Re: (Score:2)

        by Curtman ( 556920 ) *
        Thankfully the regulators were there to stop the acquisition of Shaw by Rogers.

      • Unless you can show regulation that prevented other companies from also laying down physical wires, this is not exactly the whole picture. arguably the regulation that prevented Bell from monopolizing their own infrastructure disensentivised other companies from competing in physicality laying wires, but at least in the short term helped prevent a monopoly.

    • You think there is a magical carrier out there in any other country with absolutely 0 downtime? Seriously. Issues arise. They will restore it as soon as they can, and meet whatever SLA obligations they have. Just be patient. Everyone goes down at some time.
      • SLA may apply to a single component but 25% of the country down? Anything that important should be fully redundant.

        • Re:Sparse (Score:5, Informative)

          by Strauss ( 123071 ) on Friday July 08, 2022 @03:06PM (#62685318)

          Agreed!
          And what anyone not going to look at TFA may not have noticed; this is not "just" home Internet.

          Rogers has;
          * ~1/3 of the mobile phone subscribers in the country -- who cannot make or receive calls, texts, or data.
          * Some large fraction of home internet - all offline
          * Provides network connectivity for Interac, the debit-card processing network -- offline (cash or sometimes credit only today, country-wide!)
          * Is the network provider for a reasonable fraction of businesses, including government services -- No passport office today!
          and likely more.

          The simple line is that 25% of the country's "Internet" is down -- but the catch is that this is at the consumer *and business* levels, affecting phone, data, emergency services, financial transactions, government services, and possibly others.

          In effect, whatever redundancy Rogers built in got broken here and *everything they do* isn't working.

          For the US readers; this isn't like Comcast dropping. This is like AT&T, Verizon, or Sprint dropping -- including their portions of the internet backbone.

          • Netcracker supplies some of the major systems at Rogers and other telecom companies in Canada and the USA. Most of their programmers are in Moscow. I think the problem is likely due to stupidity on someone's part, but this is one time I'm prone to going, "hmm..."

            Someone even mentioned this in an article recently, but having worked in telecom off an on over the years, I've always felt some disquiet about where stuff is coded.

            Russia's war on Ukraine looks a nightmare for Netcracker [lightreading.com]

          • Even big systems can go down, and you have no idea where the issue lies. Might be a giant fiber cable. Trench two? Maybe. But the same can be said of the end user. Are you a 911 call center? Redundant trunks with distinct pathways. Failing to plan (on your ISP going down sometimes) is planning to fail.

          • Re: (Score:1)

            by sglines ( 543315 )

            I have to wonder if this isn't the result of a good cyber attack. Trashing a few routers could do wonders. Being a Canadian company might lead the locals to think they could relax their security. I wish they were right.

        • Re: (Score:1)

          by Anonymous Coward
          So yes, 11 people are offline. BFD. Its Canader Eh.

    • Re:Sparse (Score:5, Insightful)

      by ceoyoyo ( 59147 ) on Friday July 08, 2022 @01:46PM (#62685032)

      Yeah, bull. My mobile provider is a former independent one that got bought by Rogers and is now their "we know you hate Rogers but we're friendly and nice and totally not them!" division.

      The CRTC has fallen down on the job and let the big guys buy up all their competitors.

      • Netcracker supplies some of the major systems at Rogers and other telecom companies in Canada and the USA. Most of their programmers are in Moscow. I think the problem is likely due to stupidity on someone's part, but this is one time I'm prone to going, "hmm..."

        Someone even mentioned this in an article recently, but having worked in telecom off and on over the years, I've always felt some disquiet about where stuff is coded.

        Russia's war on Ukraine looks a nightmare for Netcracker [lightreading.com]

        • Re: (Score:3)

          by ceoyoyo ( 59147 )

          Rogers did the same thing about this time last year. The Beaverton called it: "Putin determines cyber attack against Canada not necessary since Rogers is far more effective."

      • Re:Sparse (Score:5, Informative)

        by jenningsthecat ( 1525947 ) on Friday July 08, 2022 @04:22PM (#62685632)

        Yeah, bull. My mobile provider is a former independent one that got bought by Rogers and is now their "we know you hate Rogers but we're friendly and nice and totally not them!" division.

        The CRTC has fallen down on the job and let the big guys buy up all their competitors.

        True, but to be fair most or all of those "independent" "competitors" that were bought up were already using their respective ROBELUS* backbones anyway. For example, even if Fido hadn't been bought by Rogers, Fido customers would still have no service during this outage.

        *ROBELUS refers to Rogers, Bell, and Telus. These are the three major cellular infrastructure companies here in Canada, and they also provide home and business TV and internet service. There's also Shaw, but it's minor and regional, and (subject to regulatory approval) it's being bought out by Rogers. Most Canadians hope the CRTC blocks this purchase - we're sick of being fucked over by the current triopoly and want more providers, not fewer.

        • Re: (Score:2)

          by ceoyoyo ( 59147 )

          Fido used to have their own network. They were actually the first in Canada to have a GSM network. So did Wind.

          Even independents who did buy service from the big guys could buy it from more than one of them. If there was enough competition, or if the CRTC levied enough fines the big guys might even buy emergency redundancy from each other!

          • Fido used to have their own network. They were actually the first in Canada to have a GSM network. So did Wind.

            I didn't know that - thanks for the correction. I'm guessing their own networks were very limited and that they depended on the big carriers outside just a few major urban areas - am I correct in that?

            Even independents who did buy service from the big guys could buy it from more than one of them. If there was enough competition, or if the CRTC levied enough fines the big guys might even buy emergency redundancy from each other!

            Sounds like the fossils at the CRTC need to wake up and force ROBELUS to put procedures in place such that when one of them fails the other two pick up the slack. I don't think it would be too difficult - but even if it is difficult, too fucking bad. Somebody needs to bring the Big 3 to heel.

            • Re: (Score:2)

              by ceoyoyo ( 59147 )

              The FIDO network covered the big metro areas pretty thoroughly. They didn't run towers out into the sticks though.

              Canada's population is more concentrated than the US, so it's actually easier to cover the majority of people, but harder to serve nearly everybody. The obvious solution is to have a bunch of competing networks in the metro areas and one or two covering highways and rural communities, with regulated wholesale prices between carriers. That seemed to be where they were heading when the government

    • Unfortunately we are a very large country with very few customers. We don't get much choice who to use.

      You should come on down to America. For actual high speed Internet I can choose between Comcast and Xfinity. It may be a monopoly, but it’s quite competitive. /s

      • Define "actual high speed internet." The FCC defined it as 25 meg service last I knew, which means you've probably got at least one DSL provider and possibly a satellite provider as well.

        • I'm pretty sure DSL is no longer available at my address, but the speed was 10/3 last I checked.

          My options are Xfinity or T-Mobile.

          I'm actually using T-Mobile, it's about 20ms extra ping (35ms) and carrier level NAT for IP4 and no passthrough for IP6, but I get 100/50 (real world) for $50/month.

          Xfinity costs a whole lot to get more than 10 Mbps upstream.

          It's pretty sad that my best option for internet is cellular.

          • No DSLs coming to your address?

            What a shame.

            • Re: (Score:2)

              by Luckyo ( 1726890 )

              DSL is very dependent on how well cabling is maintained and distance. If DSL wasn't well maintained, it's not going to keep being good even if it was still offered.

              • Re: (Score:2)

                by Hylandr ( 813770 )

                Mos communities are allowing their telco infrastructure rot on the pole. Everything is going to fiber or cell. When storms come through our area the lines are tied up just to keep them out of the way, but never actually repaired. ~ East Texas.

          • Xfinity costs a whole lot to get more than 10 Mbps upstream.

            It's nice living in one of the few places in the US with ISP competition. I'm paying $60/month for Xfinity, and the bandwidth test I just ran says ~375/25 Mbps.

        • Possibly not, 25 meg is quite fast for ADSL, and VDSL isn't available as many places.

          I don't even have 25 meg, but that's because I don't want to pay more than $50/month, so I ordered 18MB service, but measuring it I get closer to 22. This stuff is way over priced.

        • For me there are three choices:

          Cox symmetric gig fiber
          CenturyLink symmetric gig fiber
          TMobile 200/75 (ish) fixed wireless

          This is in Phoenix. Cox is the only one I'd never use because it's $120/month with a 1.25TB cap. The other two have no data limits. CenturyLink is $65/month, which is pretty good, hence I use that one.

      • The lobbying politics are interesting here. For awhile, it was "the internet is going to be HUGE so don't kill the golden goose by regulating us!", then it morphed into "we're not a vital service, just a tiny business selling optional luxury goods, so please don't regulate us!"

      • Re:Sparse (Score:4, Insightful)

        by TechyImmigrant ( 175943 ) on Friday July 08, 2022 @03:31PM (#62685456) Homepage Journal

        Unfortunately we are a very large country with very few customers. We don't get much choice who to use.

        You should come on down to America. For actual high speed Internet I can choose between Comcast and Xfinity. It may be a monopoly, but it’s quite competitive. /s

        I moved to a house with internet supplied by the local government. 2Gig FTTH? No problem and cheaper.
        Competition doesn't work with residential internet. The two big players basically carved up the area, so while they both operate in the same city, there is no competition. This is apparently a completely legitimate form of price fixing that ISPs are allowed to get away with.

    • Makes sense (Score:4, Funny)

      by SuperKendall ( 25149 ) on Friday July 08, 2022 @02:05PM (#62685082)

      We don't get much choice who to use.

      I guess this must be where the slang term "rogered" came from then.

    • Re: (Score:1)

      by Anonymous Coward

      I am in Halifax posting using Bell Fiber. My work Cell works (Bell) . My son's cell works, (Kodo) My Wife's cell is down(rogers)

    • McDonald's and grocery store was taking cash only because of this outage.

    • I just arrived in Calgary Canada. The disruption is a big annoyance for some people, but most people and businesses get by OK. Over here, RBC bank machines may not work, but TD bank machines work OK. Some people cannot draw cash from their bank, but can get cash from a credit card with another bank. Some shops only accept cash or credit cards, no debit cards. So, people manage somehow.

    • actually we have decent choice, but it is all on the back-end of a single line. We dont have 5 different companies all bringing physical wires to the entire country. I dont have a single contract with Rogers, but did not have home internet or cellphone internet. I could not take money out of my bank or use a credit card in any store.

      Apparently the toll bridge into, Victoria I believe it was, had to fall back to a barter based economy.

  • 911 Problems (Score:4, Insightful)

    by RobinH ( 124750 ) on Friday July 08, 2022 @02:10PM (#62685100) Homepage
    This is going to give people a wake-up call about consolidating all your services through one provider. At home we have Rogers internet, a VoIP home phone provider (Vonage) and my wife has a Rogers cell phone. All of that was down today including voice calls and texting on my wife's phone. My cell phone is a pay-as-you-go through a different provider, so it worked. Also, our alarm system has police/fire/ambulance buttons and has two ways to communicate: internet and cell modem. I don't know which provider it uses for the cell modem though. So you can easily be in a situation where you lose all ability to contact anyone, including 911 services. In fact, I think a lot of emergency services rely on Rogers as well. There'll be some re-thinking after the dust settles.

    • One provider could be ok, if they are service oriented, affordable, and take their responsibilities seriously. The US had Bell/AT&T as the phone monopoly with universal service for a long time, and reliability was good (though service was lacking at times). Good enough that it was used as the basis for much of the early computer networking and its infrastucture exists still as critical internet backbone in the US.

      The problem with Rogers, is that it's like Comcast, which means they're too big to care bu

      • Re:911 Problems (Score:4, Insightful)

        by jenningsthecat ( 1525947 ) on Friday July 08, 2022 @04:27PM (#62685646)

        One provider could be ok, if they are service oriented, affordable, and take their responsibilities seriously.

        Not if they have a major fuck-up or get hit by a ransomware attack. Diversity is redundancy is additional security. All eggs in one basket is none of these. Regardless of how seriously one provider takes its responsibilities, shit happens.

    • Independent systems are a good thing. If the internet or the power goes out the cell phone still works. If the cell phone goes down I can still use the internet.

      If an earthquake takes down both everyone will know anyway.

      • Re: (Score:2)

        by RobinH ( 124750 )
        But they provide financial incentives to "bundle" your services: internet, home phone, and cell phone, for instance, all through Rogers. In this case the entire Rogers network went down, so if you had internet, home phone, and cell phone all through Rogers, you had no way to communicate. If you had a Fido cell phone, good luck because they also use the Rogers towers.

  • Mister Rogers outage (Score:3, Funny)

    by algaeman ( 600564 ) on Friday July 08, 2022 @02:26PM (#62685182)
    Am I the only one that read that as Mister Rogers outage?

  • I thought they were just getting telephones up there.

    • These old stereotypes are getting tiresome. How would you like it if canadians kept saying things like "What ? They don't have universal healthcare down there ?"

      Oh, wait...

  • Who is this Major Rogers?

  • I wondered why I didn't get my usual "Welcome to Canada" text message I often get living near the border. A signal reflection here, a signal shadow there or just overall T-Mobile crappiness, whatever, my phone often attaches to the foreign tower instead of T-Mobile's.

  • I'm in Canada on TekSavvy, so my service was unaffected. Given the widespread nature of this breakdown as well as how long it has lasted, I think this is probably a cyber-attack, possibly by a nation-state actor. I'd bet #1 on Russia and #2 on China.

    Could be wrong. I guess we'll wait and see (or more likely, wait and not see since I doubt Rogers will be very forthcoming.)

    • My home internet is with Shaw, my personal cell is with Telus, and my work cell is with Bell. Pretty lucky I have completely dodged this one. No Interac at the store today but credit card still works.

      That said Rogers had another major outage a while back due to a software update. Could the same thing happen twice? Seems just as likely.

    • Re: (Score:2)

      by Amnenth ( 698898 )

      TekSavvy is affected - at least, the Cable portion of their operations is, as they resell cable service on the Rogers network.

      If you're on a non-cable line, or in an area where the cable network is not operated by Rogers, that's why you're still good.

      • Re: (Score:2)

        by dskoll ( 99328 )

        Yeah, I'm on DSL, not cable. And apparently, the government is saying it's probably not a cyber-attack.

    • I would put China at #1. We did recently decide to ban Huawei from our country's 5G network, so breaking our existing network seems like an obvious retaliation.

      Russia has no reason that I know of beyond international hockey to dislike us anymore than any other western nation. Still, they may be trying to hack everyone and just managed to have success against Rogers.

    • I'm in Canada on TekSavvy, so my service was unaffected.

      I'm in Canada on TekSavvy and they do not use Rogers to deliver my service, so my service was unaffected.

      FTFY

      Teksavvy uses Rogers infrastructure as well, depending where you are.

  • I'm guessing some sort of platform provisioning automation or an iBGP problem.

    • Re: (Score:2)

      by ceoyoyo ( 59147 )

      Rogers does an upgrade on their Win95 systems every year in the summer. Whether they need it or not.

  • The beatings will continue (Score:3)

    by bustinbrains ( 6800166 ) on Friday July 08, 2022 @03:52PM (#62685530)

    "our technical teams are working hard to restore services"

    Translation: We're underpaying them for their hard work AND taking your money at exorbitant rates while not providing continuous 24/7 service. And, no, you can't have a refund for the outage period.

    • Re: (Score:2)

      by Luckyo ( 1726890 )

      So did the morale improve, or will there be another outage?

    • And, no, you can't have a refund for the outage period.

      The last time this happened they automatically issued credits, and they've already stated they will do so again.

  • I hope that explains why I cannot get to ycombinator, it been out all day so far.

  • It's an interesting subset of outages hereabouts.. (Score:4, Informative)

    by Petersko ( 564140 ) on Friday July 08, 2022 @04:18PM (#62685616)

    Been out to four stores today. All of them could take cash or credit. No debit.

    Thankfully as a 100% remote worker my internet connection has been just fine.

    • Re: (Score:2)

      by Rhipf ( 525263 )

      Debit and Interact rely on the Rogers network in many parts of Canada so when Rogers experiences this kind of outage debit purchases get affected. Credit cards use other providers to handle their verification systems so unless the business that you are shopping at is using Rogers Internet to contact the verifier (and don't have phone fall over authentication setup) most places can still handle credit card purchases.
      Cash is king and it is hard to bring down any system that would invalidate cash purchases.

  • measuring outages in Ontario and New Brunswick (Score:3, Informative)

    by john_heidemann ( 104993 ) on Friday July 08, 2022 @04:22PM (#62685634) Homepage

    We're seeing a huge number of networks unavailable in Ontario and New Brunswick. About 30% of Toronto's IPv4 /24s, and 45% of /24s in Moncton, NB. We describe this at https://ant.isi.edu/blog/?p=18... [isi.edu]

  • Ha! Take that Canada! (Score:3)

    by rsilvergun ( 571051 ) on Friday July 08, 2022 @04:24PM (#62685640)
    USA! USA! US... NO CARRIER...
  • Just a test of a cyber attack by China or Russia. I'm sure Rogers will call it a "software update gone wrong" or whatever the current excuse they use is.
    • When most of the US was knocked out about two years ago, they claimed it was a software glitch

  • So I'm not exactly sure when this started but it was sometime before 7am CDT. It's now 5pm CDT and it's still out. I'm surprised it has lasted this long.

  • What I'm wondering is Why Canada Services Agency doesn't have a failover, and even more so, why the heck doesn't 911 have a backup provider.

    • 911 is not out, anywhere. 911 centres in larger cities have been warning people that if they use Rogers for phone service they will not be able to reach 911, and all people are hearing is "Rogers broke 911".

  • 5pm Pacific, still out.

    This is millions of people without cell or home internet services. Anyone getting pinged with a 2FA text is going to be mighty frustrated.

    This also affects Interac, which is debit. Here in Victoria, BC, no debit was available (at retailers or ATMs). I'm assuming it's the same for the rest of the country.

    On Twitter I'm seeing posts that this may carry on until Monday, and that this is the result of a pretty serious hack. Have yet to see a new story confirming this, but that's the curre

    • Re: (Score:2)

      by ceoyoyo ( 59147 )

      In Ontario, Revenue Canada sent out a bunch of message notifications today. You can't read it without the two factor text (or phone call) of course.

      It's a tax refund, but I bet it made for some nervous mornings until people googled what OTB is.

  • Internet, phone, TV.

    I'm wondering how many people are digging out their old radios to figure out what's going on.

  • Isn't TCP/IP Resilient?

Slashdot Top Deals

Get hold of portable property. -- Charles Dickens, "Great Expectations"

Close