ISP Fights Causing Netflix Packet Drops 289
An anonymous reader writes "We've been hearing more and more reports of ISPs throttling Netflix and other high-bandwidth services lately. The ISPs have denied it, and even Netflix itself seems to believe them. If that's the case, what's going on? Well, according to this article, the blame still lies with the ISPs. While they may not be explicitly throttling connection speeds, they're refusing to upgrade network connections as they demand more money from content distributors. For example, Netflix pays Cogent to distribute their internet traffic. Cogent has an agreement with Verizon to exchange traffic — which works fine until the massive amount of traffic from Netflix makes it a lopsided arrangement. Verizon wants more money from Cogent, and one of their negotiating tactics is simply to stop upgrading their infrastructure so that service degrades. 'There are about 11 Cogent/Verizon peering connections in major cities around the country. When peering partners aren't fighting, they typically upgrade the connections (or "ports") when they're about 50 percent full, Cogent says. ... With Cogent and Verizon fighting, the upgrades are happening at a glacial pace, according to Schaeffer. "Once a port hits about 85 percent throughput, you're going to begin to start to drop packets," he said. "Clearly when a port is at 120 or 130 percent [as the Cogent/Verizon ones are] the packet loss is material."'"
Re:Can confirm (Score:4, Informative)
Oddly enough Netflix is running well for me on Time Warner in southern California. I am no fan of Time Warner, but House of Cards (the major hot button that people seem to be using as the yard stick for performance this week) is done buffering HD by about half way through the opening credits.
The majority of the articles that I have seen all seem to mention Verizon, with little to no mention of the other ISPs. Maybe the real issue is a fight between Verizon and Cogent? If that is the case, what is the solution? Do we recommend allowing Verizon to acquire more content producers so that they stand to benefit by pushing their content back the other way, thereby restoring the in/out ratio?
Re:Network vs Content providers (Score:5, Informative)
And this is how it's supposed to work (Score:3, Informative)
Look, for years and years, peering agreements have been based on the idea that peering between partners would have roughly equal inbound and outbound traffic patterns. When one partner starts pushing more traffic out than in, it signals an imbalance in the connection, and actually the burden.
Let me explain: internet routing is based on autonomous system (AS) hops. Cogent is an Autonomous System, so is Verizon, Level3, etc. If I'm connected to one AS, and I need to send a packet to someone on another AS, the router within my AS attempts to deliver that packet to the closest exit point for that AS. So for example, if I'm sending packets from Cogent to a user on Verizon, then Cogent will offload the transport of the packet to the closest point possible (usually in the same city). The idea being that Cogent doesn't know where the user is on verizon, so they just want to send it to verizon using an Exterior Gateway Protocol (e.g. BGP - which is AS hope based), and then verizon will use their Interior Gateway Protocol (e.g. something like OSPF) to deliver it most efficiently inside of their network.
And therein lies the problem with unequal inbound and outbound peering situations. If Cogent is 80% inbound and 20% outbound with a specific peer, that is usually shouldering the burden for most of the transport distance and cost for that packet. If it's 50/50, then the burden is the same, and they're equal peers.
In the internet routing world there are tier1 providers, tier 2 providers, etc etc based on how many peers they have. But of the tier 1's - not everyone is created equally. Cogent has Tier 1 status, and their POPs are all interconnected, but they don't have as many geographic POPs as say a Verizon. As a result, they dump packets to their peers as local as possible and those packets are routed across the internet by the peers. Their piers on the other hand, for the burden that *should* be cogents have fewer locations that they peer with cogent due to their limited # of POPs. So even for the outbound to cogent traffic, they end up shouldering a disproportionate amount of that traffic transporting it to one of cogent's pops.
And then you have companies like netflix. Netflix buys bandwidth from these low cost tier1 providers, who are low cost because they don't share the transport burden with the real tier 1s. And when congestion happens, and the real tier1's tell cogent "sorry you're out of bounds withe contract because you don't have an even inbound / outbound ratio" cogent decides not to pay the penalties for the uneven traffic patterns. Instead, they let congestion and packet drops happen. The Netflix comes along, and says "Oh yeah, this provider, they're terrible for netflix traffic - they don't upgrade their pipes." They do this knowing full well the economics of internet peering. How do they know full well? Because they don't even want to pay cogent. All the while that they're letting the tier1 take the heat for the crap peering situation, they're approaching said tier1 saying "Hey, nice network you got there. It would be a shame if someone publicized bad information about its performance. Hey, you know how you can prevent that from happening? Join open connect! All you have to do is host our hardware, provide power, and data center space, and cooling, oh and connectivity for free and your network will look great for netflix customers who won't have to suffer through this peering situation you have going on with Cogent."
And now, Netflix, not being a back bone at all, has managed to make you look bad and then tell you in order to look good you should host their hardware and give them free transport... And because /. is /. Netflix comes out looking like the victim here...
Re:Net Neutrality laws? (Score:5, Informative)
Yeah, but it's not like Netflix is using a free service. They are paying for that bandwidth. I assume they are paying quite a bit. More importantly, someone is selling it to them.
Re:Net Neutrality laws? (Score:4, Informative)
I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs.
All the peering details are here [netflix.com]. In short: they don't charge anything. They offer direct interconnects to Netflix's CDN for free, free peering at major internet exchange points, and free, Netflix-managed hardware caches to ISPs to avoid duplicate network traffic (the vast majority of traffic stays within the ISPs internal network). For the hardware caches the ISP needs only provide power and network connectivity.
There's really no reason for ISPs to wrangle with Netflix -- there's plenty of options to avoid congestion.
sell is the key word. Cogent not paying Verizon (Score:4, Informative)
They SELL connections to their customers. Cogent isn't paying Verizon. The peering they're talking about is an even trade with neither company paying the other. When it ceases to be an even trade, it's time for negotiations.
Re:Network vs Content providers (Score:2, Informative)
The congestion isn't on the customer-facing side of the ISP's network, it's at the peering exchange with Netflix's carriers. They make no promise about that.
Re:Can confirm (Score:5, Informative)
It has been a while since I was in the peering game - that was back when UUNet meant something and please get off my lawn - but here's the general idea:
To provide access to the Internet - either as a content host or a content consumer - you need a full view of routes to get to all other ISPs where your sources or destinations are. To get the routes, you need "peering" with the other ISPs or you need to buy "transit" from some other ISP that gets their routes via peering or transit.
Some ISPs are, frankly, more important than others because they provide access to more subscribers or more content than others - they used to be called "Tier 1" ISPs. Peering is valuable because it's traffic you're exchanging for free that you could otherwise be charging a lot of money for. Tier 1 ISPs generally agree to peer with each other because they all need each other, and it makes economic sense for them to say they're all on an equal footing - they were "peers." The economic rationale was because Tier 1 ISPs had to pay for large national or global networks, while Tier 2 or 3 ISPs had small or regional networks and that the Tier 1 ISP was bearing most of the cost of delivery. Traffic ratios were preferred to be equal (content vs. users) for peering, because if you're a content host with one datacenter and some outbound circuits, your cost is far less than having a big national network to serve end users - so web hosting/colo provider ISPs had a harder time getting peering with the big consumer/business access providers. The Tier 2 or Tier 3 ISPs would peer with each other freely because they had equivalent footprints, etc., but the big guys knew that access to their network was extremely valuable and it would be foolish to give it away for free.
So if you're a smaller ISP, and *you* need the Tier 1 more than *they* need you, don't expect to get peering. The ISP will tell you to buy transit from them, or at least buy transit from someone else who does (and the fewer "hops" to get from you to them, the better for your customers). Cogent may host much of Netflix, but they are by no means a Tier 1. This may no longer be the case - like I said, I have been out of the Tier 1 ISP world for years - but at least historically Cogent was known as a bottom feeder of the industry. They charged dirt-cheap rates but ran a crappy network and skimped on their upstream connections to cut costs.
So what's happening here most likely is that Cogent has either bought transit from Verizon and doesn't want to buy more and says "peer with us, we won't buy more." Or Cogent does have peering with Verizon but VZ has said, in effect, "you are not our 'peer'" and beyond a certain amount of peering bandwidth, you should start buying." Cogent is using Netflix to try to argue that "our content is more important to Verizon customers than the other way around," and Verizon is saying, "Um, nope." I won't say who I think is right or wrong here, but this is not the first time Cogent has had peering fights with other large ISPs [google.com] and I think you can see a pattern here.
Re:Network vs Content providers (Score:4, Informative)
Re:Netflix should get benefit from desirability (Score:2, Informative)
And if you demand your ISP gives their peering partners anything they want, the peers will take massive advantage, and dump tons of their own traffic on your ISP's links, to get delivered across the country by them.
That is not possible, and would be a violation of internet engineering rules, and a breach of contract.
A peering arrangement, in contrast to a transit arrangement, only advertises routing prefixes within the AS of the eBG router, and is not obliged route any traffic for routes it has not advertised (it is obliged to route all routes it does advertise). Any sensibly configured eBG router will have IP filtering so that it doesn't deliver traffic for prefixes it hasn't advertised, and it would take deliberate manual configuration in contravention of IER to forward traffic for routing to a peer which has not advertised route availability for the traffic in question.
On the other hand, Cogent is acting as a transit provider while pretending to be a peer, which explains why Verizon is being hostile. Cogent is using an early exit strategy, whereby they route into nationwide ISPs networks at the closest available route, knowing that nationwide ISPs have internal nationwide networks for delivering traffic within their network. They could institute route filtering so each peering point only advertises routes to nearby customer edge networks, and Cogent is forced to carry traffic cross country before dropping it on Verizon and AT&T, but this would complicate network management, create potential routing failures within their network, and not be fair to other peers who may not have nationwide delivery.
Instead I think Verizon are taking the most amicable option available to them, letting Cogent saturate their peering links and expecting Cogent to build more peering links or deliver their traffic closer to it's actual destination as every transit ISP is expected to do.