They are saying "The Verizon network is crowded right now" but in the letter they say that the problem is with "interconnection congestion". So yes the problem lies with Verizon and its hesitancy to open up peering points, but if I were a Netflix lawyer that wording would have me concerned since Verizon's network is not crowded. It's like having a crowd outside a bar waiting to get in.
Or is the interconnection part of Netflix's network?
Neither story really brings the entire truth to the table. Netflix could say "our connection to Verizon is full right now" which is more accurate but lacks the oomph that this is all 100% Verizon's fault.
Netflix could say "our pipes have a lot more room but Verizon will only accept so much" which I think also is perfectly accurate yet still paints Verizon as the ones who are lacking in capacity.
A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers.
That leaves the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content.
Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers.
I've read that. It was what I was thinking of when I wrote both the quotes in my comment. Netflix wants to get data to Verizon but the bottleneck is at the connection between them.
I've read that. It was what I was thinking of when I wrote both the quotes in my comment.
How remarkable.
What I got from that piece is that - if Verizon are one of the companies mentioned - Verizon customers are requesting data from Netflix.
Netflix have bought connectivity from Level3.
Level3 have peering arrangements with Verizon for sockets that are dropping packets and over which Verizon is refusing to take part in a joint connection upgrade.
Continually upgrading your connectivity in peering arrangements to cope with rising demand is, for nearly every other ISP in the world (outside certain monopoly markets in the US), completely standard practice as otherwise you die.
And Verizon is refusing to do this unless Level3's own customers pay it money.
Given this is a peering arrangement, now imagine it the other way around.
Say Level3 had it's own TV business and was asking Verizon's broadband customers to cough up money to them on top of Netflix and Verizon, if they wanted Level3 to upgrade the network that Netflix are already paying for, for a usable connection to Netflix...
Think about that for a bit, and then try and tell me that the whole affair doesn't look utterly hatstand.
The only thing I would add to your argument is the part about ISP customers having paid the ISP for that bandwidth but which the ISP is refusing to deliver.
Prior to capitulation, Netflix purchased access to Verizon through multiple tier 1 providers. These providers are publicly known to have private arrangements regarding interconnections.
More than one of the backbone providers in question have publicly stated that Verizon/Comcast/TWC are no longer following the tenants of their private agreements.
The bare contest here is that the established order of hierarchical internet traffic is being challenged. The reason, IMO, people on HN dislike the ground shift is that it:
a) creates communication failures where there were previously fewer
b) is based on regulatory capture and monopolistic business practices
c) is not a technology driven change
d) increases the moat for all established internet services
Interconnection points are by definition part of two networks. The question is whether the congested interconnect involves Verizon at all. I would hope that, before writing this letter, Mr. Hyman would have done his research and verified that it is indeed an interconnect that Verizon is involved with. However, it is an unanswered question.
I'm sure Netflix is carefully choosing every word, well beyond one person's responsibility. This is fire they are playing with, and they know it. They're probably treating this like a legal brief that will end up in court before a judge, because the odds of that actually happening are pretty good.
It is not an unanswered question, it is merely a question you personally have not bothered researching properly. Network engineers have thoroughly answered this question.
Heck, even I write mails to my ISP and tell them which interconnect points they're having issues at which a simple traceroute/mtr show me. And I mostly get positive feedback.
With a bit more sophisticated tools, you'll get more confidence. Or if you're Netflix, I'd assume you can just call up level3, ask them to resolve the transit issues you're having with Verizon, and get feedback and reports about it being verizon not upgrading their interconnect to level3. I'd not be terribly surprised if verizon is one of the 6 peers mentioned at http://blog.level3.com/global-connectivity/observations-inte...
Traceroutes from a single source are poor tools for positively determining a source of latency. A traceroute only tells a unidirectional story, and an incomplete one at that.
In reality in is essential to use multiple provider/observer based tools to determine point(s) of congestion. However, it is entirely possible to empirically determine who the congested peers are between, which, and when specific geographical connections are overloaded.
There is also a group of builders who have been waiting over a year to put up a large extension on the giant unused wasteland out back, however the bar owner has been telling them not to bother. Oh, and the bar owner also has a load of liquor stores and only bought the bar in the first place to stop the extension getting built.
