I don't know if these two peering partners are trying to duke it out, but I've been experiencing some horrible latency and network drops with VPN traffic to a customer of mine on Centurylink. I'm fortunate to have access on both sides, so I can trace the traffic in both directions. The traffic is from Minneapolis Comcast to Minneapolis Qwest/Centurylink, and in most ISP cases around here the traffic would peer in Chicago. The minimum latency I can get is 44ms RT, which is about 2.5 times what it should be if it were symmetric through Chicago. During peak business and evening hours this will average more around 100-150ms RT with a 1% drop rate or so. When this happens, there's no signs of latency spikes to other destinations on the Internet, so it appears to be specific with Centurylink. To put this in perspective, latency during these times from Comcast Minneapolis to Centurylink Minneapolis is on par with traffic from Minneapolis to Japan or Minneapolis to Eastern Europe.
Comcast appears to be optimally routing traffic through the Equinix carrier hotel in Chicago:
4 12 ms 11 ms 12 ms te-0-7-0-11-ar01.roseville.mn.minn.comcast.net [69.139.219.126]
5 21 ms 23 ms 19 ms he-1-11-0-0-cr01.350ecermak.il.ibone.comcast.net [68.86.94.73]
6 19 ms 18 ms 19 ms he-0-15-0-1-pe04.350ecermak.il.ibone.comcast.net [68.86.85.222]
7 100 ms 101 ms 104 ms chp-edge-01.inet.qwest.net [216.207.8.189]
However in the above traceroute you can see once it passes into Centurylink in Chicago, the latency on that edge router in Chicago spikes way high. Normally if the return path was right back to comcast at that peering point, the latency on the chicago centurylink edge router wouldn't be that much different chicago comcast edge router. The spike in latency however is better explained by seeing the rediculous return path:
4 5 ms 5 ms 5 ms min-edge-12.inet.qwest.net [63.234.166.5]
5 31 ms 30 ms 30 ms atx-edge-05.inet.qwest.net [67.14.14.214]
6 32 ms 32 ms 32 ms 65.124.97.178
7 34 ms 36 ms 35 ms he-3-11-0-0-11-cr01.newyork.ny.ibone.comcast.net [68.86.88.181]
8 34 ms 34 ms 35 ms he-2-12-0-0-cr01.chicago.il.ibone.comcast.net [68.86.89.9]
9 37 ms 35 ms 35 ms be-10406-cr01.350ecermak.il.ibone.comcast.net [68.86.84.210]
10 45 ms 46 ms 42 ms he-0-14-0-0-ar01.roseville.mn.minn.comcast.net [68.86.94.78]
According to this, the return path from the Minneapolis centurylink connected client of mine to my house in Minneapolis is via Minneapolis -> Atlanta -> NYC -> Chicago -> Minneapolis. These ping times are more normal ping times, but when congestion occurs these will stabilize up in the 100-150ms range with maybe a .5% drop rate.
What's a bit shocking is that my IP is out of Comcast's 73.37.128.0/17 netblock. All of the 73.* networks are aggregated to 73.0.0.0/8, so they're not advertising regional networks with route preferences in those regions. You would think since they aggregated this entire netblock they would ensure that their peering partners don't put non-optimal local preferences or apply metrics on them to lock the traffic through a specific peering point. I'm assuming that Comcast is advertising 73.0.0.0/8 to Centurylink in Chicago, but the most typical BGP metrics that would break this would be a higher BGP local preference or a lower MEDs value being exchanged in Atlanta. It doesn't help that Centurylink's BGP looking glass page is totally not working either, which means I'm not better able to understand if Comcast is advertising it this way with a 209:XXX community string or of Centurylink is just abusing traffic to Comcast.
I spent around 3 hours on the phone trying to find someone at Comcast's Business Class support line that could maybe address this concern or raise this issue with an engineer. All they could tell me is that my browser was broken. Yeah.. thanks for the tip. They also just want to say that the routing is working fine within Comcast, so it's not their problem. Except trying to tell someone in Technical Support that it can actually be their problem is very difficult to do. Also given the peering agreements, it's totally up to Comcast to discover and resolve this if Centurylink is adversely routing Comcast traffic. So this experience made me call a local DSL provider to get quotes in case this doesn't get resolved soon. I figured I'd document my complaint out here and let others chime in if they're experiencing it too.
I was also at another customer on Friday, and noticed their Comcast 74-netblock out of Minnesota was routing out of Centurylink in Dallas.
With net-neutraility developing more into a fist fight for more money, I've been getting a lot more sensitive to what's been happening with my traffic. Like the way Comcast specifically shapes single tcp connections not to exceed around 8mbits or so - UDP shaping is even worse. I support a lot of networks, and notice a lot of these irregularities when I'm on VPNs. To any carrier that's playing games with customer traffic - we're noticing this and hearing about it more.
-Joe
↧