Network adapter brand affects latencyChanges in IP and MAC address affect routing
2015-05-20 (updated: 2016-06-04) by Philip
Tags: NIC, OUI, latency, ping, rtt
I was enjoying a great, stable internet connection from Comcast, with the fiber running at the street about 30 feet from my house, with a strong clean signal to my SB6141 modem. I had to call the cable company to change some TV package, and we had to reboot/reprovision the modem as well. A day or two later I noticed my worry-free internet was turning into a nightmare.
I was experiencing terrible speed and latency spikes in peak hours (900+ milliseconds to the first/second hop), as well as speed drops down from ~30 to 3-4Mbps. It was happening every day between ~6pm and midnight.
Calling Comcast every day lead me through the usual mindless maze of low-level support tech recipes (even though my modem signal levels were great). I did the "reboot modem" dance a dozen times, I had to unplug my NAT router and use a client PC directly, I listened to "we don't detect a problem on our end", there is "no issue in the area", "your modem is bad and needs to be replaced", and my favorite: "your cable modem is not supported" (I do have a retail-bought Motorola/Arris SB6141 that is actually DOCSIS 3.1 compliant and on the Comcast recommended list). I couldn't convince anyone at Comcast even for a millisecond that there is possibly a problem at their end, something was wrong at their node ? I couldn't convince anyone to even look at the traces at this point.
After jumping through a few hoops, a tech came out to the house, signal levels looked ok, but he did find a somewhat problematic fiber tap on the street and moved me to another leg. Signal levels at the modem were very good throughout this whole process, there was nothing marginal. The tech was knowledgeable, I was able to show him the traceroutes, but my internet problems continued. Bellow are a couple of actual example traces to illustrate the obvious issue.
Bad connection example at peak times (speedtest: 3.5Mbps down / 1 Mbps up):
Note the bold 220.127.116.11 IP address in the second hop, I refer to it as the "bad" node below. That is where the issues start (huge lag spikes).
Good connection during the day (testing @ 30Mbps down / 6 Mbps upstream):
As you can see, the difference is night and day, literaly (some pun intended). The issue dragged on for another couple of weeks and rendered any video streaming and gaming useless, even VoIP was affected. Comcast reimbursed me $30 for the service issues after I made about 10 calls, but never admitted any issue on their end.
Being the tinkering junkie that I am, I did notice something curious after continuous rebooting and switching hardware on my end. Note that it is common knowledge that every time you plug in a different Ethernet client to a cable modem, you have to reboot it so that the MAC address of the new client is recognized and leased an IP address. To my big surprise, connecting a client with an Intel NIC fixed the connection issues, switching back to a Realtek NIC or the default Asus router MAC caused the terrible delay again. The routing changed as well, i.e. I was being issued a different IP address, going through different hops in traces. It was repeatable... Is it possible that Comcast was splitting clients based on MAC addresses, and more specifically the OUI (Organizationally Unique Identifier) part of the MAC address ?
In short, every time I got issued an IP where the first hop in traceroute was 18.104.22.168 I had a huge latency/speed issue at peak times, however, if the first hope was something else the connection was perfectly fine. I was able to get this to change by having a client with a different MAC address connect to the modem (and being issued a new IP). The difference was orders of magnitude, i.e. latency at 50ms with one and 900ms with the other. I streamlined the process of changing the client MAC address by reconnecting my NAT router and network back together, and simply doing the following:
1. Login to the admin panel of the router, and find the "Custom MAC address" setting (usually in "Advanced" or "WAN" menu)
With all that in mind, and having found the "MAC Address" setting in the router's admin interface, note the format:
AA:BB:CC:11:22:33 (where AA:BB:CC is the OUI portion of the MAC address)
MAC addresses are unique identifiers assigned to network interfaces for communication on the physical level. The most common MAC address format consists of six groups of two hexadecimal digits separated by colons or hyphens. The first 24 bits (the first three groups) being the OUI (Organizational Unique Identifier), specifying the unique vendor for that device. The OUI is assigned to vendors by the IEEE, and a single organization/vendor can have more than one OUI. Here are some common OUIs for testing:
C0:C1:C0 Cisco-Linksys, LLC
F0:7D:68 D-Link Corporation
00:AA:01 Intel Corporation
9C:8E:99 Hewlett Packard
You can use any of the above as an example of the first three hex digits of the MAC address (the OUI) to force the cable modem to assign you a new IP address, possibly in a new subnet and with different latency/speed characteristics. Here is an example traceroute after simply changing the MAC address of the NAT router, rebooting and getting a new IP from Comcast:
Note that the IP in the second hop is completely different, bypassing the bad 22.214.171.124 second hop that was causing the terrible latency/speed issues. Problem solved! Really. Really ? Intel and some Asus OUIs were getting routed on the "good" (126.96.36.199) node, others were always routed on the "bad" (188.8.131.52) node that was terribly congested for six hours every evening. The external IPs I was getting were in completely different subnets as well.
To summarize, you have some limited control of how your packets get routed, and you can possibly change the routing close to you to go around local node problems in some cases. It can even have an effect on the number of hops. This may only be true for some ISPs, if your ISP assignes IPs and uses the MAC address OUI in a similar fashion as Comcast. It may also be possible if you simply are able to have your external IP address changed in any other way (keeping modem off until DHCP lease expires, etc.). In my case the above trick solved all my issues and I haven't looked back. I am left with a bit of a sour taste from my whole experience with Comcast throughout this. Excellent techs, yet terrible customer service and I don't even want to start a rant on pricing policies ;)
I hope this helps someone, comments are always welcome.