The Broadband Guide
SG
search advanced
 Username:
 Password:
Register
 forgot password?

Network adapter MAC/OUI/Brand affect latency

Changes in IP and MAC address affect routing
2015-05-20 (updated: 2016-11-04) by
Tags: , , , ,

I was enjoying a great, stable internet connection from Comcast, with the fiber running at the street about 30 feet from my house, with a strong clean signal to my SB6141 modem. I had to call the cable company to change some TV package, and we had to reboot/reprovision the modem as well. A day or two later I noticed my worry-free internet was turning into a nightmare.

I was experiencing terrible speed and latency spikes in peak hours (900+ milliseconds to the first/second hop), as well as speed drops down from ~30 to 3-4Mbps downstream, and from 6Mbps to ~0.3Mbps upstream. It was happening every day between ~5pm and midnight.

Calling Comcast every day lead me through the usual mindless maze of low-level support tech recipes (even though my modem signal levels were great). I did the "reboot modem" dance a dozen times, I had to unplug my NAT router and use a client PC directly, I listened to "we don't detect a problem on our end", there is "no issue in the area", "your modem is bad and needs to be replaced", and my favorite: "your cable modem is not supported" (I do have a retail-bought Motorola/Arris SB6141 that is actually DOCSIS 3.1 compliant and on the Comcast recommended list). I couldn't convince anyone at Comcast even for a millisecond that there is possibly a problem at their end, something was wrong at their node ? I couldn't convince anyone to even look at the traces at this point.

After jumping through a few hoops, a tech came out to the house, signal levels looked ok, but he did find a somewhat problematic fiber tap on the street and moved me to another leg. Signal levels at the modem were very good throughout this whole process, there was nothing marginal. The tech was knowledgeable, I was able to show him the traceroutes, but my internet problems continued. Bellow are a couple of actual example traces to illustrate the obvious issue.


Bad connection example at peak times (speedtest: 3.5Mbps down / 1 Mbps up):

C:\>tracert comcast.net
Tracing route to comcast.net [69.252.80.75]
over a maximum of 30 hops:
  ...
  2   650 ms   738 ms   883 ms  c-50-159-241-1.hsd1.fl.comcast.net [50.159.241.1]
  3   765 ms   701 ms   848 ms  te-0-0-0-9-sur02.kernan.fl.jacksvil.comcast.net [68.85.94.173]
  4   752 ms   630 ms   850 ms  te-0-1-0-3-ar02.westside.fl.jacksvil.comcast.net [68.87.164.69]
  5   562 ms   595 ms  1006 ms  be-33489-cr01.miami.fl.ibone.comcast.net [68.86.95.45]
  6   688 ms   704 ms   883 ms  be-10014-cr01.ashburn.va.ibone.comcast.net [68.86.85.33]
  7   807 ms   845 ms   803 ms  be-10014-cr01.ashburn.va.ibone.comcast.net [68.86.85.33]
  8   901 ms  1081 ms   890 ms  et-2-0-0-0-ar03.newcastle.de.panjde.comcast.net [68.86.166.122]
  9   853 ms   813 ms   757 ms  te-5-3-ur01-d.newcastlerdc.de.panjde.comcast.net [68.85.159.118]
10   901 ms   946 ms   817 ms  te-5-3-ur01-d.newcastlerdc.de.panjde.comcast.net [68.85.159.118]
11   808 ms   979 ms   *   75.80.252.69.in-addr.arpa [69.252.80.75]
Trace complete

Note the bold 50.159.241.1 IP address in the second hop, I refer to it as the "bad" node below. That is where the issues start (huge lag spikes).


Good connection during the day (testing @ 30Mbps down / 6 Mbps upstream):

C:\>tracert comcast.net
Tracing route to comcast.net [69.252.80.75]
over a maximum of 30 hops:
  ...
  2   10 ms   14 ms   16 ms  c-50-159-241-1.hsd1.fl.comcast.net [50.159.241.1]
  3   23 ms   16 ms   10 ms  te-0-0-0-9-sur02.kernan.fl.jacksvil.comcast.net [68.85.94.173]
  4   18 ms   18 ms   20 ms  te-0-1-0-3-ar02.southside.fl.jacksvil.comcast.net [68.87.164.65]
  5   47 ms   25 ms   20 ms  be-33489-cr01.miami.fl.ibone.comcast.net [68.86.95.45]
  6   33 ms   39 ms   49 ms  be-10014-cr01.ashburn.va.ibone.comcast.net [68.86.85.33]
  7   46 ms   42 ms   43 ms  et-2-0-0-0-ar03.newcastle.de.panjde.comcast.net [68.86.166.122]
  8   60 ms   43 ms   43 ms  68.85.158.90
  9   47 ms   58 ms   50 ms  te-5-3-ur01-d.newcastlerdc.de.panjde.comcast.net [68.85.159.118]
10   44 ms   50 ms   45 ms  te-5-3-ur01-d.newcastlerdc.de.panjde.comcast.net [68.85.159.118]
11   55 ms   46 ms   47 ms  urlrw01.cable.comcast.com [69.252.80.75]
Trace complete.


As you can see, the difference is night and day, literally (some pun intended). The issue dragged on for another couple of weeks and rendered any video streaming and gaming useless, even VoIP was affected. Comcast reimbursed me $30 for the service issues after I made about 10 calls, but never admitted any issue on their end.

Being the tinkering junkie that I am, I did notice something curious after continuous rebooting and switching hardware on my end. Note that it is common knowledge that every time you plug in a different Ethernet client to a cable modem, you have to reboot it so that the MAC address of the new client is recognized and leased an IP address. To my big surprise, connecting a client with an Intel NIC fixed the connection issues, switching back to a Realtek NIC or the default Asus router MAC caused the terrible delay again. The routing changed as well, i.e. I was being issued a different IP address, going through different hops in traces. It was repeatable... Is it possible that Comcast was splitting clients based on MAC addresses, and more specifically the OUI (Organizationally Unique Identifier) part of the MAC address ?

In short, every time I got issued an IP where the first hop in traceroute was 50.159.241.1 I had a huge latency/speed issue at peak times, however, if the first hope was something else the connection was perfectly fine. I was able to get this to change by having a client with a different MAC address connect to the modem (and being issued a new IP). The difference was orders of magnitude, i.e. latency at 50ms with one and 900ms with the other. I streamlined the process of changing the client MAC address by reconnecting my NAT router and network back together, and simply doing the following:

1. Login to the admin panel of the router, and find the "Custom MAC address" setting (usually in "Advanced" or "WAN" menu)
2. Change the MAC address, preferably to one with a different OUI (manufacturer)
3. Reboot the cable modem and the router
4. Test latency and speed, making sure you have been assigned a new external IP address. Note any change in IPs of the first few hops in traces.

Notes:
Some cable providers may require you to release the assigned IP address to get a new one, even if your client MAC address changes. This can be accomplished by using "ipconfig /release" in Command prompt. The caveat being that you have to release the public IP assigned by your cable modem, not the internal LAN IP that your NAT router provides.
Some ISPs may also limit the number of newly assigned IPs that you can get in a certain period. If you request more than 3-4 IPs per 12 hours, you may have to wait to get assigned a new one.


With all that in mind, and having found the "MAC Address" setting in the router's admin interface, note the format:

AA:BB:CC:11:22:33 (where AA:BB:CC is the OUI portion of the MAC address)

MAC addresses are unique identifiers assigned to network interfaces for communication on the physical level. The most common MAC address format consists of six groups of two hexadecimal digits separated by colons or hyphens. The first 24 bits (the first three groups) being the OUI (Organizational Unique Identifier), specifying the unique vendor for that device. The OUI is assigned to vendors by the IEEE, and a single organization/vendor can have more than one OUI. Here are some common OUIs for testing:

Wireshark ยท OUI Lookup Tool

F0:79:59 ASUSTek COMPUTER INC.
E0:46:9A NETGEAR
C0:C1:C0 Cisco-Linksys, LLC
F0:7D:68 D-Link Corporation
00:AA:01 Intel Corporation
00:4F:49 Realtek
9C:8E:99 Hewlett Packard


You can use any of the above as an example of the first three hex digits of the MAC address (the OUI) to force the cable modem to assign you a new IP address, possibly with a slightly different routing, in a new subnet and with different latency/speed characteristics. Here is an example traceroute after simply changing the MAC address of the NAT router, rebooting and getting a new IP from Comcast:

C:\>tracert comcast.net
Tracing route to comcast.net [69.252.80.75]
over a maximum of 30 hops:
1 1 ms 1 ms 1 ms RT-AC68R [192.168.1.1]
2 24 ms 8 ms 16 ms 66.177.176.11
3 9 ms 16 ms 9 ms te-0-7-0-10-sur02.kernan.fl.jacksvil.comcast.net [68.86.170.117]
4 11 ms 19 ms 19 ms 69.139.222.185
5 24 ms 24 ms 26 ms be-33489-cr02.56marietta.ga.ibone.comcast.net [68.86.95.49]
6 27 ms 27 ms 39 ms be-10114-cr02.ashburn.va.ibone.comcast.net [68.86.85.9]
7 41 ms 41 ms 42 ms et-1-1-0-0-ar03.newcastle.de.panjde.comcast.net [68.86.94.250]
8 43 ms 43 ms 40 ms ae100-ur11-d.newcastlerdc.de.panjde.comcast.net [68.85.158.90]
9 40 ms 41 ms 42 ms ae100-ur11-d.newcastlerdc.de.panjde.comcast.net [68.85.158.90]
10 42 ms 43 ms 43 ms urlrw01.cable.comcast.com [69.252.80.75]
Trace complete.

Note that the IP in the second hop is completely different, bypassing the bad 50.159.241.1 second hop that was causing the terrible latency/speed issues. Problem solved! Really. Really ?  Intel and some Asus OUIs were getting routed on the "good" (66.177.176.1) node, others were always routed on the "bad" (50.159.241.1) node that was terribly congested for six hours every evening. The external IPs I was getting were in completely different subnets as well.

To summarize, you have some limited control of how your packets get routed, and you can possibly change the routing close to you to go around local node problems in some cases. It can even have an effect on the number of hops. This may only be true for some ISPs, if your ISP assignes IPs and uses the MAC address OUI in a similar fashion as Comcast. It may also be possible if you simply are able to have your external IP address changed in any other way (keeping modem off until DHCP lease expires, etc.). In my case the above trick solved all my issues and I haven't looked back. I am left with a bit of a sour taste from my whole experience with Comcast throughout this. Excellent techs, yet terrible customer service and I don't even want to start a rant on pricing policies ;)

I hope this helps someone, comments are always welcome.


Notes:

The above can possibly be used to improve your latency in gaming, etc. as well.
Some OUIs, even by the same vendor, were getting assigned IPs in a completely different DHCP pool, and getting routed differently.
It is not necessarily your IP address that makes the difference, rather a change in the first couple of hops that indicates connecting through different, less congested hardware.
The above is just my personal experience, the testing is by no means scientific.


Update on 2016-11-04:
I've had a pretty stable connection for over a year since this article was written. However, we had some downtime yesterday morning for about an hour, seems that Comcast was doing some work in the neighborhood. After that, my external IP address changed, and the above troubles reappeared right after 5pm - speed was way down and latency was unbearable during peak hours, to the point where webpages wouldn't load (down to 3-4Mbps downstream, 0.3Mbps upstream, 900ms+ latency to distant hosts). I reapplied the trick from this article changing the MAC address of the router slightly and rebooting the Cable modem. I had to try a couple of times with different MACs until I was able to get a new IP address with slightly different trace, and my advertised speed was back! I did a some traceroutes before and after, and seems the second hop in my traces changed. There is a definite correlation, seems like I am going through a much less congested network.

  Post your review/comments
    rate:
   avg:
comment print discuss top
News Glossary of Terms FAQs Polls Cool Links SpeedGuide Teams SG Premium Services SG Gear Store
Registry Tweaks Broadband Tools Downloads/Patches Broadband Hardware SG Ports Database Security Default Passwords User Stories
Broadband Routers Wireless Firewalls / VPNs Software Hardware User Reviews
Broadband Security Editorials General User Articles Quick Reference
Broadband Forums General Discussions
Advertising Awards Link to us Server Statistics Helping SG About