Poor Speeds from NZ to US in Windows

SamboNZ · Post by **SamboNZ** » Sat May 25, 2019 6:16 am

Hey guys,

I have a tricky issue and I need the help of your networking expertise!

Problem:
I am located in New Zealand and get virtually full line speed to the US (west coast) when running Linux.

However, with all the same hardware etc, I get relatively poor speeds under Windows 10.

I have seen reports on the net about others with similar issues, but I have yet to see anyone come up with a working solution.

Details:

Advertised Speed:
930/550Mbps (DS/US) Fibre connection

Performance (iPerf3, 10 threads):
LAN - 930/930Mbps
Auckland (2ms) - 930/530Mbps
Sydney (25ms) - 900/500Mbps
Los Angeles (125ms) - 450/150Mbps (900/500Mbps under Linux)

Current Settings:
Client OS/browser: Windows 10 (Firefox 66.0)

TCP options string: 020405b40103030801010402
MSS: 1460
MTU: 1500
TCP Window: 262656 (not multiple of MSS)
RWIN Scaling: 8 bits (2^8=256)
Unscaled RWIN : 1026
Recommended RWINs: 64240, 128480, 256960, 513920, 1027840
BDP limit (200ms): 10506kbps (1313KBytes/s)
BDP limit (500ms): 4202kbps (525KBytes/s)
MTU Discovery: ON
TTL: 52
Timestamps: OFF
SACKs: ON
IP ToS: 00000000 (0)

Router / Firewall:
Sophos SG 135 (UTM v9.602-3)

OS:
Windows 10 Pro v1809

LAN connection:
1Gbit wired

Additional Testing / Diagnosis:
- Tested with different PCs (also Windows 10) - no change
- Updated network card drivers - no change
- Tested to different iPerf3 servers - no change
- CPU utilization is nominal on all test PCs
- Checked / changed anything which looked relevant on the tweaks page (https://www.speedguide.net/articles/win ... weaks-5077)
- I have fully eliminated all local LAN network equipment & cabling, including the router / firewall.

Questions:
- Is there any way to monitor the current TCP Window size in real time as it scales?

Any input / ideas / hail Marys gratefully accepted!

Post by **Philip** » Sun May 26, 2019 5:02 pm

You can probably monitor your real-time RWIN value using a packet capture program like Wireshark, however, it seems fine from your Analyzer report. Have you tried the "optimal" settings in the TCP Optimizer? How do you measure your speed?

SamboNZ · Post by **SamboNZ** » Sun May 26, 2019 8:36 pm

Yes, I've tried optimal.

I'm concerned that the RWIN may not be scaling enough considering that in my case it needs to 11MB...

I have been measuring my speed using a number of different methods:
- Speedtest.net
- nperf.com
- iPerf3

But the most reliable is definitely iPerf3.

Post by **Philip** » Mon May 27, 2019 8:07 am

You may also have to consider that not many routers/nodes on long routes will allocate such large buffers just for your connection, the difference may not be in RWIN.

SamboNZ · Post by **SamboNZ** » Thu May 30, 2019 9:02 pm

Well, since it works fine in Linux, I expect that the routers are not to blame in this case.

Post by **Philip** » Sat Jun 01, 2019 4:12 am

Different OSes have slightly different behavior on how fast RWIN scales when the connection starts, and may use different congestion avoidance algorithms by default. I would check what congestion avoidance your Linux uses vs your Windows 10? (CUBIC, NewReno, CTCP, etc.)

SamboNZ · Post by **SamboNZ** » Sat Jun 01, 2019 4:26 am

Ok, I'll look at that.

I have however tried different congestion avoidance algos in Windows, but it didn't make any difference, at least not with 30 second iperf tests.

How long would you expect it to take for RWIN to ramp up?

Post by **Philip** » Sat Jun 01, 2019 4:38 am

It really depends on the latency/packet loss balance, it is not constant. Newer algorithms (CUBIC, CTCP, NewReno) scale up faster at the beginning of the connection, and newer OSes (updated Windows 10 versions, for example) start with 10 packets instead of 4 before waiting for an ACK, so there are many variables. 30 seconds should be enough to scale up the speed, provided there are no other issues like packet loss getting in the way.

It could also be some other simple issue, like different Network Adapter driver's allocation of buffers, regardless of TCP/IP settings.

SamboNZ · Post by **SamboNZ** » Sat Jun 01, 2019 5:14 am

I've played with NIC buffers too; up to 2048(KB?) - no change.

Post by **Philip** » Sun Jun 02, 2019 3:22 am

I would compare the Linux TCP/IP settings with the Windows OS.

The NIC send/receive buffers may be the same, but the Windows driver may not be as well optimized for higher speeds, it may have additional overhead imposed by the OS interaction, or additional options/coalescing, etc. that come into play

SamboNZ · Post by **SamboNZ** » Sun Jun 02, 2019 3:40 am

I'd be interested in hearing from anyone who has managed to get Windows working at high speed (>500mbit) with a high latency destination (>100ms), just to compare notes.

I've tested this on multiple Windows systems with different NICs and none of them are immune from this issue.

SamboNZ · Post by **SamboNZ** » Tue Jun 11, 2019 12:35 am

I just wanted to add a note here that I have just discovered that currently, iPerf3 v3.1.3 for Windows comes bundled with a version of Cygwin1.dll (2.5.1) which has a hard-coded TCP window size setting of 278,775.

This causes results to be limited to 12.6Mb/s per thread.

This is why iPerf speeds in Linux are much better that in Windows.

Updating the Cygwin dll to the latest version resolves this issue.

I have informed the iPerf devs of this issue.

Thanks to the Argon Systems article which directed me to this line of inquiry!
https://argonsys.com/microsoft-cloud/li ... -buffering

Post by **Philip** » Tue Jun 11, 2019 5:51 am

Thanks for the update, and thanks for sharing that, glad you figured it out.

SamboNZ · Post by **SamboNZ** » Tue Jun 11, 2019 5:56 am

Philip wrote:Thanks for the update, and thanks for sharing that, glad you figured it out.

No problem! I've seen many others on the net with the same issue so I figured I'd better share

I'm glad I figured it out too! It was driving me nuts!

Thanks for all the suggestions though, I appreciate it.