Upload speed problems with the new ISP (affects only Win10 1607 and newer)

[Moan]

Hello.
I'm facing a very strange problem with upload speed to certain servers. I've wrote full details about this problem at: https://superuser.com/questions/1326302 ... 7-and-newe

After hours of testing and installations of different Windows 10 versions I detected that the problem started with version 1607 (Anniversary Update).
Short problem description:
After some seconds, the upload speed decreases from 34Mb/s to ~5Mb/s. It affects only certain servers. For most services like Google Drive, OneDrive, YouTube it works fine.
It affects even VPN connections. This means: even when I can get full 34Mb/s on speed tests and uploads to some services - I still get low speed to certain servers.
I tested on both: wireless, wired (Ethernet) connections. I also tested 2 routers and my mobile phone as router.
It seems like the "autotuninglevel" doesn't work correctly, because when I set it to "disabled" I was getting low speeds from start (without the boost).
I installed Windows 8.1 on a different HDD and I didn't experience this problem when I was transferring a big file to my VPS.
Then I tested all Windows 10 versions (starting from the newest one), and detected that the last version where I get full upload speeds is: 1511. I tested on both network drivers: windows default, and the newest available for my Wi-Fi/Ethernet.

I started reading Windows 10 Anniversary Update changes, and I found: https://blogs.technet.microsoft.com/net ... rver-2016/
It seems like:
1. "Initial Congestion Window" was changed to "10".
Unfortunately I can't set it to lower value. When I type: "PowerShell.exe Set-NetTCPSetting -SettingName internet -InitialCongestionWindow 4" I get the following output:

Code: Select all

Set-NetTCPSetting : Property InitialCongestionWindow is read-only
At line:1 char:1
+ Set-NetTCPSetting -SettingName internet -InitialCongestionWindow 4

Is there any work-around to change this value?
2. "Tail Loss Probe (TLP)".
I don't know how to disable it for tests. At the website I posted it says:

Configuration: The TCP templates have the additional setting called “taillossprobe”. On client SKU switching to compat template turns TLP off. On both client and server SKUs, the Internet template has it enabled by default. The InternetCustom and DatacenterCustom templates can be used for more fine grained control for specific connections.

But I have no idea where exactly it's configured.
3. "Recent ACKnowledgement (RACK)".
Same problem as mentioned at ad. 2. I don't know where to disable this option
4. "TCP Fast Open"
Disabling it with "Netsh int tcp set global fastopen=disabled" doesn't help, so I re-enabled it.
5. "Windows Low Extra Delay BAckground Transport"
This option should be disabled by default, so not needed to disable it.

So for now I need to test changing TLP, RACK and ICW, but I don't know any methods for it.

I would very appreciate any help in resolving this issue. I can also make additional tests if needed.
Just to clarity: I have tried different TCPOptimizer settings (including CongestionProvider changes), but nothing fixed it.

Best regards.

[Philip]

You are correct in that not all settings are configurable on client versions of Windows 10.
If you choose "none" for the congestion provider, it will employ the traditional "slow start" algorithm, i.e. it wont start with a large ICW (Initial Congestion Window). That's a trait of newer congestion avoidance algorithms, i.e. CTCP, CUBIC, etc. ICW may not necessarily be your issue though, the congestion window should only play a role if there is actual congestion/packet loss on the line/node.

Newer NDIS and NIC drivers implement and tweak many newer options, so it could be related to a number of settings in the NIC driver as well, not necessarily just TCP/IP tweaks to the Internet template. I would try disabling any "Offloads" and other advanced settings in the NIC driver to see if it makes a difference, such as Large Send Offloads, number of RSS queues, etc. You can also try disabling interrupt moderation to test. I would stick to one wired adapter for testing purposes, Wi-Fi can introduce even more variables.

[Moan]

Thank you for your reply!
I changed the supplemental template with command:

Code: Select all

netsh interface tcp set supplemental template=compat

It didn't help. I suppose that the newest changes, including "Initial Congestion Window" aren't the problem in this case. (Just if somebody is testing it, changing "template=internet" restores changes).

I tried almost all possible NIC driver configurations, including disabling "Interrupt moderation" (for wired). Unfortunately nothing seems to solve the core issue. I repeated tests now.

The only work-around for this problem which I found is using a Linux Virtual machine which uses the "NAT" Network Adapter (shares host IP) + Kitty on Windows.
There are my notes, hope it will be understandable enough:

Code: Select all

Virtual Machine Local IP: 192.168.32.132
apt-get install sshpass autossh screen
nano /etc/ssh/sshd_config:
Port 777
service ssh restart

Kitty settings:
Name - > LinuxVM-Tunnel-SpeedFix (port 777 if 22 doesn't work)
Connection -> keepalives -> 30
Connection -> Data -> Autologin username/password
SSH -> Tunnels:
- Source port: 7771 Destination: localhost:8881 | Server1
- Source port: 7772 Destination: localhost:8882 | Server2
- Source port: 7773 Destination: localhost:8883 | Unused
- Source port: 7774 Destination: localhost:8884 | Unused

Connection -> Data login/pass
Connection -> Data -> Command:
screen -X -S VMTunnel1 quit; screen -X -S VMTunnel2 quit; screen -X -S VMTunnel3 quit; screen -X -S VMTunnel4 quit; screen -S VMTunnel1 -dm sshpass -p 'MyPassword' autossh -oStrictHostKeyChecking=no -L 8881:127.0.0.1:22 root@server1.example.com; screen -S VMTunnel2 -dm sshpass -p 'MyPassword' autossh -oStrictHostKeyChecking=no -L 8882:127.0.0.1:22 root@server2.example.com;

Filezilla:
Profile: LinuxVM-server1.example.com | 127.0.0.1 | 7771
Profile: LinuxVM-server2.example.com | 127.0.0.1 | 7772

So basically, when a Virtual Machine running on my laptop tunnels traffic between my laptop and selected servers I get full upload speeds (34Mb/s). It stops working when I change the Virtual Machine Network Adapter to "Bridge", so it must be on "NAT".

I suppose this is an OS level issue on Windows 10 1607+ and I'm literally out of ideas how to fix it :/ It affects other Win 10 laptops which I tested at home.

Once again thanks for your post. I would be glad to read more testing/changes ideas.

[Philip]

Just a thought, have you tried disabling any power-saving features, like the "Power Management > reduce link speed during system idle, Energy Efficient Ethernet, etc." in the NIC properties?

[Moan]

Yes, of course. I disabled all "Power Saving" options, and also played with different NIC settings.
I'm using a mobile Internet connection (LTE from T-mobile), so of course it's not perfectly stable, however, I always get stable speeds (160Mb/s download, ~34Mb/s upload). And with this Virtual Machine tricks I also get full speeds.
I tried using a VPN (which works with the full upload speed on speedtests and file transfers to other services), but still getting low speeds to certain servers. This thing actually surprises me, because in my understanding, when I can full speed to VPN, then the rest should depend on VPN, not my laptop.
I'm definitely sure that those problems started with Windows 10 1607, because I downloaded all Windows 10 versions .ISOs and installed them one by one on another hard drive. It seems like "autotuning" stops working after transferring 10-50MiB of data, and then goes with very limited speed. Also, when I pause and resume the transfer in Filezilla (clicking "Process Queue" two times) it again gets some upload boost (34Mb/s) and quickly drops to ~5Mb/s...
I can tell, this is the weirdest problem with computer I have ever experienced. Even though I found a work-around for it, I would still like to know what causes it.

[Moan]

By the way @Philip. Do you maybe know any windows software (perhaps virtual network driver) which would forward all sent/received data? Maybe something for traffic control or other filters exists?
I thought if I installed this software, maybe the windows bug wouldn't affect my transfers, so using a Virtual Machine wouldn't be required.

[Philip]

VPN providers often install a NDIS miniport driver that appears as a separate NIC driver in the Adapter list. You can look at those settings, however, consider the following:

VPN can use either TCP or UDP. It can be convigured via OpenVPN, PPTP, etc. If all of those variants experience the same issues, it is likely unrelated to TCP/IP settings and the related buffers/congestion avoidance algorithms at all.


If VPN appears to give you the full speed to most servers with some exceptions, it is possible that the capacity of the path to certain servers at certain times (either your ISP or your VPN provider) is the bottleneck at times.

[Moan]

I actually tried the UDP OpenVPN connection to the VPS (A) where I always get full upload speeds. And for my surprise, when I was connecting to the other VPS (B) - the speed was decreasing just like without using the VPN... At the same time when I used the "Virtual Machine" trick, or older Windows version (installed on the second HDD, not VM) - I was getting full upload speeds to VPS (B).
So this is something very, very strange, and it also affects "Virtual Machines" when Network Adapter is set to "Bridged" instead of "NAT". Microsoft must have changed (or damaged) something in Win 10 1607, but I have no idea what it was, or if it can be fixed by changing any settings

I thought that some software which simulates NAT (something like used by Openbox) exists. Then it could maybe "forward" all traffic from my laptop and and fix issues.

[Moan]

I managed to fix this problem on server-side.
Seems like upgrading Ubuntu Server 16.04 to 18.04 fixed this problem for all of my VPS.
So basically, Windows 10 1607+ has problems with Ubuntu 16.04 / Debian 9 (and maybe older). After Server OS upgrade the problem is gone. It explains why even VPNs didn't help to resolve this issue. Too bad I noticed it so late, would have saved a lot of time

[Philip]

Interesting, thank you for the follow-up and glad that you figured it out. I wonder if it had something to do with the TCP/IP parameters on the Ubuntu side, but that's another can of worms

[Moan]

I actually compared Ubuntu 16.04 and 18.04 settings: https://93389a29-ce5f-45f3-99c5-eeee003 ... pasta.com/
Then I set everything on 16.04 as it is on 18.04, but it didn't solve the problem. I guess the problem is somewhere deeper (maybe even in kernel).

What I can say, I experienced this problem with 4 different providers and 1 home-server computer with Ubuntu 16.04 / Debian 9. They were placed in: France, Italy, Sweden, Germany and Poland. So for sure it had nothing to do with routing. Just two OS (Win10 1607+ and older Linux) couldn't "talk to each other" flawlessly