Windows Server with 10Gbit NIC - Severe Performance Issues over Certain Routes

[biber]

Hello everyone,

we recently upgraded our Windows server (hosted by Hetzner) to a 10Gbit/s connection. The server does reach the full 10Gbit/s capacity, and our customers are not reporting any issues. However, we're experiencing a different problem from our side.

From our own network (Deutsche Glasfaser), we can only sporadically reach the full 1000Mbit/s bandwidth when accessing this Windows server. Most of the time, the transfer speed drops to around 10Mbit/s.

Some key details:

• Our client is running Windows.

• We have already enabled TCP autotuning.

• Downloads to other servers always work fine.

• Speed tests from our client to the internet consistently show 950Mbit/s.

Interestingly, when we tunnel the traffic through an SSH connection via a Linux server (which then forwards the traffic to the Windows server), everything works perfectly. This suggests the issue only occurs with direct connections to the Windows server.

A Wireshark trace shows that, when the connection is slow, a large number of TCP packets are lost and need to be retransmitted. It looks like either the client or the server is struggling to handle the connection properly. We only started seeing this behavior after switching to the 10Gbit NIC.

Does anyone have any ideas what could be causing this? We're especially puzzled why the SSH tunnel (via Linux) works fine, while direct connections don't.

Here’s a brief excerpt from Wireshark:

Code: Select all

1	0.000000	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=1 Ack=1 Win=8191 Len=1220
2	0.000000	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Previous segment not captured] 80 → 51625 [ACK] Seq=4881 Ack=1 Win=8191 Len=1220
3	0.000000	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=4294963637 Ack=1 Win=8191 Len=1220
4	0.000000	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=1221 Ack=1 Win=8191 Len=1220
5	0.000000	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=2441 Ack=1 Win=8191 Len=1220
6	0.000042	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	86	51625 → 80 [ACK] Seq=1 Ack=4294963637 Win=1024 Len=0 SLE=1 SRE=1221
7	0.000054	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	[TCP Dup ACK 6#1] 51625 → 80 [ACK] Seq=1 Ack=4294963637 Win=1024 Len=0 SLE=4881 SRE=6101 SLE=1 SRE=1221
8	0.000080	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	51625 → 80 [ACK] Seq=1 Ack=4294964857 Win=1024 Len=0 SLE=1 SRE=2441 SLE=4881 SRE=6101
9	0.000084	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	[TCP Dup ACK 8#1] 51625 → 80 [ACK] Seq=1 Ack=4294964857 Win=1024 Len=0 SLE=1 SRE=3661 SLE=4881 SRE=6101
10	0.000104	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=6101 Ack=1 Win=8191 Len=1220
11	0.000104	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=4294966077 Ack=1 Win=8191 Len=1220
12	0.000104	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=4294964857 Ack=1 Win=8191 Len=1220
13	0.000104	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=3661 Ack=1 Win=8191 Len=1220
14	0.000104	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=7321 Ack=1 Win=8191 Len=1220 [TCP PDU reassembled in 18]
15	0.000116	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	[TCP Dup ACK 8#2] 51625 → 80 [ACK] Seq=1 Ack=4294964857 Win=1024 Len=0 SLE=4881 SRE=7321 SLE=1 SRE=3661
16	0.000121	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	[TCP Dup ACK 8#3] 51625 → 80 [ACK] Seq=1 Ack=4294964857 Win=1024 Len=0 SLE=4294966077 SRE=3661 SLE=4881 SRE=7321
17	0.000149	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	74	51625 → 80 [ACK] Seq=1 Ack=8541 Win=1024 Len=0
18	0.010750	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=8541 Ack=1 Win=8191 Len=1220
19	0.010750	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=9761 Ack=1 Win=8191 Len=1220
20	0.010750	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Spurious Retransmission] 80 → 51625 [ACK] Seq=4294964857 Ack=1 Win=8191 Len=1220
21	0.010750	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=10981 Ack=1 Win=8191 Len=1220
22	0.010823	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	86	51625 → 80 [ACK] Seq=1 Ack=10981 Win=1024 Len=0 SLE=4294964857 SRE=4294966077
23	0.021622	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=12201 Ack=1 Win=8191 Len=1220
24	0.021622	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=13421 Ack=1 Win=8191 Len=1220
25	0.021622	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=14641 Ack=1 Win=8191 Len=1220
26	0.021622	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Previous segment not captured] 80 → 51625 [ACK] Seq=20741 Ack=1 Win=8191 Len=1220
27	0.021622	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=21961 Ack=1 Win=8191 Len=1220
28	0.021622	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=17081 Ack=1 Win=8191 Len=1220
29	0.021622	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=18301 Ack=1 Win=8191 Len=1220
30	0.021622	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=15861 Ack=1 Win=8191 Len=1220
31	0.021622	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=19521 Ack=1 Win=8191 Len=1220
32	0.021679	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	86	51625 → 80 [ACK] Seq=1 Ack=15861 Win=1024 Len=0 SLE=20741 SRE=21961
33	0.021689	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	86	[TCP Dup ACK 32#1] 51625 → 80 [ACK] Seq=1 Ack=15861 Win=1024 Len=0 SLE=20741 SRE=23181
34	0.021694	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	[TCP Dup ACK 32#2] 51625 → 80 [ACK] Seq=1 Ack=15861 Win=1024 Len=0 SLE=17081 SRE=18301 SLE=20741 SRE=23181
35	0.021698	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	[TCP Dup ACK 32#3] 51625 → 80 [ACK] Seq=1 Ack=15861 Win=1024 Len=0 SLE=17081 SRE=19521 SLE=20741 SRE=23181
36	0.021715	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	74	51625 → 80 [ACK] Seq=1 Ack=23181 Win=1024 Len=0
37	0.032474	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Previous segment not captured] 80 → 51625 [ACK] Seq=24401 Ack=1 Win=8191 Len=1220
38	0.032474	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=25621 Ack=1 Win=8191 Len=1220 [TCP PDU reassembled in 39]
39	0.032474	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=26841 Ack=1 Win=8191 Len=1220
40	0.032474	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Previous segment not captured] 80 → 51625 [ACK] Seq=30501 Ack=1 Win=8191 Len=1220
41	0.032474	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=28061 Ack=1 Win=8191 Len=1220
42	0.032474	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=31721 Ack=1 Win=8191 Len=1220 [TCP PDU reassembled in 43]
43	0.032474	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=32941 Ack=1 Win=8191 Len=1220
44	0.032474	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=23181 Ack=1 Win=8191 Len=1220
45	0.032474	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Spurious Retransmission] 80 → 51625 [ACK] Seq=15861 Ack=1 Win=8191 Len=1220
46	0.032474	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	[TCP Out-Of-Order] 80 → 51625 [ACK] Seq=29281 Ack=1 Win=8191 Len=1220
47	0.032513	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	86	[TCP Dup ACK 36#1] 51625 → 80 [ACK] Seq=1 Ack=23181 Win=1024 Len=0 SLE=24401 SRE=25621
48	0.032522	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	86	[TCP Dup ACK 36#2] 51625 → 80 [ACK] Seq=1 Ack=23181 Win=1024 Len=0 SLE=24401 SRE=26841
49	0.032527	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	86	[TCP Dup ACK 36#3] 51625 → 80 [ACK] Seq=1 Ack=23181 Win=1024 Len=0 SLE=24401 SRE=28061
50	0.032532	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	[TCP Dup ACK 36#4] 51625 → 80 [ACK] Seq=1 Ack=23181 Win=1024 Len=0 SLE=30501 SRE=31721 SLE=24401 SRE=28061
51	0.032537	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	[TCP Dup ACK 36#5] 51625 → 80 [ACK] Seq=1 Ack=23181 Win=1024 Len=0 SLE=24401 SRE=29281 SLE=30501 SRE=31721
52	0.032542	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	[TCP Dup ACK 36#6] 51625 → 80 [ACK] Seq=1 Ack=23181 Win=1024 Len=0 SLE=30501 SRE=32941 SLE=24401 SRE=29281
53	0.032546	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	[TCP Dup ACK 36#7] 51625 → 80 [ACK] Seq=1 Ack=23181 Win=1024 Len=0 SLE=30501 SRE=34161 SLE=24401 SRE=29281
54	0.032569	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	94	51625 → 80 [ACK] Seq=1 Ack=29281 Win=1024 Len=0 SLE=15861 SRE=17081 SLE=30501 SRE=34161
55	0.032578	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	XXXX:XXX:2b03:11a1::2	TCP	74	51625 → 80 [ACK] Seq=1 Ack=34161 Win=1024 Len=0
56	0.032590	XXXX:XXX:2b03:11a1::2	YYYY:YYYY:YYYY:2e00:b4d6:b7a:cbe4:a8c1	TCP	1294	80 → 51625 [ACK] Seq=34161 Ack=1 Win=8191 Len=1220

[Philip]

If the issue is only with your own client, that machine may need some TCP tuning.
I would try enabling "Jumbo Frames". I would go through all the Network Adapter settings, turn off any "Packet Coalescing" and "Offloads", "Green Ethernet", etc.
Look through this article:
https://www.speedguide.net/articles/net ... ation-3449

You may want to run the TCP Optimizer with the "optimal" settings as well.

I hope this helps.

[biber]

I have already tried using TCP Optimizer on the server, but unfortunately it didn’t help.
The issue occurs on all clients in our company, both Windows and Linux (over 30 devices).
What’s strange is that connections to all other servers are consistently fast — only the connection to this specific Windows server is affected.

Our router is a fritzbox 7590.

[Philip]

I would go through all the Network Adapter settings on the server. Disable all types of packet "coalescing". I would try without jumbo frames as well.