Out of order packets?

piobaire · Post by **piobaire** » Wed May 23, 2007 7:02 pm

Hi there!

I am having a strange problem lately. I just went the rounds with my ISP regarding packet loss. I FINALLY proved to them that I was having serious packet loss, and it was degrading at an exponential rate. So, yesterday, I got some guys out here to work on it. Now today, things are MUCH better. However, I am still definitely experiencing this VERY regular spikey sort of latency. About every 30 seconds I get a a spike of around 300-500 ping. While I was further trouble shooting this problem, I used the Web100 tool and these were the results.

=-=-=-=-
WEB100 Enabled Statistics:
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
running 10s outbound test (client to server) . . . . . 1.03Mb/s
running 10s inbound test (server to client) . . . . . . 4.83Mb/s

------ Client System Details ------
OS data: Name = Windows XP, Architecture = x86, Version = 5.1
Java data: Vendor = Sun Microsystems Inc., Version = 1.5.0_11

------ Web100 Detailed Analysis ------
10 Mbps Ethernet link found.
Link set to Full Duplex mode
No network congestion discovered.
Good network cable(s) found
Normal duplex operation found.

Web100 reports the Round trip time = 103.1 msec; the Packet size = 1460 Bytes; and
No packet loss - but packets arrived out-of-order 19.33% of the time
This connection is receiver limited 78.73% of the time.
Increasing the the client's receive buffer (63.0 KB) will improve performance
This connection is network limited 21.23% of the time.

Web100 reports TCP negotiated the optional Performance Settings to:
RFC 2018 Selective Acknowledgment: ON
RFC 896 Nagle Algorithm: ON
RFC 3168 Explicit Congestion Notification: OFF
RFC 1323 Time Stamping: OFF
RFC 1323 Window Scaling: OFF
Packet size is preserved End-to-End
Server IP addresses are preserved End-to-End
Information: Network Address Translation (NAT) box is modifying the Client's IP address
Server says [67.x.x.x] but Client says [x]
=-=-=-=-

I am seeing this very consistent mis-ordered packets. I have quite a bit of experience with home and small business networks, but this is over my head. Any one able to offer a word of advice on this? Anything would be MUCH appreciated!

Oh, and for the record, I think this was going on before the cable guys did their thing yesterday, but the spikes were just unmeasureably large then. My "Ping Meter" on guild wars was reading into the 30-40*K* range. I understand that a normal TCP request times out long before then though, so I am not sure how they weer measuring that.

I am on cable (not sure if it's 6Mbit or higher - we just had a few changes to our area), live in the Puget sound area and I am on Windows XP Pro (Which is actually a pro upgrade over 2k).

Thanks in advance for any help! (You guys all rock!

)

trogers · Post by **trogers** » Wed May 23, 2007 9:57 pm

Your TCP Window (RWIN) may have been set too low as test indicates your connection is receiver limited.

Try the following custom settings with TCP Analyzer:

General Settings tab:
Custom settings - check
Modify All Network Adapters - check
network adapter selection - your NIC
MTU 1500
TTL - 64
TCP Receive Window - 128480
MTU Discovery - Yes
Black Hole Detect - No
Selective Acks - Yes
Max Duplicate ACKs - 2
TCP 1323 Options:
Windows Scaling - checked
Timestamps - uncheck

Advanced Settings tab:
Max Connections per Server - 10
Max Connections per 1.0 Server -20
LocalPriority - 1
Host Priority - 1
DNSPriority - 1
NetbtPriority - 1
Lan Browsing speedup - optimized
QoS: NonBestEffortLimit - 0
ToS: DisableUserTOSSetting - 0
ToS: DefaultTOSValue - 80
MaxNegativeCacheTtl - 0
NetFailureCacheTime - 0
NegativeSOACache Time - 0
LAN Request Buffer Size - 32768
Then select "Apply Changes" and reboot to take effect

piobaire · Post by **piobaire** » Wed May 23, 2007 9:59 pm

Do you think the RWIN would affect packet re-ordering though? I'll give that a try and see how it goes.

trogers · Post by **trogers** » Wed May 23, 2007 10:03 pm

piobaire wrote:Do you think the RWIN would affect packet re-ordering though? I'll give that a try and see how it goes.

RWIN is the setting that sizes the TCP buffer to receive MSS data packets. When set too low, you will face a packet queue resulting in out-of order packets.

piobaire · Post by **piobaire** » Wed May 23, 2007 10:22 pm

Yeah, I understand what it does. Just didn't think it would affect out of order packets in such a manner.

So, after a restart and retest, I am now experiencing a 50% rise in re-ordered packets. :?

Any ideas?

=-=-=-=-
WEB100 Enabled Statistics:
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
running 10s outbound test (client to server) . . . . . 1.04Mb/s
running 10s inbound test (server to client) . . . . . . 9.00Mb/s

------ Client System Details ------
OS data: Name = Windows XP, Architecture = x86, Version = 5.1
Java data: Vendor = Sun Microsystems Inc., Version = 1.5.0_11

------ Web100 Detailed Analysis ------
10 Mbps Ethernet link found.
Link set to Full Duplex mode
No network congestion discovered.
Good network cable(s) found
Normal duplex operation found.

Web100 reports the Round trip time = 98.7 msec; the Packet size = 1460 Bytes; and
No packet loss - but packets arrived out-of-order 29.25% of the time
This connection is receiver limited 71.17% of the time.
This connection is network limited 28.79% of the time.

Web100 reports TCP negotiated the optional Performance Settings to:
RFC 2018 Selective Acknowledgment: ON
RFC 896 Nagle Algorithm: ON
RFC 3168 Explicit Congestion Notification: OFF
RFC 1323 Time Stamping: OFF
RFC 1323 Window Scaling: ON
Packet size is preserved End-to-End
Server IP addresses are preserved End-to-End
Information: Network Address Translation (NAT) box is modifying the Client's IP address
Server says [67.x] but Client says [192.x]

=-=-=-=-

trogers · Post by **trogers** » Wed May 23, 2007 10:32 pm

Notice that your speed has almost doubled to 9 mbps, meaning your ISP is using powerboost on your 6 mbps line. You need to double your RWIN again to compensate.

Try the following custom settings with TCP Analyzer:

General Settings tab:
Custom settings - check
Modify All Network Adapters - check
network adapter selection - your NIC
MTU 1500
TTL - 64
TCP Receive Window - 256960
MTU Discovery - Yes
Black Hole Detect - No
Selective Acks - Yes
Max Duplicate ACKs - 2
TCP 1323 Options:
Windows Scaling - checked
Timestamps - uncheck

Advanced Settings tab:
Max Connections per Server - 10
Max Connections per 1.0 Server -20
LocalPriority - 1
Host Priority - 1
DNSPriority - 1
NetbtPriority - 1
Lan Browsing speedup - optimized
QoS: NonBestEffortLimit - 0
ToS: DisableUserTOSSetting - 0
ToS: DefaultTOSValue - 80
MaxNegativeCacheTtl - 0
NetFailureCacheTime - 0
NegativeSOACache Time - 0
LAN Request Buffer Size - 32768
Then select "Apply Changes" and reboot to take effect

piobaire · Post by **piobaire** » Wed May 23, 2007 11:03 pm

Hehe,

It was actually THAT setting before I did this before. Let's try again. Thanks for your time!

Let's hope this pins it down!

piobaire · Post by **piobaire** » Wed May 23, 2007 11:11 pm

yay!

=-=-=-
WEB100 Enabled Statistics:
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
running 10s outbound test (client to server) . . . . . 1.03Mb/s
running 10s inbound test (server to client) . . . . . . 9.42Mb/s

------ Client System Details ------
OS data: Name = Windows XP, Architecture = x86, Version = 5.1
Java data: Vendor = Sun Microsystems Inc., Version = 1.5.0_11

------ Web100 Detailed Analysis ------
10 Mbps Ethernet link found.
Link set to Full Duplex mode
No network congestion discovered.
Good network cable(s) found
Normal duplex operation found.

Web100 reports the Round trip time = 100.15 msec; the Packet size = 1460 Bytes; and
There were 12 packets retransmitted, 350 duplicate acks received, and 170 SACK blocks received
The connection stalled 1 times due to packet loss
The connection was idle 0.32 seconds (3.2%) of the time
This connection is receiver limited 34.62% of the time.
This connection is network limited 65.35% of the time.

Web100 reports TCP negotiated the optional Performance Settings to:
RFC 2018 Selective Acknowledgment: ON
RFC 896 Nagle Algorithm: ON
RFC 3168 Explicit Congestion Notification: OFF
RFC 1323 Time Stamping: OFF
RFC 1323 Window Scaling: ON
Packet size is preserved End-to-End
Server IP addresses are preserved End-to-End
Information: Network Address Translation (NAT) box is modifying the Client's IP address
Server says [67.171.55.213] but Client says [192.168.15.2]
=-=-=-

I think that packet loss was just a hiccup. I tested three times. The first one out and out failed, the second one went through fine, and this was the third. It's that time of night though, when we do experience high volume of traffic.

trogers, Thank you VERY much for your time and help! I hope this irons out the last of my connection wrinkles!

trogers · Post by **trogers** » Thu May 24, 2007 12:58 am

piobaire wrote:yay!

=-=-=-
WEB100 Enabled Statistics:
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
running 10s outbound test (client to server) . . . . . 1.03Mb/s
running 10s inbound test (server to client) . . . . . . 9.42Mb/s

------ Client System Details ------
OS data: Name = Windows XP, Architecture = x86, Version = 5.1
Java data: Vendor = Sun Microsystems Inc., Version = 1.5.0_11

------ Web100 Detailed Analysis ------
10 Mbps Ethernet link found.
Link set to Full Duplex mode
No network congestion discovered.
Good network cable(s) found
Normal duplex operation found.

Your Nic's setting may be limiting your bandtwidth throughput.

Go to its properties and try setting 'Speed & Duplex" to 100 mbps full duplex and then reboot comp to see if speed can go higher.

piobaire · Post by **piobaire** » Thu May 24, 2007 1:16 am

Haha,

Remember the good old days when you would look at the 100 mbps and think "yeah right". (Except intranet stuff of course

I will try that right now, but I am heading to bed I think. Long day ahead of me.

legumo · Post by **legumo** » Tue Jun 12, 2007 4:20 am

believe it or not i think your problem is actually your DNS being flaky.

http://www.opendns.com

enjoy.