mTCP updates coming soon: Send me your bug reports

[mbbrutman]

Reducing the MTU will change the timing and also minimize the amount of data lost when packet start to disappear. But changing the MTU on the mTCP side won't help in this case, as the sender is using whatever they consider to be an appropriate MTU for them and mTCP is basically just sending back ACK packets.

mTCP has fragment support but getting fragments to appear is kind of rare. Back when networking hardware was more varied you could bump into a gateway that would fragment your packets, but that's not the case with the proliferation of Ethernet everywhere. (And devices that look like Ethernet.)

[mbbrutman]

If anybody has been having problems with TELNET terminal emulation I have some fixes you can try:

• Download the test version from http://www.brutman.com/mTCP/telnet.exe
• Unix users: Remember to set LANG=en_US (or the equivalent) because there is no UTF8 (Unicode) support
• Set your terminal type to 'ansi'

Mutt (an email client) was slightly broken; that has been fixed now. As have one or two other bugs.

If you still have a problem please send me a bug report. I need to know what program is causing the pain and how to recreate it. (Linux is preferred, as I have a lot of it here. Telnet BBSes also work.)


Mike

[Eudimorphodon]

Small update: I haven't had a lot of time to play with the network card since I last updated, but I did use both htget and the ftp client on several occasions to fetch multi-hundreds-of-k-size files a couple times from both local servers and an ftp server at columbia.edu and they all worked. It still seems to be a problem primarily limited to ftp.oldskool.org. Haven't gotten a chance to dump it yet but it's looking more and more like it's a server specific thing.

[mbbrutman]

This is interesting - I just caught my local Linux machine having the FTP connection/slow-down problem with DOSBox running under Windows 7.

From the trace I can see the two machines are basically arguing about what's the next expected block to send. The Linux machine grudgingly gets around to resending the missing block, but then it insists on continuing from where it thinks it should be, not from the next block after the missing block. So this constant disagreement results in something like one packet being received every few seconds, which is abysmal.

Time to do some reading. I think I'm within spec, and it's working, just slowly. But I need to find a way to convince the sender to really restart from an earlier packet. (They are about 10KB apart.)

[Trixter]

Oh cool! Glad you were able to catch it in action. It happens randomly and infrequently on some systems over here, and 50% consistently on certain very specific combinations, so if you manage a fix I'd be grateful.

The tough part is reliably reproducing it.

[Eudimorphodon]

From the trace I can see the two machines are basically arguing about what's the next expected block to send. The Linux machine grudgingly gets around to resending the missing block, but then it insists on continuing from where it thinks it should be, not from the next block after the missing block. So this constant disagreement results in something like one packet being received every few seconds, which is abysmal.

I don't think I have the dump files anymore, but it sounds about like what I remember seeing when I did the packet dump when I was running EPPPD; the server kept sending blocks of retries but the client never seemed to actually accept more than a packet or two, and eventually they'd get so out of sync the server would send the packets including the end of the transfer and decide it was done, no longer bothering to send anything while the client just sat there waiting for the packets it actually wanted. The client would just sit there waiting long enough I'd eventually see the server get bored and terminate the control connection.

[mbbrutman]

I've done some reading, and I'm pretty sure that I didn't miss anything the first time that I did this.

When a packet is sent it is put in a queue where it sits until the other side (the client) acknowledges the packet. A timer is also started. This is how TCP detects that it needs to re-transmit, and knows what to re-transmit.

In the event of a missing packet the server has to choose to resend just the missing packet or to resent the missing packet and everything after it. There is no specification here, so it is server dependent. Resending just the missing packet is optimistic.

mTCP is simple; when it sees packets that "are ahead" it knows that one got dropped, so it acknowledges the last received packet. But it is also tossing those new packets because memory is limited. This is safe because we know they will get re-transmitted. However, combine that with a server that just sends the one missing packet and everything falls apart.

The quick work around is to set a smaller TCP window. Setting it to 1460 basically only allows for one packet in flight at a time, disabling the sliding window when dealing with large file transfers. But it works much better than basically having it lose sync and never resync.

Longer term I can do the following:

• Implement TCP selective acknowledgements. This is more complex, but I don't think it actually costs more memory because only things that are in the sliding window should be sent.
• Do something clever/adaptive when we see we are getting a lot of out of order packets and drop the window size to something minimal. This should let it resync because it will only be allowed to send one packet at a time. For bonus points the window can grow again when things look safe.

Interesting problem ...

[mbbrutman]

For your FTPing pleasure, a test version of new code: http://www.brutman.com/mTCP/ftp.exe

If you have been having problems with FTP transfers freezing and never fully recovering that should help. It implements an adaptive TCP receive window; when things turn to crap the window shrinks. After things improve the window grows again. I've been testing against Jim's oldskool.org server and it's working wonderfully.

[Eudimorphodon]

So... I've been playing with it for a few minutes, and on one hand it's working a lot better when it's working; right now the score seems to be about 2 out of three. I've downloaded several files up to over a megabyte in size with no problem, but it has still on a couple attempts gotten wedged. I left it quite a long time; in the first case it wedged after "Bytes Transferred 40k(ish)" and in the end had gotten to a hair over 60k when I lost patience and ctrl-c'ed it, in the other it was 2.2MB into a 2.8MB download. That one is still sitting on the screen and, unlike the other, has actually updated the "Bytes Transferred" once, but hasn't shown any sign of un-wedging.

(I have reasonably high confidence if I kill the program and try the same file again in another session it'll work. Let me try that... yep, made it fine.)

[mbbrutman]

I downloaded the xtfiles.rar file at least 5 times; I got bored it worked so well. The file is almost 48MB, which is good because that was long enough for me to observe it stumbling and recovering two or three times in a single fetch. I also tried 8088MPH.ZIP and some files on my local machine. You will still see it slow down, but it should recover in seconds, not minutes. But I also cheated and used DOSBox; on a real machine that is slower because of the disk I/O it might stumble a lot more.

Please take notes on what works and doesn't work. (And we should probably move this to email.) I'll test on real (and slow) hardware tomorrow.

If you are adventurous turn on debug logging and set the log level to 1. (set debugging=1, set logfile=ftp.log). That just gives you warning messages; you'll see a warning message from the new code when it throttles the receive window.

Lastly, what MTU size and TCP receive window size are you using?