mTCP updates coming soon: Send me your bug reports

[mbbrutman]

I have not released an updated mTCP since July 2015. I've got a few bug fixes that I want to distribute, so I'm overdue for for another release. If you have any bug reports or feature requests, now would be the time to let me know.

So far I have the following changes:

• All: Warn about white space on the end of configuration file lines
• HTGet properly handles chunked encoding
• HTGet error checking on file writes is vastly improved.
• HTGet gets a "quiet mode" option
• HTGet HTTP 0.9 support and misc bug fixes.
• Telnet outgoing packet handling improvements. (Bug tripped by a telnet BBS sending small packets.)
• Telnet: Provide a setting to eliminate CGA snow and add a CGA_MONO color palette.
• IRCjr: Provide a setting to eliminate CGA snow.
• FTP Server: fix bug when using a sandbox on a whole drive, not just a directory.
• DHCP: More robust error checking when reading and writing the config files.

HTGet had the most work done; the lack of chunked encoding support was a major screw-up on my part.

I don't have any new programs or big changes planned. SSH and other programs that require SSL are kind of out of scope; they require too many libraries to be back ported and the libraries are too big to re-implement on my own, even if the 4.77Mhz machines that I target could handle the code. I started a Gopher client years ago but there has not been a huge demand for Gopher. ;-0

Here are some of things I am thinking about long term:

• IPv6 is partially in place but that's going to take a while to fully implement.
• FTP client: A warning when it thinks you are transferring a binary file using ASCII mode.
• Possible more optional features in the FTP client and server.
• A program that can image a hard disk over the network. (Think Ghost, but it runs on an XT class machine.)
• A telnet BBS. I started this years ago and I really should pick it up again.
• A demo program that plays "8088 Corruption" style video over the network.
• A stripped down HTTP server that is simpler and smaller.
• Multiple IRCjr profiles so you can use a different configuration for each IRC network.
• HTGet cookie support.

Cheers,
Mike

 

[maxtherabbit]

[*]A program that can image a hard disk over the network. (Think Ghost, but it runs on an XT class machine.)

that sounds amazing, maybe also imaging for floppies as well?

 

[Caluser2000]

Awesome Mike. Thank you very much.

 

[mbbrutman]

that sounds amazing, maybe also imaging for floppies as well?

No, sorry, just BIOS supported hard drives.

The intent is to allow one to image a hard drive in a machine without having to move the hard drive. Moving an MFM controller and a 20MB hard drive pair to a more modern machine to try to image it is a pain. Doing it over the network is much easier, especially if you have something like a Xircom PE3 adapter that allows you to add Ethernet temporarily without opening the case.

There are plenty of solutions for imaging floppy diskettes already, and floppy diskettes can have copy protection on them or legitimate errors so that is not a problem that I want to try to solve.


Mike

 

[Stone]

There are plenty of solutions for imaging floppy diskettes already...

Yes, I agree, but there is one specific exception to this capability. In the case of a standalone PC or XT with either one or two 360k floppy drives and no hard drive it would not be possible to image a floppy disk unless there is some other storage device available.

 

[Svenska]

Hi,

All: Warn about white space on the end of configuration file lines

Some editors (such as EDIT) make it very easy to leave whitespace around. Whitespace shouldn't matter, so a warning might be more annoying than helpful.

SSH and other programs that require SSL are kind of out of scope; they require too many libraries to be back ported and the libraries are too big to re-implement on my own, even if the 4.77Mhz machines that I target could handle the code.

I tried SSH2DOS on a 286/12 a while ago and it would time out computing the key. So... probably not worth it yet.

FTP client: A warning when it thinks you are transferring a binary file using ASCII mode.

Hmm. Is there a good reason to use ASCII mode nowadays? I only use it by accident, so simply defaulting to binary (and warn for any ASCII transfers) might be a smarter approach.

A program that can image a hard disk over the network.

Do you plan on adding write support? The ability to simply write raw hard disk or floppy images in the target machine would be very useful; even more when considering 360 KB drives or drives with alignment issues (which can't be written on a different machine).

 

[maxtherabbit]

Do you plan on adding write support? The ability to simply write raw hard disk or floppy images in the target machine would be very useful; even more when considering 360 KB drives or drives with alignment issues (which can't be written on a different machine).

that's actually what I had in mind, although transferring the floppy image to the target machine and running rawrite is trivial I suppose

 

[Svenska]

although transferring the floppy image to the target machine and running rawrite is trivial I suppose

Only if you have a hard drive. I actually had that issue (single-drive machine, drive slightly out of alignment) and resorted to writing an XMODEM disk image writer to get the machine to reliably boot...

 

[mbbrutman]

Batching several replies:


[Stone] Archiving floppies over the network:

I posit that most people have a "tweener" box to read and write diskettes with, and that tweener box has multiple floppy drives, a hard drive, and probably networking on it too. And there are other solutions, such as temporarily attaching storage via the parallel port (Backpack floppy drives, IDE hard drives, Zip drives, Compact Flash readers).

The case of a person with a lone machine without a tweener or add-ons they can use is pretty unusual.

If I write the program it won't care - it will just want a BIOS supported device. So floppies will work and I wouldn't do anything to disable that. But it's not something I expect people to take advantage of.


[Svenska]

Agreed, the whitespace warning is just a warning. None of my code cares about whitespace but somebody pointed out that it can be a problem on some of the fields so I added the warning.

FTP ASCII mode is still part of the spec and I want my FTP client to be as widely compatible as possible. Some modern FTP clients (lftp) just always transfer in BINARY mode, but I'm not willing to go that far. (They have to stuff a 'BIN' command in the command stream to do that.) For that matter no modern client supports the traditional FTP mode of opening a data connection where incoming connections could just be expected to go to the default port. But vintage FTP servers don't know anything else, so I have this support in the client even though nobody in their right mind will use it, unless they have a vintage FTP server.

Network archiving: Write support makes me a little worried; the potential for disaster is pretty high. I'd hate for somebody to wipe out their hard drive by accident, especially since I might have to require people to specifically enter their geometry.

 

[soviet9922]

it could be cool to add a ssh client to it

 

[luckybob]

+1 for the disk image bit. Even the write too.

Everything else is icing on an already delicious cake.

 

[Trixter]

If you have any bug reports or feature requests, now would be the time to let me know.

In the FTP program, all text is printed with what seems like BIOS or DOS string routines, except the actual # of bytes transferred section, which is written to the screen manually. This results in snow, but more importantly, it breaks any screen enhancement programs that speed up BIOS routines, with the effect being that the write-directly-to-screen routine writes the upwards counting numbers in a different location. This behavior was verified with both NNANSI in "fast" mode, and UltraVision. My suggestion is to not write the numbers using anything special, since they count up slowly and don't need crazy-fast writing routines.

FTP client: A warning when it thinks you are transferring a binary file using ASCII mode.

People will miss the warning. Alternate suggestions: Default to binary mode; make stop-on-ascii-binary default with an option to turn it into a warning only; make transfer-ascii-as-ascii a default with an option to disable it.

A program that can image a hard disk over the network. (Think Ghost, but it runs on an XT class machine.)

Love this. I have my own process for this but a full hard disk dump would be great. I would highly recommend you read the philosophy of ddrescue and emulate its behavior. The gist is this:

1. Read every sector
2. If you get a read error, immediately skip it, but note it for later
3. Once you hit the end, go back and try to read the bad sectors with increasing number of retries
4. Send everything out. For good reads, send the sector data. For bad reads, send NULs.

The philosphy is to get as much readable data off the drive as possible before it fails completely. Your sending/receiving protocol would have essentially four signals:

• Start reading volume of size XXXX
• Data sector NNNN (and then the sector's data, which would be written out at the correct location in the image file)
• Bad sector NNNN (and then nothing, and NULs would be written out)
• Stop reading, close image file

By following the above, your image program would also be a recovery program.

The receive program would probably need to be a java program or something else that runs on any modern hardware/OS.

A demo program that plays "8088 Corruption" style video over the network.

The 8088 Corruption/Domination methods are extremely heavy on the processing side, and even today are likely not suitable for realtime compression. But if you'd like a quick/dumb way of implementing this, you can grab aalib and port that, or if you'd like pseudo-color, libcaca. Accept any directshow source (I suggest testing with an .mp4 file since that will be the most likely use case in practice), run it through the lib to create 80x25, and send it over. There will be snow on real CGA cards but an average receive rate of 35KB/s means a framerate of 8fps. (Audio is a different problem which we can talk about offline

 

[Svenska]

I posit that most people have a "tweener" box to read and write diskettes with, and that tweener box has multiple floppy drives, a hard drive, and probably networking on it too.

I consider that true for anyone big into retro-tech, but not for most people getting access to an old system or people with limited space. I've got a USB floppy drive (720 KB capable) and some PCMCIA adapters, but not much more - which is exactly why I was having trouble with bring-up.

But it's not something I expect people to take advantage of.

You might be surprised of the uses other people find of one's solutions.

FTP ASCII mode is still part of the spec and I want my FTP client to be as widely compatible as possible.

I did not suggest removing the support. I would highly recommend defaulting to binary (with a switch for ASCII mode), though. It is always smarter to prevent accidental misuse instead of trying to detect it.

Network archiving: Write support makes me a little worried; the potential for disaster is pretty high. I'd hate for somebody to wipe out their hard drive by accident, especially since I might have to require people to specifically enter their geometry.

I don't consider that a problem. Someone writing a raw image to the disk should be well aware of the fact that doing so will definitely overwrite said disk (and if not, a warning similar to what FORMAT outputs is fine). But a backup program without restore is (in my opinion) pretty useless.

It would be great if no special program on the other machine would be required. Maybe a TFTP or FTP server to directly backup disks and partitions to and restore them from. But then, this is more of a wishlist-suggestion.

 

[Stone]

I consider that true for anyone big into retro-tech, but not for most people getting access to an old system or people with limited space. I've got a USB floppy drive (720 KB capable) and some PCMCIA adapters, but not much more - which is exactly why I was having trouble with bring-up.

You lost me there. What is 'bring-up'?

 

[mbbrutman]

More accumulated replies:

[Trixter]

FTP Client screen writing: Using BIOS routines to update the FTP client bytes transferred display was slow enough to impact performance, which is why I went to BIOS routines. I can undo them and just update the screen less often, but compatibility does have a cost. It also might not be worth it; the command recall feature is using direct screen writes too.

Demo code: My demo would just have the packet driver blast the video buffer with the raw packet data. At 80 columns that requires 4KB, or three packets. I know the TCP/IP code can sustain about 110K per second and that includes all of the checksumming overhead and a few memcpys, so UDP without checksums should be capable of at least 30 fps. The target is a PCjr, so snow is not an issue but my calculations are probably off because of the video memory sharing.


[Trixter and Svenska]

FTP ASCII vs. BIN: Nobody seems to realize this but the server controls the default transfer mode, not the client. To give the illusion of a client setting the default mode the client has to stuff a "BIN" command ahead of any transfer or at the beginning of the session. I think that's a bit intrusive but it can be done. But it's no where near as simple as just "making BIN the default."


[Svenska]

A backup-only program makes perfect sense to me. If an old hard drive really does die the chances of my having an exact duplicate unit and being able to shove the image back onto it are pretty slim. Chances are that there is a different device and the geometry will be different. I'd rather people do a backup and have their data available in an emulated system such as VirtualBox, and worry about the restore part when it happens because chances are the restore is going to be complicated.

 

[maxtherabbit]

Would it be possible to have the FTP Server write the files to the local disk while preserving their last modified date? I don't know if FTP even provides for a file timestamp so this may be stupid

 

[mbbrutman]

That's a good one, and I forgot about it previously so I'm going to add it to the list.

(There are FTP extensions that enable that behavior. They are optional but they are nice to have ...)

 

[Trixter]

FTP Client screen writing: Using BIOS routines to update the FTP client bytes transferred display was slow enough to impact performance, which is why I went to BIOS routines. I can undo them and just update the screen less often, but compatibility does have a cost. It also might not be worth it; the command recall feature is using direct screen writes too.

Understood. I guess I'd like to amend my request to "either use direct screen writes for everything, or use BIOS for everything, but don't mix'n'match because that's what causes issues".

Demo code: My demo would just have the packet driver blast the video buffer with the raw packet data. At 80 columns that requires 4KB, or three packets. I know the TCP/IP code can sustain about 110K per second and that includes all of the checksumming overhead and a few memcpys, so UDP without checksums should be capable of at least 30 fps. The target is a PCjr, so snow is not an issue but my calculations are probably off because of the video memory sharing.

The memory sharing is an upper limit, but I'm more concerned how you're estimating 110KB/s -- my xircom on my bidirectionally-modified (by you! ;-) PCjr sidecar can only pull down 35KB/s via TCP; my Intel Etherexpress card in my 5160 can pull down about 45KB/s via TCP...

Otherwise, UDP without checksums is a good idea; I'd suggest transmitting the target location of the data as a 16-bit offset, then the data in the same packet, so that if UDP gets dropped on the floor the subsequent packets will just paint around the missing area for a graceful degrade of the experience.

 

[maxtherabbit]

The memory sharing is an upper limit, but I'm more concerned how you're estimating 110KB/s -- my xircom on my bidirectionally-modified (by you! ;-) PCjr sidecar can only pull down 35KB/s via TCP; my Intel Etherexpress card in my 5160 can pull down about 45KB/s via TCP....

to system memory or to a storage device?

 

[mbbrutman]

I'm not estimating 110KB/sec - I measured it: http://brutman.com/mTCP/mTCP_Performance.html

Look at the second table for raw TCP/IP socket performance. It measures the TCP stack without any file I/O, so I can see what the upper bound is. A PCjr with a good ISA card attached to it hits 110KB/sec. That's going to be limited by the CPU in the machine. When you get to the Xircom the numbers are much lower because the bottleneck there is the parallel port. Switching to UDP and not computing the UDP checksum (legal for UDP, not legal for TCP) will still make it much faster.

For the demo it probably wouldn't be UDP either; I'd cheat and use reserved frame types:

• Packet of length C: fill in the lower 1/3rd portion of the screen
• Packet of length B: fill in the middle ortion of the screen
• Packet of length A: fill in the top portion of the screen

You want to work top to bottom because the packet driver is going to copy in the Ethernet frame header which is at the beginning of the frame, so you'll see it temporarily on the screen. But you can arrange the offsets so the next packet in overwrites it. Between this and flipping between two pages you can avoid the user seeing the extra bytes.

All of this gets done under the packet driver interrupt. One memcpy per packet with no inspection of the packet needed.