PDA

View Full Version : EtherDFS - an ethernet drive for DOS



Pepinno
April 1st, 2017, 03:12 AM
I've searched the forum, and I found no mention of EtherDFS.

So I quote here the post (Message-ID: <589f1b9e$0$5412$426a74cc@news.free.fr>) by its author, Mateusz Viste, in the Usenet group comp.os.msdos.misc:

"""
Hi all,

Today I released a new version of EtherDFS: v0.7. Since one of the major
changes of this version is that it became compatible with MS-DOS, I
thought it would be relevant to announce it here (so far it was
compatible only with FreeDOS).

EtherDFS is an 'installable filesystem' TSR for DOS. It maps a drive from
a remote computer (typically Linux-based) to a local drive letter, using
raw ethernet frames to communicate. It runs on 8086+ and consumes 7K of
RAM memory. It can be loaded high.

http://etherdfs.sourceforge.net

Mateusz
"""


Also, he then released version 0.8 of EtherDFS, with this announcement (Message-ID: <58baf5db$0$5261$426a74cc@news.free.fr>) in the same newsgroup:

"""
Today I released a new version of EtherDFS (along with its server-side
Linux companion, ethersrv-linux). EtherDFS v0.8 brings a few enhancements
and compatibility improvements:

Changelog:
- improved self-detection to avoid loading EtherDFS twice,
- added unloading support (/u),
- fixed a FindFirst regression (fixes usage under 4DOS),
- fixed SETATTR action when using a non-FreeDOS attrib command,
- implemented the 'Seek From End' call,
- minor memory optimizations,
- makes sure the redirector API is available before installing,
- support for multiple drive mappings.

EtherDFS is available for download on the project's website:
http://etherdfs.sourceforge.net

Mateusz
"""

Basically, you run the EtherDFS Linux daemon and then in the same ethernet LAN you run the EtherDFS DOS client in your MS-DOS/FreeDOS machine and map a network drive against the EtherDFS server. This is done through raw ethernet, so no TCP/IP stack is involved. You only need an ethernet NIC with a DOS-compatible packet driver on the DOS machine. It runs on 8086 and up MS-DOS machines. And it is open source.

I haven't tried it yet, but I thought it could be of interest to the folks here.

Great Hierophant
April 1st, 2017, 03:45 PM
Looks promising, but as long as the server is Linux-only, I am afraid the audience will be extremely small. If he put up a DOS server program (working through DOSBox perhaps), I bet it will get more attention.

pietja
April 1st, 2017, 04:26 PM
It looks interesting and i will see if it works on Windows 10 as it should be able to run Linux programs since the Anniversary Update.

luckybob
April 1st, 2017, 04:34 PM
WAIT WHAT?

run linux programs in win10? surely you just.

pietja
April 1st, 2017, 04:37 PM
look here: https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/

Great Hierophant
April 1st, 2017, 06:05 PM
look here: https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/

From the article : "There are some limitations here. This won’t work with server software, and it won’t work with graphical software. It’s intended for developers who want to run Linux command-line utilities on Windows." This is server software, although I don't know whether this program is what they are talking about.

pearce_jj
April 1st, 2017, 10:14 PM
grep, tail, find, dd, awk etc would all be good to have natively certainly.

pearce_jj
April 1st, 2017, 10:21 PM
Interesting.

We talked a while back about adding ATAoE to XTIDE Universal BIOS (it was probably only really me that was really interested in it!). It's a standards based layer-2 protocol.

https://en.m.wikipedia.org/wiki/ATA_over_Ethernet

AlexC
April 1st, 2017, 10:21 PM
grep, tail, find, dd, awk etc would all be good to have natively certainly.

Aren't they already, albeit not from MS? I have versions of all those for DOS dating back to the mid-1990s. Pretty sure there'd be native Windows versions too (though I haven't looked).

The 'server' limitation presumably means running a daemon, doesn't it? That would make sense as you'd need tighter integration for that to work.

Pepinno
April 2nd, 2017, 03:13 AM
I doubt the EtherDFS daemon will run on the Windows 10 "Linux environment", because the EtherDFS daemon is coded to "attach itself" to the Linux eth0/whatever interface. However, you could run the EtherDFS daemon on a Linux VM.

I hope the author publishes a DOS-based EtherDFS daemon.

Xacalite
April 2nd, 2017, 04:42 AM
Looks promising, but as long as the server is Linux-only, I am afraid the audience will be extremely small. If he put up a DOS server program (working through DOSBox perhaps), I bet it will get more attention.
I think the point is the ability to transfer files between modern machines and vintage ones, that's why the server runs on a modern OS.
And this may be very useful for low-memory clients, eg. 5150 with 64KB RAM, where one can't run SMB/NCP/NFS clients.

Xacalite
April 2nd, 2017, 04:46 AM
Aren't they already, albeit not from MS? I have versions of all those for DOS dating back to the mid-1990s. Pretty sure there'd be native Windows versions too (though I haven't looked).
Of course, it's called Cygwin, and I think there's at least one alternative project as well.

Malvineous
April 2nd, 2017, 01:42 PM
Looks promising, but as long as the server is Linux-only, I am afraid the audience will be extremely small. If he put up a DOS server program (working through DOSBox perhaps), I bet it will get more attention.

The protocol uses raw Ethernet frames, so it operates below TCP/IP and below even IPX. DOSBox doesn't emulate a packet driver (or Ethernet hardware) so there is no way to produce raw Ethernet frames from within it.

Likewise it would only work under Windows if they provide a raw packet API to their Linux compatibility layer, which I imagine they wouldn't as very few programs would ever use it. EtherDFS only uses this method in order to keep the client tiny. I think the only way to get it working under Windows would be to port it to run natively. Same for DOS, but then you'd have to have a retro server running DOS natively, which isn't really the point of EtherDFS, which is designed to link modern PCs with retro ones. Personally I plan to run a Linux server for my retro PCs anyway, so it suits me perfectly as-is! If you wanted your server to be a retro PC as well, you'd probably be better off running Novell or something on it so you are running real networking software from the same era as well.

ATA over Ethernet would be interesting but I imagine it would cause the client app to become a lot larger, having to translate BIOS calls into ATA commands. ATAoE also makes a disk image available over the network, so it's not used for sharing files. In this case, each retro PC would have its own large file on the server which is presented on the client machine as a single disk, and in order to share files you'd have to shut down the PC, mount the disk image locally, make your changes, unmount, then boot up the retro PC again. With EtherDFS it's a remote filesystem rather than a remote disk image, so multiple PCs can access the same network share at the same time, creating files that the other machines can see immediately.

PhantomEigh
April 2nd, 2017, 01:48 PM
I run a hyper-V server (homelab) here at the house that's the primary storage and compute box. It runs a bunch of junk like Emby, a DC, a torrent server, Server Essentials 2012 R2, all kinds of stuff for the house.

One of the things I run is a Server 2003 box so that Windows 3.1 and 98 have a place to log in and store files over the network. Those OS's are just to old to log into the newer OS's via SMB. For DOS, the Microsoft network stuff uses way too much RAM especially if you want to build a boot disk and do some installs over the network. I'll fire up an Ubuntu VM for this and try it out, I'll mount the Server 2003 share on the Ubuntu box and then try to share it out with the ethersrv-linux package.

Trixter
April 2nd, 2017, 03:07 PM
This is essentially a lightweight netware that uses linux as the file server. If it works it's an incredibly lightweight way to make a linux directory just show up under DOS as a drive letter. This can breathe new life into systems that can use a parallel-port xircom adapter but can't have a hard disk, like early laptops. I keep meaning to test it out, and will report back when I do.

For those worried about "I don't run Linux", you can easily run Linux as a VM inside whatever you are running (ie. use virtualbox, qemu, etc.). And you can also cross-mount between host OS and VM OS. It's not as elegant as running Linux by itself but it does work.

keenerb
April 2nd, 2017, 04:17 PM
I just set this up, it's working 100% as described.

It allows mtcp utilities and WATTCP tools (at least the ones I tested) to share the packet driver, I was able to FTP files from an ftp server directly to my mapped network drive.

This is a big deal to me...

mateuszviste
April 4th, 2017, 06:45 AM
Hi guys, I'm glad you like the idea behind EtherDFS. About running on Linux and not Windows: I stopped using Windows somewhere around 1996, so it's unlikely that a Windows version of ethersrv (the EtherDFS server) will ever appear - unless someone else does it of course :)
That being said, and as mentioned in this thread already, you can easily run a Linux VM inside a Windows host. I heard about people running ethersrv inside a Linux VM under VirtualBox with excellent results. Also, it runs perfectly on a Raspberry PI. EtherDFS allows not only to share data between an old PC and a modern machine, but also (and that was my main goal) share data between multiple old PCs. No more swapping floppies with all those bad sectors around!

archeocomp
April 5th, 2017, 02:55 AM
This is essentially a lightweight netware that uses linux as the file server. If it works it's an incredibly lightweight way to make a linux directory just show up under DOS as a drive letter. This can breathe new life into systems that can use a parallel-port xircom adapter but can't have a hard disk, like early laptops. I keep meaning to test it out, and will report back when I do.
.
Wow very promising as I have both Xircom and a Bondwell B200 Laptop with two 720k floppy drives. Let us know please.

Tor
April 5th, 2017, 03:18 AM
Looks promising, but as long as the server is Linux-only, I am afraid the audience will be extremely small. ? Since when is the Linux audience "extremely small"? I would say it's rather large. In any case, it comes with source, if you don't have a Linux box or you don't want to try one of the already suggested VM methods, just get a Pi and build and run it. I don't see anything in the source which would prevent that. [Edit: I see Mateus already said it works.] Cheap small solution.

dieymir
April 5th, 2017, 08:54 AM
Wow! This is what I like about these forums. Having the developer posting here is a real treat. It works like a charm but I want to ask for a couple of featurettes:;)

* A DOS server to be able to have a peer to peer vintage computer network.
* Make it work with PC/MS-DOS 3.1 and higher. Actually, I have not tested it with those DOS versions but you said it requires DOS 5.0+

Stone
April 5th, 2017, 09:38 AM
I like U - N E T II for a Server/Workstation based network. It can use anything from a PC on up through 486 and PS/2. And it's parallel port so no NICs are required. DOS 3.3 or higher is recommended.

archeocomp
April 5th, 2017, 10:18 AM
Stone would you mind posting a link? I can not find anything with that name.

Trixter
April 5th, 2017, 11:48 AM
Let us know please.

Unfortunately my initial tests with 8088 systems (which work 100% with mTCP) have not been very promising. On a PCjr with a Xircom PE3, the driver appears to load, but it occasionally truncates a byte from the stream, which means I can't rely on it for anything. When I copy a file to the local drive, then copy it again, then run a file compare, they don't match up. Example:

https://goo.gl/photos/55Zhg6UBdceihdga8 (https://photos.google.com/photo/AF1QipOvM9aqBiU5PfmnaZwzerR_sd9SuXkGTxy-EpG6)

Looking at PROTOCOL.TXT, it appears that etherdfs's protocol does not contain any error-checking/checksum whatsoever, so until this is added, etherdfs is unusable for me. And, honestly, I don't think it should be relied on for anything other than tinkering until this is added. No network is 100% perfect.

On a real IBM PC 5160 with an Intel Etherexpress 8/16, the driver hangs while initializing. This might be because the NIC is connected to a wireless adapter, but again it works with mTCP perfectly so I'm not sure where the trouble is. I've double-checked the MAC is correct, but it just sits there:

https://goo.gl/photos/ZjHbgZWuczGAVWUX9 (https://photos.google.com/photo/AF1QipP4pn1MeuzyWsgTDugaCdLe8r5qJ5VPyJTv2F9C)



* A DOS server to be able to have a peer to peer vintage computer network.
* Make it work with PC/MS-DOS 3.1 and higher. Actually, I have not tested it with those DOS versions but you said it requires DOS 5.0+

The author already stated no DOS server will be written by him. Also, the DOS network redirector interface wasn't really usable until 4.x, so you should be using 4.x or higher with this.

My suggestions for any features or enhancements:


Add a checksum and retransmit if it fails. At a bare minimum, using the UDP 16-bit checksum (https://en.wikipedia.org/wiki/User_Datagram_Protocol#Checksum_computation) should be enough, and it is simple enough to be optimized easily in assembly.
Rewrite the client in assembly. Not necessarily to reduce its size, but rather to make it faster and easier to read and maintain without C or compiler/stack tricks.

eeguru
April 5th, 2017, 12:01 PM
* A DOS server to be able to have a peer to peer vintage computer network.

A more immediate compromise maybe to recompile the server for modern Windows under Cygwin or MinGW32. That is something that could be done and tested by someone other than the original author. It's been a while since I last looked at networking under DJGPP. But depending on what is there, that might be a possibility as well for 32-bit DPMI DOS.


* Make it work with PC/MS-DOS 3.1 and higher. Actually, I have not tested it with those DOS versions but you said it requires DOS 5.0+

Redirectors prior to v5.x are troublesome. 3.3 initially added redirector support and while it is very similar to 5.x support, there is enough difference that chasing quirky bugs is way more than a trivial effort. 4.x redirector support uses an entirely different interrupt and calling convention. It's so dissimilar that you might as well write an entire different implementation just for 4.x.

Stone
April 5th, 2017, 12:56 PM
Stone would you mind posting a link? I can not find anything with that name.It doesn't seem to be out in the wild so I'll send you a link.

keenerb
April 5th, 2017, 05:00 PM
Unfortunately my initial tests with 8088 systems (which work 100% with mTCP) have not been very promising. On a PCjr with a Xircom PE3, the driver appears to load, but it occasionally truncates a byte from the stream, which means I can't rely on it for anything. When I copy a file to the local drive, then copy it again, then run a file compare, they don't match up. Example:

https://photos.google.com/photo/AF1QipOvM9aqBiU5PfmnaZwzerR_sd9SuXkGTxy-EpG6

Looking at PROTOCOL.TXT, it appears that etherdfs's protocol does not contain any error-checking/checksum whatsoever, so until this is added, etherdfs is unusable for me. And, honestly, I don't think it should be relied on for anything other than tinkering until this is added. No network is 100% perfect.

On a real IBM PC 5160 with an Intel Etherexpress 8/16, the driver hangs while initializing. This might be because the NIC is connected to a wireless adapter, but again it works with mTCP perfectly so I'm not sure where the trouble is. I've double-checked the MAC is correct, but it just sits there:


I haven't seen any data integrity issues yet. I wonder if it's more or less likely to lose data than say FTP?

Your photo isn't loading for me, also.

mbbrutman
April 5th, 2017, 07:54 PM
Ethernet frames are protected by a CRC that the hardware computes and checks. In theory the packet driver will not ever see an Ethernet frame that was corrupted in flight because it will fail the CRC check. So for a local network your biggest danger is packet loss, not packet corruption. Assuming everything is working correctly.

FTP is going to be far more reliable than just blasting raw packets onto the wire, but at the expense of more computational overhead. In addition to the Ethernet CRC you get an IP header checksum and a TCP checksum that covers parts of the IP header, the full TCP header, and the full payload.

You can get really good throughput with this technique, assuming no packet drops. To make it more robust you have to start adding sequence numbers, re-transmission, and error detection. That leads to dead time on the wire, which then leads to implementing sliding windows for the packets. By the time you get to that you might as well have just implemented TCP/IP which features all of those things and is a universal standard.

Side note: For UDP data transfers the checksum is optional. You still get the IP header checksum, but the UDP packet does not need to have a checksum. Nobody uses that feature because it is a terrible idea.

mateuszviste
April 6th, 2017, 02:40 AM
Looking at PROTOCOL.TXT, it appears that etherdfs's protocol does not contain any error-checking/checksum whatsoever, so until this is added, etherdfs is unusable for me. And, honestly, I don't think it should be relied on for anything other than tinkering until this is added. No network is 100% perfect.

That's quite a bold claim. Don't you know that every Ethernet frame is protected by a hardware CRC? Of course it's possible that your PE3 adapter doesn't bother computing and/or checking this mandatory CRC field, but then I guess you should complain to Xircom about that. I don't have any PE3 so I cannot test, though. I do, however, use EtherDFS on my 80C86 laptop through a PLIP link with 100% success.
That being said, I do plan to add a checksum to EtherDFS before v1.0 (for fun, mostly), but it's really low priority, since it shouldn't ever be required anyway.


On a real IBM PC 5160 with an Intel Etherexpress 8/16, the driver hangs while initializing. This might be because the NIC is connected to a wireless adapter, but again it works with mTCP perfectly so I'm not sure where the trouble is. I've double-checked the MAC is correct, but it just sits there:

Sadly I am unable to open your photos. I understand you pass the MAC address (argument) in its full form to EtherDFS - in such case, the hang is surely not network-related, because in such situation EtherDFS doesn't send nor expect any packet. What's the exact version of DOS you are running there? Perhaps this version does things slightly differently... I'd be willing to test it, but I need to know what is the exact OS. Also, do you you have any other INT 2F TSRs running there? (like CDROM, networking drive, or similar). If you'd be willing to perform a few tests, I could prepare a few debug versions of EtherDFS so we could, perhaps, at least pin-point the issue (if not solving it).


The author already stated no DOS server will be written by him.

I don't think the author ever said that. He did say, however, that he don't do Windows, but that's another story. I might port ethersrv to DOS at some point, but cannot say for sure - it's not a trivial thing to do, even though I'd really like it, and my hobby time is very limited.


Also, the DOS network redirector interface wasn't really usable until 4.x, so you should be using 4.x or higher with this.

That's very true indeed. I target MS-DOS 5.0+ for this reason exactly - covering versions below becomes very messy, very fast. Is there any reason why one would want to use MS-DOS 3.x instead of 5.x?



Rewrite the client in assembly. Not necessarily to reduce its size, but rather to make it faster and easier to read and maintain without C or compiler/stack tricks.


It would hardly make EtherDFS faster, and probably not smaller either. It's perhaps a personal thing, but reading and maintaining C always has been significantly easier than assembly to me. Also, the amount of bugs and amount of time needed for development is very much in favor of developing in C. Shortly said - sorry, won't happen. My project, my time, my rules. Feel free to fork an assembly version, though - I'd very much welcome such initiative, and would be happy to assist during the struggle.

mateuszviste
April 6th, 2017, 02:55 AM
In theory the packet driver will not ever see an Ethernet frame that was corrupted in flight because it will fail the CRC check.

That's correct. One could argue that the Ethernet CRC could still fail detecting some corruptions because of CRC collisions, but I believe this to be anecdotal at best.


So for a local network your biggest danger is packet loss, not packet corruption.

Especially true when running over wifi (which I do myself - my 8086 PC is linked through a PLIP link to a wifi AP). EtherDFS handles packet loss without any troubles, and performs retransmissions whenever necessary, in a way that is totally transparent both to the user and his applications.


You can get really good throughput with this technique, assuming no packet drops. To make it more robust you have to start adding sequence numbers, re-transmission, and error detection. That leads to dead time on the wire, which then leads to implementing sliding windows for the packets. By the time you get to that you might as well have just implemented TCP/IP which features all of those things and is a universal standard.

The above is perfectly true, but I have to state that performance is not part of EtherDFS' goals. I use raw Ethernet frames mostly for simplicity, and also for being able to share the packet driver with other networking tools. EtherDFS doesn't come with sliding windows - in this regard it's more like TFTP than FTP (ie "send single frame, expect single ACK"). Sliding windows are great when there is latency, but on a LAN (where EtherDFS is supposed to be used), the latency should not be that important.

keenerb
April 6th, 2017, 05:14 AM
I ran a few tests last night with my 1000SL and Etherlink III card.

I copied a 13k file from remote to local 5,000 times and computed a checksum after each copy.

There were no data errors or mismatched md5sums for the entire time period. That's probably good enough for me. I really wouldn't want to try this over anything wireless or with any packet loss, though...

Also, according to the protocol.txt file, the packet does include a sequence number:


57 | S | a single byte with a "sequence" value. Each query is supposed to
| | use a different sequence, to avoid the client getting confused if
| | it receives an answer relating to a different query than it
| | expects.

In the event a packet is lost, the client should simply re-send the request right?

mbbrutman
April 6th, 2017, 07:30 AM
While the Ethernet CRC should protect you, it is still inadequate.

I've helped people debug quite a few old machines where the packet driver presented corrupted data to the application. Being that it was somebody else's machine I could not do a root cause analysis, but it was very clear that they were dealing with bad hardware. What you really need is end-to-end protection. The TCP/IP checksums get you closer to that by ensuring that not only is the Ethernet CRC working, but the interface to the host machine and the host machine itself are working properly. A real end-to-end solution is using MD5 to compare checksums after it is written to disk. Zip files are great for data transfer in this regard because their CRC will detect pretty much any error in the entire process.

Trixter might have bad hardware, but given that FTP is working well it can't be that bad. Even with good hardware you are going to have errors at various stages of the data transfer, which is why it is advisable not to rely on just the Ethernet CRC. You have to assume the hardware is trying to sabotage you.

mateuszviste
April 6th, 2017, 08:36 AM
While the Ethernet CRC should protect you, it is still inadequate. (...)

I totally agree with you. I also had the occasion to observe a (high-end, corporate) switch that was corrupting Ethernet frames, and forwarding them with a *recomputed* (hence seemingly valid) CRC32. But this, along with the case that you mention, is something I consider odd-ball situations. They are also the secondary reason why I do plan to implement an (optional, ie. that can be disabled) CRC-like mechanism in EtherDFS.
In trixter's case, I wonder if the problem comes from the PE3 thing that do not bother validating the CRC, or perhaps his parallel port is misbehaving and dropping some bits here and there. In any case, the next version of EtherDFS will be able to catch such misbehaving hardware. It's worth noting though, that there is no silver bullet for bad hardware - should a single RAM cell go bad and 'forget' to flip from time to time, transmissions will misbehave no matter how many CRCs there are on the wire.

Trixter
April 6th, 2017, 09:17 AM
I corrected the photos in my previous post.

So there are two concerns:

This https://goo.gl/photos/fRUzQKaEG5PWqMny6 shows that sometimes a byte is silently lost. The photo shows the same 512K file transferred only twice. If you want to argue that my hardware is bad, then that's the end of that conversation, although I'm not sure how I'm expected to fix it.

My second concern is that the driver completely hangs on initialization on a real PC with a "real" ethernet card (Intel Etherexpress 8/16). That system is not running any other network redirectors at all (no other network sharing, no mscdex, no cdrom drive, no parallel-port devices), and it runs IBM PC DOS 2000 (ie. IBM PC DOS 7.00, revision 1). The cards in that system are an IBM CGA, Intel NIC, Sound Blaster Pro, ADP-50 IDE hard drive adapter, floppy controller, Central Point option Board. I will trace through driver initialization to see if I can pinpoint where it is locking up.

On the system where it is hanging, I have the NIC connected to a wireless bridge. I will test connecting it wired to see if that fixes the problem.

EDIT: I found on my PC DOS 2000 system, it was hanging on exit (on return to the OS). I then booted completely clean with F5 and found that it no longer hung on return to the OS, but the drive that was created shows "invalid directory" when queried. I will try with MS-DOS 6.22 and see if I get the same results.

EDIT2: Same result with MS-DOS 6.22. I'll try a wired connection next.

EDIT3: A wired connection worked, but the resulting network drive created appears to be read-only. On the other system where it is working with corruption, the drive is working correctly as read-write. I'll try going back to PC-DOS 7 to see if I can narrow down what TSR or configuration is causing trouble with etherdfs.

mbbrutman
April 6th, 2017, 09:37 AM
I wouldn't look at the PE3 too closely; I'm sure it is validating the CRC. Just like they don't let you set the CRC, they also take responsibility to enforce the CRC.

I would be more inclined to blame an occasional bit flip on the rest of the hardware, especially if a parallel port is involved. However, even with a 5 nines correctness rate (99.999%) per packet you will still get 1 packet in 100,000 that is corrupted. On the conservative side that would mean a bad bit every 150MB at best, assuming no protocol overhead and full packets. (100,000 * 1500). If you are only sending 512 bytes per packet because you are emulating a block device at a low level that would leave you exposed to corruption every 50MB or less.

It is not software's job to fix bad hardware. It is software's job to work around it. Which is why multiple layers of error detection are so important. Especially on older hardware that is well past its expected operating lifetime. For older hardware assume that it is out to kill you and code defensively.

keenerb
April 6th, 2017, 09:45 AM
I corrected the photos in my previous post.

So there are two concerns:

This https://goo.gl/photos/fRUzQKaEG5PWqMny6 shows that sometimes a byte is silently lost. The photo shows the same 512K file transferred only twice. If you want to argue that my hardware is bad, then that's the end of that conversation, although I'm not sure how I'm expected to fix it.

My second concern is that the driver completely hangs on initialization on a real PC with a "real" ethernet card (Intel Etherexpress 8/16). That system is not running any other network redirectors at all (no other network sharing, no mscdex, no cdrom drive, no parallel-port devices), and it runs IBM PC DOS 2000 (ie. IBM PC DOS 7.00, revision 1). The cards in that system are an IBM CGA, Intel NIC, Sound Blaster Pro, ADP-50 IDE hard drive adapter, floppy controller, Central Point option Board. I will trace through driver initialization to see if I can pinpoint where it is locking up.

On the system where it is hanging, I have the NIC connected to a wireless bridge. I will test connecting it wired to see if that fixes the problem.

Is it reproducible reliably?

I have a few PE3 adapters, I could fire one up on a non-Tandy box and see what

mateuszviste
April 6th, 2017, 10:24 AM
This https://goo.gl/photos/fRUzQKaEG5PWqMny6 shows that sometimes a byte is silently lost. The photo shows the same 512K file transferred only twice. If you want to argue that my hardware is bad, then that's the end of that conversation, although I'm not sure how I'm expected to fix it.

Thanks for the additional information! I absolutely don't want to argue about bad hardware, nor anything, really. Still, I would be surprised if EtherDFS was loosing bytes in such manner (because it works very well for many people), and I'd be also surprised if the problem was coming from the network (because of the Ethernet CRC32 and because "loosing a byte" is a somewhat unusual kind of network corruption, and I've seen many). The most probable culprit in my opinion is between your NIC and EtherDFS - perhaps your parallel port, perhaps something else. Or perhaps I'm wrong about everything. Doesn't matter to me really. The important fact is that having some kind of CRC on EtherDFS frames might solve (or mask at least) the problem. I will do some prototype very soon - would be great if you could test it out to see if it makes things better for you.


My second concern is that the driver completely hangs on initialization on a real PC with a "real" ethernet card (Intel Etherexpress 8/16). That system is not running any other network redirectors at all (no other network sharing, no mscdex, no cdrom drive, no parallel-port devices), and it runs IBM PC DOS 2000 (ie. IBM PC DOS 7.00, revision 1).

I didn't test PC DOS at all - the only OSes I tested EtherDFS on were FreeDOS and MS-DOS (v5.x, v6.x and v7.x), hence it may be very possible that PC-DOS does something slightly differently. I will grab a copy of PC DOS 2000 and try to reproduce the problem. But first I need to close the CRC thing.

Trixter
April 6th, 2017, 06:23 PM
I would be surprised if EtherDFS was loosing bytes in such manner

I've never accused EtherDFS of losing bytes. I'm sure something about my hardware is losing bytes, and I also think this is normal and expected with the type of hardware that EtherDFS targets, so that's why I was arguing for some form of error detection beyond layer 2. The TCP and UDP RFCs were created around the same time as this hardware, and they felt it was necessary to have their own protocol checksum...


I will do some prototype very soon - would be great if you could test it out to see if it makes things better for you.

I am happy to test any prototype. I will be busy this weekend at a gaming convention, but after that I am available to test a CRC'd version on my dodgy hardware. I have a strong, vested interest in seeing this get better and succeed, as it solves a lot of problems for myself and my colleagues.


I didn't test PC DOS at all - the only OSes I tested EtherDFS on were FreeDOS and MS-DOS (v5.x, v6.x and v7.x), hence it may be very possible that PC-DOS does something slightly differently.

I tested with both PC DOS 7 and MS-DOS 6.22 with a clean boot, and I directly connected that system to the network with a cable, and now it doesn't hang on exit to DOS. Unfortunately, the resulting drive is not fully functional; the resulting network drive created appears to be read-only. Any attempt to change an existing file, or create a new one, results in odd errors from DOS such as "not enough space" (there is) or some other unspecified error that I don't get when using etherdfs on my other system. I will try to find out exactly what in my autoexec/config.sys was causing it to hang on exit and report back.

Trixter
April 6th, 2017, 08:05 PM
Update: I found what was causing the hang on my 5160 system: 4DOS. (I actually use an older version branded by Symantec called NDOS.) Switching to stock COMMAND.COM fixed the problem and etherdfs now returns to DOS and appears to run.

However, it is still not fully functional. Here is the behavior I'm seeing on the 5160 with an Intel EtherExpress 8/16 card:


Trying to create a text file on the network drive at the command line with "copy con test.txt" results in "Insufficient disk space" when there is plenty of free disk space.
Editing a text file on the network drive and saving it appears to make a copy of the last character in the file, and also appends a junk character. This is 100% reproducible. The source filesystem on the ethersvr is an MS-DOS filesystem.


Here is a quick video of me reproducing the bugs: ("sled" is a text editor I've used on this system for 30 years, I know it isn't producing the extra characters)


https://youtu.be/rkbULGAnB5A

evildragon
April 6th, 2017, 09:09 PM
While the Ethernet CRC should protect you, it is still inadequate.

I've helped people debug quite a few old machines where the packet driver presented corrupted data to the application. Being that it was somebody else's machine I could not do a root cause analysis, but it was very clear that they were dealing with bad hardware. What you really need is end-to-end protection. The TCP/IP checksums get you closer to that by ensuring that not only is the Ethernet CRC working, but the interface to the host machine and the host machine itself are working properly. A real end-to-end solution is using MD5 to compare checksums after it is written to disk. Zip files are great for data transfer in this regard because their CRC will detect pretty much any error in the entire process.

Trixter might have bad hardware, but given that FTP is working well it can't be that bad. Even with good hardware you are going to have errors at various stages of the data transfer, which is why it is advisable not to rely on just the Ethernet CRC. You have to assume the hardware is trying to sabotage you.

I think I might have been one of those people with such hardware, my SMC card, because I remember trying to troubleshoot that with you, and ever since switching to a Xircom PE-3, everything's been great.

mateuszviste
April 6th, 2017, 11:44 PM
Trixter, could you please test this version of EtherDFS on your "corrupting" PC?

http://mateusz.viste.fr/temp/etherdfs-test/etherdfs081beta.zip

The above version comes with a checksum implementation on each frame.
Note, that for this version to work, you need to upgrade ethersrv-linux to version 20170406.

About the "read-only" and "additional char" troubles you have on the other PC: could you compile ethersrv-linux in debug mode, and provide me with the console output of ethersrv-linux when the problem occurs? To enable debug mode on ethersrv-linux, you would need to edit the file debug.h and set the "DEBUG" definition to "1" (and then run "make" to recompile the program).

Trixter
April 9th, 2017, 04:27 PM
I just got back from traveling, but will definitely do this for you in the next day or two. Thanks for helping to locate the problem.

keenerb
April 11th, 2017, 04:33 PM
Seems to be working fine for me.

I haven't been having packet issues; should I enable debug?

Pepinno
April 12th, 2017, 11:18 AM
Seems to be working fine for me.

I haven't been having packet issues; should I enable debug?

The consensus, currently, seems to be that EtherDFS is safe to use, unless you have a flaky hardware (like a flaky ethernet card or a flaky parallel port attached to a Pocket Ethernet Adapter). Given that all retro-hardware is past its expected life expectancy, it means...

...that YMMV.

Anyway, EtherDFS looks to me as a major breakthrough in DOS/PC retro-computing. Any machine can now easily transfer files if it has a ethernet NIC or a parallel port with a Pocket Ethernet Adapter - no more sneakernet or ad-hoc serial cables to do it "laplink-style".

Stone
April 12th, 2017, 11:24 AM
ny machine can now easily transfer files if it has a ethernet NIC or a parallel port with a Pocket Ethernet Adapter - no more sneakernet or ad-hoc serial cables to do it "laplink-style".Hey... any machine with a parallel port -- period! That's it. Full parallel network accessibility has been around for over 25 years! :-) No Pocket Ethernet Adapter is necessary.

Pepinno
April 12th, 2017, 11:45 AM
Hey... any machine with a parallel port -- period! That's it. Full parallel network accessibility has been around for over 25 years! :-) No Pocket Ethernet Adapter is necessary.

A parallel port by itself is a problem, as modern machines usually don't have one to interface with your retro PC, whereas modern machines tend to have some kind of ethernet in them.

mateuszviste
April 12th, 2017, 10:58 PM
Seems to be working fine for me. I haven't been having packet issues; should I enable debug?

I understand you tested the EtherDFS "test" (0.8.1) version, and it works for you. There is no point in activating debug if all works fine for you, really. And it would be surprising if the test version did not work for you while the "normal" (0.8) was doing fine :)
The test version only adds a checksum on frames and validates everything it receives from ethersrv. It detects sneaky corruptions that might happen elsewhere than on the Ethernet wire itself (for example bits that would flip on the LPT port, if the NIC is LPT-connected, or bytes that go missing). Unfortunately I didn't get any feedback from Trixter, so I can only hope it solves the problem on his hardware. Soon I will release a formal v0.8.1 with this patch. It will make EtherDFS slightly slower, but I guess it's worth it, even if only for the 0.1% of people that run it on flaky hardware.

mbbrutman
April 13th, 2017, 08:41 AM
<snip>

Anyway, EtherDFS looks to me as a major breakthrough in DOS/PC retro-computing. Any machine can now easily transfer files if it has a ethernet NIC or a parallel port with a Pocket Ethernet Adapter - no more sneakernet or ad-hoc serial cables to do it "laplink-style".

Er, Novel Netware, FTP, NFS and MS style file and printer sharing have existed for a long time on PC class hardware. Let's not go overboard with the superlatives ...

keenerb
April 13th, 2017, 09:13 AM
Er, Novel Netware, FTP, NFS and MS style file and printer sharing have existed for a long time on PC class hardware. Let's not go overboard with the superlatives ...

It's also only 7k memory.

It deserves a few superlatives for that. Smallest alternative I've seen was around 50k.

Pepinno
April 13th, 2017, 09:45 AM
Er, Novel Netware, FTP, NFS and MS style file and printer sharing have existed for a long time on PC class hardware. Let's not go overboard with the superlatives ...

Novell Netware's IPX is a layer of complexity over raw ethernet, FTP and NFS run over TCP/IP which is two layers of complexity over raw ethernet, LAN Manager/NetBIOS is a layer of complexity over raw ethernet. Those layers of complexity eat up scarce conventional memory on old PCs, and slow them down.

EtherDFS is just the sweet spot: nimble, fast, easy, modern-world compatible, and open-source.

I don't know about you, but I am breaking the champagne about it!

mbbrutman
April 13th, 2017, 09:51 AM
Novell Netware's IPX is a layer of complexity over raw ethernet, FTP and NFS run over TCP/IP which is two layers of complexity over raw ethernet, LAN Manager/NetBIOS is a layer of complexity over raw ethernet. Those layers of complexity eat up scarce conventional memory on old PCs, and slow them down.

EtherDFS is just the sweet spot: nimble, fast, easy, modern-world compatible, and open-source.

I don't know about you, but I am breaking the champagne about it!

This doesn't take away from EtherDFS, but EtherDFS by definition is a layer of complexity over raw Ethernet too ... Otherwise, we would just call it Ethernet.

keenerb
April 13th, 2017, 11:01 AM
This doesn't take away from EtherDFS, but EtherDFS by definition is a layer of complexity over raw Ethernet too ... Otherwise, we would just call it Ethernet.

I really like that I can still apparantly use mtcp over it. I've FTP'd data from an external server directly onto my mapped drive, which is neat.

keenerb
April 14th, 2017, 07:04 AM
It's not working with a Xircom pocket ethernet for me either, I just set up a test



Received frame of 80 bytes (cksum = ENABLED)
00 0C 29 0A 26 B3 00 80 C7 2E 64 C6 ED F5 00 00 | ..).&... ..d.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 8A D9 82 11 02 1B 13 5C 47 41 | ........ .....\GA
4D 45 53 5C 44 4E 44 5C 44 4E 44 2E 3F 3F 3F 00 | MES\DND\ DND.???.
CHECKSUM MISMATCH! Computed: 0x6CC5h Received: 0xD98Ah
Received frame of 80 bytes (cksum = ENABLED)
00 0C 29 0A 26 B3 00 80 C7 2E 64 C6 ED F5 00 00 | ..).&... ..d.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 8A D9 82 11 02 1B 13 5C 47 41 | ........ .....\GA
4D 45 53 5C 44 4E 44 5C 44 4E 44 2E 3F 3F 3F 00 | MES\DND\ DND.???.
CHECKSUM MISMATCH! Computed: 0x6CC5h Received: 0xD98Ah
Received frame of 80 bytes (cksum = ENABLED)
00 0C 29 0A 26 B3 00 80 C7 2E 64 C6 ED F5 00 00 | ..).&... ..d.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 8A D9 82 11 02 1B 13 5C 47 41 | ........ .....\GA
4D 45 53 5C 44 4E 44 5C 44 4E 44 2E 3F 3F 3F 00 | MES\DND\ DND.???.
CHECKSUM MISMATCH! Computed: 0x6CC5h Received: 0xD98Ah
Received frame of 80 bytes (cksum = ENABLED)
00 0C 29 0A 26 B3 00 80 C7 2E 64 C6 ED F5 00 00 | ..).&... ..d.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 8A D9 82 11 02 1B 13 5C 47 41 | ........ .....\GA
4D 45 53 5C 44 4E 44 5C 44 4E 44 2E 3F 3F 3F 00 | MES\DND\ DND.???.
CHECKSUM MISMATCH! Computed: 0x6CC5h Received: 0xD98Ah
Received frame of 80 bytes (cksum = ENABLED)
00 0C 29 0A 26 B3 00 80 C7 2E 64 C6 ED F5 00 00 | ..).&... ..d.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 8A D9 82 11 02 1B 13 5C 47 41 | ........ .....\GA
4D 45 53 5C 44 4E 44 5C 44 4E 44 2E 3F 3F 3F 00 | MES\DND\ DND.???.
CHECKSUM MISMATCH! Computed: 0x6CC5h Received: 0xD98Ah


I tried to run a simple text game on my mapped drive.



Received frame of 92 bytes (cksum = ENABLED)
00 0C 29 0A 26 B3 00 80 C7 2E 64 C6 ED F5 00 00 | ..).&... ..d.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 65 64 82 2C 02 2E 00 00 01 01 | ......ed .,......
00 00 5C 47 41 4D 45 53 5C 41 4E 47 42 41 4E 44 | ..\GAMES \ANGBAND
5C 41 4E 47 44 4F 53 2E 43 46 47 00 | \ANGDOS. CFG.
CHECKSUM MISMATCH! Computed: 0xB232h Received: 0x6465h
Received frame of 92 bytes (cksum = ENABLED)
00 0C 29 0A 26 B3 00 80 C7 2E 64 C6 ED F5 00 00 | ..).&... ..d.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 65 64 82 2C 02 2E 00 00 01 01 | ......ed .,......
00 00 5C 47 41 4D 45 53 5C 41 4E 47 42 41 4E 44 | ..\GAMES \ANGBAND
5C 41 4E 47 44 4F 53 2E 43 46 47 00 | \ANGDOS. CFG.
CHECKSUM MISMATCH! Computed: 0xB232h Received: 0x6465h
Received frame of 92 bytes (cksum = ENABLED)
00 0C 29 0A 26 B3 00 80 C7 2E 64 C6 ED F5 00 00 | ..).&... ..d.....
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ........ ........
00 00 00 00 00 00 65 64 82 2C 02 2E 00 00 01 01 | ......ed .,......
00 00 5C 47 41 4D 45 53 5C 41 4E 47 42 41 4E 44 | ..\GAMES \ANGBAND
5C 41 4E 47 44 4F 53 2E 43 46 47 00 | \ANGDOS. CFG.
CHECKSUM MISMATCH! Computed: 0xB232h Received: 0x6465h


"type angdos.cfg"

Trixter
April 14th, 2017, 07:42 PM
Trixter, could you please test this version of EtherDFS on your "corrupting" PC?

http://mateusz.viste.fr/temp/etherdfs-test/etherdfs081beta.zip

The above version comes with a checksum implementation on each frame.
Note, that for this version to work, you need to upgrade ethersrv-linux to version 20170406.

About the "read-only" and "additional char" troubles you have on the other PC: could you compile ethersrv-linux in debug mode, and provide me with the console output of ethersrv-linux when the problem occurs? To enable debug mode on ethersrv-linux, you would need to edit the file debug.h and set the "DEBUG" definition to "1" (and then run "make" to recompile the program).

I finally got around to testing this, but I'm afraid the results were worse than before. I am limiting my testing to one platform to keep things simple and consistent for troubleshooting. I compiled the 0406 version with debugging on, downloaded and ran the beta client, and now I cannot copy files. At the DOS side, I see this:


c:\tdl.net>copy g:\tdl\titles.idx
File not found - G:\TDL\TITLES.IDX
0 file(s) copied

On the server side, I get the following output which is attached.

Based on the error output, I suspect your checksum routine has an error?

mateuszviste
April 15th, 2017, 03:14 AM
Based on the error output, I suspect your checksum routine has an error?

If it would, it would be buggy in all situations, with all hardware, not only with the PE3 gimmick.

Brian already sent me his results off-forum yesterday, and from these I noticed that for some reason the PE3 appears to append an additional 1-byte to the frame. I have no clue why it does that, but when I recomputed by hand the checksums based on the debug output Brian sent me, the checksum magically started to match. Long story short, I uploaded a new test version today, that is able to detect when a frame is padded, and trim the additional garbage. The latest test version requires ethersrv-linux version 20170415. Since I don't have a PE3 myself, I count on you guys to test it out :-)

http://mateusz.viste.fr/temp/etherdfs-test/

mbbrutman
April 15th, 2017, 08:45 AM
If it would, it would be buggy in all situations, with all hardware, not only with the PE3 gimmick.

Brian already sent me his results off-forum yesterday, and from these I noticed that for some reason the PE3 appears to append an additional 1-byte to the frame. I have no clue why it does that, but when I recomputed by hand the checksums based on the debug output Brian sent me, the checksum magically started to match. Long story short, I uploaded a new test version today, that is able to detect when a frame is padded, and trim the additional garbage. The latest test version requires ethersrv-linux version 20170415. Since I don't have a PE3 myself, I count on you guys to test it out :-)

http://mateusz.viste.fr/temp/etherdfs-test/

You are blaming the Xircom PE3? That's a pretty bold move ... they were in use all over the place in the mid 90s on corporate networks, which was a pretty expensive environment at the time. If the PE3 was adding extra bytes to a packet it would have never been sold.

I don't have anything in the mTCP code that deals with padding on packets, except for ensuring that all packets that I send are a minimum of 60 bytes long to make Intel gigabit adapters happy. (And I think that number probably should be 62, not 60, so I might fix it after re-investigating it.)

Can you post or send me your source code? I wouldn't mind looking for the off-by-one error no matter where it is.

mateuszviste
April 15th, 2017, 09:27 AM
You are blaming the Xircom PE3? That's a pretty bold move ... they were in use all over the place in the mid 90s on corporate networks, which was a pretty expensive environment at the time.

I am sure of nothing, and since I cannot reproduce the problem at hand, I have to rely on assumptions. Note, that "adding an extra byte" is not as bad as it sounds - in most cases it's totally harmless, since IP doesn't care anyway, as it knows exactly the length of payload to read.

What I do know, is that:
- EtherDFS works perfectly with all NICs and packet drivers I could test
- It's reported as "not working" with PE-3
- On the PE-3 thing, based on the ethersrv debug output provided by both Brian and Jim, I see that EtherSRV seem to receive one extra byte of data

Clearly the above observations aren't enough to lead to a "PE3 sends one extra byte" conclusion, but the fact that the problem appears so far only with this one NIC makes it suspect.


I don't have anything in the mTCP code that deals with padding on packets, except for ensuring that all packets that I send are a minimum of 60 bytes long to make Intel gigabit adapters happy.

Not only gigabit adapters, there are more stuff that expects >= 60 bytes frames. My WiFi bridge doesn't tolerate anything less than 60 bytes, which was very troublesome with WatTCP, as the latter tends to generate smaller frames sometimes (the workaround I used is activating the TCP timestamp option in WatTCP configuration to force the segments to be bigger - adding the 14-bytes long TCP timestamp is enough to pass the 60-bytes Ethernet barrier).


Can you post or send me your source code? I wouldn't mind looking for the off-by-one error no matter where it is.

Of course. EtherDFS is an open-source project, and its source code is publicly available on the project's SVN:
https://sourceforge.net/p/etherdfs/code/HEAD/tree/

keenerb
April 15th, 2017, 11:58 AM
Everything seems to be working OK on a XIrcom PE-3 test system.



As a presumably separate issue, I have also noticed that EtherDFS does NOT like multiple clients attached to same server, very strange things seem to happen like "MKDIR TEMP" actually made TEMP6, and "mkdir ASDF" failed completely.

This debug output on my server was during a time when both my Tandy 1000TL and my Tandy 2500SX were attempting to access the same C drive share. This is probably just something that I shouldnt' be doing to begin with, but it happened and I got some error output so it can't hurt to mention it.



CHECKSUM MISMATCH! Computed: 0xC6BDh Received: 0x8CFDh
CHECKSUM MISMATCH! Computed: 0xC6BDh Received: 0x8CFDh
CHECKSUM MISMATCH! Computed: 0xC6BDh Received: 0x8CFDh
CHECKSUM MISMATCH! Computed: 0xC6BDh Received: 0x8CFDh
CHECKSUM MISMATCH! Computed: 0xD312h Received: 0xA427h
CHECKSUM MISMATCH! Computed: 0xD252h Received: 0xA427h
CHECKSUM MISMATCH! Computed: 0xD312h Received: 0xA427h
CHECKSUM MISMATCH! Computed: 0xD252h Received: 0xA427h
CHECKSUM MISMATCH! Computed: 0xD312h Received: 0xA427h
CHECKSUM MISMATCH! Computed: 0x47D4h Received: 0x8F2Ah
CHECKSUM MISMATCH! Computed: 0x47D4h Received: 0x8F2Ah
CHECKSUM MISMATCH! Computed: 0x47D4h Received: 0x8F2Ah
CHECKSUM MISMATCH! Computed: 0x47D4h Received: 0x8F2Ah
CHECKSUM MISMATCH! Computed: 0x47D4h Received: 0x8F2Ah
CHECKSUM MISMATCH! Computed: 0xCDC4h Received: 0x9A3Bh
CHECKSUM MISMATCH! Computed: 0xCE1Ch Received: 0x9A3Bh
CHECKSUM MISMATCH! Computed: 0xCDC4h Received: 0x9A3Bh
CHECKSUM MISMATCH! Computed: 0xCE1Ch Received: 0x9A3Bh
CHECKSUM MISMATCH! Computed: 0xCDC4h Received: 0x9A3Bh
CHECKSUM MISMATCH! Computed: 0xA10Fh Received: 0x4021h
CHECKSUM MISMATCH! Computed: 0xA10Fh Received: 0x4021h
CHECKSUM MISMATCH! Computed: 0xA10Fh Received: 0x4021h
CHECKSUM MISMATCH! Computed: 0xA10Fh Received: 0x4021h
CHECKSUM MISMATCH! Computed: 0xA10Fh Received: 0x4021h
MKDIR Error: Invalid argument
CHDIR Error (/mnt/dosdisk//asdf?): No such file or directory
MKDIR Error: Invalid argument
CHDIR Error (/mnt/dosdisk//templ): No such file or directory
MKDIR Error: Invalid argument
RMDIR Error: No such file or directory
MKDIR Error: Invalid argument
MKDIR Error: Invalid argument
CHDIR Error (/mnt/dosdisk//temp5): No such file or directory

Trixter
April 15th, 2017, 12:09 PM
MKDIR TEMP" actually made TEMP6

This sounds very much like my reproducible issue of a random junk byte being appended to frames.

keenerb
April 15th, 2017, 12:14 PM
This sounds very much like my reproducible issue of a random junk byte being appended to frames.

Did you have multiple clients connected? When I terminated all other clients it functioned normally.

mateuszviste
April 15th, 2017, 12:27 PM
Everything seems to be working OK on a XIrcom PE-3 test system.

Good. So trimming out the extra byte makes it work.


As a presumably separate issue, I have also noticed that EtherDFS does NOT like multiple clients attached to same server, very strange things seem to happen like "MKDIR TEMP" actually made TEMP6, and "mkdir ASDF" failed completely.

As Jim pointed out already, this does look very much like still the same problem: a garbage byte trailing the query and not trimmed. There shouldn't be any problem with running multiple clients on the same drive (that's one of the goals behind EtherDFS after all). Three questions:
- are both Tandy PCs running a PE3 adapter?
- when things stop working, do they stop working for one of the PCs, or both?
- are you 100% sure that you run the latest "test" EtherDFS release on both PCs? (old EtherDFS clients will still work with ethersrv, but without the data-validation features)

keenerb
April 15th, 2017, 05:12 PM
Everything seems to be running OK now. I tracked down an old copy of etherdfs.exe in a PATH folder, I was probably running an old version by mistake.

mateuszviste
April 15th, 2017, 11:30 PM
Everything seems to be running OK now. I tracked down an old copy of etherdfs.exe in a PATH folder, I was probably running an old version by mistake.

That's great news - means that when used with a Xircom PE3, at least one extra byte is being sent, at least in some situations, and perhaps some corruptions happen down the road, too. Both troubles are fixed in the (to be released later today) version 0.8.1 of EtherDFS, which includes an internal checksum (to eliminate corrupted frames) as well as a way to detect padded frames (to "fix" frames which have been trailed with some extra garbage, instead of just failing them because of the checksum).
I don't have the means to investigate the exact causes, which seem highly related to the PE3 hardware itself (or its packet driver), but I'm glad that EtherDFS will work in such harsh environment as well now.

mateuszviste
April 16th, 2017, 02:40 AM
I released a new public version of EtherDFS, the version 0.8.1.
Available for download at the usual place: http://etherdfs.sourceforge.net

mbbrutman
April 16th, 2017, 12:08 PM
I spent about three hours reading code and debugging this morning. Here are my conclusions:


The Xircom is adding padding to Ethernet Frames.
EtherDFS was in error when using the return code from the recv syscall directly.


I can't find anything in the Ethernet specifications regarding padding except that runt frames are not allowed; your payload (including MAC address but not the CRC) has to be a minimum of 60 bytes. (Add 4 bytes for the CRC to get to the full 64 bytes required, but software generally can't see those four bytes, hence the usual notation of 60 bytes for the frame.)

If you are running 802.3 then you are supposed to put the payload length in bytes 13 and 14, right after the MAC address fields. In Ethernet II you put a type field in there instead. Ethernet II types are supposed to be greater than 0x600 so that it is not ambiguous when trying to figure out if bytes 13 and 14 are a length or a type. If you are using an Ethernet II type then you are responsible for encoding the payload length in the payload somewhere, as IP does it.

Whether you are using 802.3 or Ethernet II the length is available to you, either in bytes 13 and 14 or in your Ethernet II payload. Using the results of the recv syscall seems dangerous when you consider that the length is already available and can be protected by your own checksum routine. It would not surprise me if other cards or drivers were adding padding for performance reasons.

mbbrutman
April 16th, 2017, 12:43 PM
And just for giggles, here is a sample frame where the Xircom adds padding:


12:41:25.332788 00:80:c7:30:81:0b (oui Unknown) > 00:01:29:d4:14:c6 (oui Unknown), ethertype IPv4 (0x0800), length 76: PCjr.3255 > darth.telnet: Flags [P.], seq 29:50, ack 37, win 4096, length 21 [telnet WONT ECHO, WILL NAWS, SB NAWS IS 0x50 0 0x19 SE, DONT STATUS, WONT LFLOW [|telnet]
0x0000: 0001 29d4 14c6 0080 c730 810b 0800 4500 ..)......0....E.
0x0010: 003d 0008 0000 ff06 35d4 c0a8 0270 c0a8 .=......5....p..
0x0020: 021e 0cb7 0017 43b6 3326 8cae 59ab 5018 ......C.3&..Y.P.
0x0030: 1000 19bb 0000 fffc 01ff fb1f fffa 1f00 ................
0x0040: 5000 19ff f0ff fe05 fffc 210b P.........!.

Note the IP header says the frame length is 0x3d, which is 61 bytes. Add 14 for the Ethernet header and you get to 75 bytes. Yet it's reporting the frame length is 76 bytes.

Once again, not in violation of any specification though. The length of the IP payload is clearly set to be 61 bytes, and this is an Ethernet II frame so the payload has to include the length.

mateuszviste
April 16th, 2017, 02:01 PM
hi Mike, thanks for looking it up. I see you confirm my "bold" findings.


The Xircom is adding padding to Ethernet Frames.

If I may nit-pick - that's not entirely correct. They do not do "padding", they rather send one byte too many (presumably word-aligned?). This is a quite serious bug, as it means they send over the wire some information that was never meant to leave the PC. Sure, it's only one byte - but still. "padding" would be if they would actually pad the frame with zeroes, or any other static value.


EtherDFS was in error when using the return code from the recv syscall directly.

EtherDFS was not in error - the length information could be obtained only in this way, given the protocol design. Was the protocol naive about assuming that no extra stuffing of frames will occur? Probably. Although I did not observe such behaviour myself on the various hardware combinations I tested - so far I only heard about that in relation to the Xircom dongles. Obviously it's better for the protocol to not trust the hardware too much, even if it costs a few additional CPU cycles. Since v0.8.1 this corner case is covered.


Using the results of the recv syscall seems dangerous when you consider that the length is already available and can be protected by your own checksum routine.

The length is not "already available", that's the root cause of the problem. EtherDFS works around this since v0.8.1 by encoding the length in its own payload. Ethernet II does not provide this information at all. Applications have either to trust that whatever they receive is correct, or transfer the length separately.

mbbrutman
April 16th, 2017, 02:42 PM
You were bold to blame Xircom for a bug. It is unexpected behavior, but not a bug; it is totally compliant with the specifications. It is padding Ethernet frames with an odd number of bytes to make them an even number of bytes, and that is correctly being reported to Linux. It is not a bug to pad the frame, but it would be a security leak if you could get anything from that extra byte. It is definitely a case of padding (although insecure) because they only do this on packets with odd numbers of bytes.

However, EtherDFS was very much in error. There is nothing anywhere that says you can rely on the return code of the recv to tell you what your payload length is. That is why in either 802.3 or Ethernet II you have to explicitly encode the length in the payload. recv will tell you how many bytes were received, not how many bytes are in the payload. There is a subtle but important difference there.

The length was supposed to be in the payload; EtherDFS was wrong to [1] not encode the length explicitly in the Ethernet payload as required by the specs and done by every other protocol and [2] not to bother with any error checking. It's good that both of these are corrected now.

mateuszviste
April 17th, 2017, 01:46 AM
You were bold to blame Xircom for a bug.

Which ultimately proves correct, even though you disagree on the exact interpretation.


It is unexpected behavior, but not a bug; it is totally compliant with the specifications.

Bollocks. The Packet Driver Specification is very clear and non-ambiguous: "Transmits length bytes of data, starting at buffer."
Any different behaviour is a bug. Sending "length bytes of data +1" is certainly a different behaviour, thus a bug.


It is padding Ethernet frames with an odd number of bytes to make them an even number of bytes

We are disagreeing on the definition of what "padding" means in this context, obviously.
It's clearly a bug in their packet driver. I assume they found it's faster to transfer data two bytes (WORD) at a time, and instead of detecting the "odd bytes" case, they simply added one additional byte "'cause the application layer behind will surely sort it out". And that's true for things like IP or IPX, but it's no golden rule.


However, EtherDFS was very much in error.

In assuming that it will be used with non-buggy packet drivers? Surely, yes. :)


That is why in either 802.3 or Ethernet II you have to explicitly encode the length in the payload.

Is it your personal opinion, or have you found (and could share) some formal specification that says that? I, myself, do not have access to any IEEE specs, so I'd be very much interested in knowing what it says exactly in this matter.


EtherDFS was wrong to [1] not encode the length explicitly in the Ethernet payload as required by the specs

Please quote said specs, if possible. Anyway - again, we are not discussing any sort of hardware optimization here, nor proper padding (which is necessary only to avoid runt frames), but an obvious bug in Xircom's PE3 packet driver.


and done by every other protocol

That's a very "bold" claim :-) - which happens to be untrue. While it holds true for IP and IPX, these weren't designed to be transmitted exclusively over Ethernet, hence they had to be extremely cautious. There are Ethernet-based protocols out there that rely on Ethernet actually doing its job. ATAoE comes to mind, as well a Fiber Channel (FCoE), iSCSI, Spanning Tree and WoL, and surely these are not the only ones.


and [2] not to bother with any error checking. It's good that both of these are corrected now.

I understand you are referring to the lack of checksum again. I am still convinced that this is overkill, and that's the reason why I made it optional in the 0.8.1 implementation of EtherDFS. The Ethernet CRC is already very strong and does its job perfectly. If a bit flip happens because of klunky hardware elsewhere than on the wire itself then, well, that's klunky hardware. A checksum will detect only a small subset of things that could go wrong, and hardware can fail in any place: DRAM memory, bad disk sector, ISA bus, LPT port, etc. I can't save them all, and I simply do not like wasting CPU cycles on things that are already done very efficiently in hardware.

At the end of the day, we are probably arguing for nothing - if one thing is certain, it's that it's much easier to make new software work around pre-existing bugs than to fix 30-years old proprietary code. Implementing an optional checksum + length check in EtherDFS was undoubtedly a necessary thing to do. You may call it a bug fix, I will keep calling it a new feature.

mbbrutman
April 17th, 2017, 05:06 AM
Packet Driver != Ethernet

The packet driver specification is not the same as the Ethernet specification. Ethernet is a physical access medium. The Packet driver is a software specification. Packet drivers exist for Ethernet, Token Ring, and serial. Packet drivers exist to provide a common programming API to the underlying hardware. They do not have any control over how the physical layer does framing. You can't blame the Xircom device or the packet driver for an extra pad byte that was within their right to add.


Ethernet 802.3: The length of the payload needs to be in bytes 13 and 14. As per the specification.


Ethernet II: Bytes 13 and 14 encode the type of the packet. You, the implementer of the protocol, and supposed to encode the length of your payload in the payload of your packet unless you can infer it, like for example, fixed length packets. Your packets are not fixed in length. You can not encode the length of your payload in the payload at your own peril.


It was a bug to assume that the recv( ) syscall was giving you your payload length. Sorry, people get bitten by strange implementations all of the time. Padding frames is perfectly normal to expect in communications physical layer implementations. It was also a bug to design a new protocol that didn't encode the length in the payload.


No existing software had a problem with the Xircom. It runs TCP/IP from Novell, Microsoft, WatTCP, etc. It runs other protocols as well. EtherDFS is the first program/protocol to trip on it, and it's clearly due to your assumptions about the way Ethernet framing and packet drivers are supposed to behave, which are clearly wrong.

At this point I'll send you my other observations about possible problems in the code in private, as we're just talking in circles here.

Pepinno
April 17th, 2017, 11:52 AM
Black cat or white cat, what matters is that it caches mice.

Or some such thing.

Thank you to everyone: the developer, the beta testers, the debuggers, and the forum site itself for hosting all this!

Trixter
April 17th, 2017, 07:12 PM
I'd like to point out that the original bug I reported and demonstrated in the video I provided (not handling an additional byte properly) was not limited to the Xircom device. It also showed up on the Intel Etherexpress 8/16 ISA card I have. I am very busy this week and cannot test the new code until the weekend, but I will test only on the Intel hardware from now on to eliminate the argument that a specific device or driver was at fault.

I've written a few drivers. When given the option of using the length of the received data, or the length that was specified in a header, you should always use the length specified in the header. I'm not sure about modern hardware practices, but in the 80s and 90s it was common for hardware to deliver the contents of the full register used (AX/EAX) despite 1-3 bytes of that not being necessary; you used the data specified by the protocol, not the data you actually got.

keenerb
May 2nd, 2017, 05:52 AM
Just as an update, this has worked really well for me.

Something odd is still going on with a FEW apps when executed directly from the mapped drive, mostly very advanced apps like Geoworks Ensemble, but I haven't had any time to track down precisely what's going on there...

trmg
May 2nd, 2017, 08:53 AM
Is there any reason this wouldn't work with newer machines? Say 386/486 class systems running MS-DOS 6.22?

keenerb
May 2nd, 2017, 09:02 AM
Is there any reason this wouldn't work with newer machines? Say 386/486 class systems running MS-DOS 6.22?

It works fine on my 386SX-20 machine. I get much better performance, in fact.

trmg
May 2nd, 2017, 09:12 AM
It works fine on my 386SX-20 machine. I get much better performance, in fact.

Awesome. I will definitely need to give this a try. Today when I need to copy files to/from these systems I run Windows (3.11), do the copy, then exit Windows. It works, but this would be much nicer.

trmg
May 6th, 2017, 11:53 PM
So, I'm giving EtherDFS a spin and am running into an issue. The server is running on my Fedora 25 box. The client is running on my 486 using an Intel PRO/100 NIC. I was able to successfully mount the drive on the 486. However, whenever I try to interact with said drive I get the error "file not found".

If I do "mkdir test", I get the error "File not found - TEST". However, if I look at the DOSDRIVE on the Fedora box I see that the directory was in fact created.

If I do "edit g:\test.txt" (I have the drive mounted as G:) and then try to save the file, edit throws the error "File not found." However, if I look at the DOSDRIVE on the Fedora box, I see a zero byte file named "test.txt"

If it makes any difference, EtherDFS would not auto detect the server. I had to specify the MAC address of my Fedora box manually.

I'm sure I'm missing something. Any ideas?

keenerb
May 8th, 2017, 09:31 AM
So, I'm giving EtherDFS a spin and am running into an issue. The server is running on my Fedora 25 box. The client is running on my 486 using an Intel PRO/100 NIC. I was able to successfully mount the drive on the 486. However, whenever I try to interact with said drive I get the error "file not found".

If I do "mkdir test", I get the error "File not found - TEST". However, if I look at the DOSDRIVE on the Fedora box I see that the directory was in fact created.

If I do "edit g:\test.txt" (I have the drive mounted as G:) and then try to save the file, edit throws the error "File not found." However, if I look at the DOSDRIVE on the Fedora box, I see a zero byte file named "test.txt"

If it makes any difference, EtherDFS would not auto detect the server. I had to specify the MAC address of my Fedora box manually.

I'm sure I'm missing something. Any ideas?

I had this problem with an older version. Make sure you're running the latest client AND server version.

I'm using a FAT back-end volume, so maybe that's causing a problem still if you're not. I'll test that this evening...

trmg
May 8th, 2017, 01:10 PM
I took the lazy route and used the pre-compiled RPM as linked on the EtherDFS site. I guess I'll need to compile it myself and give it another go. :D

Shadow Lord
December 20th, 2017, 08:13 AM
Ok. I read through the whole thread and I understand that the author is not going to write a Windows server (a DOS server may be forthcoming at some point). So any brave souls out there ported this to Windows 32bit or 64bit code yet? While at some point I'd like to play with Linux I rather not be learning Linux and working with new SW at the same time. Plus spending all the time learning Linux would seriously take time away from just using a simple program to get it done.... :D

GrizzlyAdams
December 20th, 2017, 08:54 AM
Ok. I read through the whole thread and I understand that the author is not going to write a Windows server (a DOS server may be forthcoming at some point).

Reason being that you would need something like libpcap or write your own protocol handler (which would need to be signed) to gain access to the layer of the networking stack you need. For most people, installing a virtual machine or Pi with EtherDFS and Samba is easier and faster. You could actually make a DOS server easier than a windows port of the linux server, from my experience with the windows platform sdk.

alank2
December 20th, 2017, 09:17 AM
Wireshark uses winpcap/libpcap. I too would love to see a windows server for this. I'm not sure how difficult it would be to port.

pearce_jj
December 29th, 2017, 07:44 AM
I still don’t understand why we don’t use ATAoE for this...