I always thought there was a cluster chain for it to still follow somehow, but I guess this isn't the case.
Basically did it check the starting cluster and the size of the file against the FAT and make an assumption that the file was contiguous? So if you delete a fragmented file you could really forget about getting it back?
If so, (I don't think it is). The second FAT was intended as backup, as it holds the file structure. You can lose the root directory and still recover the contents if the FAT is intact, but not the other way around.
DOS 5 and later did undelete tracking by using a special file, PCTRACKR.DEL. It can hold a certain (configurable) number of entries. DR-DOS took a similar approach, but also included a FAT snapshot utilitiy, called DISKMAP.
Yup, that was a big drawback of using FAT vs. CP/M's "flat" scheme.