NOTE 2: VERY IMPORTANT - I do NOT know if this bug exists in Gentoox as I have not been able to test it yet. It might, hence this is FYI.
The Background History
I am working on a fatx utility (it's a long way off), so I've been spending time learning all things FATX.
I wanted to make a full copy of my xbox HD, so I booted xebian on a PC with both the old and new HD installed. Both drives have been formatted in the XBOX. I mounted both old and new F drives ala:
Code: Select all
mkdir /mnt/f1 mkdir /mnt/f2 mount /dev/hda55 /mnt/f1 mount /dev/hdc55 /mnt/f2
Code: Select all
cp -a /mnt/f1 /mnt/f2
Code: Select all
md5deep /mnt/f1 > md5f1.txt md5deep /mnt/f2 > md5f2.txt
But it wasn't the contents of the files that were different (all the MD5 checksums were a match). It was that there were files on F1 not on F2 and similarly, there where files on F2 not on F1.
I'll now cut to the meat of the situation
There appears to be a bug in the linux 2.4.xx FATX driver that causes DIRECTORY SEARCHES to every once in a while MISS A FILE COMPLETELY. It just doesn't show up in the search. So doing stuff like "ls -lR" or using "find" or "tar -c" or anything that can recurse through the directory tree can MISS FILES. The file is there, but the search doesn't find it. If you access the file IMPLICITLY BY NAME, it will work.
Look at the following session as an example.
Code: Select all
> ls -la premiere_* -rwxr-xr-x 1 root root 2743 Sep 21 07:08 premiere_austria.png -rwxr-xr-x 1 root root 2220 Sep 21 07:08 premiere_direkt.png -rwxr-xr-x 1 root root 3731 Sep 21 07:08 premiere_erotik.png -rwxr-xr-x 1 root root 2450 Sep 21 07:08 premiere_filmclassics.png -rwxr-xr-x 1 root root 2941 Sep 21 07:08 premiere_filmfest.png -rwxr-xr-x 1 root root 1851 Sep 21 07:08 premiere_krimi.png -rwxr-xr-x 1 root root 3395 Sep 21 07:08 premiere_music_studio.png -rwxr-xr-x 1 root root 2434 Sep 21 07:08 premiere_nostalgie.png -rwxr-xr-x 1 root root 2142 Sep 21 07:08 premiere_serie.png -rwxr-xr-x 1 root root 1906 Sep 21 07:08 premiere_sport_portal.png -rwxr-xr-x 1 root root 1446 Sep 21 07:08 premiere_win.png > md5sum premiere_win.png 59119ee9070ed5e6cc8cc4279f89e729 premiere_win.png > md5sum premiere_doesnotexist.png md5sum: premiere_doesnotexist.png: No such file or directory > md5sum premiere_start.png ac303faf26142cd70e97973ee539f6c3 premiere_start.png
If I was to copy this folder to another folder using "cp -a" then this file would not get copied because it doesn't show up on directory searches.
Further investigation indicated that the FATX driver seems to MISS A FILE about every 400 files or so. So if you have a directory < 400 files (give or take) you'll be fine. For every directory with > 400 files, a directory search will miss a file about every 400 files.
With this in mind, I examined ALL the missing files I had from my big copy of F1 to F2. First I sorted all my directories by number of files in each directory. Then I found I was missing 31 files. I compared which files where missing from which directories, and the correlation was that I was missing files from ALL directories that had ~ > 400 files. Furthermore, I missed 2 files from directories that had ~ > 800 files. One big folder had 2969 files in it, and that folder was missing 6 files. There were NO folders with > 400 files that was NOT missing files. That means that ANY folder with > 400 files would start missing files.
I can reproduce this problem with xebian and with the XBOXHDM boot disc. I do not know if this fatx problem exists in other distros such as gentoox or xdl nor do I know if this problem exists with some of the kernel 2.6.xx fatx drivers. Other distros may or may not have this problem - I have yet to test them.
What does this mean?
Well, if you use something like XBOXHDM or xebian boot discs to do fatx copying you MAY LOSE DATA because files you thought you copied were missing and not found therefore they will not get copied. If you don't check, you could lose data if you deleted the source data/folders after the copy/backup.
If you use only xbox linux to list files, it may not be readily apparent if you are missing files because you won't know what is missing. You would need to boot the xbox with something like Evox, avalaunch, unleashx, or XBMC and then FTP into your xbox and use the ftp client to see what files SHOULD be there in large directories. Pick a large folder ( > 400 files - the bigger the better), and do an "ls -lR" from linux and save the listing. Then boot your favorite dashboard (not linux) and use FTP to get a listing of the same large folder and save the listing. Now compare the results and see if you get the EXACT SAME NUMBER OF FILES. If not, then your linux fatx driver has this bug.
I've looked through the FATX driver source code, but I have yet to find the "smoking gun". Something isn't working right on large directory searches but it may take some time to figure out where the bug is in the fatx driver code.
Basically this is a "HEADS UP" issue and it would be nice if someone else could reproduce this bug just to make sure I haven't messed up in some strange way. I have tested 3 different xbox HDs and they ALL exhibit the same issue on large directories (using xebian boot CD or XBOXHDM boot CD).