[kwlug-disc] 16GB flash-memory pen transfer speeds up then corrupts files

[Bob Jonkman]

Are you copying a single file that's greater than 4 GiBytes?  That will cause corruption 
on 32-bit file systems (FAT32 for sure, ext3 maybe) since the directory entry for file 
size is only a 32-bit word.  Windows needs NTFS to support files larger than 4 GiBytes.

If you're copying multiple small files or folders totalling greater than 4 GiBytes then I 
don't know what goes on.

-- -- -- --
Bob Jonkman <bjonkman at sobac.com>         http://sobac.com/sobac/
SOBAC Microcomputer Services              Voice: +1-519-669-0388
6 James Street, Elmira ON  Canada  N3B 1L5  Cel: +1-519-635-9413
Software   ---   Office & Business Automation   ---   Consulting



On 10 Dec 2008 at 18:27, Rick wrote:

>I got this 16GB flash-memory pen-drive from eBay
>
>   http://cgi.ebay.ca/ws/eBayISAPI.dll?ViewItem&rd=1&item=350126058238
>   &ssPageName=STRK:MEWN:IT&ih=022
>
>and I have been trying to use Linux to copy files. At first, all
>seemed well.  But, then, I did some hard testing by copying over 5GB
>to 15GB all at one go. When I do larger copying, I always get file
>corruption at some point during the copy. I have tried the following
>partitioning:
>
>   (1) with /dev/sdX1 as Linux ext2
>   (2) with /dev/sdX1 as FAT32
>   (3) with /dev/sdX as a single FAT32 "superfloppy"
>
>By the way, the "superfloppy partitioning" is when the memory-card or
>pen-drive doesn't have any partitions. The entire media is a single
>partition. You format with:
>   # mkfs.vfat -F 32 -I /dev/sdX
>      ... note the "-I" argument
>and you mount with:
>   # mount -t vfat /dev/sdX /mnt/...
>Notice that there is _no_ partition table number.
>
>The superfloppy setting was how the pendrive had arrived and, then,
>"fdisk -l /dev/sdX" spits out a weird partition table (which, based on
>my readings, is acceptable). In fact, I have a new "MP4 player" device
>which also reports a similarly weird partition table.
>
>I'm not sure if this file corruption is from having older drivers or
>if this pen-drive is broken.
>
>I read that I should set /sys/block/sd.../device/max_sectors down from
>240 to, say, 32. That setting helped and, once, I managed to get 12GB
>copied. (Or, at least, "du /mnt/..."  says that 12GB got copied and
>the "ls" didn't show obviously corrupt files.)
>
>I reformatted this 16GB pen-drive from a WinXP machine. Then, while
>still on WinXP, I tried copying a 5GB directory. But, again, I got
>corruption ... this time, after only around 2GB of copying.
>
>Now here is the weird part. Both on Linux and on WinXP, copying
>proceeds at a slow rate but, then, suddenly speeds up. That is, WinXP
>said that it would take 40 minutes to copy everything and, halfway
>through, the remaining time dropped from under 30 minutes to just 4
>minutes ... which, I believe, is when corruption started happening.
>
>As for Linux, I believe that the transfer rate suddenly sped up
>because I was monitoring disk i/o with the "gkrellm" system-monitoring
>tool. Paralleling what happen in WinXP, the "gkrellm" graph shows a
>sudden, sharp jump in the transfer rate partway through the copying.
>
>I'm not sure if the WinXP has the latest drivers for pen-drives
>(or do drivers remain unchanged at the 16GB level for flash-memory
>pen-drives?) Also, perhaps my version of Linux might not have the
>newest drivers.
>
>I'm guessing that I most like have a broken pen-drive ... but the eBay
>seller has hundreds of buyers and no negative reviews (at least in the
>time frame that eBay is reporting which, I've read, is 30 days, down
>from 90 day).
>
>Also, I did this formatting operation at one point:
>   # mkfs.vfat -c -v -n 16gPen -F 32 -I /dev/sdj
>Note the "-c" flag ... which will "Check the device for bad blocks
>before creating the file system". (I assume that "badblocks" is being
>run.) And "mkfs.vfat" did not report any errors.
>
>Any opinions on the state of the pen-drive ... or, better yet,
>comments on what I'm doing wrong? Maybe my version of Linux is lacking
>something to properly handle 16GB of flash memory? If so, maybe my
>initial efforts on Linux have changed formatting in the flash-memory
>and WinXP isn't smart enough to fully reformat what Linux did?
>
>_______________________________________________
>kwlug-disc_kwlug.org mailing list
>kwlug-disc_kwlug.org at kwlug.org
>http://astoria.ccjclearline.com/mailman/listinfo/kwlug-disc_kwlug.org