Pool file duplication causing file corruption under certain circumstances

[Catch-22]

I am using StableBit DrivePool and noticed that I am getting file corruption that is somewhat reproducible. I particularly noticed this with FLAC files as I was attempting to verify my library of music using flac -t which checks the MD5 signature of the decoded WAV file.

My setup is DrivePool with Folder Duplication enabled on specific folders, "Bypass filesystem Filters" checked, "Read striping" checked and "Real-time duplication checked".

Note that it appears to be "Read striping" that is the culprit for this but I am not 100% sure. Particularly concerning to me is that this happens even with "Verify after copy" checked.

Steps for me to reproduce are to download a FLAC to a DrivePool location that matches the above parameters, verify it with flac -t (ensure that it verifies OK), copy the file to a different location (doesn't even have to be a DrivePool location), verify this copied file with flac -t and see that the file does not verify anymore.

Checking with a hex editor, I can see it's not even just 1 byte difference, usually something like the first 32kb is fine, then I get random jumbled up data for a 128kb or so then the remainder of the file is correct (and the file size is correct).

Here's something I would never expect to see in a working filesystem:

PS F:\test\> flac -t .\test.flac

flac 1.3.2
Copyright (C) 2000-2009  Josh Coalson, 2011-2016  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

test.flac: ok
PS F:\test\> copy .\test.flac test2.flac
PS F:\test\> flac -t .\test2.flac

flac 1.3.2
Copyright (C) 2000-2009  Josh Coalson, 2011-2016  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

test2.flac: *** Got error code 2:FLAC__STREAM_DECODER_ERROR_STATUS_FRAME_CRC_MISMATCH


test2.flac: ERROR while decoding data
            state = FLAC__STREAM_DECODER_ABORTED

Edit:

It seems I am not the only one running into this problem, see this recent Reddit thread:

 

[GoreMaker]

100%

Once I disabled it, all my data corruption issues vanished. I haven't turned it back on since. It was a nightmare restoring all my backups and re-doing days of work from scratch.

[AtariBaby]

thanks guys i disabled it

[Catch-22]

Circling back, I don't believe this is fixed. I have left it turned off for a while. I agree, the lack of urgency on this is alarming, corruption should set off serious alarm bells for the maintainers and the fact that it has been nearly 5 years since I gave reproducible instructions on this (and that it had to be discovered by a user in the first place) and we still have no response is disappointing.

I still have my DrivePool but I am slowly migrating away from it (it's prohibitively expensive for me to replace everything at once).

[baChewie]

We really need something from the devs here:
- Closure on how/why this is happening
- Assurance that new versions of Drivepool will have read-striping disabled by default
- Since it's an attractive function, and data corruption with it enabled has been confirmed, the hover/tool-tip must include a very stringent warning that it is 'alpha/beta', "at your own risk" and should not be used in production or for any data that is not 1:1 replicated/backed-up elsewhere
- The hover/tooltip is not enough, when the user attempts to enable it, a serious warning must be shown with similar language as the above ^
- Confirmation that the issue is being actively investigated, included in both the aforementioned tool-tip and pop-up warning

I love the product, but this shakes my confidence.

[GoreMaker]

💯

[Christopher (Drashna)]

I know there have been a few tickets about this.  Unfortunately, neither Alex nor I have had any luck at reproducing the issues locally. 

If you haven't already, open a ticket at https://stablebit.com/contact

If you have previously opened a ticket about this, then if you could, open a new ticket.

Getting drive tracing from when this is happening would be very helpful, and can be done when opening the ticket (eg grab the logs, upload and attach during ticket creation): 
https://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

And afterwards, run the StableBit Troubleshooter.

And as much information about the files that this is most commonly happening on.  And exact steps to reproduce the issue, if possible. 

Because issues like this, when we can't reproduce them, are much more difficult to track down.  Let alone fix. 

[GoreMaker]

I'm not inclined to go through the process again just for the sake of testing. I don't have the spare hardware or the time to test for it. I can create a ticket, but I will not follow up on it afterwards.

Here's a description of what I experienced, which is what I'll include in my ticket, but I feel it should be publicly viewable.

I was using a read-striped duplicated pool (as outlined in my post here) to host a vast quantity of DNG photo files and Adobe Lightroom catalogs. Thousands upon thousands of photos from countless events spanning many years, all distributed across dozens of catalogs. I was in the process of editing hundreds of photos from a recent event, which initially involved flagging all the keepers and then "Enhancing" those raw photo files for noise reduction. "Enhanced Denoise" is a VERY resource-intensive AI feature in current versions of Lightroom Classic that creates brand new raw files from existing ones, typically hammering the GPU at 100% the entire time. This also generates a ton of disk I/O since these raw files are pretty large and brand new, bigger ones are being generated. I do these in large batches, typically dozens at a time, then I walk away and find something else to do for a few minutes because the computer becomes very slow to respond during the process, even with a Core Ultra 7, 64GB of fast DDR5 RAM, an RTX 4070 Super and very fast NVME SSDs.

The result was that at least 10% of the newly created raw files were corrupt, all very randomly. No error message was ever generated, Lightroom simply completed the process "successfully", but when I tried to edit the photos, Lightroom refused to load the corrupt ones, displaying only the embedded thumbnail. It would give me an error message at this point stating "This file is corrupt" [not those actual words]. Initially I thought Lightroom was at fault, so I spent a lot of time finding the corrupt files (they're not identified until I try to edit them), deleting the corrupt version, and re-enhancing the original. Again, I got about a 90% success rate, with 10% of those being corrupt. It never occurred to me that DrivePool might be at fault, I assumed it was either my GPU or Lightroom that was messing up. I had a deadline for these photos so I just powered through it and got my work done.

When I read on this forum about this potential corruption issue, I turned off read striping on DrivePool and restored a recent backup of all my files to overwrite everything on the pool. Then I tried "Enhancing" a batch of 500 photos. There was zero corruption. All the files were generated successfully without any problems. Everything was otherwise exactly the same: hardware, software versions, drivers, etc. It was the exact same Windows 11 installation, just with read striping disabled. Of course, all my hours of work for the recent photo event was lost since I needed to restore a backup that preceded the use of DrivePool. Luckily I haven't needed to create new prints of those photos since the project was completed.

I've since reconfigured my system and no longer use DrivePool at all. The file corruption left a sour taste and I no longer have faith in the product. This file corruption issue wasted many hours of my time and made the job much more stressful unnecessarily. It's entirely possible that there were other corruption issues in the rest of the filesystem that I wasn't aware of. I can't be taking that risk with my data. I now run an Intel RSE array with RealTimeSync to a separate volume, instead. I'm not pursuing a refund for the product, it's not worth my time and effort, but I also have no plans to use the product again.

[number_one]

I also don't really have the bandwidth to set up a test at the moment, but my issues were easily reproducible by using rsync through a Git for Windows bash environment.  I was using rsync to copy files from a DrivePool volume to a remote machine via SSH.  Granted that's not a super common scenario, but it's also not at all rare for developers and/or those who also use Linux and/or Mac environments.

I don't doubt that this is a tricky issue to debug, but with all the scenarios detailed I really find it hard to believe that no one on the development team can reproduce the issue at all.

[moskeale]

I just experienced file corruption on a video file while I was working in After Effects.  One minute it was working and then it was hanging up the program.  Took a half hour of troubleshooting to figure out which file was causing the hangup, and then upon trying to open that file in other programs realized it was completely corrupt and unreadable.  Luckily it was a file I could easily re-download, which isn't the case for most of my files.  This could be catastrophic for my work, I rely on Drivepool to ensure the integrity and safety of my data.

Not sure how to reproduce such a random event, and I don't want to, given the potential for damage.  I've disabled read-striping in fear of this.  Hopefully I don't find any other affected files.  Let me know if I can provide any logs or data that might be helpful.

[MitchC]

To reproduce this for them I would use a program that can hash all the files it sounds like you could just run it a few times on a striped drive and cause the issue by comparing the results.

 

Here is powershell script that can do it for you: https://gist.github.com/mitchcapper/e2a20ac82d5e7da69deee2eeda7a65a4

Run it with two scans where D is your drivepool striped drive first IE:

.\hash.ps1 scan -FolderPath "d:\" -OutputFilePath "C:\Temp\DrivePool_$(Get-Date -Format 'yyyyMMdd').md5"

Then:

.\hash.ps1 compare -File1Path "C:\Temp\DrivePool_20230101.md5" -File2Path "C:\Temp\DrivePool_MD5_20230115.md5"

If you get any changes that were not expected you likely had a read corruption issue.  It sounds like this should be pretty reproducible, especially with larger files.   Once you have it then capture w/e logs covecube wants.

Also note, script has AI generated parts and I only did mild testing and review.

[AtariBaby]

Any chance this was fixed in the last update?

[RJGNOW]

I just ran into "this" problem, not quite to the extent of corruption (at least none that I've found yet) but definitely causing problems with Plex streaming of audio files (pauses, skips, all sorts of issues). Turning off "Read Striping" seems to have corrected the problems. 

[AK96SS]

Lordy, Lordy.

I've spent days trying to figure out what the heck was happening with my .mkv files. For over a month.

I thought I had a bad drive (or three), even though Stablebit scanner reports no issues. Then I thought I may have network latency issues, so I went from Windows Explorer moving files across the network to using TeraCopy and performing file integrity checks. Removed the latest drives (since the problem seemed to appear after I added them). I thought that had resolved it, but then...

I *just* ripped a blu-ray, moved it to the NAS, then went to watch an episode and it froze.

I've got a lot of disks that I ripped over the last decade - to the tune of over 4000 movies (with extras) and over 600 TV shows. Yes, I still have a lot of the original media (but I've sold off a lot of it as well) -  but the HOURS of time spent ripping, sorting media files, identifying extras, chapter titles, tagging everything...  then I open a file to watch an episode and it freezes on me.

When I go back to the source file, a newly ripped .mkv reports "No valid Matroska level 1 element found" - even though I used TeraCopy to perform a hash comparison of the file when it was written to the NAS. Re-rip, recopy, same thing...  and "Read Striping" was enabled.

I can't even begin to imagine how much work I have ahead of me - first in identifying what is bad, then in restoring it. Not to mention that for some of the older stuff, I do NOT have the original media anymore. None of this is torrented - I did it all manually (I know, but it's a thing for me).

My wife was pleased that I was finally going to be boxing up the DVD/Blu-Ray collection and getting it out of the living room... I can't even begin to express my displeasure. I've been using DrivePool for years, recommending it to everyone who will listen (and cares) and now here we are.

[Shane]

1 hour ago, AK96SS said:

When I go back to the source file, a newly ripped .mkv reports "No valid Matroska level 1 element found" - even though I used TeraCopy to perform a hash comparison of the file when it was written to the NAS. Re-rip, recopy, same thing...  and "Read Striping" was enabled.

Just to explicitly confirm, you did then disable Read Striping and further attempts to re-rip and recopy have all been successful since?

[AK96SS]

So far, that is the case.

I disabled Read Striping and am in the process of re-ripping/recopying, and the first ones appear to have been successful.

I rip everything on the desktop, since it is faster to come off the blu-ray onto an NVME drive, then copy it across the network to the NAS. On the NAS it is written to the SSD drives, then overnight it will be parceled out to the spinners.

As I said, so far, it appears to be working - I am going to force the balancing to move it off the SSDs now.

[AK96SS]

37 minutes ago, AK96SS said:

So far, that is the case.

I disabled Read Striping and am in the process of re-ripping/recopying, and the first ones appear to have been successful.

I rip everything on the desktop, since it is faster to come off the blu-ray onto an NVME drive, then copy it across the network to the NAS. On the NAS it is written to the SSD drives, then overnight it will be parceled out to the spinners.

As I said, so far, it appears to be working - I am going to force the balancing to move it off the SSDs now.

OK, balanced (and off the SSDs) and it appears to be working so far... knock on wood.