Will Deduplication Repair Read Errors?

[fjih]

Apologies if this has been answered, but I couldn't find it in the forums or manuals. The closest question I found was this one, but it has more to do with corrupted but readable contents.

If I have two drives in a 2x duplication setup and a read of one of the drives results in an unrecoverable read error, will DrivePool repair the damage? In other RAID type setups, if a URE is encountered the failed read can be rebuilt from parity and written back to the failed drive.

What steps does DrivePool take when encountering a read error of a duplicated drive?

[VapechiK]

hello

 

see here:

 

https://community.covecube.com/index.php?/topic/15289-drivepool-logic-when-2x-pool-file-duplication-goes-out-of-sync/#comment-48403

whenever there is a data or file  'compare'  error, DP will pop-up a box notifying you of it, and ask what you wish to do about it.

 

cheers

 

[fjih]

Thanks VapechiK, but I'm looking at a slightly different question. I understand that DrivePool might not know what to do in case the contents of the two copies differ, but I'm wondering what it does when one of the copies is completely unreadable and produces a read error from the drive.

[VapechiK]

hello again lol

 

same difference, as far as i can see.  the files may just differ slightly, as in a recently modified document or whatever, or they could be media files, one of which is corrupt and unplayable.  it will pop-up a box, and if you follow the path(s) listed to the file(s), you can attempt to play (execute) both files and see which one is corrupt.  as far as 'fixing' the corrupt file, i would say that's a hard NO.  if i remember correctly, all it does is ask if you want to keep the most recent version of the file in question, or fix it yourself.  if you choose the most recent version, and that's the one that's corrupt, oh well i guess you're SOL.

 

cheers

 

Edited January 8, 2024 by VapechiK
edit for clarity

[fjih]

The difference is in determining the source of truth for the file.

If both copies are readable and differ it's difficult to determine which file is correct. One might want to assume that the file with the newer modification time is more recent and therefore the correct one, but this is a dangerous assumption (see the reference post in my original post where Rob had to chkdsk a drive resulting in a file with a newer modification time but was 0kb). Doing the wrong operation here will result in data loss, and so it's in DrivePool's best interest to pose the question to the user and have them manually resolve it.

In the case where one of the drives flat out fails to read the file the situation is different. The safest course of action is to again ask the user what to do, but this is not what other RAID type of systems do. If all the other drives do not produce a read error of the file we might assume that their copy of the data is correct. If the other drives' view of the file is correct, we can "heal" the bad drive's copy by writing the file back from the good drives back to the bad drive.

Many parity based RAID systems take this approach. This silently handles a bad sector on the drive and we can continue operating without having to involve the user. The window of data loss is mostly closed; if we write the contents of the readable disk over to the unreadable disk, no extra data was lost other than what was already lost from the bad drive.

Whether or not this is the correct course of action is debatable. My question is what DrivePool is implemented to do in this scenario. It may be that they do as you say and always leave it as a manual operation for the user to resolve. However, DrivePool could match what other parity RAID systems do and silently repair instead.

[VapechiK]

 

StableBit DrivePool is NOT a RAID solution/implementation.  AFAIK, RAID is block based, while DrivePool has always been and always will be file based.  therefor, NO silent automatic repair of the file(s) in question.  sorry if that's not what you want to hear, but thems the facts lol...  you can have 4 identical drives, pair them off as 2 RAID 1 arrays, then add them both to the pool and enable duplication on them.  as soon as 2 (or more) duplicated files differ for whatever reason, you are still gonna get the pop-up box asking how you want to proceed.  perhaps when/if Shane replies to your question, they will have more to add.

 

cheers 

Edited January 9, 2024 by VapechiK
pc :)

[Shane]

VapechiK is correct; StableBit's DrivePool is not a parity RAID system, it will not "self-heal" a damaged file.

StableBit's Scanner is able to detect and attempt to repair damaged files, and if you have that plus DrivePool's duplication you can manually replace a non-repairable file with its good duplicate when alerted.

Some users combine DrivePool with SnapRAID to get parity healing capability (albeit not fully automated).

As VapechiK indicates, you can also pool sets of RAID volumes to let those provide duplication/parity.

[fjih]

Thanks for the clarification.

Just wanted to point out that I'm not asking for an implementation. I'm just looking for clarification of what is implemented. I understand DrivePool is not RAID. I'm making a statement that other redundant filesystems do offer healing and that DrivePool could implement a version of it.