worrying issue with pictures etc

[AMCross]

hi all

 

 

not sure if its drivepool related but I am having issue with some pictures stored having the following effect as per attached

 

looks like pixels have moved

 

 

now im not sure how drivepool works but can it be the cause ?

running latest r c and latest scanner beta

[Christopher (Drashna)]

In theory, yes, it could, but it's very very very unlikely. We do a LOT to make sure that data is written correctly, and in sync between disks, to prevent this from happening.

 

Specifically, any "IO" sent to the pool is redirected to the disks in the pool instead. So like I said, it is *possible*....

 

 

 

The problem is that it's very hard to tell after the fact. If you're able to reliably reproduce this issue, that would be one thing (and if so, please enable file system logging/tracing so we can capture the issue)...

 

Also, if you access the file directly from the "PoolParts" folder, does it have the same corruption? If it doesn't, and it is corrupted only when accessing it from the pool, and especially if duplication is enabled, then the pool could be at fault.

Also, is this over the network or locally?

[AMCross]

checked over the network

 

all folders for photos and video are duplicated

 

 

reproducing the issue maybe an problem

 

 

just noticed pictures from last years holiday which I know 100% were fine weeks ago are now like this

 

will check poolparts folder

[Christopher (Drashna)]

If that's the case, and you have StableBit Scanner installed, I would recommend forcing it to scan the disks (click on the "+" next to the disk, and find the button with the green circle and the yellow circular arrow in it, and click that. Then select "mark all readable as unchecked")

 

And yeah, I know that reproducing the issue may be a problem. As I said, it's hard to troubleshoot after the fact.

Though, during the last week, was there any unexpected reboots or BSODs, or other disks issues?

[dbailey75]

This is concern I've had for the last couple of years,  Mentioned it a few times here and on other forums as well, many reasons why file(S) could maybe become corrupt, bad sectors, bit rot, HDD failure, hardware failure, power outages while DP is balancing/moving data, etc,  luckily you've identified your images, and I hope that you can restore them from a backup.  But when you have thousands of photos that you may not view for some years, you need a way of tracking their integrity and maintain a inventory of your files.    While I did purchase the integrity checked for WHS, it's not quite what I was looking for, I was able to do something a bit manual with a tool called exactfile,  you can run an inventory of all your files and create a digest,  I'm sure some one smarter than me can add some automation to this process, but you can run a digest check monthly to ensure the hash has not changed on your files and no files have gone missing, as you add new files, create a new monthly hash.  So far it seems to work, as you get a few pesky .tb files that have changed from time to time and fail the verification, and I've done test were I delete files, and they were flagged in the verification.  So doing this monthly, you have a chance to restore the file from a backup, if you never knew there was an issue, the backup would get replaced by the curpt file, or the missing file, may go unnoticed as old backups are purged.

 

Exactfile is a great tool, would be awesome if this tool or similar could be intergraded with Drivepool/scanner with more automated features.

[Umfriend]

I second that suggestion by dbailey75. Scanner already performs a scan every month and reads the entire disk (I know, sector based, not file based) but I'd love a second whole disk read monthly to ensure no files have changed that have not been saved in the past month. It's fine that I have backups but retention policy only keeps backups for so long.

[Christopher (Drashna)]

..... I run the surface scan every 7 days..... personally..

 

 

@dbailey,  that's a very good suggestion and a good link.

And if you don't mind, what did you not like about the WHS Integrity Checker add-in?

 

@everyone:

Backups. If you know me from WGS/HSS, then you've probably heard me say it before, but it's always worth repeating: There is no such thing as too many backups. If the data is important, you should have it stored in multiple mediums, and multiple locations. The HSS guys suggest a "3-2-1" backup strategy, in fact. 

3 Backup copies of anything you want to keep. 

2 different storage media. 

1 offsite storage site.

 

 

This helps prevent any significant data from being lost or corrupted.

[dbailey75]

..... I run the surface scan every 7 days..... personally..

 

 

@dbailey,  that's a very good suggestion and a good link.

And if you don't mind, what did you not like about the WHS Integrity Checker add-in?

 

@everyone:

Backups. If you know me from WGS/HSS, then you've probably heard me say it before, but it's always worth repeating: There is no such thing as too many backups. If the data is important, you should have it stored in multiple mediums, and multiple locations. The HSS guys suggest a "3-2-1" backup strategy, in fact. 

3 Backup copies of anything you want to keep. 

2 different storage media. 

1 offsite storage site.

 

 

This helps prevent any significant data from being lost or corrupted.

The biggest issue with integrity checker is you can't schedule verification checks,  they went through the effort of making this add in, but left out this very import feature.  it will add the a hash to files automatically as they are added to the watch folders, but there needs to be a way to schedule the verification, monthly. or as often as one would like.  The other issue is that you can't export a list of your files, "an" inventory so to speak, so you have a report of all your files, file name, directory path, and hash,  I can pull a report from exact file to get the inventory, but I don't believe their report includes the hash either.  I was playing around with another program called FileVerfier++, which did a better job on the inventory report, but I settled on Exact file for its ease of use.  I hope between exact file, and integrity checker, I can identity missing or corrupt files while I still have a chance to restore from backup.

 

And now that I think about it, I believe I submitted this as a feature request some time last year to stablebit.  Hum, can you check on that?  I'm not sure how.

[Christopher (Drashna)]

Well, I've flagged the request for alex (if it's again, that's fine)

https://stablebit.com/Admin/IssueAnalysis/5243

 

And I was talking with Alex about it again. I do try to keep pushing for it, because it is a great idea. Hopefully, Alex will decide to whip something up.

[Umfriend]

..... I run the surface scan every 7 days..... personally..

 

 

@dbailey,  that's a very good suggestion and a good link.

And if you don't mind, what did you not like about the WHS Integrity Checker add-in?

 

@everyone:

Backups. If you know me from WGS/HSS, then you've probably heard me say it before, but it's always worth repeating: There is no such thing as too many backups. If the data is important, you should have it stored in multiple mediums, and multiple locations. The HSS guys suggest a "3-2-1" backup strategy, in fact. 

3 Backup copies of anything you want to keep. 

2 different storage media. 

1 offsite storage site.

 

 

This helps prevent any significant data from being lost or corrupted.

It does not. With pictures, home movies etc, files may be untouched for years. I have DP x2, 3 Backup Disks of which 1 is offsite at all time. But Server Backup retains backups back to, say 4 months or so (or, it may be I started anew with Server Backups in which case it is my bad), anyway, there is a limit to how far one can go back. As there is no *active* check on file integrity, by the time I'd open an old file, it may be corrupt and I may only have backups of that corrupted file.

 

Not sure if DP actually does a regular _compare_ of duplicates (so not just meta-data but the actual content) which would, for me, make this all a non-issue, it ends somewhere.

I had scanner run weekly until a week ago then thought I was maybe being a bit, uhm, anal? about it? Default was 30 after all so I settled for bi-weekly now. But would it sense an accidental/incidental flipping of a bit?

[dbailey75]

It does not. With pictures, home movies etc, files may be untouched for years. I have DP x2, 3 Backup Disks of which 1 is offsite at all time. But Server Backup retains backups back to, say 4 months or so (or, it may be I started anew with Server Backups in which case it is my bad), anyway, there is a limit to how far one can go back. As there is no *active* check on file integrity, by the time I'd open an old file, it may be corrupt and I may only have backups of that corrupted file.

 

Not sure if DP actually does a regular _compare_ of duplicates (so not just meta-data but the actual content) which would, for me, make this all a non-issue, it ends somewhere.

I had scanner run weekly until a week ago then thought I was maybe being a bit, uhm, anal? about it? Default was 30 after all so I settled for bi-weekly now. But would it sense an accidental/incidental flipping of a bit?

 

Agreed. most backup tools, especially those for home use that are affordable, will not touch a file that has not been "modified" since the last backup,which is why I went with syncback, as part of the backup profile for my irreplaceable data, it compares the hash of each file on the server to the hash of the file in the backup set,   a new hash is generated at each backup from both sources, again, this is another check to ensure the data does not change over time with out me knowing.  doubles the backup time, even when there is no new data, but it gives me that warm and fuzz feeling.

[Umfriend]

Agreed. most backup tools, especially those for home use that are affordable, will not touch a file that has not been "modified" since the last backup,which is why I went with syncback, as part of the backup profile for my irreplaceable data, it compares the hash of each file on the server to the hash of the file in the backup set,   a new hash is generated at each backup from both sources, again, this is another check to ensure the data does not change over time with out me knowing.  doubles the backup time, even when there is no new data, but it gives me that warm and fuzz feeling.

Actually dbailey, if I understand you correctly, as long as I do not reformat ,my backup HDDs and start anew, I should be fine? A file that has not been modified for a long time would only be written to the backup disks _once_ at initial backup? I understand you _actively_ look for changed/corrupted files by comparing against backup but I could simply wait until a file becomes corrupted and then restore that file from the backup, no?

 

Anycase, bit-by-bit compare of DP-duplicated files would suffice for me.

[Christopher (Drashna)]

I do believe that StableBit DrivePool does do a hash check of files when it runs a duplication pass.

 

Also, there are a couple of reasons I mentioned accessing the file via the PoolPart path directly. If it's duplicated, it's possible that only one version is corrupted, and the service hasn't run a pass on it yet. This way, you can verify and resolve the issue yourself.

Also, it could be an interaction with the pool causing the issue, and the actual files are fine. And by "interaction", I mean with something like Avast or other antivirus software. They install file system filters and could cause complications if they don't handle the Pool drive properly.

[dbailey75]

Actually dbailey, if I understand you correctly, as long as I do not reformat ,my backup HDDs and start anew, I should be fine? A file that has not been modified for a long time would only be written to the backup disks _once_ at initial backup? I understand you _actively_ look for changed/corrupted files by comparing against backup but I could simply wait until a file becomes corrupted and then restore that file from the backup, no?

 

Anycase, bit-by-bit compare of DP-duplicated files would suffice for me.

I'm not a tech guy by trade, just on the weekends, and maybe I've read too many articles on bit rot, file corruption, etc, but how do you know when this occurs?   I put in some checks in place to help flag these issues should they occur. Maybe I'm a little anal, but i'm the CIO and CTO of the family,  I lose those pictures, and I'm in big trouble. lol.

 

I do believe that StableBit DrivePool does do a hash check of files when it runs a duplication pass.

 

Also, there are a couple of reasons I mentioned accessing the file via the PoolPart path directly. If it's duplicated, it's possible that only one version is corrupted, and the service hasn't run a pass on it yet. This way, you can verify and resolve the issue yourself.

Also, it could be an interaction with the pool causing the issue, and the actual files are fine. And by "interaction", I mean with something like Avast or other antivirus software. They install file system filters and could cause complications if they don't handle the Pool drive properly.

 

So if the original file becomes corrupt,  would drive pool not just duplicate this new, bad file?  

 

hi all

 

 

not sure if its drivepool related but I am having issue with some pictures stored having the following effect as per attached

 

looks like pixels have moved

 

 

now im not sure how drivepool works but can it be the cause ?

running latest r c and latest scanner beta

OP, I apologize for hijacking your thread, but at the very least I hope this information is beneficial.  

[AMCross]

no problem

 

trying to get my head round it

[Christopher (Drashna)]

I'm not a tech guy by trade, just on the weekends, and maybe I've read too many articles on bit rot, file corruption, etc, but how do you know when this occurs?   I put in some checks in place to help flag these issues should they occur. Maybe I'm a little anal, but i'm the CIO and CTO of the family,  I lose those pictures, and I'm in big trouble. lol.

 

 

... Neither was I actuall. I just have a passion.

 

But as I said above, for files that are very important, or irreplaceable, you should definitely use that "3-2-1" strategy. Because what happens if you accidentally delete the files? Sure, there is data recovery, but that may not get it back 100% intact.

 

And I do think that I really did get Alex to seriously considering creating some sort of integrity checker. Hopefully.

So if the original file becomes corrupt,  would drive pool not just duplicate this new, bad file?

No. Normally, when it checks the duplication, it checks that the there are the expected/correct number of copies, and compares the file hash of both. I'm not sure if you've seen it, but the "file mismatch" error? That's what it's complaining about. The contents don't match. If you want, test this out with a simple text file. The notification may not be immediate, but it WILL happen.

OP, I apologize for hijacking your thread, but at the very least I hope this information is beneficial.

I don't consider it hijacking, actually. This is all very related information. And if AMCross didn't ask, I'm sure it was probably on his mind.

 

 

As for the silent data corruption, or "bitrot".... well... that's a funny term. Because there is no real good definition for it. And better yet, for what is happening, theoretically, between NTFS and the disks' Error correction routines in it's firmware..... it shouldn't happen.

Also, it's very possible that the corruption is much less esoteric. Windows Search tries accessing the files periodically, and other tools may as well. If they open the file as writable, or updates it's metadata... or any other program does the same.... it could cause the issue as well.

[Umfriend]

No. Normally, when it checks the duplication, it checks that the there are the expected/correct number of copies, and compares the file hash of both. I'm not sure if you've seen it, but the "file mismatch" error? That's what it's complaining about. The contents don't match. If you want, test this out with a simple text file. The notification may not be immediate, but it WILL happen.

I don't consider it hijacking, actually. This is all very related information. And if AMCross didn't ask, I'm sure it was probably on his mind.

I did not know about the hash-total check. How often does is check? I have immediate balancing and all other just at default settings. I have set up a test just now to see how that works. A text file with "This is line one with the letter A.", which was stored on both disks. Changed one of them to letter B.

"

[dbailey75]

 

 

... Neither was I actuall. I just have a passion.

 

But as I said above, for files that are very important, or irreplaceable, you should definitely use that "3-2-1" strategy. Because what happens if you accidentally delete the files? Sure, there is data recovery, but that may not get it back 100% intact.

 

And I do think that I really did get Alex to seriously considering creating some sort of integrity checker. Hopefully.

 

No. Normally, when it checks the duplication, it checks that the there are the expected/correct number of copies, and compares the file hash of both. I'm not sure if you've seen it, but the "file mismatch" error? That's what it's complaining about. The contents don't match. If you want, test this out with a simple text file. The notification may not be immediate, but it WILL happen.

 

I just completed a some sales training for Veeam Backup, and the 3-2-1 strategy was on the test, lol.  

 

I was not aware of this feature, this is good to know.

[Christopher (Drashna)]

I did not know about the hash-total check. How often does is check? I have immediate balancing and all other just at default settings. I have set up a test just now to see how that works. A text file with "This is line one with the letter A.", which was stored on both disks. Changed one of them to letter B.

"

In theory, it should happen once a day, I believe. Specifically, it's part of the duplication pass, that should be happening once a day.

 

I just completed a some sales training for Veeam Backup, and the 3-2-1 strategy was on the test, lol.  

 

I was not aware of this feature, this is good to know.

Well, I'm glad I could help out there.

[Umfriend]

23 hrs and 19 minutes have passed and DP has not signalled any issues. Is there a duplication pass when I have immediate balancing? Something else I could have done/should have done that disables/enables such a check? Running v2.0.0.420.

[Christopher (Drashna)]

To be honest, when I've tested this in the past, it's flagged it right away (within minutes) but that was on a very, very small pool (for testing).

 

I've flagged Alex, to well, get verification for this, and more info.

Though, I want to say that this happens at 2am, usually.

[Christopher (Drashna)]

Worst case, change the duplication, and then change it back. This will FORCE a duplication pass to occur.

[Umfriend]

Uhm, I'll await official confirmation from you on this. Still no message from DP on the A/B difference in the two files. I mean, I would not care if it turns out a compare of some sort is done weekly (perhaps even monthly) but I would like it to do it and know when it does it.

 

I'll instruct a re-measure NOW, see what that does.

[Christopher (Drashna)]

I need to apologize, I misunderstood some things, or made some assumptions that were incorrect.

So let me correct that here.

 

 

When "Real-time duplication" is enabled, it doesn't make a periodic duplication pass. 

However, it does do one forcibly, when you change the duplication status, or level. 

If you change a folder's duplication status, it checks just that folder. If you change the "Pool File Duplication" status, it will check he entire pool.

 

However, if real-time duplication is NOT enabled, then it will make a duplication pass, every day at 2AM (or whenever it is configured to). 

 

Additionally, the background duplication pass looks for any 3rd party app/user errors (access/modifications). 

It also compares each file part using the file size and times. If there is a mismatch, then it performs a hash check. If they're identical, then the file times are synchronized.

 

 

 

In theory, between this, and StableBit Scanners surface scan, there should be no disk related corruption. Or at least, you should be notified of it. 

Specifically, the scan should "exercise" or "scrub" the disk to help identify disk errors before they affect your data. And should identify any issues with the disk itself.

 

 

 

Though, talking with alex (in a large part because of this thread), it sounds like he is definitely seriously considering creating an integrity checking app. That would integrate with DrivePool.

[Umfriend]

I'd really like such an app/add in. It may be the absolute best worthless app (worthless in that it may never ever find anything) but the comfort would be so great.

 

OT: Early days I was a console operator at a Unisys mainframe (or, as the IBM-operaters called it, a mini ;-). Late 80s. Everything was double and could survive one failure: 2 CPUs, 2x5 Harddrives (huge machines), 2x2 tape drives, 2 consoles of course, 2 printers, 2 communication controllers etc. The database was duplexed as well. Every Wednesday evenening we would run the Compare. It'd check the contents of both databases. Had been done since the start in 1985, went until the system was replaced in, 1998 or so. Wednesday was overtime day (evening) because the compare ran for 2 to 3 hours and exceeded the last shift. Paid well that and was very reliable. It never ever found a thing. Everyone was happy.

[Christopher (Drashna)]

I wouldn't say "worthless". Peace of mind is worth a heck of a lot.

But it is definitely more of a "program you hope you'll never need", though.