Assistance with dead drive cleanup

[mr_yellow]

Hi!

So I unfortunately had a drive throw SMART errors and eventually gave up it's ghost.  While it was giving errors and I had a replacement on order, I did the following:

1) I setup new placement rules that moved some more "important" data off the drive and moved some more non-essential stuff only to that drive

2) My pool was pretty full (drives were well past their 95% limit) so rebalancing would complain of not being able to move X amounts of GBs 

3) eventually the drive got worse (more and moreunreadable sectors) at which point I tried to remove the drive from the drive pool (but only checking off skipping duplication for faster drive removal)

4) in that process, the drive *really* died and is now not even mountable in windows.  sometimes it'll spin up but the the drive name defaulted to "local disk" in windows and it complains that the drive is not readable.   During this time, the drive showed up in drivepool with the generic label and I clicked "Remove" and it disappeared.  It didn't even have a prompt on how to remove like before.  Poof, it vanished.

Okay, so the drive is dead and it took along with it a few TB of data.  I can live with that since I managed to pull a list of files and nothing critical was amongst it.  

Now, the drive pool is giving a Duplication Warnings saying ~15 files are missing:

10:26:38.1: Warning: 0 : [FileDuplication] Error checking file \\?\GLOBALROOT\Device\HarddiskVolume15\[REDACTED] for duplication consistency. The system cannot find the path specified
10:26:40.3: Warning: 0 : [FileDuplication] Error checking file \\?\GLOBALROOT\Device\HarddiskVolume15\[REDACTED] for duplication consistency. The system cannot find the path specified
10:26:40.4: Warning: 0 : [FileDuplication] Error checking file \\?\GLOBALROOT\Device\HarddiskVolume15\[REDACTED] for duplication consistency. The system cannot find the path specified
...
10:27:03.6: Information: 0 : [FsControl] Set overall pool mode: PoolModeNoReportIncomplete (lastKey=FileDuplication, pool=4538dc0f-4f53-419c-a2bf-63406948c107)

I've tried clicking "Duplicate now" but after it re-measures, I get the exact same error message.  Afraid to click Rebalancing...   

What's also curious is when I spin up the drive and it mounts as "local disk", it DOES actually show up in Stablebit Scanner under the custom drive name I gave it and I can see there's a lot of activity (80-100+ MB/s at some points).  What's going on here?  Is the drive self scanning itself?  SMART is showing a slew of realloc sectors (25k+) and the unreadable sector is 100+ now, vs 24-32 before...  Should I leave the drive plugged in and let whatever is scanning it finish?  

From what I've read in earlier posts, I've enabled file system logging in troubleshooting but not sure where that log is.  

Any advice?

[gtaus]

I really like DrivePool, but I will admit that bad things have happened to me when my pool reaches its max storage over 95% like you mentioned. Most frequently, I get Duplication Inconsistent errors and I don't know how to fix them - other than adding more storage and/or removing data from the pool. Rebalancing does not help much if your HDDs are already over 95%.

As a rule of thumb, IMHO, DrivePool should never be considered a backup solution. So if you lose a pool drive, and lose data on the pool, you should have already had a backup offline stored somewhere else for recovery. In my case, I use DrivePool as a media storage server and if I lose a pool drive, and some or all of that data on the failed drive, I might not even bother to recovery the loss with my backups. Once I watch a movie, for example, I don't really care if it stays on my DrivePool or not. So I would probably not restore a copy from my backup copy.

I try to keep as much free space on DrivePool as my largest drive in the pool. That way, if a drive starts showing signs of failure, then DrivePool (via Scanner) has a chance of offloading the failing drive into the pool before it completely dies. I have had some drives report a problem and I was able to offload the drive into the pool over a day or two. That's when things go good. Other times, a failing drive may die within hours, and there is not time to fully transfer all files off the drive. And, of course, I have had some drives just up and die without any warning....

DrivePool offers duplication, but to be honest, I have never really understood the method it duplicates the files. Worse yet, if a file becomes corrupt, DrivePool cannot tell you which is the good copy and which is the corrupted copy. From what I understand, DrivePool at best can only tell you that one copy is older than the other. DrivePool itself cannot repair any corrupt files or tell you which copy is good.

Over a period of time, I use DrivePool Duplication less and less. What I do for my media files is use a free program called MultiPar which creates .par2 files for verification and recovery. I have MultiPar set to create 10% recovery files, which is usually enough to repair a track or two on a music album folder. In DrivePool, any album folder, for example, may have music tracks on more than one pool drive. If you lose a HDD, you might lose a track or more on a failing HDD. With the .par2 verification, you can tell within a few moments if your data set is all there, if the data can be recovered from your .par2 recovery files, or if you have to pull out that backup copy from the closet like I mention earlier. Anyway, the verification and recovery features of MultiPar have saved me lots of time and effort after a HDD failure.

Here is a link for the free MultiPar program, if you are interested.

As for connecting your failing drive and see it spinning up.... well, I guess if you have already given up the idea that you will get anything else off the drive, you could just let the program run either until the program completes whatever it is doing, or the drive dies and becomes unreadable. When I first got into DrivePool, I had some HDD issues with older drives and decided to buy Hard Disk Sentinel after trying the free version. The free version will monitor your drives and alert you to possible failures, and if also offers a bunch of free diagnostic tools, but the paid version has a feature that allows you to "fix" certain HDD issues and continue to use that drive. I was able to "save" a couple of my HDDs using Hard Disk Sentinel's paid version, so I figure I got my money's worth out of the program with my first HDD "save." A couple other drives died hard and there was no saving them. Even though I was able to "save" a couple of my drives with Hard Disk Sentinel, I still replaced them as soon as possible. In my experience, a failing drive will continue to fail until it crashes completely. But I was able to buy time using Hard Disk Sentinel and did not lose data on those "saved" drives before I replaced them.

Finally, the Service Log can be found by clicking on the upper right icon Settings>Troubleshooting>Service Log... Hope that helps.

[mr_yellow]

Thanks for the tip with MultiPar..  I will definitely try this out because I've already discovered a few corrupted JPGs of my kids in my photo library and that's with not even trying so I'm definitely worried about data integrity about the rest of my "backups". 

I've been running drive pool for so many years now, I've had thoughts of going unraid and setting up a proper RAID pool for the sake of data integrity.  I'm not too worried anymore about maximizing the space that I already have with the selective duplication.  But I'm not sure what is the best migration path.  

I'm curious if anyone else here has transitioned from drivepool to windows spanned volumes (which contain their own parity volumes).  

The duplication errors eventually went away on their own after a few more reboots and rebalances.    Maybe the migration question is best asked in it's own thread.