Create new pool from recovered data of old pool

[smark36]

Hi,

I had a very severe issue on my home server (Windows 10) where all of a sudden all the data on my DrivePool drive and some data on another disk, which was not part of any pool, was suddenly deleted. The entire pool suddenly showed as completed empty. All individual drives of the pool also showed as empty. I don't know what caused this. I'm honestly a bit shocked that something like this can even happen at all.

In any case I have now done a fresh install of my server. (previously I had upgraded it already a few times from one Windows version to another and I replaced the internal hardware including mainboard and cpu a few times... I guess it was not in the best shape anymore after all these changes over the years) And I used (expensive) data recovery software to recover as much of my data as I could.

So now I have two 20 TB disks with about 30 TB of data on them and I would like to create a new pool for those two disks.

I understand that the fastest way of doing this is to create a new pool with those two disks which means that DrivePool only uses the available empty space of those drives and creates a pool with that. Then I could move my data into the hidden PoolPart folder to fill the pool with my data efficiently.

But I now have two disks with the same folder structure each. So can I add my data to the PoolPart folder of each of the disks like this which would mean that I create the same folder structure on both disks? E.g. I would have a "BACKUP" folder in PoolPart of disk 1 and disk 2? (I assume yes, cause that's what DrivePool also does)

And what happens if I add a file with the same name (and same path) on disk 1 and on disk 2? This can be the case because the data recovery software may have detected the same file on multiple disks of the old pool. Maybe the file was on one disk at first but then was moved to another becasue of balancing. In any case it can be that I now have a file of the same name (maybe different file sizes, maybe on of the even has filesize 0) on both of my 20 TB disks.

I would be grateful for any help or past experiences any of you may have. With this amount of data it's not so easy to back up everything quickly. It always takes days.

[Shane]

Condolences on the data loss. Sounds like you got hit by something nasty.

14 hours ago, smark36 said:

But I now have two disks with the same folder structure each. So can I add my data to the PoolPart folder of each of the disks like this which would mean that I create the same folder structure on both disks? E.g. I would have a "BACKUP" folder in PoolPart of disk 1 and disk 2? (I assume yes, cause that's what DrivePool also does)

Yes, that's correct.

14 hours ago, smark36 said:

And what happens if I add a file with the same name (and same path) on disk 1 and on disk 2? This can be the case because the data recovery software may have detected the same file on multiple disks of the old pool. Maybe the file was on one disk at first but then was moved to another becasue of balancing. In any case it can be that I now have a file of the same name (maybe different file sizes, maybe on of the even has filesize 0) on both of my 20 TB disks.

DrivePool considers a file that is in the same folder tree (e.g. folder\subdolder\name.ext) on multiple poolparts inside a pool to be a duplicated file, because that's how it  does duplication. However if you weren't using duplication then yes it is most likely due to balancing and the recovery software un-deleting the moved files:

1. If the "duplicated" file has different last-modified dates on those disks DrivePool will consider that an error and offer to keep only the newest file or let you resolve it manually.

2. If the "duplicated" file has the same last-modified dates on those disks but different sizes DrivePool will also consider that an error but leave it up to you to resolve manually.

3. If the "duplicated" file has the same dates and sizes on those disks DrivePool will presume there is no problem even if the files are not identical in content, and proceed normally - which if you have duplication turned off means it will delete one of the "duplicates" to free up space in the pool.

In theory so long as the what happened to the old lost pool was only the deletion of its files, the recovery process restored the deleted files in full, and the files going into the new pool are only from the old lost pool, then you could simply seed the new pool and you should only run into issue #1.

In practice it's possible that the recovery software may have only partially recovered some file contents, leading to issues #2 and #3.

Another issue you'll run into is that recovery software can't tell the difference between deleted files you wanted deleted and deleted files you didn't want deleted - so you'll have to get rid of the former all over again.

14 hours ago, smark36 said:

I would be grateful for any help or past experiences any of you may have. With this amount of data it's not so easy to back up everything quickly. It always takes days.

If possible make a back up of the recovered content before you do anything else with it (including recreating your pool).

Then perhaps use something like DigitalVolcano's Duplicate Cleaner (I used this as an example because it's the one I know best) to identify duplicates and uniques by date, size, name and content so you can resolve the above issues in an interface that's designed for it, again before you recreate your pool. While there's a bit of a learning curve, as a dedicated tool it has a lot more capabilities (such as being able to detect files that look identical but have different content) than DrivePool's basic same-name-but-different-date detection.

P.S. If you do plan to use duplication in the new pool, remember to turn off read-striping.

[smark36]

Thanks a lot for the detailed explanation! In the meantime I did go through my two disks and removed all the duplicates I found. I did this with FreeFileSync. Everytime a file existed on both disks I deleted it from one. And I resolved all conflicts where the file had the same name (and modified date?) but different filesize.

Last thing remaining is to back up my data. Which will again take a lot of time but it is inevitable.

After that I'll create the new pool and move everything into the PoolPart folder.