Background: DrivePool is used as my unduplicated home media storage server for Plex/Kodi. Original media files are backed up and stored on HDDs in the closet. No duplication is used on DrivePool for the media files. I recently lost a 5TB HDD in a pool of 17 drives, all of different brands, sizes, ages....
Problem: My disk monitoring software warned me about an imminent HDD failure. I immediately attempted to remove the disk from DrivePool, but all attempts to move data off that disk failed and the drive completely crashed in less than 12 hours. Now, like many before me, I am wondering what was actually on that failed drive and do I want to recover the lost pool files from my backups.
Solutions considered:
SnapRAID might work for some people as adding a parity solution to the pool for recovery, but that means dedicating additional disks to the pool just to hold the parity data. Also, from what I understand, SnapRAID accesses drives via drive letters. With 17 drives in my pool, I have turned off the drive letters because DrivePool does not need them. It appears that there are ways to mount drives in empty folders so SnapRAID can bypass the Drive Letter requirement, but it looks to me that the SnapRAID solution gets more and more complicated as you add pool drives to DrivePool.
Duplication on DrivePool seems to be the easiest solution to recover from a HDD failure within DrivePool, but, who can afford to have a 1:1 solution on a 70TB+ pool? I already have backups of the data on HDDs stored in the closet. I just don't have the budget to both backup files in offline storage and then again duplicate the files in DrivePool.
WinCatalog 2020 is what I started to use just before my HDD failed. In theory, if you have DrivePool cataloged in WinCatalog 2020 and you suffer a HDD failure, you can run an update on the DrivePool volume and WinCatalog 2020 will report the "deleted" files compared to its last update. From what I understand, this still would require me to manually look up and recover lost files from my back ups in the closet. However, at the point, I may or may not decided to recover some or all of those files from my backups. Problem is, with a 5TB drive failure, that gets to be a lot of lost files. As I said, I just started using WinCatalog 2020, have not learned how to use it well yet, and and had not even gotten to the point of cataloging my DrivePool before I had a HDD failure in the pool.
Solution I am searching for:
I would like some kind of cataloging program, like WinCatalog 2020, that I could constantly use to catalog my DrivePool automatically, like maybe every night. If I had a HDD failure in DrivePool, and lost an entire 5TB HDD of unduplicated media files, I would like the cataloging program to tell me which files were lost and prompt me to see which files I would want to recover from my storage backups in the closet. I could select the files I wanted to recover, and the program would tell me which storage HDD(s) I needed to get online (HDD caddy in my case) and then it would automatically restore those files to DrivePool from my original storage backups, prompting me to get each backup HDD as needed for the recovery task.
It would be great if DrivePool had such a cataloging feature built into it, but from what I have read on past threads in this forum, there does not seem to be much interest in working in that direction. Over the years my media collection has grown and now I currently have 70TB of files in DrivePool. As HDDs continue to increase in size, a loss of even one pool HDD means lots of unduplicated files are potentially lost to the pool. I don't expect DrivePool to do everything, but I can see HDDs at 12TB+ being more common and the loss of just one 12TB+ pool HDD could have a significant negative impact on unduplicated media files.
If anyone has addressed this issue and developed a working solution to recover from a HDD loss in an unduplicated media storage pool, please let me know. I have no desire to reinvent the wheel if a solution exists. Thanks for any comments.
Question
gtaus
Background: DrivePool is used as my unduplicated home media storage server for Plex/Kodi. Original media files are backed up and stored on HDDs in the closet. No duplication is used on DrivePool for the media files. I recently lost a 5TB HDD in a pool of 17 drives, all of different brands, sizes, ages....
Problem: My disk monitoring software warned me about an imminent HDD failure. I immediately attempted to remove the disk from DrivePool, but all attempts to move data off that disk failed and the drive completely crashed in less than 12 hours. Now, like many before me, I am wondering what was actually on that failed drive and do I want to recover the lost pool files from my backups.
Solutions considered:
SnapRAID might work for some people as adding a parity solution to the pool for recovery, but that means dedicating additional disks to the pool just to hold the parity data. Also, from what I understand, SnapRAID accesses drives via drive letters. With 17 drives in my pool, I have turned off the drive letters because DrivePool does not need them. It appears that there are ways to mount drives in empty folders so SnapRAID can bypass the Drive Letter requirement, but it looks to me that the SnapRAID solution gets more and more complicated as you add pool drives to DrivePool.
Duplication on DrivePool seems to be the easiest solution to recover from a HDD failure within DrivePool, but, who can afford to have a 1:1 solution on a 70TB+ pool? I already have backups of the data on HDDs stored in the closet. I just don't have the budget to both backup files in offline storage and then again duplicate the files in DrivePool.
WinCatalog 2020 is what I started to use just before my HDD failed. In theory, if you have DrivePool cataloged in WinCatalog 2020 and you suffer a HDD failure, you can run an update on the DrivePool volume and WinCatalog 2020 will report the "deleted" files compared to its last update. From what I understand, this still would require me to manually look up and recover lost files from my back ups in the closet. However, at the point, I may or may not decided to recover some or all of those files from my backups. Problem is, with a 5TB drive failure, that gets to be a lot of lost files. As I said, I just started using WinCatalog 2020, have not learned how to use it well yet, and and had not even gotten to the point of cataloging my DrivePool before I had a HDD failure in the pool.
Solution I am searching for:
I would like some kind of cataloging program, like WinCatalog 2020, that I could constantly use to catalog my DrivePool automatically, like maybe every night. If I had a HDD failure in DrivePool, and lost an entire 5TB HDD of unduplicated media files, I would like the cataloging program to tell me which files were lost and prompt me to see which files I would want to recover from my storage backups in the closet. I could select the files I wanted to recover, and the program would tell me which storage HDD(s) I needed to get online (HDD caddy in my case) and then it would automatically restore those files to DrivePool from my original storage backups, prompting me to get each backup HDD as needed for the recovery task.
It would be great if DrivePool had such a cataloging feature built into it, but from what I have read on past threads in this forum, there does not seem to be much interest in working in that direction. Over the years my media collection has grown and now I currently have 70TB of files in DrivePool. As HDDs continue to increase in size, a loss of even one pool HDD means lots of unduplicated files are potentially lost to the pool. I don't expect DrivePool to do everything, but I can see HDDs at 12TB+ being more common and the loss of just one 12TB+ pool HDD could have a significant negative impact on unduplicated media files.
If anyone has addressed this issue and developed a working solution to recover from a HDD loss in an unduplicated media storage pool, please let me know. I have no desire to reinvent the wheel if a solution exists. Thanks for any comments.
Link to comment
Share on other sites
10 answers to this question
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.