Jump to content
Covecube Inc.
  • 0

File Placement, how to add new HDD for only backup files


Question

I have a Seagate USB 6TB HDD that I would like to add to my existing 16 HDD DrivePool. However, I don't have 100% confidence in the HDD because I ran across a few corrupted files on it that were causing problems. I offloaded the entire HDD and moved those files to DrivePool. I ran a few MS diagnostic short tests on that HDD and it passed. I ran the Seagate Long Generic Seatools (almost 24 hours) test on it and it failed, but gave me no report what was wrong. I then loaded up HD Tune and ran the disk sector complete scan (about 18 hours) and it detected no errors and passed the drive as good. As of now, I am thinking\wishing that the Seagate 6TB drive is probably good. It could have just been those corrupted files that were causing some problems and maybe not any bad sectors on the HDD itself.

At this point, I would like to add that 6TB Seagate HDD to my DrivePool, but only let my backup media files to be placed on that HDD. That way, if the drive has problems, it would only affect files that I have archived elsewhere so I could rebuild the data. In other words, I only want that 6TB HDD to have the \movies folder on it an no other folders or files. Also, since I have \movies folder spread over the existing 16 HDDs in DrivePool, I cannot use File Placement to only check the one 6TB for that folder. In File Placement, I see how to check which drives the folder(s) can be placed on. Is there a way to select the HDD, and limit it to only accepting files for the \movies folder while still allowing the other HDDs to continue to also use the existing \movies folders.

Any help appreciated. Thanks.

Link to post
Share on other sites

15 answers to this question

Recommended Posts

  • 0

I have never tried to use File Placement Rules but as I understand it, if you tell DP to place \movies on that 6TB HDD then it will and also spill over to other drives when the 6TB HDD is full. DP will not, I think, through a no space error. Not sure how to ensure that non other data arrives on that drive, that might need a lot of FPRs, but may be easy.

Having said that, I would not add a drive I am suspicious about if, unlike me, you do not use duplication. Do you have Scanner? Maybe run the Seagate tool again.

Link to post
Share on other sites
  • 0

another way would be if you dont need duplication

1 dont add the disk to the pool

2 use backup software - set a job to copy the changed/added files to it - nightly job or similar

simpler and you dont have to mess with FPR's that may get complex

also you can detach it and move it to another pc without having to remove it from DP as it would throw an error

Link to post
Share on other sites
  • 0

In the File Placement Rules section:

  • your FIRST (uppermost) rule should be that all files matching "\movies\*" are to be placed on all drives INCLUDING that one (check all drives)
  • your LAST (lowermost) rule should be that all files matching "*" are to be placed on all drives EXCLUDING that one (check all the others, leave it unticked, and in your particular case you also want to select "Never allow files to be placed on any other disks" for this rule).
  • Any other rules you might have should be placed ABOVE the LAST rule, and should not have that drive checked (and again, you may wish to select "Never allow...").

This is because FPR checks the rules from uppermost to lowermost until it finds a matching rule for the file being balanced and then uses only that rule.

NOTE that File Placement is only performed when balancing is triggered instead of in real time; you might wish to use the SSD Optimizer balancer plugin to mark at least one of the other disks as "SSD" so that new files are never even temporarily placed on the 6TB HDD, which is otherwise possible even if you have "Balance immediately" selected in the Settings tab.

Link to post
Share on other sites
  • 0
On 11/6/2020 at 12:30 AM, Umfriend said:

Having said that, I would not add a drive I am suspicious about if, unlike me, you do not use duplication. Do you have Scanner? Maybe run the Seagate tool again.

That's my dilemma. I don't have complete confidence in the drive because it failed the Seagate Tools long generic test - but it has passed all other testing and Scanner shows no problems, Hard Disk Sentinel reports 100% health, etc.... Would not be an issue with a smaller HDD, but a 6TB HDD still is worth $$$ if it is good.

Link to post
Share on other sites
  • 0
On 11/7/2020 at 2:51 PM, Spider99 said:

another way would be if you dont need duplication

1 dont add the disk to the pool

2 use backup software - set a job to copy the changed/added files to it - nightly job or similar

simpler and you dont have to mess with FPR's that may get complex

also you can detach it and move it to another pc without having to remove it from DP as it would throw an error

I am considering that path. Thanks.

Link to post
Share on other sites
  • 0
15 hours ago, Shane said:

In the File Placement Rules section:

  • your FIRST (uppermost) rule should be that all files matching "\movies\*" are to be placed on all drives INCLUDING that one (check all drives)
  • your LAST (lowermost) rule should be that all files matching "*" are to be placed on all drives EXCLUDING that one (check all the others, leave it unticked, and in your particular case you also want to select "Never allow files to be placed on any other disks" for this rule).
  • Any other rules you might have should be placed ABOVE the LAST rule, and should not have that drive checked (and again, you may wish to select "Never allow...").

This is because FPR checks the rules from uppermost to lowermost until it finds a matching rule for the file being balanced and then uses only that rule.

NOTE that File Placement is only performed when balancing is triggered instead of in real time; you might wish to use the SSD Optimizer balancer plugin to mark at least one of the other disks as "SSD" so that new files are never even temporarily placed on the 6TB HDD, which is otherwise possible even if you have "Balance immediately" selected in the Settings tab.

Thanks. Before I read your post, I wrote rules as you suggested, but when I started transferring files I noticed that all files were being sent to the empty 6TB "suspect" drive. I was not aware of your "NOTE that File Placement is only performed when balancing is triggered instead of in real time;..." So that probably explains why I saw all those files being written to the empty 6TB HDD. Also, the SSD Optimizer balancer plugin tweaks make sense. Those are the important things I did not know when I wrote my initial rules. When I saw all my new file transfers going to the 6TB HDD, I stopped the transfers and removed the 6TB from DrivePool. As you mentioned, they were probably only temporary files and might have been cleaned up upon a balance transfer.

I am currently leaning towards using this suspect 6TB HDD only as a offline backup storage HDD. I can use TeraCopy and move files to the 6TB HDD using the verify command to ensure all files are 100% verified upon transfer. Probably a better solution for me than to risk using a suspect HDD in DrivePool as others have already mentioned. Also, DrivePool is working just fine, so I don't need to muck it up with a drive that I don't completely trust.

Thanks for all the responses.

Link to post
Share on other sites
  • 0

 

6 hours ago, gtaus said:

I can use TeraCopy and move files to the 6TB HDD using the verify command to ensure all files are 100% verified upon transfer.

For whatever it's worth, in the past I have encountered problems with "copier" software silently missing files. Admittedly I was dealing with very large file sets, very long paths, and unicode names, all back when a lot of software would have trouble with just one of those let alone all three, and the less-than-reliable hardware I was (ahem) relying on at the time probably didn't help, but the important takeaway is that if you're working with "irreplaceable" data you might want to stress-test your copier and verify that it is actually doing what it says it's doing.

Link to post
Share on other sites
  • 0
16 hours ago, Shane said:

 

For whatever it's worth, in the past I have encountered problems with "copier" software silently missing files. Admittedly I was dealing with very large file sets, very long paths, and unicode names, all back when a lot of software would have trouble with just one of those let alone all three, and the less-than-reliable hardware I was (ahem) relying on at the time probably didn't help, but the important takeaway is that if you're working with "irreplaceable" data you might want to stress-test your copier and verify that it is actually doing what it says it's doing.

Good points. However, after using Teracopy for awhile, I prefer to using it over MS Explorer for transferring large file sets. I usually set up my large file set transfer to occur over night. In MS Explorer, if it hits a problem of any kind, it stops at that point and waits for you to respond to a pop up error window. Obviously, if you have a 10 hour transfer set to perform, and MS Explorer hits a problem 30 mins into the transfer, you just wasted that night for transfers because you still have 9.5 hours uncompleted. Teracopy, on the other hand, will take that 10 hour transfer and when it detects a problem with a file, it will simply tag it as an error and continue on to the other files. In the morning, all 10 hours of the transfer are complete and maybe only one file was in error and is left in the queue as an error.

Teracopy also has a verify checkbox where it calculates a hash code for each file transferred and then reads back that file and compares the hash code. If they match, all good. If they do not match, that file goes into the error list and remains in the queue. However, using the verify command doubles your transfer time as you would expect.

As to the "suspect" 6TB HDD, which I think might be OK, I decided to just create a new DrivePool for my \movies\ folder starting with that HDD. I have all my original movie files saved offline on other HDDs, so if, for any reason, this 6TB HDD fails or starts to lose files, it will be no big loss as I have the original files safely stored on HDDs in the closet. I will just add the new drive letter to my sources in Kodi and Plex. Should work just fine. Also, since I am creating a new - standalone DrivePool - just for these media files, I will be adding my other older drives to that pool and use them until they die. Good use for old hardware. Also, solves the problem of using that suspect drive ONLY for the \movies\ files which are non critical.

Link to post
Share on other sites
  • 0
On 11/11/2020 at 2:45 PM, Umfriend said:

I think a 2nd Pool for non-critical data using old hardware is an excellent idea.

Well, my excellent idea has proven successful, but maybe in an unexpected way. I created a 2nd DrivePool for my non-critical \movie\ data using my suspect 6TB USB HDD. Everything seemed to be working fine as I used Teracopy to move 6TB of \movie\ files to the new DrivePool using the verify command to ensure all files were 100% correct upon transfer. Everything worked fine for a day or two. Thinking I was good to go, I was about to add some more older HDDs to that 2nd DrivePool when, for whatever reason, that 2nd DrivePool went offline and I could not access it (only the suspect 6TB USB HDD on the pool at that time). Fortunately for me, I only had that one suspect 6TB HDD on the 2nd DrivePool at that time, so troubleshooting was fast and easy. Turns out, that suspect 6TB HDD has a tendency to shut itself down, go offline, and then throw the DrivePool into an error state. I was able to unplug the 6TB, plug it back in, and then both the drive and the 2nd DrivePool came back to life.

This happened a couple more times yesterday, with that 6TB USB HDD going offline seemingly random, requiring me to unplug it and then plug it back in, so I decided it was just not worth the trouble trying to salvage use out of that 6TB HDD anymore. I am currently in the process of using Teracopy with verify to move all the \movies\ file off that 2nd DrivePool with only the 6TB USB HDD back to my original DrivePool. 

Turns out that creating a 2nd DrivePool using only that suspect 6TB USB HDD was a good idea because I quickly was able to determine there is indeed something wrong with it (despite all the diagnostic programs that scan it and report it is working fine). The only program that correctly determined the HDD was failing was Seagate Seatools using the long generic test (it passed the short tests). Unfortunately, the HDD is out of warranty so I will have to just eat the loss and move on.

But, I would like to shout out to Stablebit Drivepool because, even though that 6TB HDD is failing, my data seems to be recoverable. I am currently about 50% complete on the file transfer off that suspect HDD and all files are intact. I can guarantee you that if that same suspect HDD had gone bad in my MS Windows 10 Storage Spaces setup, using data packets spread all over the pool, it would have most assuredly crashed the entire Storage Space and all my data would have been lost. How do I know that? Because I used Storage Spaces for years and after my 3rd catastrophic loss of data due to a HDD failure, I moved over to Storage Spaces. Despite having 2 and even 3 HDD failure protection on Storage Spaces, I lost entire pools of data when only 1 drive out 20 drives went bad. I guess I can attest to the fact that when a HDD goes bad in DrivePool, I have been able to recover my data off that failing HDD. For my few folder/files that require duplication, I have 2x or 3x duplication set and with DrivePool I don't worry that any 1 drive will crash my entire pool.

Thanks to everyone for their responses and helping me work through this issue.

Link to post
Share on other sites
  • 0

... Would that 6TB USB HDD happen to be a Seagate Backup Plus Hub by any chance?

Because I bought one a ways back (P/N 1XAAP2-501) and it behaves exactly the same way you've described. The drive inside might actually be okay, just with a lemon enclosure. Since yours is out of warranty and you're planning to ditch it, consider instead shucking it and using the HDD as an internal drive (run the Seagate long test on it again of course).

BTW just FYI, Seagate has used SMR drives in its Backup Plus Hubs and I don't recommend using SMR drives in a pool, so also check the part number of the drive itself to see if it is one, with (if you don't want to open it up to check) a utility like Crystal Disk Info or similar.

Link to post
Share on other sites
  • 0
2 hours ago, Shane said:

... Would that 6TB USB HDD happen to be a Seagate Backup Plus Hub by any chance?

Yes, indeed, it is a Seagate 6TB USB Backup Plus Hub. It would be great if the HDD was good and that maybe only the enclosure is a "lemon" and causing my problems. Like I said, the HDD passed all my HDD diagnostic programs and reports Good Health at 100%. It did fail the Seagate Seatools long generic test, but if the enclosure firmware is randomly disconnecting the HDD, maybe that is why it fails that long test.

CrystalDisk Info was: Model: ST6000DM003-2CY186 6001.1 GB   Serial Number: WG500ZFK  Health Status: Good   S.M.A.R.T.: No alerts on anything. I don't know how to tell if this HDD is SMR or not, but since it is no longer under warranty, I have no problem cracking open the case and using the bare HDD in an empty slot in my MediaSonic ProBox. 

2 hours ago, Shane said:

Because I bought one a ways back (P/N 1XAAP2-501) and it behaves exactly the same way you've described.

My enclosure is P/N 1XAAP2-500. 

I am currently moving all files off the 6TB HDD, but once done, I I'll crack open the case and remove the drive. I'll test the bare drive in my desktop HDD caddy, and if it passes, then I'll put it into my MediaSonic ProBox and add it DrivePool. Thanks for your response and for giving me some hope that the actual drive might still be good.

Link to post
Share on other sites
  • 0
On 11/16/2020 at 4:46 AM, Shane said:

Checking, it seems the ST6000DM003 is a SMR drive. I don't recommend putting it in any pool where you want decent rewrite performance, but if you're only wanting good read performance it's fine.

Yes, I think the ST6000DM003 is a SMR drive. However, after your post about the USB Seagate Backup Plus Hub HDD enclosure case probably being the problem with that HDD, I did remove the HDD from the enclosure and put it in my MediaSonic ProBox about a week ago and it is working flawlessly. So I am very happy that I was able to "save" the HDD for use in DrivePool.

I am not a big fan of the SMR technology and understand why you don't want SMR HDDs in a pool. However, in my DrivePool, I have a 248GB SSD in front of the queue as cache. I have DrivePool set not to move any files until there is at least 100GB of files to optimize. In my mind, any file(s) written from the SSD cache to the SMR drive will not be fragmented, which would be better for the file(s) sitting on the SMR drive. Also, my DrivePool is set up mainly as a media storage server for Plex and KODI, so rewrite is not really an issue for me. It seems to work just fine for reads from the SMR drive.

Sorry for the late reply, but for some reason I am not getting any notice of replies to my threads from this forum. I have checked the box to "Notify me of replies", I have checked my account profile for the correct email address, and I have even checked my Junk box in Hotmail, - but no notices from this forum.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...